Posts on Digi Hunch

Training and Inference in SageMaker AI

Sun, 20 Apr 2025 07:20:00 -0400

To automate and secure machine learning operations, one needs to know about programming with SageMaker SDK. While algorithms are data scientists’ wheelhouse, infrastructure engineers like myself need to understand the skeleton of the code in order to orchestrate the activities, optimize performances, and secure the communications.

It was overwhelming to begin with. Programming in machine learning heavily depends on specific frameworks, SDKs, and is highly experimental. The experimental nature entails the followings in the notebook workload:

data engineers may want to load data once, and play with different data processing logics
if the program fails halfway through, the data scientist can rerun from the failed step, or even execute the cells out of order
some lines of code (for training or inference), may demand significantly more resource than other lines;

These makes machine learning workload more challenging and require more flexibility at the infrastructure layer. However, most of the tutorials target data scientist aspirant and lack the perspective of infrastructure operations.

In this post I’ll go over what I’ve learned about training and inference workload. I will not focus on specific algorithms. Instead, I’ll use the most easy-to-understand algorithm, solving a typical problem. I’ll also use a simple framework (SK Learn) but try to reveal what is common amongst other supported frameworks.

Frameworks and Data

Amongst many programming frameworks, I need one that is generic, beginner friendly and compatible with SageMaker AI. The candidates are:

Scikit-learn: good for classical machine learning activities;
TensorFlow and PyTorch: industry standards supporting a wide range of tasks; from simple models to advanced deep learning applications;
Hugging Face, PyTorch and TensorFlow can handle NLPs;
XGBoost excels at handling tabular data

The use case statements above might be over-simplifications but at least Scikit-learn is a solid choice for beginners. It’s also good to keep Pytorch and TensorFlow in mind. As to the machine learning problem, I want to go easy with a simple classification problem with Logistic Regression. Despite the word “regression” in the name, logistic regression is an algorithm to solve classification problems (whereas linear regression solves a regression problem).

I use the simplistic iris flower data set to solve the classification problem: identifying the species (setosa, versicolor, or virginica) based on the length and width of sepal and petal. The dataset is very clean and does not reflect real-world messiness, making it a good for testing machine learn concepts, although not for training in production. The input is four numbers. The prediction output is a number (0,1, or 2) representing the species identified.

Training and Inference

Machine Learning concerns with indeterministic results. We first train the models with input data along with known results. Once trained, we can store the model as artifact with version control. Then we serve the model so we can feed it with unseen data and use the output as predictions. This machine learning flow has a lot in common with SDLC. Training resembles building of an application, the output artifact is the trained model, which we store as flat file or tarballs. In a model lifecycle, we track versions of trained models, quality control the models, decomission them and so forth. To perform inference, we serve the model in different ways, such that consuming application can use it for predictions.

To get started, let’s look at the following simple snippet, where we load data, train a simple model and use it for inference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Step 1: Load the Iris dataset
data = load_iris()
X = data.data # Features (sepal length, sepal width, etc.)
y = data.target # Target (species labels)

# Step 2: Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Step 3: Create and train a logistic regression model
model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

# Step 4: Predict using the test set
y_pred = model.predict(X_test)

# Step 5: Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Model accuracy: {accuracy * 100:.2f}%")

Note that the snippet does not use any SageMaker SDK libraries yet. The example is simple enough to run anywhere. In real life however, the invocations of .fit() method and sometimes .predict() method can be resource-demanding. In such experiments, even a simple change to the hyper-parameter may dramatically increase the resource requirement. This experimental nature calls for a mechanism to allow execution of certain lines of code, in a different runtime environment. The SageMaker SDK provides two mechanisms: a @remote decorator and a RemoteExecutor class. Take the decorator as an example, it employs Python’s decorator to implement a wrapper of the function in the code so that the function can execute remotely on a different machine. The code looks like this:

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Step 1: Load the Iris dataset
data = load_iris()
X = data.data # Features (sepal length, sepal width, etc.)
y = data.target # Target (species labels)

# Step 2: Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Step 3: Create and train a logistic regression model: put the action in a function and mark it to execute remotely
from sagemaker.remote_function import remote

@remote(instance_type="ml.m5.large")
def train(X_train,y_train):
 model = LogisticRegression(max_iter=200)
 model.fit(X_train, y_train)
 return model
model=train(X_train, y_train)

# Step 4: Predict using the test set: put the action in a function and mark it to execute remotely
@remote(instance_type="ml.m5.large")
def predict(model):
 return model.predict(X_test)
y_pred = predict(model)

# Step 6: Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Model accuracy: {accuracy * 100:.2f}%")

Note that this snippet requires a SageMaker SDK library (sagemaker.remote_function) and must run on a Jupyter Lab notebook. That comes with the flexibility to run any function on any type of instance allowed in the SageMaker domain. When a function runs remotely from Jupyter workspace notebook, SageMaker has to spend time provisioning and bootstrapping an EC2 instance, which usually takes a noticeable delay and the stdout gives a summary of billable seconds. SageMaker also capture these remote executions as training job uniformly.

SageMaker SDK Abstractions

We can summarize the activity pattern from the two example scripts as below:

This pattern however, does not fully integrate with other SageMaker AI capabilities. In a full-blown integration, we use the abstraction classes in SageMaker SDK, so that the platform has insights into our activities in the script, i.e. when we’re training models, building pipelines, running inference, etc. This allows SageMaker AI platform to manage these activities. The verb manage represents many specific operations to keep track of outside of the Python scripting runtime. For example, we can keep the training runs, even after the Jupyter workspace runtime is stopped or deleted.

We have to know the abstraction classes for different frameworks. They all follow the same pattern regardless of the framework name. Here is a diagram of the relations between the base classes in SageMaker SDK:

In this programming model, the users offload activities like training and loading a model to a separate Python script called entry point script. Then the users reference the script in the constructor of the estimator. The platform requires the script to follow a particular pattern. For example, to load a model, the function named model_fn() must exist. The training code goes under main() function. A simple entry point script looks like this:

# entry_point.py script for training and loading models on SageMaker AI
import argparse
import joblib
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import os

def main(args):
 # Load data
 df = pd.read_csv(args.train)
 X = df.drop(columns='species')
 y = df['species']
 
 # Train-test split
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
 
 # Train the logistic regression model
 model = LogisticRegression(max_iter=200)
 model.fit(X_train, y_train)
 
 # Evaluate the model
 y_pred = model.predict(X_test)
 accuracy = accuracy_score(y_test, y_pred)
 
 print(f'Model accuracy: {accuracy * 100:.2f}%')
 
 # Save the model to a file
 joblib.dump(model, '/opt/ml/model/model.joblib')

if __name__ == '__main__':
 parser = argparse.ArgumentParser()
 parser.add_argument('--train', type=str, default='/opt/ml/input/data/train/iris_train.csv')
 args = parser.parse_args()
 main(args)

# function to load a model
# https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/using_sklearn.html#load-a-model
def model_fn(model_dir):
 """Load the trained model from the model_dir"""
 model_path = os.path.join(model_dir, "model.joblib")
 return joblib.load(model_path)

This script takes an argument for the location of the training data. The main function performs the model training. Let’s name this script entry_point.py and in the next couple sections we can discuss two patterns to perform inference in a scalable fashion: Batch Transform and Inference Endpoint.

Batch Transform

Training is an expensive activity that we cannot afford to repeat upon receiving every inference requests. That’d be too slow. To serve the model for inference request in a scalable way, we hold the inference requests in batches, and sometimes in the form of a multi-line input file from S3 bucket. Below is an example:

# Import necessary libraries for SageMaker
import sagemaker
from sagemaker import get_execution_role
from sklearn.datasets import load_iris
import pandas as pd
import os
import joblib
# Get the execution role for the SageMaker notebook
role = get_execution_role()
########################################################
import boto3
from sagemaker.inputs import TrainingInput
# Load the Iris dataset
iris = load_iris()
X = iris.data
y = iris.target
# Create a DataFrame to easily export
df = pd.DataFrame(X, columns=iris.feature_names)
df['species'] = y
# Save dataset to a CSV file and upload to S3 bucket
train_file = 'iris_train.csv'
df.to_csv(train_file, index=False)
s3_bucket = 'sagemaker-ca-central-1-train' # Make sure the bucket exists
s3_data_path = f's3://{s3_bucket}/iris/'
s3 = boto3.client('s3')
s3.upload_file(train_file, s3_bucket, 'iris/iris_train.csv')
# Define the location of training data in S3
s3_input_train = TrainingInput(s3_data=s3_data_path, content_type='csv')
########################################################
# Define the SKlearn Estimator and start training the model
from sagemaker.sklearn import SKLearn
sklearn_estimator = SKLearn(
 entry_point='train.py', # Your script
 role=role,
 instance_type='ml.m5.large', # Choose an appropriate instance
 instance_count=1,
 framework_version='0.23-1', # Use a compatible version of sklearn
 base_job_name='sklearn-iris-job',
 hyperparameters={},
 source_dir='.', # If using any other source code files, include them here
)
# Start training the model
sklearn_estimator.fit({'train': s3_input_train})
########################################################
# Preparing inference input file and output directory
input_file='batch_input.csv'
pd.DataFrame(X).to_csv(input_file, header=False, index=False)

# Upload the CSV file to S3
s3_bucket = 'sagemaker-ca-central-1-train' # Make sure you replace with your bucket
s3_data_path = f's3://{s3_bucket}/iris_input/'
s3 = boto3.client('s3')
s3.upload_file(input_file, s3_bucket, 'iris_input/batch_input.csv')
s3_output_path = f's3://{s3_bucket}/iris_ouput/'
########################################################
transformer = sklearn_estimator.transformer(
 instance_count=1,
 instance_type='ml.m5.large',
 output_path=s3_output_path,
 strategy='MultiRecord',
 assemble_with='Line',
 accept='text/csv',
)
transformer.transform(
 data=f'{s3_data_path}{input_file}',
 content_type='text/csv',
 split_type='Line',
 input_filter="$[0,1,2,3]"
)
transformer.wait()
########################################################

We specify the instance size while creating the estimator with constructor. The later transform step takes long and requires provisioning of a computing resource. After the transformer run, the result is in the S3 bucket in the output directory.

Inference Endpoint

If inference requests come in at arbitrary times and expect timely responses, it makes sense to introduce client-server architecture. We use web service framework to host trained models behind a passively-open inference endpoint, allowing client applications to make REST API calls with inference input in the payload, and get inference output in the HTTP response. Essentially, we serve the model behind web servers.

In SageMaker AI, we can set up inference endpoint in three ways. The most common is real-time inference endpoint. We can configure a fleet of web servers in an autoscaling group to serve the inference requests. Below is an example:

# Import necessary libraries for SageMaker
import sagemaker
from sagemaker import get_execution_role
from sklearn.datasets import load_iris
import pandas as pd
import os
import joblib
# Get the execution role for the SageMaker notebook
role = get_execution_role()
########################################################
import boto3
from sagemaker.inputs import TrainingInput
# Load the Iris dataset
iris = load_iris()
X = iris.data
y = iris.target
# Create a DataFrame to easily export
df = pd.DataFrame(X, columns=iris.feature_names)
df['species'] = y
# Save dataset to a CSV file and upload to S3 bucket
train_file = 'iris_train.csv'
df.to_csv(train_file, index=False)
s3_bucket = 'sagemaker-ca-central-1-train' # Make sure you replace with your bucket
s3_data_path = f's3://{s3_bucket}/iris/'
s3 = boto3.client('s3')
s3.upload_file(train_file, s3_bucket, 'iris/iris_train.csv')
# Define the location of your training data in S3
s3_input_train = TrainingInput(s3_data=s3_data_path, content_type='csv')
########################################################
# Define the SKlearn Estimator and start training the model
from sagemaker.sklearn import SKLearn
sklearn_estimator = SKLearn(
 entry_point='train.py', # Your script
 role=role,
 instance_type='ml.m5.large', # Choose an appropriate instance
 instance_count=1,
 framework_version='0.23-1', # Use a compatible version of sklearn
 base_job_name='sklearn-iris-job',
 hyperparameters={},
 source_dir='.', # If using any other source code files, include them here
)
# Start training the model
sklearn_estimator.fit({'train': s3_input_train})
########################################################
# Create a model based on training result
model = sklearn_estimator.create_model()
predictor = model.deploy(
 role=role,
 initial_instance_count=1,
 instance_type='ml.m5.large',
 endpoint_name='TESTENDPOINT1',
 wait=True,
)
########################################################
# Make predictions with the deployed model
test_data = [[5.1, 3.5, 1.4, 0.2]] # Example test data (sepal length, width, petal length, petal width)
prediction = predictor.predict(test_data)
print('Prediction: ', prediction)
# Make predictions with the deployed model again
test_data_2 = [[6.7, 3.1, 5.6, 2.4]] # Example test data (sepal length, width, petal length, petal width)
prediction_2 = predictor.predict(test_data_2)
print('Prediction_2: ', prediction_2)
# Clean up by deleting the endpoint when done
predictor.delete_endpoint()

The real-time inference endpoint is backed by specialized hardware and remains up all the time. That is expensive, especially if inference requests are only sporadic and can tolerate latency. To lower the cost, SageMaker AI platform supports two variants of inference: serverless and asynchronous inferences. The former stems from the (psudo-)concept of serverless computing, with the ability to scale to zero when idle. The latter uses a queue to hold up requests until a meaningful cumulation of jobs. Then it notifies the requestor of the processing results.

To use asynchronous inference, you specify the serverless_inference_config parameter when creating a model. To use asynchronous inference, you need to use the AsyncPredictor class. Whether every model (self-trained, JumpStart, etc) can work with all three patterns of inference, has to be reviewed case by base.

Summary

Most of the machine learning training materials were created by data scientist focusing on the models and algorithms. For infrastructure engineering like myself to learn the topic, I need to understand different ways that one can perform training and inference without getting swapped into the algorithmic details. In this post I took some simplistic examples to look at what training and inference looks like, as local script and as integrated with SageMaker AI platform using its SDKs, to familiarize myself with MLOps operations.
There is also a Udemy course I used to familiarize myself with the topic. The material in the course does not rely on SageMaker AI at all and built model (with scikit-learn) and app (host on Docker and Kubernetes) ground up. In the next few posts, I’ll continue explore training activities on Kubernetes based infrastructure.

SageMaker AI and Networking

Tue, 01 Apr 2025 00:53:00 -0400

Machine learning workflows are highly experimental. To smooth out the processes, Amazon SageMaker AI packages many features as managed services. As an infrastructure specialist, I want to remain compliant. At a basic level, compliant architecture means multi-account structure and hub-and-spoke VPC topology in a landing zone. While the multi-account operating model for SageMaker is well documented, these ML managed services obscure the network configuration. I’m always looking for low-level insights on where the computing activity is happening and how the application traffic flows in and out of our VPCs. I don’t always get straight answers and I often have to experiment them out. This post is a review of the networking aspects of SageMaker AI I recently learned.

Naming Shenanigans

Unfortunately, I have to start with how AWS has renamed these services, so the terms remain clear throughout the post.

In Dec 2024, AWS renamed SageMaker to SageMaker AI. The name of SageMaker going forward represents the overarching AWS service for machine learning, data, analytics and generative AI. Here is a video for clarification. I’m not a fan of how they repurpose the names. As of date there are still a lot of content referencing SageMaker AI as SageMaker, whose meaning has changed.

If that’s not enough, here’s another one. The SageMaker Studio launched originally in 2019 for model development. In 2023 that became SageMaker Studio classic, in favour of the newly launched studio, taking the name of SageMaker studio. In Feb 2025, AWS deprecated SageMaker Studio classic. You can only create SageMaker Studio in SageMaker AI.

At the SageMaker level, AWS launched SageMaker Unified Studio, the all-encompassing development environment for data analytics, generative AI, and so on. In this post though, we talk about many features under SageMaker AI and SageMaker Studio. While the service UIs are picturesque, we remain focused on two questions: how these services interact with resources on our VPCs, and how they connect to the Internet.

Workload Categories

We divide machine learning workload into three categories, based on network connectivity pattern: notebooks, model hosting, and pipeline jobs.

Notebooks are where data scientists carry out experiments by running experimental scripts on performing hardwares (depending on the tasks), usually within IDE application as Jupyter Labs, Code Editors. Data scientist users may perform any machine learning related activities such as model evaluation, etc. It is possible that one part of a notebook only requires consumer grade CPU and another part of the notebook program requires a performant GPU. It all depends on the nature of the program code.

Model is the key artifact in Machine Learning workflows. Models themselves are files stored in S3 buckets. The machine learning engineers performs two most common activities. They train the model, and feed the model with unseen data for new output (inference). In simple workflows, data scientists may build, train a model and run inference all from within the same notebook. As the experiment concludes and the team wants to operationalize the inference, it makes senses the run inference in a client-server architecture. This calls for a inference endpoint acting as the server, backed by the trained model, operating on a single or an autoscaling group of instances. The client application feeds the endpoint with unseen data, often using REST API calls, and expects inference results.

Pipeline steps like training do not operate on a server. They are similar to Notebook workloads. The difference is that pipeline steps are headless executions. The steps are non-interactive without engaging the Studio GUI. Many other types of activities in machine learning are in similar pattern, such as model evaluation, model optimization or any general processing such as a Python script. I consider them similar to training activities. Since we orchestrate these headless activities with pipelines (e.g. SageMaker pipeline), and each step may execute on some specialized instance depending on the computing requirements. Collectively, I call these activities the pipeline jobs.

Let’s look at these workloads through the networking lens.

Studio Notebooks

The most common Studio app is some kind of notebooks, such as Jupyper Lab. However, this category can generally include all kinds of SageMaker Studio apps, e.g. Canvas, Code Editor. I use the term Studio app and Studio notebook interchangeably but the APIs mostly refer to these as apps, such as AppNetworkAccessType.

In the app, a user may create one or more spaces each specifying the backing instance type. The configuration that influences the instances’ networking setup is in SageMaker AI domain’s Network Setting. There are two parts of this AppNetworkAccessType setting:

Network Mode (also called AppNetworkAccessType in AWS SDK):
- PublicInternetOnly (default): only EFS traffic goes through the specified VPC and subnets. Other studio traffic (e.g. API calls) goes through the Internet Gateway of the VPC that the studio manages internally
- VpcOnly: all studio traffic goes through the specified VPC and subnets. This delegates the responsibility of connectivity to endpoints to the VPC’s owner.
VPC and Subnet: to place EFS mount points on. Also route other studio traffic in VpcOnly mode.

There are one diagrams on the documentation for each network mode (PublicInternetOnly on the left; VPCOnly on the right):

The diagrams (as of March 2025) are not accurate because the PublicInternetOnly mode also has a domain managed ENI per app space. The VpcOnly mode is when the ML do not like the idea that a Notebook instance can bypass centrally managed Internet path. The team must ensure the endpoints are reachable, either via Internet, or via routable VPC endpoint (e.g. Gateway Endpoint for S3 and Interface endpoint for the rest). In a hub-and-spoke setup it might be another dedicated VPC that provides the interface endpoints centrally.

The domain settings include a few configurations on the underlying instance. For example, SecurityGroupIds specifies the security groups associated with the ENIs. DockerSettings enables Docker daemon on the instance, allowing users to test container workload in local mode. Note that in the more constraint VPC-only mode, Docker pull and push operations outside of Amazon Elastic Container Registry aren’t supported. To pull or push from ECRs users also need to white-list account IDs of the private ECRs in the VpcOnlyTrustedAccounts setting.

In comparison with the other two types of workload, the network traffic for studio notebooks are the easiest to control because they are all configured at the SageMaker AI domain level. Once users with user profiles under a domain creates a notebook, the domain or user profile determines the network mode, and subnet values and security groups. Users themselves cannot change these settings.

Inference Endpoint

The machine learning realm has a few established programming frameworks to host a model file behind an inference endpoint. For example, TensorFlow, PyTorch, Scikit-learn, and even Fast API. Amazon SageMaker AI supports many such frameworks and makes it straightforward. Managing the frameworks requires complex dependency management, a typical use case of containers. Apart from choosing a proper container image, user also selects instance types. These machine learning special purpose instances are pricier than their commodity counterpart. In low-traffic workflows, users may provision inference endpoint on-demand and use it in a controlled time-window, or just use serverless inference endpoint if the model supports it.

When creating inference endpoint, the CreateModel API is used. Under the VpcConfig attribute, two parameters are at play: Network Isolation and VPC-Subnet configuration. I summarize them as below based on the documentation:

Network Isolation	VPC-Subnet Configuration	Description
Disabled	Not specified	SageMaker AI containers are able to access external service and resources on the public Internet; but not able to access resources inside your VPC
Disabled	Specified	SageMaker AI containers communicate with resources inside your VPC through an ENI (Elastic Network Interface). Users are responsible for managing network access to your VPC and Internet.
Enabled	Not specified	SageMaker AI container cannot communicate with resources inside your VPC or on the public Internet
Enabled	Specified	The download and upload operations are routed through your VPC, but the inference (and training) containers themselves continue to be isolated from the network, and do not have access to any resource within your VPC or on the internet.

The Network Isolation option governs the container connectivity option. If we do not expect the inference activity to make outgoing network calls (except for downloading artifact and packages), then we should enable network isolation. On the other hand, if the inference container needs resources on VPC or on the Internet, disable network isolation. Either way, we specify the VPC so that we manage the routing through VPC.

In a compliant networking setup where Internet access must be centralized, the VPC-Subnet configuration must always be configured. The network isolation value depends on the nature of inference workload. However, what seems to be missing in the Studio UI is the activity to enforce that VPC-Subnet configuration is always specified.

As a workaround, we can use the SageMaker Domain’s IAM role to contain such attempt at API level. Below is an example of deny policy:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": [
 "sagemaker:CreateModel"
 ],
 "Condition": {
 "BoolIfExists": {
 "sagemaker:VpcSubnets": "false"
 }
 },
 "Effect": "Deny",
 "Resource": "*",
 "Sid": "DenyModelcreationIfNotOnVPC"
 },
 {
 "Action": [
 "sagemaker:CreateModel"
 ],
 "Condition": {
 "ForAnyValue:StringNotEquals": {
 "sagemaker:VpcSubnets": [
 "subnet-999999999999999",
 "subnet-111111111111111"
 ]
 }
 },
 "Effect": "Deny",
 "Resource": "*",
 "Sid": "DenyModelcreationIfAnySpecifiedSubnetIsNotIntended"
 }
 ]
}

We can use this policy in conjunction with the AmazonSageMakerFullAccess managed policy. The request to create a mode gets denied, either if VpcSubnets are not specified, or they are but not from the preset list of subnet IDs. Once the user selects subnets, corresponding ENIs will get created in the subnets too and user needs to specify security groups for the ENIs.

Enforcing in IAM policy requires that the user who creates endpoint either on SageMaker Studio UI or SageMaker SDK must know the exact subnet IDs as well as appropriate security groups. This requires access to the VPC and can turn into an operation pain point if the users are not well versed with networking. Ideally subnet configuration should also be enforceable at the domain level.

Pipeline Jobs

Most machine learning jobs do not need to function behind an endpoint (i.e. server-side), for example, training, labeling job, model optimization, hyper parameter tuning, etc. In operation, we often use a pipeline to orchestrate these short-lived, non-interactive, headless jobs. Therefore, I simply refer to them as pipeline jobs. They sometimes rely on special purpose instance types. In most cases, they need access to either the Internet or other resources available via customer VPC, a connectivity pattern similar to that of interface endpoints.

To create such resources, SageMaker AI domain user either operate on SageMaker Studio, or program with SageMaker SDK. To make it easy to specify network isolation and subnet configurations, the SDK even has a class for NetworkConfig that can pass to many types of processors (steps).

from sagemaker.network import NetworkConfig

security_group_ids = ['sg-#']
subnets = ['subnet-#']
enable_network_isolation = True

network_config = NetworkConfig(
 security_group_ids=security_group_ids, 
 subnets=subnets, 
 enable_network_isolation=enable_network_isolation
)

script_processor = ScriptProcessor(
 image_uri='my-script-processor-image',
 command=['python3', 'script.py'],
 instance_type='ml.m5.large',
 role=role,
 network_config=network_config
)

The network_config parameter exists as an argument in the creation method of many other resources via SDK. However, the SageMaker UI domain does not have a mechanism to enforce it one way or another. Leaving this option open to users is not what every organization wants either. We could exercise control as much as we can with condition keys such as sagemaker:VpcSubnets in the deny policy for SageMaker IAM role as the example above shows. The Service Authorization Reference document lists out in which SageMaker SDK calls the sagemaker:VpcSubnets condition key (or equivalent) exists. A proactive IAM policy to safeguard all the applicable SDK calls would be helpful as a workaround to the missing enforceability at domain level for SageMaker AI.

Note that on the SageMaker Studio’s Pipeline tool there is a Network configuration seemingly for the pipeline. However, the CreatePipeline SDK call does not have an argument about network configuration. The PipelineDefinition argument requires a JSON format input to define the pipeline configuration and the network configuration is defined per step in the definition.

Another perspective to look at this issue is how we can give a pipeline step (or in general any job runtime) the flexibility to connect to Internet, and in the mean time remain in control of its network connectivity. It depends on the intended security posture but we mainly look at these two questions on the requirement:

can the job runtime access the VPC?
can the job runtime access the Internet on its own path?

Depending on the answer, we can configure a Job in three ways:

	Configuration	Description
a	Neither #1 or #2 are allowed	Most secure but might be overly restrictive because the job runtime may need to download artifacts. This requires enabling Network Isolation and specify VPC-Subnet configuration.
b	Either #1 or #2 is allowed, but not both at the same time.	Exclusively allowing #1 is more secure because the VPC can manage access to Internet for the job runtime. Exclusively allowing #2 breaks the central Internet access pattern and should not be allowed if central Internet access is a compliance requirement.
c	Both #1 and #2 are allowed at the same time	This configuration should not be made possible due to exfiltration risk.

This classification of network configuration for ML pipeline job, is quite similar to DevOps pipeline job agent (think of Azure DevOps agent or Terraform agents). The user may use service provider’s agent which come with its own Internet access, or choose to self-host the agent to allow access to VPC but the VPC’s owner is then responsible for managing Internet routing through the custom VPC.

SageMaker AI makes c impossible, which is good. SageMaker administrator needs to evaluate the requirement between a and b and determine how to enforce it with IAM policy.

Summary

As an infrastructure security specialist, I investigated networking options in SageMaker AI. When any user is performing any task in SageMaker AI, I am concerned with two questions:

How does the instance behind the endpoint connects to the internet and to our VPCs;
How do we enforce the connectivity pattern (enforceability);

For the studio notebooks workload, the control is through SageMaker domain. For Inference endpoint and pipeline jobs, the control is reactive. We use IAM policies and role to deny attempts to create resources with bad network configuration. The limitation is that the SageMaker users have to know what network configuration options are available.

As a result, the infrastructure security specialist must clearly define the required network configuration, and communicate it out to SageMaker AI users.

The Leanest Web and Email Hosting

Thu, 27 Feb 2025 22:14:11 -0500

This site has been quiet for a while. During this time, I migrated the hosting platform again, and refactored email solution. This post, is another note about how I finally came to the most cost-effective web and email solution for a small business, with a solid security posture, and at nearly no cost.

For background, I have had this website for more than five years now, and revamped it a couple times. I have been with Amazon Lightsail at an already low cost. I didn’t have a lot of emails. So I have been relying on the email forwarding feature from my old domain registrar. There have been a few challenges. The content has grown a lot. The speed gradually slows down. WordPress needs reinstall. The outgoing emails get flagged as unverified.

WordPress Hosting

Worried about too many plugins, I debated about changing content management platform. Unfortunately, it’s not easy. There are some alternatives but none has all the features from all the plugins that I’ve been using. Even if they do, it would be too much work: I’d have to migrate the code snippets, highlights, etc, find similar themes and verify the new theme and new plugins work together. Despite of the criticism, WordPress today is still a predominant content management platform. So I stick to it.

Staying with WordPress isn’t maintenance-free. When PHP version has major update once in a while, I still had to re-install and migrate to a new server. Since I’ll exercise migration anyways, why not also shop around for a new virtual private server. Surprisingly I learned that Google Cloud Platform has an always-free tier including a decent size virtual machine.

	Amazon Lightsail 7USD/mo	GCP Free tier
Class	micro	e2-micro
CPU	2 vCPUs	2 vCPUs
Memory	1 GB	1 GiB
Disk	40GB SSD	30 GB (standard persistent disk HDD)
Data Transfer	1TB	1GB egress

The vCPU in different platforms do not represent exactly the same compute capacity, but should be generally comparable. The egress data transfer usage was below 1GB on Lightsail so the new service is sufficient. By going with GCP’s free tier, the only downgrade is the disk speed.

The GCP platform provides Bitnami package image and it’s fairly simple to spin up a new VM. The setup is as simple as Amazon Lightsail and I don’t have to deal much with networking. It doesn’t work out to be completely free but the net cost is negligible. I also picked the Nginx-based WordPress image because I had wasted a lot of time dabbling with Apache which I never use elsewhere.

WordPress Migration

With the new server up in a few minutes the next step is migration. The All-in-One WP Migration and Backup plugin (by ServMask) is the Swiss army knife. There are some pitfalls though.

First, the plugin has a size limitation in the free version. Luckily I can bypass it with changes to the PHP and Nginx configurations. For PHP configuration, change the line in /opt/bitnami/php/etc/php.ini, to upload_max_filesize = 512M. For the other change on Nginx as the reverse proxy, in /opt/bitnami/nginx/conf/nginx.conf, under http configuration, add:

client_max_body_size 512M;

The second pitfall is trickier. I have to temporarily host the new site, either by an public IP, or by a different DNS name (such as new.digihunch.com), so I can log in and verify the functions. Once I log on to the new site, the new DNS name or IP will start to be written to the database. I’m not sure which exact service does that. But once I put the new site on the production domain name (i.e. www.digihunch.com), the database still contains many entries referencing the temporary DNS name. This causes many insidious broken links.

There is a plugin for this kind of issue: Better Search Replace by WP Engine. The plugin goes through all tables in the database and allows you to find and replace strings. It also supports dry-run mode and I managed to fix tens of thousands of bad references. In addition, I managed to change username using a plugin called Easy Username Updater.

Another important activity after migration, is to ensure the MFA by WordFence plugin continues to work, and re-scan the entire file system.

CloudFlare Security

With less egress data and an HDD, the VPS in GCP is slower. I need to speed it up with CDN. It is hard to resist the freebies by CloudFlare. I’ve already been using it for security. Now it’s time to dig deeper.

CloudFlare greatly simplifies TLS certificates. It manages your certificate automatically. On the other side, it issues an origin certificate (and key, with long expiry) for communication with origin server. In full encryption mode, all I need to do is deploy origin certificate to Nginx. There is no manual renewal effort.

Note this setup requires CloudFlare to manage DNS so I just transferred my domain to CloudFlare. Therefore I consolidate domain registrar, certificate vendor and CDN to one vendor. CloudFlare can manage DNS record in proxy mode, to hide the public IP of the origin server from clients. I can also control the security group of the origin server to only open port 443 to CloudFlare IPs.

Another annoying but necessary configuration is the redirect of zone apex and http requests. Specifically most websites needs the following redirects:

If request from client is http://www.digihunch.com, enforce https;
If request is https://digihunch.com, redirect to https://www.digihunch.com;
If request is http://digihunch.com, enforce https, then redirect following rule #2.

I used to manage the these redirects in the reverse proxy (Apache). Now that I changed to Nginx, I offload these settings to CloudFlare, instead of configuring another reverse proxy again. For #1, I simply turned on “Always use HTTPS” and “Automatic HTTPS Rewrites” under edge certificates. For #2, I need a CNAME for zone apex to alias to www, along with a redirect rule. As a result, there is no need to ever open port 80 on the VPS, just to let the reverse proxy redirect URL.

CloudFlare Cache Rules

As to caching, I had to turn on a couple Cache rules to enable the caching. Cache hit was at about 50% and there are rooms to go higher if I enable reserve cache. Once cache is on, there are other considerations, such as expiring the cache, and by passing the cache during server deployment. CloudFlare allows you to purge cache by rules. There is also a Development Mode that temporarily allows you to bypass all cache for testing. With cache enabled, and the WordPress plugin activated, the page load time for recent posts appear to be faster than before.

One challenge is that the admin bar of WordPress went into the CDN and is served to visitors. To get rid of this behaviour I have two catch rules:

A cache-everything rule at order 1, to make everything eligible for cache;
A bypass rule at order 2 to bypass cache conditionally. The condition expression reads:

(http.cookie wildcard "wp-.*") or (http.cookie wildcard "wordpress_logged_in_*") or (http.cookie wildcard "wordpress.*")

To make use of the cache space. It is important to have small image sizes. Last year I added many AI generated feature images, most of which are 2MB in PNG format. I have to convert many PNG images into webp format, which only takes a fraction of the space and is widely accepted by most browsers today.

Sometimes there are other annoyances when it comes to speeding up page loading. For example, I noticed that with integration between Google Tag Manager and Microsoft Clarity, the home page issues a call to https://www.clarity.ms/tag/tag-id-xyz. This call blocks the rendering of a big chunk of home page by a minute! I didn’t notice it because my Brave browser’s ad-block removes that call. Nonetheless, the issue has been there for any visitor without built-in ad-block in their browser. The lessons learnt is that you always test with standard browsers, or at least in Brave browser with shields down.

SMTP service

I have been using email forwarding for incoming emails, and Gmail’s feature to send email from a different address or alias, for outgoing emails. This feature works but oftentimes the email gets marked as unverified on the recipient side. That gives it a good chance to appear as spam.

I need similar email forwarding mechanism and sending service. The free tier of CloudFlare goes a long way. Receiving is simply about creating Email routing rules, with a few MX and TXT records on DNS created and managed by CloudFlare. These records ensure CloudFlare routing rules captures all emails coming to the domain.

For outgoing emails, I have to get rid of the alias-based mechanism by Gmail. I need a proper SMTP server but CloudFlare doesn’t offer one. Resend seems like a popular choice with a good amount of free tier usage. However I’m hesitant to introduce another single-purpose platform so I just resort to Amazon SES for a full-feature, low-cost SMTP service. I do have to request production access, stating the server is for transactional email. This allows the SMTP service to arbitrary recipient address.

The other benefit of having my own SMTP server is to enable email integration in WordPress, which enables other important features such as web form, WordFense security alert and admin password reset by email.

Sending Reputation

To keep outing email from being marked as spam, it is important to understand what mechanisms are at play (SPF, DKIM, DMARC) to maintain sending reputation. At minimum, we configure SPF, DKIM and DMARC. SPF (Sender Policy Framework) identifies which mail servers are allowed to send mail on behalf of your custom MAIL FROM domain through a DNS TXT record that is used by DNS. The receiving mail server checks the SPF record of the sender’s domain to see if the email came from an authorized server. For example, when I use the SMTP server by Amazon SES, and customize the MAIL FROM field to mail.digihunch.com, I need an MX and a TXT record for mail.digihunch.com to tell receivers that it authorized amazonses.com to send email on its behalf.

DKIM (DomainKeys Identified Mail) is an email authentication method used to verify that an email message was sent by an authorized mail server and that the message content hasn’t been altered in transit. When an email is sent, the sending mail server generates a unique digital signature (based on the content of the email) using a private key. This signature is added to the email header. DNS Lookup: The receiving mail server looks up the sender’s domain in DNS to find the corresponding public key for DKIM authentication. Amazon SES configuration requires three TXT records to store the sender’s key information.

With SPF and DKIM configurations, the Amazon SES setup complies with DMRAC authentication protocol, making it less likely to be flagged as spam. The email comes off as mailed by amazonses.com and signed by digihunch.com.

In addition, there is Brand Indicators for Message Identification (BIMI) protocols that enables logo for your email on the recipient’s inbox. I managed to set up BIMI with SES, without Verified Mark Certificate (VMC), a protocol that requires evidence of ownership of the logo.

Although SPF, DKIM and DMARC cannot guarantee the email is not marked as junk, this is the best thing you can do. There are other techniques with diminishing return on effort.

Summary

This post summarizes my journey to land on the leanest web and email hosting solutions for small business. After five years of improvement, the tech stack that brings me low cost, solid security posture and lightening-fast speed, consists of: virtual machines from Google Cloud Platform (or Lightsail from AWS), web security and CDN by CloudFlare with Email supported by Gmail and Amazon SES (or Resend).

Initially with GCP, I tried to stick to the free tier but I did notice the standard persistent disk struggling, especially when I run system scan from WordFence. To overcome that I upgraded the disk to balanced disk (SSD backed) out of the free tier. There might be even cheaper alternatives but the marginal value isn’t worth the effort. Also, I configured the caching in CloudFlare for read, adjusted the home page, and schedule IO-intensive activities such as bi-weekly snapshot to quiet hours. The combination gave the site a 100 score in the CloudFlare speed testing.

Firewall Deployment Patterns

Sat, 16 Nov 2024 16:24:15 -0400

The Hub-and-Spoke topology is the most common topic in the discussion for building cloud infrastructure design. This topology appeared in both AWS and Azure design papers and had been around as a very important option in physical networking design.

The AWS whitepaper Building a Scalable and Secure Multi-VPC AWS Network Infrastructure has thorough discussion on the topology. This topology often feature a Transit Gateway as the hub. In addition to workload VPCs, the network topology often includes some special-purpose VPCs, such as interface endpoints VPC, or shared tooling VPCs. One of the special-purpose VPC is the inspection VPC. It is a key design area to suit the need of inspection and traffic management for the business and the design may vary a lot depending on the available inspection tools such as a Firewall appliance.

Inspection Requirements

The most common situation with an enterprise is connecting with on-prem networking. Options including Direct Connect, site-to-site IPsec or SD-WAN overlay. The business decides whether and at what level they would like to inspect the traffic between on-prem and their VPCs. Here is an example.

Connectivity	Inspection Requirement
Between Workload VPCs (East-West)	No inspection
Between a workload VPC and a special-purpose VPC	Normal Inspection
Ingress Traffic from Internet to Workload VPC	Deep Packet Inspection
Egress Traffic from Workload VPC to Internet	Deep Packet Inspection
Between Workload VPC and on-prem networking over Direct Connect	Normal Inspection
……

With a normal inspection, the firewall appliance only checks the information in the packet’s header, such as the source and destination IP addresses, port number, etc. With deep packet inspection, the appliance examins a larger range of metadata as well as the data in each packet. DPI provides a more effective mechanism to perform network packet filtering and find otherwise hidden threats. It is however an expensive operations from a performance standpoint. Ultimately the business makes the call but it is important to identify ALL connectivity scenarios in this phase and explicitly document the decision and rationales. They can choose from an NGFW product or the Network Firewall service from AWS, depending on capability required.

Inspection Architecture

At minimum, inspection is required for ingress and egress traffic to and from workload VPC. The design must account for both routing and inspection. Many would use the same VPC for ingress/egress traffic and for inspection.

It is also possible to separate these two purposes into two different dedicated VPCs: an inspection VPC that hosts firewall services or appliances, and an ingress/egress VPC that directs traffic from and to the Internet but we must route the traffic to the inspection appliance. If all traffic to be inspected has to be routed through the Transit Gateway both ways, the cost would be high. In 2020 AWS introduced Gateway Load Balancer (GWLB) to address this use case.

The recommended pattern using GWLB allows you to place firewall appliance and a GWLB in one VPC, and place the GWLB endpoint (GWLBE) in a different VPC. The connectivity between GWLBE and GWLB is backed by HyperPlane, a technology that also enables other endpoint service such as PrivateLink. The connectivity between GWLB and the appliance take place with Geneve encapsulation. This pattern places any appliance behind an endpoint, so long as the appliance supports Geneve protocol.

The GWLB technology enables a number of inspection patterns based on distributed ingress paths, as summarized in this document. Distributed ingress/egress means each workload VPC can have their own Internet Gateway and NAT gateways. They must configure their route table so as to send the traffic via GWLBEs to inspection appliances. In general, I recommend this pattern over the centralized ingress/egress patterns where only the inspection VPC can take ingress traffic from Internet Gateway. The Network architectures for ingress traffic inspection presentation from 2021 ReInvent covered this topic as well, especially about the scaling benefit of distributed ingress.

Firewall deployment patterns

The firewall deployment pattern available differ between vendors and the requirements. Since the GWLB pattern places appliances behind the GWLB, the appliances rely on Geneve traffic that GWLB forward over. Some vendors may argue that this pattern keeps the NGFW product from performing other tasks that do not support Geneve traffic. One example is Network Address Translation. The native NAT gateway services is very expensive (consider fck-nat as an alternative for NAT). Many clients want to use the NAT feature of the NGFW product. The architecture therefore has to be adjusted in favour of centralized egress. Review this post about one-arm mode and two-arm mode.

If we have to go with central ingress/egress anyways, there are still numerous options. Take FortiGate for example, while the GWLB pattern of deployment is supported, other available options include:

Traditional pattern with multiple interfaces across different subnets in the inspection VPC (L3 mode)
Integration with Transit Gateway using Transit Gateway Connect Attachment
Integration with Transit Gateway using Transit Gateway VPN Attachment

I regard the first option as traditional because it does not directly integrate with Transit Gateway and it is very similar to how we deploy them in a physical networking environment. Fortigate refers to it as L3 (NAT/route) mode. In this mode the Firewall appliance can also influence network routing.

The second and the third options are similar except for different types of Transit Gateway attachments are used. The reason to directly integrate with Transit Gateway is so that the Transit Gateway can route the traffic for inspection therefore no need for a Gateway Load Balancer, and thus no dependency on the firewall features supporting Geneve.

The second option builds a GRE (Generic Routing Encapsulation) tunnel over a Transit Gateway Connect attachment as the transport tunnel, and uses BGP to exchange routes between the Transit Gateway and the appliance. It treats the firewall instances as SD-WAN appliance and has performance benefit. The third option uses VPN attachment with the main benefit of encryption if it is part of compliance requirement.

Rules

The rule configuration for Firewall is critical to the operation of the entire multi-VPC network configuration. Unfortunately, there is no standard with the rule syntax across majore NGFW vendors, leading to challenges for customers to swap vendors. Most flavours of rules have common elements such as

Action (Pass, drop, or alert)
Source and Destination
Protocol and Port

A very common open-source firewall rule syntax is the Suricata-compatible format. One important adopter is the AWS Network Firewall, which supports both stateful and stateless rule groups. With stateful rule group, there are two options for how the Suricata engine evaluates rules.

With “Action Order” option, Suricata engine evaluates the rules in the order of: pass, drop, reject and alert. You can use the priority attribute to influence evaluation;
With strict order, the rules are evaluated in the order of the rule definition;

It is important to be aware of the rule evaluation order since it impacts the firewall behaviour deeply.

Summary

In the networking infrastructure design, ingress and egress routing are the most critical one-way door decision. This decision must account for both routing and inspection. While there are many options, we usually start with capturing the key requirements. In this post we reviewed how to approach the requirement, a key technology Gateway Load Balancer and some firewall deployment patterns with FortiGate as an example. The approach is similar for other NGFW vendors, such as Palo Alto, Check Point or Cisco Secure Firewall.

Cloud Certifications for Learning?

Wed, 09 Oct 2024 16:03:29 -0400

Certification count is a measurement of capability of a consulting practice. At individual level it sometimes serves to navigate the learning journey, especially for engineers aspiring to advance to an adjacent technical area, for example, a network engineer moving to cloud engineer, or a data engineer transitioning to machine learning engineer. Personally I don’t find certifications the most efficient way to learn. However, it is the most measurable way. The market has a plethora of choices for certifications but the quality is not very consistent. Through years of studying for different certificates, I have learned to treat certificates as necessary evil and be extremely cautious before investing time to it. After I sat another exam yesterday, I decided to write down my views of certifications and alternative ways of studying a topic in cloud technologies.

Dosage of Marketing

A few years ago, I sat the Azure exams on AI fundamentals (AI-900) and AI engineer associate (AI-102). Out of the two I found the fundamentals one better, despite of lower level. That is because AI-900 focuses on the core concepts that are broadly useful. In contrast, the AI-102 exam, at least at that time, felt like a marketing exam all about what managed AI services Azure has for what kinds of problems. I’m not a fan of marketing exams but admittedly all these exams by cloud service providers contains some dose of marketing. You just can’t completely avoid marketing from these exams. It’s just a matter of how much. My first principle: be wary of the marketing scheme behind the exam.

Why is this important? We choose an exam because we want to dedicate precious time to learn. Unfortunately, the time we, aspiring IT engineers starting to study a new topic, is the time we are the most vulnerable to marketing schemes. Not only are we eagerly open to devouring the course contents, we’re also laser-focused and repeating a lot. This tunnel vision while cramming the exam allows biases to sneak in and reinforce in favour of the product names in the exam.

If you’re about to build a container platform and your first flash of thought is Elastic Container Service, you should congratulate Amazon on the marketing win. If your decision process relies too much on these mental shortcuts that the exam study has covertly built, you are developing a tendency of not being analytical to engineering problems. This makes you a worse engineer than you could otherwise have been.

A wise learner has be cautious about the dosage of marketing in these exams. One indicator of high dose of marketing, is the portion of questions focusing managed and unique services in the particular cloud service provider. The certifications by non-profit organizations such as CompTIA and Linux Foundations tend to have low dose of marketing in the exams.

Stability of Topics

The time studying an exam is an investment. We want the certification to remain stable over the course of several years. New certification exams should only launch under rigorous review of the necessity, maintainability and the direction should not change drastically. Unfortunately, this is not always the case. Let me pick on AWS specialty exam this time. If you have been in the certification game for long, you might remember the following specialty certifications:

Big Data specialty — launched in Oct 2016, and deprecated in Apr 2020 in favour of Data Analytics specialty and Database specialty
Data Analytics specialty — launched in Apr 2020, retired in Apr 2024
Database specialty — launched in Mar 2020, retired in Apr 2024
SAP on AWS specialty — launched in Apr 2022, retired in Apr 2024
Alexa Skill Builder — launched in May 2019, retired in Mar 2021

These changes are a main reason I am very hesitant to go for specialty certifications with AWS. Some even had a lifespan of less than two years. In my opinion, learning is worth it but spending time on those short-lived certificates is a completely waste of time. In Lindy effect, the longer a non-perishable item has been around, the longer it’s likely to persist into the future. Technologies such as TCP/IP, Unix, Object Oriented Programming, have been around for decades and yet still profoundly impact a production system today. Only lasting knowledge is worth more time.

As of spring 2024, AWS introduced two certificate level exams: Data Engineer Associate and Machine Learning Engineer Associate. So it seems that some specialty certifications are being consolidated into the associate-level. All previous associate-level exams had been around for more than a decade. Given that record of stability, I decided to venture the AWS Certified Machine Learning Engineer Associate certification.

Breadth and Depth

The next important factor to consider is the breadth and depth of certification exams.

Some certifications are targeting different audiences. For example, I personally regard the AWS Certified Cloud Practitioner as targeting non-technical folks and would not carry much weight for an experienced cloud engineer. On the other hand, I chose AWS Certified Machine Learning Engineer Associate over the Machine Learning Specialty because the latter targets data scientists and is a lot more demanding on theories. Even if I grind it out there wouldn’t be opportunity in my day-to-day responsibility to digest and reinforce what I have jammed in my head.

I’m dubious on certifications for a single tool but there seems to be more and more of them these days. The only exception is if the tool plays such a crucial and fundamental role in the IT systems, and it is complex but there is no alternatives. For example, Red Hat as a classic distribution of Linux operating system, Kubernetes as a complex container orchestration service. However, certifications on single tools such as Prometheus, Istio, Cilium, and Terraform, are just too narrow to worth the effort. In my opinion these certifications provide very limited return on investment. We just need to learn these tools as we go with documentations, or go through the essential workshops.

Upkeep

Renewal requirement is another consideration, especially if the organization requires your active certification status. Having a streamlined renewal process is a big plus because once certified our main duty is on the actual work. Kudos to the Azure certifications that only require a shortened free on-line exam for renewal. It is quite a hassle to have to retake the exam again. The validity of the certification is also subject to arbitrary changes. For example in 2024 Linux Foundation changed the validity of CKA exam from 3 years to 2 years.

Certification inflation is real. There might have been a time when they measure one’s competency in the technical field. But today very few of them can differentiate one’s technical competency, thanks to Goodhart’s law. In addition to the certification inflation, another threat to the value is the question leaking. The longer the exam has been administered, the more likely the questions are leaked all over the place. This gives an unfair advantage for users of question dumps. And because the scores are scaled results, this systematically penalize honesty, and ultimately impairs the value of the certification.

More ways of learning

Last year I had a number of certifications all up for renewal, I gave up many of them that are time-consuming. They did help with learning but only up to certain extent. Unfortunately, so long as I work in consulting I still have to live with it. As a result, I learned to be very picky about which ones to pursue to get the most value of my time.

At the end of the post, I must reiterate that I don’t personally find certifications are best way to learn. In the preparation, I always tend to focus on acing the exam itself. Over the years, I’ve also used a few other ways in my learning, and can summarize them as such:

certification exam: most measurable but least effective in my experience.
online courses: highly dependent on the quality of the individual course. Low-quality ones are just workshops. The best ones walk you through a classic project to fill the knowledge gap.
workshops: my preferred way going into a new topic. Vendor managed ones may over-emphasize on tools and under-emphasize on concepts.
side projects: most effective but requires a real-life problem and can be time consuming
quality white paper: an effective way of learning going into new topic in a systematic way
read a book: as long as the book content is up to date, this is the most systematic approach
blogging: effective but time-consuming
talk: to someone using the technology hands-on, free of fluff

I hope every cloud engineers can also choose certifications wisely and maximize their leanring.

Debating between count and for_each in Terraform

Tue, 27 Aug 2024 22:36:39 -0400

In Terraform, we often have to create an array of resources of the same type but similar attribute values. For code reusability, manageability and for DRY principle, it’s better to use loop. Terraform HCL supports loop via the use of meta-argument. Currently, there are two options to drive a loop: count and for_each .

Problem with `count` loop

The book Terraform Up and Running (Chapter 5 Terraform Tips and Tricks) regards count as Terraform’s oldest, simplest and most limited iteration construct. One of the big limitations is the shifting of index if the length of resource array changes. The point comes with a good example:

variable "user_names" {
 description = "Create IAM users with these names" 
 type = list(string)
 default = ["neo", "trinity", "morpheus"]
}
# The Example from the book Terraform Up and Running
resource "aws_iam_user" "example" { 
 count = length(var.user_names)
 name = var.user_names[count.index]
}

As you execute Terraform apply, three IAM users will be created, with the plan looking like:

# aws_iam_user.example[0] will be created
+ resource "aws_iam_user" "example" {
 + name = "neo"
 (...) 
 }
# aws_iam_user.example[1] will be created
+ resource "aws_iam_user" "example" {
 + name = "trinity"
 (...) 
}
# aws_iam_user.example[2] will be created
+ resource "aws_iam_user" "example" {
 + name = "morpheus"
 (...) 
}

Then if you remove “trinity” from the variable user_names, and run terraform plan, the plan would look like:

Terraform will perform the following actions:
 # aws_iam_user.example[1] will be updated in-place
 ~ resource "aws_iam_user" "example" {
 id = "trinity"
 ~ name = "trinity" -> "morpheus"
 }
 # aws_iam_user.example[2] will be destroyed
 - resource "aws_iam_user" "example" {
 - id = "morpheus" -> null
 - name = "morpheus" -> null
 }
Plan: 0 to add, 1 to change, 1 to destroy.

In this plan, instead of deleting the second user, it renames the second user and deletes the third user. While the plan matches the code logic, it is often an unwanted result, considering the resource could be one that many other resources depends on, such as a subnet.

This is a good example of the problem with count. Terraform identifies each resource in the generated list of resource by position(index) . When the length changes, the index shifts. If you remove an item from the middle of the list, Terraform will delete every resource after the deleted item, then re-create all the resources that come after the deleted one. As a consequence, you may loose availability or even worse, lose data.

Embrace `for_each` loop

If we modify the example above to use for_each, the code looks like:

variable "user_names" {
 description = "Create IAM users with these names" 
 type = list(string)
 default = ["neo", "trinity", "morpheus"]
}
# The Example from the book Terraform Up and Running
resource "aws_iam_user" "example" 
{ 
 for_each = toset(var.user_names) 
 name = each.value
}

This results in the creation of three IAM users. If you remove the “trinity” user from the middle of the input collection and apply, the plan looks like this:

Terraform will perform the following actions:
 # aws_iam_user.example["trinity"] will be destroyed
 - resource "aws_iam_user" "example" {
 - arn = "arn:aws:iam::123456789012:user/trinity" -> null
 - name = "trinity" -> null
 }
Plan: 0 to add, 0 to change, 1 to destroy.

The plan suggests that Terraform will delete the very resource that was taken out from the middle of the input collection and no existing resources in the array are impacted.

Note that in the code snippet above, we use function toset() to convert the input list to a set (ordered and de-duped list of string). This is because we can only loop over a set or map when creating an array of resource. If the array of resource being created have another attribute whose value needs to be individualized, we can loop over a map and store the individualized attribute values as key-value pairs.

A few pages down, the book discusses an important limitation for both count and for_each. The length of the resource array that you are creating with count or for_each meta-argument must not be computed from other resources. Terraform must be able to compute count and for_each during the plan phase, before any resources are created or modified. The length of the resource array can be from hardcoded values, data sources, or even a list of other resources to create in the same file, so long as the length can be determined during the plan, instead of not being computed from other resource outputs.

A real-life example with classic pattern

The book then touches on another advantage of for_each: the ability to create multiple inline blocks within a resource. The guide from Hashicorp documentation also has a section on when to use for_each Instead of count, with a similar example. The section merely mentions when to use count in the opening sentence: If your instances are almost identical, count is appropriate.

That makes for_each sound like a no-brainer, after reading all the literatures about this topic. In my experience with a specific use case at the beginning, count feels more efficient. The example from the book is too simplistic. To better compare the two options, I need a realistic example. Let’s consider this use case where, after creating a VPC, I need to create the followings:

one NAT gateway for each availability zone (each NAT Gateway maps to one subnet and one allocation ID)
one public subnet for each availability zone
one public IP allocation in each availability zone

We can summarize the relationships between resources in the following diagram:

I deliberately pick this example because they are self-contained. So are all code snippets in this post. The example also demonstrate a classic relation between resources that we can find everywhere in infrastructure automation. Here’s another example off the bat:

create an array of aws_subnet, each has a subnet_id attribute;
create an array of aws_route_table, each has a reout_table_id attribute;
now, create an array of aws_route_table_association, each referencing one aws_subnet (by subnet_id) and one aws_route (by route_table_id);

If we address the NAT gateway example, we’re good with many other resources that shares the same relation pattern. In the next section, we’ll first implement the NAT gateway example, using count loop.

Implementation using `count`

The use case exemplifies the pattern where we have multiple types of resources related to each other. We need a loop in each type of resources, resulting in multiple arrays of different resource types. Moreover, the elements in the array for aws_nat_gateway has 1-to-1 mappings with both the array for aws_subnet, and the array for aws_eip.

With count, I created Terraform code with everything in a single main.tf file for the convenience of illustration, like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}
variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"]
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"]
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

resource "aws_subnet" "public_subnets" {
 count = length(var.public_subnets_cidr_list)
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = var.public_subnets_cidr_list[count.index]
 map_public_ip_on_launch = true
 availability_zone = data.aws_availability_zones.this.names[count.index]
}

resource "aws_eip" "nat_eips" {
 count = length(var.public_subnets_cidr_list)
}

resource "aws_nat_gateway" "nat_gws" {
 count = length(var.public_subnets_cidr_list)
 subnet_id = aws_subnet.public_subnets[count.index].id
 allocation_id = aws_eip.nat_eips[count.index].id
 depends_on = [aws_internet_gateway.internet_gw]
}

The intent is to create subnet, public IP and NAT gateway for two availability zones. I also want to add one more AZ in the future and have the code to handle the addition gracefully. To add the new AZ, I uncomment line 10 and comment out line 9. The plan after this code change looks like this:

Terraform will perform the following actions:

 # aws_eip.nat_eips[2] will be created
 + resource "aws_eip" "nat_eips" {
 (...) 
 }

 # aws_subnet.public_subnets[2] will be created
 + resource "aws_subnet" "public_subnets" {
 + availability_zone = "us-east-1c"
 + cidr_block = "147.206.8.0/22"
 + id = (known after apply)
 (...) 
 }

 # aws_nat_gateway.nat_gws[2] will be created
 + resource "aws_nat_gateway" "nat_gws" {
 + allocation_id = (known after apply)
 + subnet_id = (known after apply)
 (...) 
 }

Plan: 3 to add, 0 to change, 0 to destroy.

The plan creates a set of resources required for the new availability zone without touching any existing resource, which is expected.

Why do I only focus on the use case of adding a new subnet in new AZ, and not deleting or modifying CIDR on an existing subnet? That’s because we rarely do that with production. We rarely remove the use of an availability zone. Nor do we modify the CIDRs on an existing subnet. In fact, AWS SDK does not even have an API to change CIDRs on a subnet or a VPC. In our infrastructure operation, we make such decisions upfront so they are immutable once provisioned. We simply don’t need to consider all the possible CRUD actions on a resource.

So, the count loop does just the job. Now, what about for_each?

Implementation with `for_each`: first attempt

Since for_each takes a set or map, I have to make some adjustment. My first attempt looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

locals {
 subnet_config = [
 for i in range(length(var.public_subnets_cidr_list)) : {
 cidr = var.public_subnets_cidr_list[i]
 az = data.aws_availability_zones.this.names[i]
 }
 ]
}

resource "aws_subnet" "public_subnets" {
 for_each = { for idx, rec in local.subnet_config : idx => rec }
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
 tags = { Name = "PUBLIC-SUBNET" }
}

resource "aws_eip" "nat_eips" {
 for_each = toset(var.public_subnets_cidr_list)
 tags = { Name = "NATEIP" }
}

data "aws_subnets" "public_subnets" {
 filter {
 name = "tag:Name"
 values = ["PUBLIC-SUBNET"]
 }
 depends_on = [aws_subnet.public_subnets]
}

data "aws_eips" "nat_eips" {
 filter {
 name = "tag:Name"
 values = ["NATEIP"]
 }
 depends_on = [aws_eip.nat_eips]
}

locals {
 nat_gw_config = [
 for i in range(length(var.public_subnets_cidr_list)) : {
 subnet_id = data.aws_subnets.public_subnets.ids[i]
 alloc_id = data.aws_eips.nat_eips.allocation_ids[i]
 }
 ]
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = { for idx, rec in local.nat_gw_config : idx => rec }
 subnet_id = each.value.subnet_id
 allocation_id = each.value.alloc_id
 depends_on = [aws_internet_gateway.internet_gw]
}

Note that I have to create a couple of data resources (nat_eips and public_subnets) and local variables (subnet_config and nat_gw_config) in order build the required map data structures and feed them to the for_each parameters.

After Terraform apply, let’s edit public_subnets_cidr_list with the additional subnet CIDR for the 3rd AZ. The plan looks like this:

Terraform will perform the following actions:

 # data.aws_eips.nat_eips will be read during apply
 # (depends on a resource or a module with changes pending)
 <= data "aws_eips" "nat_eips" {
 + allocation_ids = (known after apply)
 (...) 
 }

 # data.aws_subnets.public_subnets will be read during apply
 # (depends on a resource or a module with changes pending)
 <= data "aws_subnets" "public_subnets" {
 + id = (known after apply)
 + ids = (known after apply)
 (...) 
 }

 # aws_eip.nat_eips["147.206.8.0/22"] will be created
 + resource "aws_eip" "nat_eips" {
 + allocation_id = (known after apply)
 (...) 
 }

 # aws_nat_gateway.nat_gws["0"] must be replaced
-/+ resource "aws_nat_gateway" "nat_gws" {
 ~ allocation_id = "eipalloc-0ffe32519d20b9b7f" # forces replacement -> (known after apply) # forces replacement
 ~ subnet_id = "subnet-0b4b202056c75bb0a" # forces replacement -> (known after apply) # forces replacement
 (...) 
 # (1 unchanged attribute hidden)
 }

 # aws_nat_gateway.nat_gws["1"] must be replaced
-/+ resource "aws_nat_gateway" "nat_gws" {
 ~ allocation_id = "eipalloc-0ff1d32fb271b5cf4" # forces replacement -> (known after apply) # forces replacement
 ~ subnet_id = "subnet-09b6486fbe44e9795" # forces replacement -> (known after apply) # forces replacement
 (...) 
 # (1 unchanged attribute hidden)
 }

 # aws_nat_gateway.nat_gws["2"] will be created
 + resource "aws_nat_gateway" "nat_gws" {
 + allocation_id = (known after apply)
 + subnet_id = (known after apply)
 (...) 
 }

 # aws_subnet.public_subnets["2"] will be created
 + resource "aws_subnet" "public_subnets" {
 + availability_zone = "us-east-1c"
 + cidr_block = "147.206.8.0/22"
 (...) 
 }

Plan: 5 to add, 0 to change, 2 to destroy.

Wait a second, I expect the template to create a new subnet, a new elastic IP and a new NAT gateway in that new AZ. But why does it plan to delete the two existing NAT gateways and recreate two? This doesn’t make for_each an appealing option at all.

Apart from the interruptive plan, there are also other problems. First, since the aws_subnet resources requires cidr_block and availability_zone values, I have to build a map (subnet_config) for its resource array to consume. Similarly, I have to build a second map (nat_gw_config) to create resource array for aws_nat_gateway, which requires subnet_id and allocation_id. This map takes more work to build. Because of the 1-to-1 relationship between subnet_id and alloc_id, I have to fetch the values from two data sources (line 47-61), use a common index (line 63-70). Can I neat it up and combine two maps into one? Not really. Because the second map (nat_gw_config) uses a data source depending on the subnets, which depends on the first map (subnet_config). Trying to combine the maps causes circular dependency!

Also, the additions of data sources makes the code less readable. As Marcel L pointed out in his post, two cons with for_each are: complexity and requiring a map (to store multiple attribute values). Now we seem to have one more: it may cause unintended deletions

Is `for_each` a bad idea?

Let’s find out why for_each could destroy two existing NAT gateways.

Notice that I built the map nat_gw_config by looping through the list of variable public_subnets_cidr_list. After apply, we appended it one more string at the end, without changing the existing order. However, the devil lies in the order of the string lists returned from the data sources. By printing this map, we found that the originally value before the AZ addition is:

index	alloc_id	subnet_id
0	eipalloc-0ffe32519d20b9b7f	subnet-0b4b202056c75bb0a
1	eipalloc-0ff1d32fb271b5cf4	subnet-09b6486fbe44e9795

Based on this, NAT Gateway with index 0 is created with eipalloc-***b7f and subnet-***b0a. NAT Gateway with index 1 is created with eipalloc-***cf4 and subnet-***795. After we add the third AZ, and apply the run, the new map, with a new alloc_id and a new subnet_id looks like this:

index	alloc_id	subnet_id
0	eipalloc-0b70460721596e33f (new)	subnet-0b4b202056c75bb0a
1	eipalloc-0ffe32519d20b9b7f	subnet-0335071ced2dc9922 (new)
2	eipalloc-0ff1d32fb271b5cf4	subnet-09b6486fbe44e9795

There are two factors at play. When the data sources return the ids (data.aws_subnets.public_subnets.ids and data.aws_eips.nat_eips.allocation_ids), the return is sorted. It doesn’t matter whether the order alphabetical or the opposite. Because in any given order, the randomly generated new ID, can fall anywhere in the list. In this particular result, the new alloc_id falls at the beginning, and the new subnet_id falls in the middle. As a result, NAT Gateway with index 0 and 1 are both changed. Therefore they have to be destroyed and replaced.

All these come from having to build a map. The values of each object in the map come from two different data sources. The values are not predetermined and contain a random part. When we add more AZ, the entire map get shuffled, leading to deletion of existing resources. Yikes.

Implementation with `for_each`: second attempt

The first draft of this post drew some ideas on Reddit. One redditor pointed out that the snippet above with for_each isn’t the optimal way. With some tricks to we can manage the map so that it maintain relative order if we have to add new AZ. The strategy is:

Avoid using data sources to retrieve attribute values
Use a unique key to identify objects in the map;
Directly look up from the resource by the unique key

We’re able to do #2 and #3 because when a resource has the for_each argument set, the resource itself becomes a map of objects. We can then locate that resource by the key. We can determine what that key is so long as it uniquely identifies the resource. Below is the revised code snippet with for_each:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

locals {
 subnet_config = {
 for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 cidr = cidr
 az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)]
 }
 }
}

resource "aws_subnet" "public_subnets" {
 for_each = local.subnet_config
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
}

resource "aws_eip" "nat_eips" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => cidr }
}

locals {
 nat_gw_config = {
 for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 subnet_id = aws_subnet.public_subnets[md5(cidr)].id
 alloc_id = aws_eip.nat_eips[md5(cidr)].allocation_id
 }
 }
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = local.nat_gw_config
 subnet_id = each.value.subnet_id
 allocation_id = each.value.alloc_id
 depends_on = [aws_internet_gateway.internet_gw]
}

In this example, I use the MD5 hash of CIDR as the unique identifier key to ensure we have a consistent mapping between allocation id and subnet id. When a new AZ is created, the new allocation-subnet id pair will have its own new key. The unique key can be any identifier (even the CIDR itself) as long as it is unique and we do not change the selection of unique key after the first apply.

One more shot with `for_each`

The code snippet above got rid of data sources, but still have to leverage two local values (subnet_config and nat_gw_config) as helpers. Are they absolutely necessary?

Not really. The Terraform documentation has a page about References to Values, where it states:

If the resource has the count argument set, the reference’s value is a list of objects representing its instances.
If the resource has the for_each argument set, the reference’s value is a map of objects representing its instances.

In other words, using for_each with a map as input, we’re also creating a map as output, which is the resource array itself. The key is the same as the input map. Therefore, we can reuse the key. I know that sounds too abstract. Here’s the code refined:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 #default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

resource "aws_subnet" "public_subnets" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 cidr = cidr
 az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)]
 }
 }
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
}

resource "aws_eip" "nat_eips" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => cidr }
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = aws_subnet.public_subnets
 subnet_id = aws_subnet.public_subnets[each.key].id
 allocation_id = aws_eip.nat_eips[each.key].allocation_id
 depends_on = [aws_internet_gateway.internet_gw]
}

Voila. I use md5 of the CIDR as the key again, first to create both aws_subnet and aws_eip. I also followed the example of chaining for_each between resource types. This way, when creating aws_nat_gateway, I can reference an instance in each resource array by the same key. Chaining for_each is very handy. But admittedly, it takes several iterations for me to get there. The code is neater, but not as straightforward to read due to the list/map comprehension.

Conclusion

I came across a team where the code review guideline favours for_each strongly. I see where that comes from after reading the book. But I don’t find count to be evil. That triggered my initiative to dive deep into this topic.

In this post we brought up a classic pattern of relationship between resources, and examined several ways to implement them using count and for_each. Using count can be straightforward but carries the risk of index shifting if additional element is added in the middle of the resource array. On the other hand, for_each is more powerful, but it requires some crafting with the Python-style list/map comprehension.

My recommendation is, start with a holistic look at the types of resources to create with loop, and how they are related with each other. Go with count if index shifting isn’t a risk. For example, when you need to create one instance of a resource conditionally. Otherwise, use for_each loop if the team is comfortable with the list/map comprehension. In some cases where we need to conditionally create several instances of the same resource, we can use a technique such as:

for_each = variable.disabled ? {} : data.any_resource.map
for_each = variable.disabled ? toset([]) : data.any_resource.list

In fact, the recommendation from AWS Terraform best practice is highly in favour of for_each.

Test Open ID Connect Flows Locally

Mon, 05 Aug 2024 13:24:34 -0400

Earlier this year, I had to integrate an application with an identity provider. Both claim to be compliant with Open ID Connect. But when they don’t get along, I must find out where it breaks to determine which party isn’t compliant. Therefore, I had to really get to the transaction-level details. As a result, I was eager to find out a way to test Open ID Connect Flows locally. Considering that I discussed OIDC in several past articles and use cases (authenticating to kube-api server, Kubernetes workload, ROSA), I feel it is important to be able to test Open ID Connect flows locally.

Architecture

What makes local testing difficult is that there are several parties involved and they act in different roles. Take Authorization Code Flow as an example, the main actors are:

User Agent: the browser;
Authorization Server: an identity store, also referred to as identity provider;
Resource server: an HTTP server that returns protected resource;
OIDC client application: the component that communicates with identity store on behalf of the resource server;

Apart from the bowser (obviously running on my laptop), we can group the the other actors in different patterns. Here I illustrate some options below:

In essence, we need an OIDC client app and an authorization server. And I find the most simplistic test architecture to be pattern #2 (without reverse proxy) on the client side, and pattern #1 on the server side. In the next section let’s discuss why I prefer this test architecture.

Choice of Tools

The principle of this lab is that we can focus on the OIDC flows itself and simplify every other aspects as much as we can.

The authorization server needs to access my client app. As a result, if I use a public authorization server, such as Azure or Google, I’d have to host my client app on a public IP with domain name as well. This creates churns. To see how much hassle this can involve, review my article Istio External Authentication lab. Instead, I need a tool to host the authorization server on my laptop. And yes, it’s KeyCloak. It is a well-renewed open-source project for identity and access management. It is also the upstream project of RedHat SSO. Another reason is it operates on PostgreSQL database which is a common relational database technology with release in Docker images.

On the client side, we can have our web service with built-in OIDC capability, which usually requires development work in a language with OIDC library. Alternatively, if the client app lacks such capability, we would address this capability in a different component on the client side. It can either be a standalone, purpose-built proxy, or a generic reverse proxy with OIDC capability. For this, I examined a few options. Nginx and Traefik have OIDC support for a fee in their Enterprise product. Apache has a module mod_auth_oidc for free but it requires building the plug-in on my own for Mac platform. Eventually I landed on the purpose-built option, using the oauth2-proxy open source project. This project can act as both actor #3 (with a minimal HTTP server) and actor #4 (OIDC client). So it also saves me from hosting a separate web server. Also, I have used it in the past in the Istio External Authentication lab.

In reality, each message in the OIDC flows must be TLS encrypted. In local testing though we don’t really care. Similarly we don’t necessary need a reverse proxy if it plays no role in the OIDC flow. Both Keycloak (with PostgreSQL) and oauth2-proxy are released in Docker images. As a result, we are ready to roll with only three tools: the browser on local host, Keycloak and oauth2-proxy in the Docker daemon, which provides great portability.

To get started, let’s have two fictitious domains, web.digihunch.com, and keycloak.digihunch.com. In the Docker compose manifest, I name the services based on their hostname so that they can reference each other from within the container network namespace. To access the service from the host, I force the DNS resolution to localhost in /etc/hosts on my Mac, and make sure to declare the same host port in the port mapping (4180 for dummy web service; and 8080 for Keycloak).

Configuration

I created a Docker compose file as below, to set up my test:

services:
 web.digihunch.com:
 container_name: oauth2-proxy
 image: quay.io/oauth2-proxy/oauth2-proxy:latest
 command:
 - --http-address
 - 0.0.0.0:4180
 environment:
 OAUTH2_PROXY_COOKIE_SECRET: NYZaClZinINKwxNGzEDeFGh64W6tmq1eB6uHQPa4S5o 
 OAUTH2_PROXY_CLIENT_ID: dh-user-client
 OAUTH2_PROXY_CLIENT_SECRET: pmkwBjkVesrj7fw1MY7h5s9e3cmAKXgc 

 OAUTH2_PROXY_PROVIDER: oidc 
 OAUTH2_PROXY_OIDC_ISSUER_URL: http://keycloak.digihunch.com:8080/realms/digihunch-users

 OAUTH2_PROXY_PASS_ACCESS_TOKEN: true 
 OAUTH2_PROXY_EMAIL_DOMAINS: '*' 
 OAUTH2_PROXY_REDIRECT_URL: http://web.digihunch.com:4180/oauth2/callback
 OAUTH2_PROXY_PROVIDER_DISPLAY_NAME: DHCKC

 OAUTH2_PROXY_COOKIE_CSRF_EXPIRE: '5m'
 OAUTH2_PROXY_COOKIE_CSRF_PER_REQUEST: true
 OAUTH2_PROXY_COOKIE_SECURE: false # Needed for HTTP connection

 #OAUTH2_PROXY_UPSTREAMS: file:///var/www/static/#/home/ # serve page at /home path
 OAUTH2_PROXY_UPSTREAMS: static://202 
 volumes:
 - ./config/oauth2-proxy.cfg:/etc/oauth2-proxy.cfg
 # - ./config/www:/var/www/static/
 ports: 
 - 4180:4180
 networks:
 - oidc_network
 restart: unless-stopped
 depends_on:
 - keycloak.digihunch.com
 
 postgres-db:
 image: postgres
 container_name: postgresdb
 restart: always
 shm_size: 128mb
 ports:
 - 5432:5432
 networks:
 - oidc_network
 volumes:
 - ./data/pgdata:/var/lib/postgresql/data
 environment:
 - POSTGRES_USER=master
 - POSTGRES_PASSWORD=masterpass
 - POSTGRES_DB=keycloak

 keycloak.digihunch.com:
 image: quay.io/keycloak/keycloak
 command: start
 environment: # Based on Hostname:v2 https://www.keycloak.org/docs/25.0.0/upgrading/#migrating-to-25-0-0
 KC_HOSTNAME: http://keycloak.digihunch.com:8080
 #KC_HOSTNAME_ADMIN: For simplicity, no separate management URL or port
 KC_HOSTNAME_BACKCHANNEL_DYNAMIC: true
 KC_HTTP_ENABLED: true ## Otherwise HTTPS is the enforced by default.
 KC_HEALTH_ENABLED: true
 KEYCLOAK_ADMIN: admin 
 KEYCLOAK_ADMIN_PASSWORD: kcadminpass
 KC_DB: postgres
 KC_DB_URL: jdbc:postgresql://postgres-db/keycloak
 KC_DB_USERNAME: master
 KC_DB_PASSWORD: masterpass 
 ports:
 - 8080:8080
 networks:
 - oidc_network
 restart: always
 depends_on:
 - postgres-db

networks:
 oidc_network:
 enable_ipv6: false
 driver: bridge

For OAuth2-Proxy, the official image for oauth2-proxy is distroless. So if you have to troubleshoot its container file system, you need to access it via an assistant container:

docker run --rm -it --name debugger --privileged --pid container:oauth2-proxy --network container:oauth2-proxy busybox sh
# to see target container's file system, go to: ls -l /proc/1/root/

Alternatively, we can use the one that Bitnami releases but be vary of some nuances.

In the Keycloak part, I use environment variables and had to watch out for the recent changes on hostname v2. We should first start up the Keycloak service. From http://keycloak.digihunch.com:8080, we can login using the credential specified in the environment variables, then we can create a client app:

Create a new realm (dropdown-> Create realm) with name digihunch-users
Switch to this realm from the dropdown, create a couple users (e.g. dhadmin@www.digihunch.com), and set password.
Create a group (e.g. myadmin) and join the user to the group
Under the same realm, create a client, with OpenID Connect as type, client ID being dh-user-client. turn on client authentication (without Direct access grants)
Save the client for now and grab the client secret. Note that the OIDC discovery document (http://keycloak.digihunch.com:8080/realms/digihunch-users/.well-known/openid-configuration) should come online.
Update the docker compose file:
- The value for OAUTH2_PROXY_CLIENT_SECRET is from the client secret;
- The value for OAUTH2_PROXY_OIDC_ISSUER_URL should be http://keycloak.digihunch.com:8080/realms/digihunch-users;
- The value for OAUTH2_PROXY_CLIENT_ID is dh-user-client;
Restart all services including the web. Log on to keycloak and go back to the client configuration in the realm. Under settings. Put in valid redirect URIs as http://web.digihunch.com:4180/oauth2/callback and save the client.

Now, let’s start an private browser session, and browse to http://web.digihunch.com:4180/. The browser should redirect you to keycloak’s login page. Once log in is successful, it should redirect you to the static response with 202 code as the manifest configured.

Summary

In this post, I laid out the steps to test the login for OIDC authorization code flow locally as a starting point. For a bullet-proof solution, I recommend taking a look at KeyCloak administration guide. For example, we typically disable the master realm for security.

There is a similar test setup on Otka blog but I simplified all the aspects that I regard as distractors. There are many other flows that can be tested. However, some of the testing still requires client pattern #1 if we need to initiates an activity from the client application.

IAM Roles for any workload

Sun, 14 Jul 2024 23:53:49 -0400

Background

A few month back a client of mine wanted to use GitLab pipeline to deploy infrastructure on AWS with Terraform. The key question is how to authenticate the Terraform process running in the pipeline to AWS with temporary credential. Having worked it out on GitHub, my proposal at time was to add OIDC provider to represent the GitLab runner.

After a few months, they told me that they are self-hosting their GitLab instance. The idea above was based on exposing an identity provider document on the public Internet, which the client is unable to do.

Now, I have an idea: IAM Roles Anywhere.

IAM Role Introduction

Many 101 tutorials asks beginners to create standalone IAM users (or group) with IAM policies directly attached. For programatic access they also include creating a pair of access key and secret access key and pass them along to an external application. The keys are long term credentials, and worse, never expires. The leakage of these long-term credentials had been such a headache that AWS strongly discourage the use of long term credentials. You can feel the discouragement when trying to create an access key through the web console, or by the banners on top of the documentation page about how to do so.

The recommendation is use temporary security credentials. In the context of AWS that means IAM roles. The users must assume an IAM role by issuing an API call, and the Security Token Service (STS) grants temporary credential in response.

This diagram has several variations. For example, the request can be AssumeRole, AssumeRoleWithSAML and AssumeRoleWithWebIdentity, depending on whether and how the user info is federated with external identity store. The returned response, a triplet of three values, makes the temporary credential that we should use in any secure environment. They must be renewed before expiry. This model works not only for human identity (e.g. SAML integration, OIDC integration, cross-account access) but also for workload identity (e.g. EC2 instance profile, Lambda execution role, ECS task role, etc).

Another good example is IAM Role for Service Account (IRSA), where a web identity represents a Kuberentes Service Account to gain role credential using the AssumeRoleWithWebIdentity API. In this post however, I’d like to explore more about the IAM role for EC2 instance profile.

The IMDS service

For EC2 instance, we all know that we can associate an IAM role as the instance profile and grant the process using AWS SDK running on the instance with permissions associated with the IAM role. At a lower level, this relies on the IMDS (Instance Metadata Service) running on the instance.

If an EC2 instance’s profile points to an IAM role, a process running on the instance using AWS SDK will also need to get the triplet from the STS. It is summarized in this diagram:

The IMDS is a service available on one of the link-local IP address (169.254.169.254) on the EC2 instance. Requests made to this IP address are not routed elsewhere. The Instance Metadata Service (IMDS) is a means for the cloud service provider’s virtualization layer to share information with the processes on the operating system of a virtual machine. It responds with information related to the instance itself, such as the subnets, IAM role, instance ID, AMI ID, security group. The instance metadata also includes user data script for cloud init process to consume, and most relevantly, the role credential for the instance. This also requires that the IMDS service to have connectivity to the STS endpoint, either via interface endpoint or over the Internet. The AssumeRole calls are logged in CloudTrail.

All major cloud vendors (AWS, Azure and GCP) uses the IMDS mechanism, and this mechanism obviously draws the attention of bad actors. I find some good articles on this here and here.

IAM Role for EC2 Workload

For EC2 instances at AWS, the initial IMDS v1 was introduced in 2012 and allows a GET method to fetch instance metadata. The IMDS v1 is subject to attacks such as SSRF (Server-side request forgery). In 2019 AWS introduced IMDS v2 which tackles those vulnerabilities. As of date, the recommendation is to use IMDSv2. Here is an example of how to fetch instance metadata, including the credential:

# Grab a token
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
# Get top-level instance metadata information
curl http://169.254.169.254/latest/meta-data/profile -H "X-aws-ec2-metadata-token: $TOKEN"
# Get the name of the role
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials
# Get the credential for the role session
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/InstanceProfileRoleName

These commands emulate how the SDK library fetches the credentials to assume the instance profile role. You can also find similar commands on the documentation. However, there isn’t much details about how the instance metadata service interacts with the STS service, except a general statement:

These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials.

When creating a new EC2 instance, make sure that the instance metadata option has http_endpoint enabled, to enable the IMDS service. Also set http_tokens to required, which would run IMDSv2 exclusively. With that setup, the application does not have to mana to use an SDK version that supports IMDSv2. Another metadata option is http_put_response_hop_limit, with default of 1. This limits the number of hops in the metadata request. If the process runs from a Docker container with bridge networking mode, set it to 2 or the process cannot even secure a token.

The diagram above illustrates this scenario with two hops.

IAM Role Anywhere Concept

As this point, we know that the process of workload assuming an IAM role, is essentially using SDK to gain role credentials from instance metadata. In addition to using SDK and instance metadata, AWS also supports using X.509 certificate to gain role credentials. As a result, workload no longer needs AWS SDK, and it doesn’t rely on instance metadata from an EC2 instance. This mechanism is known as IAM Role Anywhere, and it greatly expands the use cases for IAM Role.

To make this work, we first have to provide a certificate authority (CA) to AWS as a trust anchor. It can be any X.509 CA including AWS Private CA. The IAM Roles Anywhere will allow any end-entity endorsed by this trust anchor, to assume an IAM role as specified. We also need to create a profile, in which we can add IAM policies directly, or link to an IAM roles with a trust policy for service principal rolesanywhere.amazonaws.com.

To gain role credential, the requestor must provide both the private key, and its end-entity certificate. The certificate proofs the endorsement of the CA as Role’s trust anchor specifies. The private key proofs the requestor’s identity. The requestor uses the aws_signing_helper utility to request role credentials.

The utility is compatible with the credential_process feature in AWS config, which passes the returned role credentials to the AWS config profile for AWS CLI or SDK running on external virtual machine.

IAM Role Anywhere Lab

Let’s tweak the three commands from this old post of mine to create the test materials: a self-signed CA and a certificate signed by the CA:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority' -addext basicConstraints=critical,CA:TRUE,pathlen:1 -addext keyUsage=keyCertSign

cat > ext.cnf <<EOF
[v3_leaf]
keyUsage = digitalSignature
basicConstraints=CA:false
EOF

openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.digihunch.com'

openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt -extfile ext.cnf -extensions v3_leaf

I tweak them to add the X.509 extensions to meet the requirement for signature validation. We need the following files from the output.

ca.crt -> the certificate of the CA. We provide this file as the trust anchor
server.crt -> the certificate of the server, we need it as the end-entity certificate
server.key -> we need to present this file to proof identity of the requestor

First, we go to AWS console and create a new trust anchor. Copy the content of ca.crt as the certificate. Then we can create a profile with an IAM role, with the trust policy looking like this example. Then we can request the role credential with one command:

aws_signing_helper credential-process --certificate server.crt --private-key server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN

Moreover, if the workload supports AWS SDK or can use CLI but not an EC2 instance, we can bake this in the AWS profile on the external machine:

[profile myprofile]
output = json
credential_process = aws_signing_helper credential-process --certificate /path/server.crt --private-key /path/server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN

We can even configure this in any pipeline as code to allow deployment from a non-AWS pipeline.

Summary

In summary, apart from native AWS services, an IAM role can trust the following types of principals:

Native IAM identity such as an IAM user or an IAM group
Authenticated identity from SAML identity provider that IAM is configured to trust
Authenticated identity from OIDC identity provider that IAM is configured to trust
Validated identity endorsed by a Certificate Authority that IAM designate as a trust anchor

The first type is rarely used because few organizations uses AWS IAM as their identity store. Most organizations have their identity store with federation capability via SAML. On the other hand, a lot of modern applications adopts identity stores with OIDC compliance. Now with IAM Role Anywhere, any entity with X.509 identity can also assume an IAM role. It works with any CI/CD pipeline, whether it is self-hosted. Also, it is now more important to keep the keys safe.

Managing EC2 instances across accounts with Ansible

Mon, 27 May 2024 14:00:22 -0400

I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.

Prerequisites

This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I’ll start with the choice of the tools.

Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we’ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we’d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance’s SSH port unless the bastion is also in a connected network.

As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we’ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture.

That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts.

Configure AWS CLI

This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:

[toplevel]
whoami = sts get-caller-identity --no-cli-pager --output yaml
profile = configure list-profiles

With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we’ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:

We better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable;
As a security best practice, we should not configure profiles with long-term IAM credential in the config file;
We must ensure the CLI doesn’t prompt for log-in every time we switch profile

To satisfy #3 there are many ways but we’ll discuss two: using cross-account IAM role, and using AWS SSO.

Bonus point if you enable auto-complete for AWS CLI.

Configure AWS CLI Profiles

With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:

[profile jump_account]
credential_process = /opt/bin/awscreds-custom --username helen

[profile target_account_1]
role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole
source_profile = jump_account

[profile target_account_2]
role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole
source_profile = jump_account

In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you’re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.

If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:

[sso-session sso]
sso_start_url = https://myorg.awsapps.com/start/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile target_account_1]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess

[profile target_account_2]
sso_session = sso
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess

This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command “aws sso login” . Then you can use all profiles by specifying --profile switch without having to log in again.

Configure SSH to EC2 via SSM

The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.

SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you’d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you’d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked.

Configure Ansible Inventory

We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:

mytest:
 hosts:
 instance1:
 ansible_host: i-00aabbffcc7755221
 ansible_user: ubuntu
 ansible_ssh_common_args: -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1"
 instance2:
 ansible_host: i-eedd88ff66aa22442
 ansible_user: ubuntu
 ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand="sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile target_account_2 \""

Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile’s account.

Another way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.

Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:

plugin: amazon.aws.aws_ec2

# Attach the default AWS profile
aws_profile: target_account_1

compose:
 ansible_host: instance_id
 ansible_user: "'ubuntu'"
 ansible_ssh_common_args: "'-o ProxyCommand=\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \"'"

Then we can display the rendered inventory list, and Ansible-ping the instances.

ansible-inventory -i aws_ec2.yaml --list -y | less
ansible all -i aws_ec2.yaml -m ping

For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.

Conclusion

In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.

Public Key Infrastructure 3 of 3 – PKI Implementation

Sat, 30 Mar 2024 00:13:00 -0400

After the last two post, now we can focus on PKI implementation. The use case is software testing, where we need to create and recycle a lot of short-lived certificates. Typically, we don’t have to create public certificates because testing workload is internal. Also, hosting a public CA is much more involving. In this post we go over some options to host private CAs.

Root CA

I’m assuming we are creating a self-signed root CA for the organization. Large organizations usually keep their root CA offline and store the key material in highly protected configuration such as HSM. That is not something we can easily emulate. Nor are we concerned with the details of protecting root CA. In order to enable configuration of intermediate CAs, all we need for root CA, is to create a self-signed certificate along with the key. We can do this with OpenSSL following the first section of this post. Now we’ll use open-source Step CA. In my opinion, this tool is handier. For example, we can create a root CA this with a single command:

step ca init

This will start a prompt with a few questions. Then it will create a root CA and an intermediate CA. The keys and certificates are stored in ~/.step/ directory.

Note that by default, the root CA certificate has a path length of 1. To make the path length more than 1, we’d have to customize the creation. We can specified the desired path length in a certificate template and create a self-signed certificate with the template:

cat << EOF > root.tpl
{
	"subject": {{ toJson .Subject }},
	"issuer": {{ toJson .Subject }},
	"keyUsage": ["certSign", "crlSign"],
	"basicConstraints": {
		"isCA": true,
		"maxPathLen": 2
	}
}
EOF

step certificate create --kty=RSA --size 4096 --template root.tpl "DigiHunch Root CA" root_ca.crt root_ca.key

The command outputs the certificate and key of our test root CA. With that, next, we’ll create subordinate CAs with a few different tools.

AWS Private CA as intermediate CA

I’ll start with AWS Private CA. It was a spin-off service from AWS Certificate Manager and is fairly simple. However, we need to understand its capacity limit. AWS Private CA documentation has a page on RFC Compliance, which lists what in RFC 5280 are supported and what are not. It performs very basic CA functions. It does not perform domain validation, hence no ACME support. We can create root CA as well but here we’ll create an intermediate CA:

In AWS Console, create a Private CA. Indicate you want to create a subordinate CA. You may specify CRL distribution and OCSP endpoint. However, you’re responsible for the providing CRL and/or hosting OCSP service;
In the Private CA, we need to create CA Certificate. We need to provide CSR to our root CA to sign this CA’s certificate. The root CA can be either another AWS Private CA or external CA. In this case we choose external CA and export the CSR to a file (e.g. dh.csr).
We can use Step CA to sign the request. Note that we have to give reasonable value for expiry date. Because ACM create certificate with 13 month validity period by default, the expiry date must be at least 13 months from the current date. We also need to set path length. In this case, I put 0 so the Private CA can only issue end-entity certificate. The command looks like this:

step certificate sign pca.csr \
 /Users/digihunch/.step/certs/root_ca.crt \
 /Users/digihunch/.step/secrets/root_ca_key \
 --profile intermediate-ca \
 --not-after=2027-01-24T07:20:50.52Z \
 --path-len=0

In the AWS console, we paste the generate certificate content in the as certificate body, and the root CA’s certificate as certificate chain.

If the Private CA serves as parent CA of one more layer of CA, set the path length to 1. This requires the path length to be at least 2.

Once the CA certificate is installed, we have completed creating a private CA. We can reference this CA from ACM (AWS certificate manager) when creating a private certificate. Since there is no validation support, ACM will allow you to claim any domain. Currently it does not support ACME-based certificate automation. However, you can use AWS CLI or any AWS based automation tool to create your private certificate.

Step CA

As a cheaper alternative, we can host private CA using step CA on virtual machines. One of the benefits is ACME support (http-01 challenge). In the init command in this post, we already create the intermediate CA certificate and configuration. In this part of the lab, we’ll have two servers: the CA server (pki.digihunch.internal) and the web server (web.digihunch.internal). Make sure the DNS resolution works for both servers. The architecture looks like this:

The CA server runs the Step CA process acting as ACME server. We deploy the CA in standalone mode (instead of linked or hosted deployment), meaning it’s not connected to any cloud services. We can host the service on port 443 so make sure the process has port binding permission within the operating system, and the firewall (security group) allows traffic via port 443.

The Web Server runs Step CLI acting as ACME client. On the web server, we run Step CLI in standalone mode (instead of webroot). During the challenge-response phase, the CLI will get the required random number from ACME server, and host it the as the response on port 80. Make sure that Step CLI process has port binding permission in the OS, and the firewall (security group) allows traffic via port 80.

Make sure the DNS resolution works for both servers. On the CA server, we fetch the fingerprint (in preparation for setting up Step CLI on web server). Then we add an ACME provisioner, and host the CA server with a single command:

step certificate fingerprint $(step path)/certs/root_ca.crt
step ca provisioner add myacme --type ACME
step-ca $(step path)/config/ca.json

The CA server is listening on port 443 (by default). Now we can test ACME process with any ACME compatible client. Let’s use step CLI on the web server:

step ca bootstrap --ca-url https://pki.digihunch.internal:443 \
 --fingerprint <fingerprintvalue>
step ca certificate web.digihunch.internal acme.crt acme.key \
 --acme https://pki.digihunch.internal/acme/myacme/directory

The bootstrapping step is to establish trust on the CA server. Then the provisioning process should complete automatically:

Here the step CLI command uses standalone mode by default so it is important to ensure reachability (port 80, DNS name) from the CA server when step CLI hosts the response. You can also go with webroot mode, where Step CLI generate the file to the web root directory so the response becomes available. This is helpful when you already run another web hosting process such as Nginx on the web server. This blog post covers the usage of other ACME-compatible tools with Step CA as private CA server.

Another well-known tool is HashiCorp Vault. There is already a good tutorial here and all the OpenSSL command in the tutorial can be replaced with Step commands.

Cert Manager on Kubernetes

The Cert Manager project on Kubernetes makes PKI work simple. In my discussion about self-signed certificate on Kubernetes, I covered how to use Cert Manager to create self-signed certificate. However, a corporate with multiple clusters may need to chain each cluster-wide issuing CA to an intermediate CA outside of the cluster. Let’s look at this architecture as an example of a full PKI implementation:

In this architecture, we extend corporate on-premise root CA to the cloud. The Ops account hosts the AWS Private CA, and shares it out to workload account(s) using Resource Access Manager. In each EKS cluster, the Private CA serves as the cluster-level issuer, which can issue CA certificates across Kubernetes namespaces. In each namespace, there is a Cert Manager issuer responsible for issuing certificates within the namespace. Cert Manager supports many issuers and we’re using the AWS Private CA issuer.

To demonstrate the gist of this architecture, in our lab we’ll create one Kubernetes cluster in the same AWS account as the Private CA, and create a Cert Manager certificate in one namespace (e.g. ingress). To get started, create a private CA with the instruction in this post and ensure that the private CA has path length of 1. Then create an EKS cluster. You may use my CloudKube project to provision this cluster in Terraform or any other ways. We use the IRSA model to grant a service account access to the Private CA. First, we create an IAM policy and reference the ARN of the private CA:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "awspcaissuer",
 "Action": ["acm-pca:DescribeCertificateAuthority", "acm-pca:GetCertificate", "acm-pca:IssueCertificate"],
 "Effect": "Allow",
 "Resource": "arn:aws:acm-pca:<region>:<account_id>:certificate-authority/<resource_id>"
 }
 ]
}

Name this policy ‘PCA-Access’. It is the minimum required permission. Now let’s create a service account, along with an IAM role that uses this policy:

eksctl utils associate-iam-oidc-provider \
 --region $AWS_REGION \
 --cluster $CLUSTER_NAME \
 --approve

eksctl create iamserviceaccount \
 --cluster=$CLUSTER_NAME \
 --namespace=cert-manager \
 --name=aws-pca-sa \
 --role-name EKSCertManagerPrivateCARole-$CLUSTER_NAME \
 --attach-policy-arn=arn:aws:iam::123456789012:policy/PCA-Access \
 --approve

In order to run the command successfully, you need the IAM permission to create IAM role, as well as API access to the cluster. Now we can install both Cert Manager and the Private CA Issuer.

helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --version v1.13.3 \
 --set installCRDs=true \
 --create-namespace

helm install aws-ca awspca/aws-privateca-issuer \
 --namespace cert-manager \
 --version v1.2.7 \
 --set serviceAccount.create=false \
 --set serviceAccount.name=aws-pca-sa

Note that when installing Private CA issuer with helm, specify the service account that we created earlier (aws-pca-sa). We install both to the cert-manager namespace, where we’ll create a ClusterIssuer. Now we can declare the following manifest:

apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
 name: pca-cluster-issuer-rsa
 namespace: cert-manager
spec:
 arn: arn:aws:acm-pca:ca-central-1:383500642091:certificate-authority/7f2d7b38-2508-4492-81f0-b5b85427c99c
 region: ca-central-1
---
apiVersion: v1
kind: Namespace
metadata:
 name: ingress
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: ingress-ca-cert
 namespace: ingress
spec:
 isCA: true
 commonName: ingress-ca
 secretName: ingress-ca-secret
 privateKey:
 algorithm: RSA
 size: 2048
 issuerRef:
 name: pca-cluster-issuer-rsa
 kind: AWSPCAClusterIssuer
 group: awspca.cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: ingress-ca-issuer
 namespace: ingress
spec:
 ca:
 secretName: ingress-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: ingress-cert
 namespace: ingress
spec:
 commonName: web.digihunch.com
 secretName: web-digihunch-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.digihunch.com
 privateKey:
 algorithm: RSA
 size: 2048
 issuerRef:
 name: ingress-ca-issuer
 kind: Issuer
 group: cert-manager.io

In this manifest, we create a CA certificate, along with a CA in the ingress namespace. With that, we create an end-entity certificate. As a result, we should find the certificates ready:

kubectl -n ingress get certificate
NAME READY SECRET AGE
ingress-ca-cert True ingress-ca-secret 24s
ingress-cert True web-digihunch-secret 24s

The certificate can be reference by the workload in the namespace. Because an Issuer can only issuer Certificate in the same namespace, we need a issuer in the namespace where the certificate will live. The other benefit that Cert Manager brings, is the support of ACME challenges, which is an enhancement to AWS Private CA.

Summary

We discussed several approaches in PKI implementation. PKI is a fundamental requirement in a software testing environment today. Having an internal public key infrastructure enables many other use cases too. For example, the team may use SSH user certificate. You can also host a CA with Step. PKI allows an enterprise to configure SSL inspection on their Next Generation Firewall. The IAM Role Anywhere feature on AWS also operates on an organization’s own PKI.

Public Key Infrastructure 2 of 3 – Certificate Automation

Fri, 08 Mar 2024 00:14:00 -0400

Following the last post on PKI, we’ll discuss automation of certificate issuance. Two key activities to automate are: validation of the requestor and issuance of the certificate.

Validation

Validation isn’t always required. For private CAs, the trust boundary does not go beyond the internal engineering team, there is little incentive to perform any validation. AWS Private CA is based on this idea. The requestor can claim to be any identity. The private CA, when issuing the certificate, does not perform any validation. Neither is there a need to convince any entity outside of the trust boundary of the validity of the certificate. Validation is optional. For public facing certificate however, validation is a must because we’re convincing every browser in the world of the validity of the certificate requestor. Common validation levels include:

Domain Validation (DV): the certificate requestor must demonstrate the right to administratively manage the affected DNS domain.
Organization Validation (OV): in addition to the DV criterion, the issuer verifies the actual existence of the requestor’s organization as a legal entity.
Extended Validation (EV): the certificate requestor must persuade the certificate provider of its legal identity, including manual verification checks y a human. Unlike DV and OV certificates, only a subset of CAs can issue EV certificates.

For both OV and EV, a certificate provider publishes its vetting criteria through its certificate policy. They require human validation of any registrants. At corporate level, EV certificates are required for sensitive public-facing workloads (e.g. banking, financial, health information). For non-sensitive public-facing workloads, DV certificates may be sufficient. For non-public facing workloads, such as software testing, they may go with DV certificates or no validation at all, depending on the specific use case. Since I set up PKI for the latter, I’ll focus on DV. DV is the most basic level and can be fully automated.

DigiCert, a well-known trusted third party, has a detailed page on the differences among DV, OV and EV.

Certificate Automation

As to automation, there are some common certificate automation protocols:

ACME (Automated Certificate Management Environment): commonly used in web server automation.
SCEP (Simple Certificate Enrollment Protocol): commonly used in enterprise environments for managing certificates in the network devices such as routers, switches and IP phones.
EST (Enrolment over Secure Transport): a more secure alternative to SCEP suitable for various use cases beyond network devices.
CMP (Certificate Management Protocol): more comprehensive protocol with a wide range of functionalities for complex certificate management scenarios.

SCEP is common in network industry. EST and CMP target very specific scenarios. We’ll examine ACME as it’s most relevant to the use case of web service. The biggest advocate of ACME is Let’s Encrypt, a non-profit CA run by ISRG that provisions X.509 certificates at no charge. Let’s Encrypt is the world’s largest CA, aiming to secure all websites with HTTPS. ACME only issues DV certificates, since they can be fully automated.

The ACME Protocol

The ACME protocol automates validation and issuance. The certificate requestor will have to use an ACME-capable client. The certificate provider (CA) needs to act as ACME server. At a high level, the flow looks like this:

I came across this good diagram on the ACME flow from a post from small step. It has all the transactions in detail. As it shows, the delivery (issuance) of certificate material is based on HTTP POST method. The domain validation process is based on a challenge-response model. The ACME specification makes this an extension point, with the following most comment challenge types:

HTTP-01 (HTTP Challenge): the domain in question needs to host a random number at a random URL under /.well-known/acme-challenge on port 80. The CA will fire an HTTP GET request to that URL. This is easy to configure because we usually have full control on the web server. There must be network connectivity between the web server and the CA to allow HTTP traffic.
DNS-01 (DNS Challenge): the requestor provisions a TXT record with random value. The ACME server does not need to connect to the web server. It only needs to perform a DNS lookup to confirm the challenge. However, the certificate requestor needs the privilege to modify DNS record.
TLS-ALPN-01 (TLS ALPN Challenge): ALPN is the protocol during TLS negotiation. The client presents a self-signed TLS certificate containing the challenge response as a special X.509 certificate extension. This challenge type is useful when a security policy requires the CA to reach the client via a TLS connection.
DEVICE-ATTEST-01 (Device Attestation Challenge): This is for Apple Managed Device Attestation (ADA) and other secure zero-touch provisioning (SZTP) applications as part of your device management (MDM) strategy. Certificates identify specific hardware devices, via permanent device IDs. These are typically client certificates that can be used for device authentication.

At the implementation level, Let’s encrypt drives its public CA with Boulder. It supports two challenge types. When hosting a private CA, you can use Boulder too. Some feel Boulder is complicated and you can consider the following alternatives:

LabCA: based on Boulder and supports hosting in docker.
Step CA (open source): a simple CA solution
Cert Manager: very popular choice on Kubernetes
Hashicorp Vault: a secret management solution including certificate management capability with ACME support.

You may combine different solutions for all level of CAs. For example, use Step CA for internal root CA, and Cert Manager for intermediate CAs for Kubernetes workloads. On the client side, Let’s Encrypt recommends Certbot. However, there are many choices. Step CLI (by Step CA), acme.sh, etc. Let’s Encrypt compiled a list here.

If we’re not seeking automation with ACME in our process, and just want to manually sign certificates, we can use generic tools (e.g. openSSL, cfssl, easyRSA, etc). They act both as client (gingnerate CSR) and server (signing CSR) using different command switches.

Renewal and Revocation

Lifecycle management involves renewal and revocations. Renewal is essentially re-issue certificates closer to expiration date. In software testing, we often use short-lived certificates, to ensure that our test scenario covers automated certificate renewal as well. It is the responsibility of requestor to initiate the renewal, and distribute the renewed certificates. With Let’s Encrypt, the renewal process will challenge the requestor again for validation purpose. However, in some cases, the certificate provider may choose not to perform validation on every renewal. For example, short-lived certificate gets renewed every week, while validation is performed every year. During the renewal process, the private key of the website does not change. Note the difference between renewal and rekey. If the website’s private key is compromised, then instead of renewal, we should re-issue a private key and request a new certificate.

Revocation is a challenging process. To declare that a certificate should no longer be trusted, there are currently two ways: CRL and OCSP but both have drawbacks. CRLs are lists of all the certificates that a CA has issued but revoked. This list can grow very large. It is not feasible for the application (e.g. Browser) to download the giant list for each CA regularly and check for every website that matches the CA. OCSP provides a query-based method. The application can query the revocation status against the OCSP endpoint. It however brings its own challenges. The OCSP server is subject to downtime. The network connectivity between application and OCSP server causes latency. Many applications simply treats query timeout as not revoked. To reduce the load, application may cache OCSP responses, leading to potentially out-dated status. Worse, a malicious CA can track website of the application user.

Let’s encrypt has a page on these challenges, and it proposes a new browser-summarized CRLs. It was still a recent effort so we’ll see how that plays out.

Summary

Following the first post on the PKI concepts, we discussed the automation of certificate issuance in this post. In the next one, let’s go over some labs.

Public Key Infrastructure 1 of 3 – Basics

Tue, 20 Feb 2024 00:05:00 -0400

In 2021, I wrote an intro to Public Key Infrastructure (PKI). Now that I have to host my own certificate authority, I decide to dive a little deeper into PKI in this series of posts. In software testing scenario, we need to issue (and recycle) a lot of certificates, and manage their lifecycle events such as (renewal, revocation). As a result, the corporate should establish its own private key infrastructure (PKI). This effort includes hosing their own Certificate Authority.

Concepts

IT professions often use the acronyms PKI and CA interchangeably in the context of implementation. However, strictly speaking, PKI is the entire framework (strategy, policy, etc) around managing certificate at corporate level, and CA is the specific entity that issues certificates. When it comes to architecting PKI, a key design decision is the hierarchy of CAs. Microsoft has a good page explaining the common options.

In a single-tier CA hierarchy, the root CA is also issuing CA. The root CA as the anchor of trust of the PKI also issues certificates to the end entities. Obviously, this does not scale. It is only for sandbox testing, and not suitable for any environment that needs to scale. It also carries significant risk because the private key at the anchor of trust has to remain online.

With a two-tier CA hierarchy, the root CA only issues certificates to its subordinate CAs. Therefore the root CA can go offline. As a result, the subordinate CAs issue certificates to end entities. Since the root CA can remain offline (except for issuing certificates for new CAs), the chance compromise is reduced significantly. This option also scales better with multiple issuing CAs. If one of them gets compromised, then only the end entities of the compromised CA are impacted. The root CA still needs to be online from time to time. Apart from granting new subordinate CA, the team may also need root CA online to sign CRLs, which is a management overhead.

The two-tier hierarchy introduces another problem. The root CA also needs to restrict the certificates that its subordinate CAs can issue. For example, one issuing CA can only issue certificates in the *.dev.digihunch.com domain, and another CA can only issue *.ops.digihunch.com. No subordinate CA shall issue “rogue” certificate beyond their authorized scope. In order to enforce issue boundaries on issuing CAs, RFC 5280 defines multiple ways to express constraints, such as basic constraints (including path length), name constraints, policy constraints, and EKU.

In a three-tier CA hierarchy, the top-level is still a root CA that stays offline. One level below, is a layer of CAs that also stay offline and we refer to them as intermediate CAs. Going down one more level down, there are the issuing CAs for end-entity certificates. Oftentimes, we use the intermediate CAs as policy CAs where we introduce restrictions to the subordinates. In this model, the root CA can remain offline nearly all the time because we can issue CRLs at the intermediate level. This is more flexible, but also more management overhead.

Implementation Options

The two-tier hierarchy is good in most scenarios but the three-tier hierarchy is also common for large organizations. The next consideration is implementation strategy. Take two-tier model for example. We can think about these options:

Implement a self-managed PKI, with an internal root CA. The internal subordinate CAs are the issuing CAs. They are chained to the internal root CA.
Implement a self-managed PKI, with its certificate purchased from a commercial CA. The internal subordinate CAs are the issuing CAs. They are chained to the external root CA, as the TPP (trusted third party). The external CA may or may not be a public root CA.
Purchase certificates from a commercial CA that are chained to a public root CA

Option 3 essentially delegates the PKI to a commercial provider. This is usually not a favourable option due to the hefty charge and minimal control. In option 2, when the third party is a public root CA, you can have all your certificates with public trust, although this is a pricey option too. Even if the third-party isn’t a public root CA, there is still a benefit of delegating the management of Root CA to commercial provider.

Option 1 is for use cases where we need a lot of certificates quick, for example, in agile development iterations. The certificates do not need trust beyond the organization, and the risk of root CA being compromised is manageable. In this option all CAs are private CAs. Option 2 on the other hand, is flexible in terms of trust boundary. You can just rely on the commercial provider as a TTP. If the TPP supports public root CA, you can issue certificates for public facing workload. In other words, you can choose either a private CA or a public CA. When we work with a private CA, we have to import the certificate to the trust stores of the organization.

Some PKI topologies splits a CA into two sub-components: the Certificate Authority Service (CAS, or simply CA) and the Registration Authority (RA), as the diagram above shows. With many locations, each location has an RA that communicates with a with a central CAS. The RAs receive requests from local clients and are responsible for authenticating these requests and pass the validated ones along to the CAS. The validation can take place in ACME protocol. The CAS implicitly trusts RAs and will sign the validated requests from RAs before sending them via API calls. The RA then relays the certificates back to the local requestors. This topology is more scalable with the CAS focusing only on signing and the RAs on validation and passing the requests along. Large organizations may have their central CAS hosted on-prem, and remote RAs in the CSP regions.

Components for X.509 Certificate

X.509 is the standard for digital certificate. The X.509 standard has the following most important fields:

Subject: The name of the subject (e.g. a user, service, device), commonly represented as X.500 formate distinguished name (DN). For website, the value can be CN=digihunch.com
Serial Number: A unique identifier for each certificate that a CA issues.
Issuer: DN of the CA. For a self-signed root CA, the issuer is the subject. They are different otherwise, such as in subordinate CA certificates and end entity certificates.

These fields were introduced in version 1. In addition, X.509 version 3 certificates introduces extensions that provide additional functionality and features to the certificate. Each extension comes in two flavours: critical and non-critical. A certificate-using system MUST reject the certificate if it encounters a critical extension it does not recognize or a critical extension that contains information that it cannot process. A non-critical extension MAY be ignored if it is not recognized, but MUST be processed if it is recognized.

Here are some common extensions that a lot of implementations use:

Subject Alternative Name (SAN): only used in end entity certificates, not in CA certificates. The format of SAN is flexible and it does not have to be X.500 DN. For website certificate, we often place alternative DNS names here.
Key Usage: The intended scope of usage for a private key is specified through the Key Usage and Extended Key Usage (EKU) extensions in the associated certificate. Example: “Certificate Sign, CRL Sign”
Basic Constraints: Used to distinguish between end-entity cert and CA cert. You should also specify path length. The value can be: “CA:TRUE, pathlen:1”. If the path length constraint is 0, the CA may have one more level of subordinate CA. But these subordinate CAs must have path length of 0 on their own certs, and cannot extend one more level as their own subordinates. Those subordinate CAs can only issue end-entity certificates. If the path length value is none, then there is no restriction on the levels of subordinate CAs.
Name Constraints: for CA certs only, defined in RFC5280, to limit the scope to certain names on the certificate that the CA issues. Client must verify that a certificate is allowed to be signed by CA.
CDP (CRL distribution points): URL(s) where the application or service can retrieve the certificate revocation list (CRL).
AIA (Authority Information Access): URL(s) where the application or service can retrieve the revocation list for CA’s certificate
SKI (Subject Key Identifier): the SHA-1 hash of the subject’s public key
AKI (Authority Key Identifier): the SHA-1 hash of the issuer’s public key
EKU (Extended Key Usage, also Enhanced Key Usage): an object identifier (OID) for each application or service a certificate can be used for. It needs to align with Key Usage
Certificate Policy: reference to the certification practice statement (CPS) of the issuer. During exchange, any relying party can access the assurance level associated with the certificate, and decide on the level of trust to put in the certificate.
Policy Constraints: for path validation, it can be used to prohibit policy mapping or to require that each certificate in a path contain an acceptable policy identifier.
Policy Mappings: in CA certificates to restrict the certificates that the CA can issue.

This isn’t a complete list. Plus, X.509 V3 also supports custom extensions. Here is a sample certificate. In real life, the way each implementation uses these extensions may vary slightly, so interoperability issues between PKIs exist. When designing your own PKI, it is important to examine the usage of extensions to comply with the standard. For example, Microsoft has the following recommendations to restrict certificates:

For subordinate CA certificates, the Basic Constraints extension should be present and marked as critical
The cA field should be set to TRUE
The pathLenConstraint field should be set to the minimum value required to enable the business scenario (i.e. 0 if that CA will issue certificates only to End Entities)
The EKU extension should be present and contain the minimum set of EKU object identifiers (OIDs) to enable the business scenario. Furthermore, the anyExtendedKeyUsage OID (2.5.29.37.0) should not be specified.

Summary

This post discussed the basic concepts in public key infrastructure. In the next post, I’ll cover the automation of certificate issuance.

Workload Identity on Kubernetes 2 of 2 – EKS

Mon, 08 Jan 2024 00:05:00 -0400

I discussed in my previous post on workload identity and dived into how it works in AKS (Azure Kubernetes Service). In this post I will continue the topic with AWS as the example. From the perspective of CSP, we consider any running process on the cloud resource as workload. Therefore, I’ll start with control plan and node identities. From the perspective of a Kubernetes platform, the term workload mostly refers to applications running in Pods. So later in this article I’ll distinguish two mechanisms for Pod Identity: IRSA and EKS Pod Identity.

EKS Control Plane and Node Identity

AWS directly associate an IAM role with EKS control plane and an IAM role with each node group. We don’t need an extra step of assigning a “managed identity” (as in Azure) to a cluster or to a node group ( and then bind a role to the identity). You can find this pattern from Terraform code. Each aws_eks_node_group resource has a node_role_arn attribute to link to its IAM role, and a cluster_name attribute to link to the cluster. Each aws_eks_cluster resource has a role_arn attribute for cluster’s permission.

The cluster’s IAM role is usually bound to managed policies like AmazonEKSVPCResourceController and AmazonEKSClusterPolicy. The IAM role that is assigned to the node group is the exact IAM role of the instance profile of each node. The kubelet process on the nodes are the main users of this role and the permission should not be broader than what it needs to do. This role usually have a few managed policies such as AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonSSMManagedInstanceCore and AmazonEC2ContainerRegistryReadOnly.

The node role applies to self-managed node and managed node. When using Fargate to provide computing capacity, each Fargate profile will use its own IAM role, to connect to the cluster and pull container images. This IAM role is known as Pod Execution Role. For a private cluster, the place to run the command would be a bastion host with connectivity to the cluster’s API endpoint. Refer to this post about the connectivity to private cluster.

IAM Role for Service Account (IRSA)

When AWS launched EKS in 2018, Kiam was a popular open-source project to grant Pods access to AWS resources. In 2019, AWS introduced the official mechanism, IRSA (IAM Role for Service Account). IRSA ties a Kubernetes identity (in the form of Service Account) to an IAM role in AWS. IAM allows creation of web identity based on OIDC. EKS can act as an OIDC issuer.

This requires a few points of configurations, via the cluster API and via cloud the endpoint. The eksctl utility makes it simple with two commands:

$ eksctl utils associate-iam-oidc-provider \
 --cluster $CLUSTER_NAME \
 --approve

$ eksctl create iamserviceaccount \
 --cluster=$CLUSTER_NAME \
 --namespace=kube-system \
 --name=aws-load-balancer-controller \
 --role-name AmazonEKSLoadBalancerControllerRole \
 --attach-policy-arn=arn:aws:iam::112233445566:policy/AWSLoadBalancerControllerIAMPolicy \
 --approve

The first command creates an OIDC web identity integrated with the EKS cluster, and the second creates a Service Account in Kubernetes and links it to the identity, and assign an IAM policy. These two commands must run under certain conditions. The AWS CLI identity for first command requires the the permission to add OIDC provider. The second needs the permission to create an IAM role. In addition, it requires kube API access to the cluster. So the command needs to run from an environment that can access both the cluster’s API and AWS API.

The IAM identity provider is somewhat similar to a managed identity with OIDC federated credential in Azure. However, unlike managed identity, here in AWS we cannot create the OIDC identity until after the cluster creation. In other words, the OIDC web identity’s lifecycle does not decouple with the cluster lifecycle. We have to create a new web identity every time we create a new EKS cluster. In large organizations, the permission to create a new web identity is highly restricted.

EKS Pod Identity

There are a few other limitations with IRSA. As this blog post suggests:

Further, cluster administrators have to update the IAM role trust policy each time the role is used in a new cluster during scenarios like blue-green upgrades or failover testing. Additionally, as customers grow their EKS cluster footprint, due to the per cluster OIDC provider requirement in IRSA, customers run into the per account OIDC provider limit. Similarly, as they scale the number of clusters or Kubernetes namespaces in which an IAM role is used, they run into IAM trust policy size limit, which makes them duplicate the IAM roles to overcome the trust policy size limit.

AWS brings the new mechanism “EKS Pod Identity” at reInvent 2023. In this mechanism, user can hook up an IAM role directly to a Kubernetes service account, without having to resort to a web identity and OIDC integration. Users just need to create a Pod Identity Association, using the CreatePodIdentityAssociation API, with the following parameters:

Cluster name
Namespace
ARN of the IAM role
serviceAccount

Both AWS CLI and ekscli already support the CreatePodIdentityAssociation API. Before creating a Pod Identity Association, we need to install the add-on “Amazon EKS Pod Identity Agent”, and ensure that the node roles have the permission. That is because the agent needs to use AssumeRoleForPodIdentity API. We also need an IAM role, with the trust policy principal being “pods.eks.amazonaws.com” and our own choice of resource tags as condition. Note that another implicit prerequisite is that the programming running in the Pod use a newer version of AWS SDK to access cloud resource.

This blog post has good details, including a diagram and a walk-through.

Comparison

Both EKS Pod Identity and IRSA are here to stay. I’m afraid this is going to create confusions. I put the following table for their comparision:

	IRSA	EKS Pod Identity
Pros	– in use since 2019 – support EKS, EKS-A, ROSA – support all EKS versions	– support role session tags – no dependency on OIDC identity provider – create an IAM role once for all clusters. the role can be created before cluster – cross account access through resource policies and chained AssumeRole operation
Cons	– Cannot create OIDC identity provider, until the cluster is ready – One OIDC provider per cluster, with the risk of hitting quota – Trust policy sprawl as more clusters are created	– the program has to use newer version of SDK. – ony support EKS – Pod Identity Agent (DaemonSet) can’t run on Fargate

Compairing IRSA with EKS Pod Identity

The blog post also contains a long table for their comparison. In the near future, I will have to check the SDK version of a workload in order to assess whether EKS Pod Identity will function. This is a restriction because it depends upon software builder disclosing the SDK version used. The EKS cluster also needs to host daemonSet on a node agent. On the other hand, go with IRSA if portability between EKS and EKS-A and ROSA is of concern, because the IAM service principal pods.eks.amazonaws.com is dedicated to EKS.

The blog post also gives the migration step as follows:

Ensure EKS cluster is above 1.24, and install the add-on for EKS pod identity agent.
Ensure the SDK running in pod meets the version requirement.
Update the IAM role’s trust policy with the new principal “pods.eks.amazonaws.com”

So the EKS Pod Identity mechanism still requires an IAM role. It does not required an OIDC identity. The service account connects to IAM role via an agent on the node.

Summary

A good design concerns not only functionality, but also streamlined configuration experience. EKS Pod Identity is a great improvement over IRSA heading the right direction. It just came out two months ago so still too early to adopt, especially without knowing the workload details. For now I tend to use pod identity as a backup mechanism when IRSA isn’t available for some reason. However, I recommend starting to introduce the Pod Identity mechanism for all new EKS clusters and new workloads.

Workload Identity on Kubernetes 1 of 2 – AKS

Sat, 23 Dec 2023 16:43:00 -0400

As applications are moved to the cloud, the application workload hosted on virtual machines need to interact with cloud resources. For this, we need an IAM solution with two mechanisms:

a (non-human) identity in the cloud service platform (CSP), to represent the application;
a way to grant permission to this identity, so it can manage resources

CSPs such as Azure and AWS have their own implementations of the two mechanism. In Azure, we have Entra workload identity (including service principal and managed identity) for the first mechanism, and Azure roles for the second. On AWS, they are the identity pool capability of Amazon Cognito and IAM role. Next, what about the workload on managed Kubernetes service? Essentially, we will need to more mechanisms:

a native Kubernetes identity to represent the workload (Pod);
a way to map the Kubernetes identity to the identity in CSP

Kubernetes Service Account is designed for the first item. The second mechanism is for the CSP to address. In this post, let’s examine this in Azure. Specifically, how does Azure manage workload identity with Azure Kubernetes Service (AKS).

Node Identity and Cluster Identity in AKS

Let’s define what exactly is a workload identity. In Azure we think of it as one type of non-human identity. In our context, workload identity in the broader sense contains:

the identity that represents the control plane (or the whole cluster)
the identity that represents the node (or kubelet process)
the identity that represents the application in a Pod (workload identity in the narrow sense);

So it is important not to confuse these identities. In this section, I’ll focus on 1 and 2 since they are part of workload identity in the broad sense. In the rest of the sections, I’ll discuss 3, and use the narrow sense of workload identity.

When we create an AKS cluster, we create both a cluster control plane and a node pool. Both the control plane and the nodes need to provision cloud resources using Cloud API from Azure. For example, if we use Terraform’s AzureRM provider to create an azurerm_kubernetes_cluster resource, then we specify the cluster’s identity using service_principal or identity block. We specify the nodes’ identity using the kubelet_identity block, because kubelet is the process that runs on each node.

Even though a cluster builder might be tempted to assign the same identity to both Control plane and kubelet, the security best practice is to keep them separated. It is also the responsibility of the cluster builder to distinguish activities by the control plane and by kubelet process on each node, and attache an Azure Role with minimum privilege to each of the identities.

These two types of identities (control plane and kubelet) are relatively straightforward. In order to use them, we don’t have to play with Kubernetes objects. In the next section, we’ll continue to discuss the identities that represents each Pod in Azure. We now refer to them as workload identities, but the first available technology was pod managed identity.

Pod Managed Identity in AKS

When I first worked on Azure Kubernetes, Pod managed identity was in preview and was the recommendation. However, Microsoft renamed it (to Microsoft Entra pod-managed identities) and then deprecated it after a couple years of preview. As of Oct 2022, the recommended mechanism becomes Microsoft Entra Workload ID. For simplicity, we refer to the deprecated mechanism as “Pod Identity”. We discuss pod identity only for the purpose of understanding why it is no longer recommended and what is missing in it. For new workload deployment, we should always use workload identity.

For Pod Identity to work, a feature flag EnablePodIdentityPreview must turn on. Pod Managed Identity operates on a Kubernetes controller called MIC (Managed Identity Controller) and a DaemonSet called NMI (Node Managed identity). You start with an Azure managed identity with appropriate roles. Once you installed Pod Identity, there will be two CRDs AzureIdentity and AzureIdentityBinding. To grant Azure permissions to a Pod, you create a CR for each CRD. The AzureIdentity CR connects to your Azure managed identity. You also create an AzureIdentityBindign CR. When declaring a Pod, you link to AzureIdentityBinding by using the label aadpodidbinding.

There are two problems with pod managed identity. First, there is a vulnerability when it works with kubenet as network plugin. This vulnerability requires an additional mitigation step. Second, it does not make use of Kubernete’s Service Account. Let’s discuss in the next section why it’s favourable to use Kubernetes’ ServiceAccount.

Kubernetes Service Account

In Kubernetes RBAC model, Service Account can bind to Roles to gain access to other Kubernetes resources. The most common use case is allowing the running application in a Pod to access other Kubernetes resources. When it comes to letting an application in Pod access cloud resources in the CSP, it makes sense to use Service Account, for a a consistent pattern.

A service account must carry a token to function. Each namespace has a default service account with the token mounted automatically. Each Pod created in a namespace uses the default service account of the namespace, unless otherwise specified. However, many security organizations do not considered this default behaviour as the best practice. For example, CIS Kubernetes benchmark 1.8 has these two recommendations:

Ensure that the default service accounts are not actively used (5.1.5)
Ensure that Service Account Tokens are only mounted where necessary (5.1.6)

In other words, we should create non-default service account with automountServiceAccountToken set to false. Then when declaring a Pod, we explicitly specify the service account and where to grab the token for the service account. One way to pass ServiceAccount token is through volume projection.

To allow a Pod to access Azure resources, we use the combination of Kubernetes Service Account and Microsoft Entra workload identity.

Workload Identity for AKS

Microsoft introduced Entra Workload Identities in late 2022 to address IAM issues around machine identities. It comes with some modern features such as conditional access (e.g. location-based access, anomaly sign-in detection, etc). A workload identity can be:

application: an abstract entity as the global representation of your application for use across all tenants;
service principal: the local representation of a global application object in a specific tenants;
managed identity: a special type of service principal that eliminates the need for developers to manage credentials

In our use case for AKS workload, we also make use of Azure’s Managed Identity. This part is the same as the pod identity mechanism. However, here we create a federated identity credential for managed identity. The OIDC federated identity credential is issued by the AKS cluster. Within the AKS, the service account references the identity by client_id. Here is the documentation for the whole process.

One of the improvements in Entra workload identity for AKS, is the use of service account, which obviates the use of CRDs. Another improvement is the use of federated identity, whose lifecycle is tied to the cluster. This pattern is not only neater, but also standard. We map a service account to a managed identity with federated credential.

Summary

On managed Kubernetes services, we need an integration mechanism to grant Kubernetes workload access to cloud resources. We discussed what’s needed in this integration mechanism and looked at Azure Kubernetes as an example. In the next post, we’ll discuss how this issue is addressed in Elastic Kubernetes Service on the AWS side.

WordPress Security Basics

Fri, 17 Nov 2023 12:02:00 -0400

Background

In 2019, I moved this site to WordPress hosted on an Amazon Lightsail instance. There were few visits at that time so I lived with the single-server architecture. The website traffic has since been in steady growth but I have been too busy to catch up with the WordPress security setup. In July 2023, a malware impacted this site as well as the web traffic. It took me several months to fix a few related issues but the traffic still has not fully recovered. This post is about the lessons.

The Incident

I first noticed the issue when I clicked on links to my web page from Google result and got redirected to some spam site. It did not happen 100% of time, but it is annoying enough. In the mean time, from Google search analytics I noticed traffic volume going up with a lot of traffic going to URLs that I did not recognize or create. Somehow these URLs have a lot of clicks and impression counts. These are signs of artificial traffic.

Obviously the site was hacked. The first thing to determine is whether the server access was compromised. From the audit log (/var/log/auth.log and auth.log.gz) I can see a lot of brute force attempts to connect but fortunately none was successful. That also prompt me to change the default SSH port and use ECDSA key pair. Since the OS access is safe, the hack happens at the WordPress level. I suspected the sideloaded plugins from a few days ago. So I immediately removed all sideloaded plugins.

The attack is called malicious redirect. The plugin puts creepy pages in WordPress directory without my awareness and direct user traffic via my website. To clean up the damage, I looked into my WordPress directories at /opt/bitnami/wordpress and found many suspicious signs:

There are directories with weird names, such as rexall-vitalmin, or q4lee3, etc
In each of those directories there was an index.php file and .htaccess file;
Those directories also have other files which look like red herrings;
All those files have the same date time (from July 6);

Other directories to look at are /bitnami/wordpress/wp-content/plugins, where I noticed two directories (named gokyfozaxy and q199n071) that are not accounted for; and /bitnami/wordpress/wp-content/themes/, which contains unknown directories.

Clean up and hardening

In one of the .htaccess file I noticed segments of mojibake (garbled texts). I first tried to manually remove those files, but the problems stayed. Because the malicious redirect did not happen consistently on every single click, I sometime had false impression that the problem went away.

However, the challenges with manual cleaning are: 1. there are too many bad files (.htaccess and php files containing mojibake segments); 2. some existing files are impacted with mojibake segments too. I found a free plugin called WordFence to scan the file directory for malicious chagnes, and delete the bad files or bad segments. I also tried a paid scanner (Malcare) which found an bad file in /bitnami/wordpress/wp-content/themes/. However, it also blocked my site so I removed Malcare right away. Using the combination of WordFence and Malcare appears to have cleared up the offending files. After restarting apache, the bad URLs are no longer redirecting to spam sites.

This time, I decided to harden the WordPress system given the evidence of brute force attack at different point of entries. At OS level, I mentioned the changes to SSH daemon configuration. At WordPress level, I used WordFence to perform several levels of scans for problems and and improved posture such as admin user’s MFA. I also noticed a few unrecognized wordpress users and used wordpress CLI to delete those and other unused users.

The wordpress.org website has some general guidance on what to do when a site is hacked, and a general guidance on hardening WordPress.

Back Links

Another clean up work I had to do is dealing with back links. Back links are URLs from other sites that references this site. There are several situations:

If it’s a made-up URL, then it returns 404. In my case, these are URLs that stopped working once I cleaned up my server from the incident. However, the sources are still using these bad URLs. They are bad back-links;
If it’s a legit URL, look at if it’s hot linking, such as another site directly access an image from my site. These are bad back-links;
If it’s legit URLs, and the referrer site has a good domain authority score. These are likely to be good back links

Generally, it is painful to deal with bad back links because I’m not in control. I used a few free backlink checker tools (e.g. Links report on Google Search Console, SEOMATOR, SEMRush free) and found a lot of spammy sites that I had to request Google to disavow. Otherwise, they may negatively impact the search performance.

Repercussions

In the next few months, my pages are no longer a stop for their redirect. However, web request for those invalid URLs keep coming. The bad pages are still in Google’s cache. There are a lot of page request with 404 return code, and we consider this an HTTP flood. The problem now is that the amount of 404 return code is impacting how my site ranks in search engine. To make it worse, the amount of these requests with invalid URL increase since August.

To fix this, there are two measures. First, in Google search console, I have to tell Google to remove those URLs from its cache. I have identified a number of prefix patterns, and submitted a request for each URL pattern. It takes google a day to have them cleared.

After that, the bad request will no longer come from Google users clicking on bad URL. In my case, the requests did not reduce significantly, suggesting that most of the requests come from bots. Therefore I had to figure out a way to prevent those bad request hitting my server, which is a typical web application firewall requirement.

Looking for such a solution for my WordPress Security I landed on Cloudflare. Cloudflare is pretty user-friendly with an easy-to-understand reference architecture. When I started, Cloudflare can import my DNS records, and guided me to change my name servers so I delegate my DNS management it. When I first move to Cloudflare the website gives ERR_TOO_MANY_REDIRECTS. I ended up having to go to SSL/TLS and set encryption mode to Full (strict) to get rid of this error. I also have to re-configure email forwarding as a result of name server change.

CloudFlare

Even for a self-hosted single-server site, it is very beneficial to place an Application Firewall upfront for WordPress security. I find CloudFlare are very useful service that provides everything else you need to host the web site. For example, it contains a domain registry itself. It manages DNS and allows email forwarding. In addition, it helps generate TLS certificate etc. The free tier covers everything for a small website, with the Application Firewall as the core feature. Within the free tier I can have these features:

domain registrar and name servers (not for free but at a reasonable cost)
SSL certificate (not for free but at a reasonable cost)
Request event tracking
redirect rule: zone apex to www, and /status to uptime status page
return code 409 for obsolete URLs (using routes and workers)
email routing and forwarding
WAF rules (path, parameter, rate, etc)
DDoS protection and Bot Fight mode
hot-linking prevention (i.e. other sites references images on your site directly)

I am still exploring features for CloudFlare. One stunning feature is routes and workers. Essentially you can serve a function in response to HTTP request at a specific route. This is particularly useful in scenarios where it is not straightforward to add web pages on the backend server. For example, I want requests with certain paths to return HTTP code 490 and do not want to mock with the WordPress server, we can make use of CloudFlare worker.

Lessons Learned

For WordPress security, never use suspicious plugins. Keep an additional layer of defense in WordPress such as Wordfense. It helps block malicious traffic that went through the first layer. It also helps configure MFA for administrators.

AWS Systems Manager is an Omnipotent Hodgepodge

Sun, 29 Oct 2023 21:32:49 -0400

Introduction to Systems Manager

AWS Systems Manager addresses a lot of SysOps requirements for configuration management, including server automation. In this domain, there is another AWS service called OpsWorks. However, with OpsWorks Stack, OpsWorks Chef and OpsWorks Puppet all coming EOL in 2024, the entire OpsWorks service is mostly deprecated. By partnering with leaders such as Chef and Puppet, OpsWork services represent the era when AWS needed to mirror the configuration management capability on premise, in an effort to convince customers migrating to the cloud. Today, AWS Systems Manager has evolved to fill a lot of gaps around configuration management for servers in the cloud.

Although AWS Systems Manager sounds like a single service. It consists a collection of many seemingly disparate capabilities that serves similar requirements around configuration management. In fact, many of the Systems Manager capabilities are built on top of a couple of what I call core capabilities, such as Session Manager, RunCommand, Automation. This post will review these core capabilities and how Systems Manager employs them to expand with other capabilities.

SSM Agent and Session Manager

What enables all other capabilities is the SSM agent installed on the EC2 instances. The agent running as a systemctl task by ssm-user on EC2 instances. Most of AMIs come with this agent pre-installed. It stores the logs in /var/log/amazon/ssm/. This agent works with an instance profile with a role with the AmazonSSMManagedInstanceCore managed policy, in order to communicate with AWS Systems Manager (ssm.<region>.amazonaws.com) backend. Because of that, you also need to provide a network path to the backend endpoint, either via Internet, or interface endpoint.

This communication also allows an IAM user to connect to an instance’s shell. A common use case is for private instance that do not have Internet access but do have access to SSM backend endpoint. In a previous post I discussed using Session Manager to replace a bastion host to connect to EKS nodes.

When launching an instance using an AMI with SSM pre-installed, the SSM agent should launch after all the config sets from Cloudformation Init are finished. As a result, the Cloudformation Init script is not able to communicate with SSM backend via the agent, unless you install and start SSM agent first on your own, in CloudFormation Init. To troubleshoot SSM, it is important to review its logs.

Through Systems Manager Hybrid Activation, the SSM agent can also work on virtual machines out of AWS and report back to with SSM backend. This gives on-prem servers the identities (instance tags, instance profiles) required for Systems Manager to manage them as if they were EC2 instances. As a result, extend Systems Manager capabilities to on-prem fleet (requiring advanced instances tier).

Types of SSM Documents

There are several types of document that SSM uses, including:

Command Document
Automation Document
Package Document
Session Document
Policy Document
Change Calendar Document

The AWS documentation has a table on what they each are for. Here I’ll focus on three types of documents: Command Document, Automation Document and Session Document.

The Command Document is for the RunCommand capability. It executes on EC2 instances usually performing tasks relating to the operating system or application. I think of a Command Document as an Ansible Playbook that consists of Ansible tasks. We can author Command document that runs configuration steps using plugins, such as aws:downloadContents, aws:runShellScript, etc. This feature directly competes with Ansible. To troubleshoot why a command fail on an instance, check the file ssm-document-worker.log in the ssm agent log directory. Each log entry should have a command ID as reference.

The Automation Document (aka runbooks) is for the Automation Capability. You can define sequence of actions for automation. There are many pre-defined actions such as executing AWS API calls (aws:executeAwsApi), run commands (aws:runCommand), or executing a Lambda function. Therefore a runbook requires an IAM role (Automation Role). The schema of action sequence (YAML or JSON) looks very similar to an Ansible playbook. The web console comes with an UI to visualize the action sequence but most of the time I’d rather .

Session Document is for Session Manager capability. AWS Systems Manager Session Manager uses Session documents to determine which type of session to start, such as a standard session, a port forwarding session, or a session to run an interactive command. In most cases, automation developers do not need to create their own Session document, because the pre-built ones are sufficient:

AWS-PasswordReset
AWS-StartInteractiveCommand
AWS-StartPortForwardingSession
AWS-StartPortForwardingSessionToSocket
AWS-StartSSHSession

In my experience, I use the AWS-StartSSHSession and AWS-StartPortForwardingSession documents most often. To establish SSH connection for forward port to connecting host for Remote Desktop session.

To author your own document, reference the schema correctly and use the latest SSM agent. However, I would explore if any existing shared document in the library already covers what you need. For example, the command document AWS-JoinDirectoryServiceDomain help join a Windows server to a managed Active Directory domain. The command document AWS-RunPatchBaseline is used by Systems Manager Patch Manager capability to check and apply operating system patches. They include steps for Windows, MacOS and Linux instances. The automation runbook AWS-AttachIAMToInstance helps you add IAM role to an EC2 instance.

RunCommand and Automation

The Run Command capability run on top of SSM agent. You can specify one or more target instances. You also specify other other options such as command parameters, rate control and where the output goes. This capability allows an IAM user to run command directly on the OS of an instance (using an OS user ssm-agent) and centrally keep track of those command runs on the AWS side. The most common commands to run on the OS is packaged into Command Documents. There is even a Command Document that allows you to run a pre-built Ansible playbook.

Another way this capability is extremely helpful, is that we can reduce the load of cloud init process. Traditionally, we put a log of logics in the user data script for the cloud init process to execute. The cloud-init mechanism comes from Linux OS and the execution of the user data script is not very transparent to troubleshoot. You have to check the cloud-init-output log from the OS. The use of the UserData script should be reserved for establishing communication with CloudFormation endpoint and SSM endpoint. From there, other automation tasks should be done using SSM capabilities (e.g. State Manager) for better manageabilities.

Take an example of joining a newly provisioned Windows server to a domain. If we do this in user data script, we will have a few problems. First, we can only tell success/fail state from logs in the OS. Second, if an OS user inadvertently removed the instance from domain, there is no mechanism to capture that. If we use Systems Manager’s RunCommand capability, along with State Manager association, the AWS management console will be able to tell whether domain joining is successful, and the association can detect when the instance is removed from domain, report this finding as out of compliance, and remediate the issue. We’ll discuss State Manager in more detail in the next section.

As part of automation, we often have to invoke AWS API calls, which happens outside of any target VMs. The Automation capability of Systems Manager is for this scenario. You can orchestrate your API calls using Automation runbooks. These automation steps do not execute on any target EC2 instance, so they do not rely on SSM agent. However, it needs its own IAM role to perform API tasks. This capability saves you from having to run API calls by creating a new Shell environment to run AWS CLI, or from your own Lambda function using the boto3 SDK library.

When we combine Automation and RunCommand capabilities, we can perform most of the automation orchestration steps. They are the core capabilities that further enable a variety of other Systems Manager capabilities.

Maintenance Window and State Manager

Maintenance window is a very straight forward capability to schedule RunCommand activity with a cron or rate expression. You can specify target by instance tags, define one or more tasks, and define a window of activity and at what point prior to the end of Windows should the agent stop performing more activities (cutoff). Each task can be a type of a RunCommand command, Step Function, Lambda function and automation runbooks.

State Manager is a similar capabilities with a lot of feature overlap with Maintenance Window. State Manager operates on the concept of associations. An association connects target instances to command document or automation runbook to execute. Similar to Maintenance Window, you can specify a schedule expression, document parameters and instance tags. State Manager was brought in to combat configuration drift. The associated document should consist of idempotent scripts so that a State Manager association can repeatedly execute these documents to ensure compliance.

Maintenance Window is more about scheduling one or more tasks. On the State Manager side however, association failure by default will be reported as out of compliance compliance. This is useful in scenarios such as keeping a Window instance in the domain, or keeping SSM agent up to date. You can choose either capability for many common setups but they have subtle differences. For example, for Patch management, you can use State Manager to detect missing patches and report compliance, and Maintenance Window to actually apply the missing patches. In fact, there is a document page on choosing between State Manager and Maintenance Windows to distinguish their best use cases.

Fleet Manager and Inventory

Fleet Manager presents a centralized view for all instances for users to perform common administration tasks, such as exploring file systems and logs, admin users and groups, manage registry and events on Windows instances, check processes and performance metrics. It also gives shortcuts to patch nodes, run commands, start session, etc. I think of Fleet Manager as a minimalist configuration management UI. It is not as sophisticated as those from Ansible Tower or Puppet but it comes at no additional cost.

A very useful feature of Fleet Manager is to run a web-based remote desktop to connect to Windows Instances. This saves the need for a bastion host as long as the instances have SSM connection. You will need the RSA private key to decrypt the Administrator password, which I would not recommend. If the Windows server is on a domain, you can enter your domain credential via Fleet Manager. If the users logged in via IAM identity center, Fleet Manager also has the login option for them via SSO using IAM Identity Center identity. When a user logs in this way, Fleet Manager uses RunCommand capability to execute AWSSSO-CreateSSOUser document against the server to create a local admin user.

Another aspect of configuration management is the inventory management. Unlike in Ansible, the term inventory in the context of Systems Manager refers to the metadata of instances, which includes installed applications, AWS components, network configurations, instance details, services, Windows registry and roles, etc. The full list of what is part of metadata is in the document and you can even define your own inventory item. To gather inventory data, we can makes use of a State Manager association to execute the AWS-GatherSoftwareInventory document. Once we set up the association, the agents will report inventory data back to Systems Manager. More importantly, we can create Resource Data Sync objects to write inventory data (along with compliance data) to S3 buckets, allowing downstream applications to consume. A common use case is to run Athena query against those bucket and produce QuickSight dashboard.

Patch Manager and Compliance

The Patch Manager also operates on State Manager associations. The automation runbook is AWSRunPatchBaseline, where you can just scan for missing patches or install them as well. The SSM document can run on all three platforms (Windows, Linux and MacOS) and determines which patches are missing relative to a the Patch Baseline. There should be at least one default Patch baseline. Each OS (e.g. Ubuntu, Debian, Amazon Linux, etc) classifies patches differently, and a patch baseline is a configuration that defines whether a patch is approved based on operating system and their classifications. The automation document also allows you to override the patch baseline. When executing the document to scan for patches, it records patch compliance information using the PutInventory API command. When using the document to install patches, you can run the document from a Maintenance Window and specify whether you need to reboot the target instance if required.

The compliance capability reports compliance status for instances. By default there are two types of compliance: association and patch. The association compliance detects whether a state manager association is failed on certain instances. The patch compliance, as just mentioned, checks whether patches are up to date relative to the specified patch baseline. You can also define custom compliance item (with put-compliance-items API) but the documentation isn’t clear on what exactly it can achieve and where on the instance does it pull the compliance status. From the example in put-compliance-items, custom compliance type seems to check the installation of additional software package in the inventory.

Other capabilities

Amongst the other capabilities, the one I use the most often is parameter store, which is a way to store a variable for different services to consume.

In the domain of change management, the change manager is a mini change management system. Organization can use it to manage their change process such as approvals. More importantly, you can fire automation runbook from change manager and tie it back to the change control item. Change calendar allows you to block changes during specific period. Both of them are organization level capabilities.

When it comes to operations management, the Incident Manager capability allows you to create response plan for incidents. Response plan can execute runbook actions once an incident is logged. It also helps you notify the on-call incident response team. On the other hand, OpsCenter capability allows you to create OpsItem, which also includes a way to execute runbook. The OpsData can aggregate to Explorer, which is a centralized dashboard for operations data. The Explorer, OpsCenter and Incident Manager capabilities can operate at organization level.

These capabilities around change management and operations management come nowhere close to full-fledged ITSM solutions such as ServiceNow or SMAX. However, they have the ability to trigger runbooks and natively integrate with other AWS services.

There is also a quick setup capability which uses pre-baked CloudFormation template to configure other services. For Patch manager the current recommendation is to use quick setup to configure patch policy.

Summary

Systems Manager has so many capabilities that I cannot cover everything in a single post. Here is a good walk-through. Some capabilities like session manager, fleet manager and state manager, are extremely helpful. However, in my opinion, there are two problems with grouping all these capabilities under Systems Manager. First, With too many different capabilities, this service lacks focus, which makes it difficult to learn. Second, some capabilities have overlap with other capabilities, or another AWS services, which also makes it confusing. I try to sort out how these capabilities enable each other in the diagram below:

This diagram may not be 100% accurate but it demonstrate the dependencies and can assist troubleshooting. For example, when compliance is missing data, check the execution history of run command. It also illustrates the key role of SSM agent as the underlying enabler of most of the other capabilities.

Overall, Systems Manager is extremely powerful. You can try to replace your server management solutions (e.g. Ansible, Chef and Puppet) with Systems Manager configurations. With a good understanding of its capabilities, you can build your fleet automation in an efficient and scalable way.

Orchestrate Landing Zone with Landing Zone Accelerator on AWS

Fri, 22 Sep 2023 23:05:04 -0400

As a continuation to the last post, we explore the Landing Zone Accelerator on AWS (LZA) as an orchestration tool in this post. LZA borrows a lot from the ASEA, an accelerator project to deploy the security reference architecture (SRA). LZA is a multi-purpose project that consists of both the orchestration engine (the accelerator itself) and a few reference architectures (as configuration files).

Comparison with Control Tower

First, let’s sort out how LZA is related to Control Tower. Control Tower’s main functionalities are available as an AWS service, with some customization capabilities available as a standalone solution on top of the service, as I discussed in the last post. Unlike Control Tower, LZA as a whole is a standalone solution. Luckily, the installation of the solution itself is highly automated.

I see LZA both as an extension of Control Tower, and as a complement to Control Tower. It is an extension of Control Tower because LZA can co-exist with Control Tower. We can configure LZA to enable Control Tower and use its Account Factory to provision new accounts (alternatively but not recommended, we can opt out of Control Tower and manage account creation on our own). I also see LZA as a complement to Control Tower because it comes with full end to end automation scheme for networking infrastructure and most of the services involved. This is missing in Control Tower, which leaves it with users to provision networking infrastructure in the customization.

Thanks to the infrastructure automation capability, even if you do not have a strong regulatory requirement, there are still good reason to go with LZA for its low-code automation capability. Below is a table that summarizes the differences:

Control Tower	Landing Zone Accelerator
– Multi-account management tool – Governance layer – Customization Framework to bring your own infrastructure automation	– can manage Control Tower – low-code automation engine for infrastructure automation and service deployment based on CDK – reference configurations based on common industry profiles and regulatory requirements

Comparison between Control Tower and Landing Zone Accelerator

As the name suggests, LZA is an accelerator so there is no expectation of its user knowing how to program infrastructure as code. However, it still expects its users to know YAML very well. Knowing how CloudFormation and CDK works can greatly help the users troubleshoot deployment.

Reference architectures in LZA

The input of LZA is configuration as code in YAML format. The LZA repository comes with a number of sample configurations to implement some industry-based best practices. The reference architectures currently include:

General best practices reference configuration: for clients other than the categories below;
Government customers: US Gov Cloud (FedRAMP compliant, on aws-us-gov partition), US State and Local Government, China (on aws-cn partition), Canada Federal (CCCS compliant) TSE-SE (Highly Trusted Secure Enclave Sensitive Edition) on commercial partition for governments, national security, defence, and law enforcement customers reference architecture;
Election: for election customers including elections agencies, committees and campaigns;
Healthcare: for healthcare customers. However, the document does not mention HIPAA compliance or anything related to the HIPAA Reference Architecture;
Finance and Taxation: for tax workload to secure Federal Tax Information (FTI) data;
Education: for education industry customers.

Many of these reference architecture shares a few common traits in the networking design. Take the CCCS reference as an example, the networking involves the followings:

Workload VPCs: consisting of a number VPCs for production and test environments;
Shared services VPC: hosting common services such as pipelines, Active Directories, etc
Endpoint VPCs: centrally hosting interface endpoints
Perimeter VPCs: acting as ingress, egress and inspection VPCs.

The Perimeter VPC hosts firewalls (either AWS Network Firewall or NGFW appliances behind Gateway Load Balancers). All the VPCs are centrally managed in an AWS network account, and are shared to other accounts using Resource Access Manager. The reference architecture document keeps the details of this architecture, which was derived from the security reference architecture (SRA).

Special Purpose VPCs

I consider the non-workload VPCs as special purpose VPCs. The shared services VPC is the most straight-forward. The Endpoint VPC is the most standardized. It is used to centrally host VPC interface endpoints for security and cost reasons. Unlike Gateway endpoint which is only available for S3 and DynamoDB, interface endpoint carries a standing charge and therefore should be consolidated. In addition, since interface endpoints are based on interfaces, we can centrally control the security group and interface policy.

To integrate the endpoint VPC, not only do we need to create those interface endpoint. We also need to account for routing (using Transit Gateway route tables) and name resolution. For name resolution, we need to create a Route53 private hosted zone for each DNS name, such as ec2.us-east-1.amazonaws.com and associate them with each workload VPC. Note that the interface endpoints DNS name may not always follow the same format. See the exceptions in my old post. Also note that this would create a lot of associations (between Private Hosted Zone for each Interface endpoint and each workload VPC). For example, 20 workload VPC with 30 private hosted zones will create 600 associations. To overcome this, use Route53 profile (introduced in April 2024).

Another special purpose VPC is the perimeter VPC. This VPC vary greatly between customers because of different requirement and historical preferences. One of the key design areas is the placement of NGFW, which is discussed in this post.

LZA Orchestration Engine

The installation process may feel complex at the beginning because we have to first install the pipeline to that installs the pipeline. The initial setup consists the following steps:

CloudFormation installs the installer. We start with a CloudFormation template to deploy the LZA installer itself. It deploys resources such as CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject). These resources are in the INSTALLER circle in the diagram below;
The installer installs the accelerator core. In the LZA installer, the CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject) drive the installation of the LZA. The input is the official LZA GitHub and we need a GitHub token for this step. The output is the actual LZA orchestration engine, including CodePipeline (AWSAccelerator-Pipeline) and CodeBuild (AWSAccelerator-BuildProject and AWSAccelerator-ToolkitProject). The user may specify their own GitHub repo as the configuration repo. Otherwise, a CodeCommit repo will be created. The LZA resources are shown in the CORE circle in the diagram below;
The acceleration core configures the landing zone. The LZA orchestration engine deploys actual resources in the landing zone, with the CodeCommit repo (aws-accelerator-config) or the specified GitHub repo as input.

If we enable Control Tower with LZA, we should first log in to management account and configure Landing Zone with Control Tower. we can also create (and register) the required OUs and accounts from Control Tower. Then we can deploy Landing Zone Accelerator with default configuration.

After the initial setup, we will need to iterate over the aws-accelerator-config repo. We implement our landing zone design in YAML configuration following the schema documentation. Changes in the configuration repo will trigger the pipeline (aka LZA’s orchestration engine) to redo step 3, whereas step 1 and step 2 are performed only once. The duration of step 3 is significantly longer than the first two steps.

Pitfalls

If LZA manages Control Tower, it expects existing OUs registered in Control Tower or it will report error. For account, LZA can create accounts listed in the manifest but not yet created. However, with the lengthy account vendor process for multiple account we run the risk of task time out in the pipeline.

During the installation, some account may run into quota limit. For example, the Networking Account usually have more than five VPCs whereas the quota is 5 VPCs per region per account. We need to increase the quota on those accounts.

The full deployment usually creates some SCPs. However, if we ever need to re-deploy a configuration, some steps steps might be blocked by certain SCPs. Attempts to temporarily detach SCPs from OUs, or modify SCPs often get reverted. The cause is an EventBridgeRule in us-east-1 region named RevertScpChangesModifySc. The rule should be disabled temporarily to perform the troubleshooting activity. We can do this with the following steps:

Disable the EventBridgeRule RevertScpChangesModifySc , which is only present in us-east-1 region;
Detach SCPs and note down what are detached, one OU at a time;
Go to the failed CF stack in the region, delete the failed stacks (after turning off termination protection);
Rerun the pipeline step from where it failed. This time it should go past the failure to the end, if SCP is the cause as per our assumption;
Re-attach SCPs. Suppose Security and Infrastructure OUs share one group of SCPs, and Dev, Test, and Prod OUs share a different group of SCPs;
Re-enable the EventBridgeRule;

Even with the EventBridgeRule RevertScpChangesModifySc disabled, when you re-run LZA deployment pipeline, the Accounts step will re-attach the SCPs using the AWSAccelerator-AccountsStack in the management account in us-east-1 region.

In general, how SCP works with organization structure is something to be very careful about, especially when the hierarchy consists of multiple layers of OUs. It is important to keep in mind, that deny statements in SCP take effect down the hierarchy, where as allow statements only affects the immediate child account of the OU where the SCP is attached to, as per the evaluation logic. As a result, an SCP with allow * statement (in the LZA-AWSFullAWSAccess managed policy) must be applied to Root, every OU at each level, and every account, for LZA to function.

In addition, there are some hard limits for SCP. Each SCP has a size limit of 5120 characters, and each OU can attach a limit of 5 SCPs.

Challenges

Powered by CDK, LZA automates the creation of a lot of resources. The configuration files uses the deploymentTargets attribute to allow users to specify to which accounts or OUs the declared resources will be deployed to.

Supporting many resources is a double-edge sword. Because the accelerator needs to go through every aspect of a landing zone, it is very slow to run. The accelerator pipeline may take as long as 40 minutes without any change to the configuration code. This is extremely slow if you just want to make some small changes in the configuration (e.g. update route table, add IAM role).

Even though LZA supports many resources, it’s not flexible with every resource. For example, today we can deploy IAM roles using RoleSet. However, in the trust policy of the IAM role you can only specify a two types of principals under the assumedBy attribute: account and service types. On the other hand a trust policy can support many other types of principals such as another IAM role.

AZ Mapping

Another important ability that LZA does not support is consistent AZ mapping across accounts. (Correction: this is now supported in LZA v1.5 as of Oct 2023). In some example LZA configurations, we deploy VPCs across multiple accounts using two or three availability zones referenced by their logical ID, such as us-east-1a and us-east-1b. However, AWS maps logical ID to physical ID and the mapping may be different in each AWS account. Using the same logical ID cannot guarantee the physical AZ are the same across account. As of LZA 1.5, the ability to reference physical ID in availabilityZone is supported in LZA configuration file.

Summary

I’ve spent a lot of time on LZA recently. It is extremely powerful. LZA streamlined the landing zone deployment process with configuration as code. It also allows users to customize their landing zone towards their own architectural needs and compliance requirement. For example, you can declare arbitrary SSM parameters in each account.

On the down side, the LZA deployment is time consuming through the pipelines. It tries to automate too many aspects of the infrastructure, which makes itself quite a complex project. Expect lots of changes in each new version.

The idea of being a low-code solution is to make it simple for end users but it often sacrifices flexibility. For example, if you want to create an IAM role in each new account that references the Management account ID, it is not possible until such feature is implemented in the accelerator. When the accelerator pipeline fails, it still requires deep CloudFormation knowledge to troubleshoot.

Orchestrate Landing Zone with AWS Control Tower

Sat, 19 Aug 2023 17:25:00 -0400

Following an introduction to AWS Landing Zone, I’ll dive deeper into Control Tower as an orchestration tool in this post.

More on Landing Zone

In data center operation, there are numerous tasks that other teams have to complete before the the deployment of an application. For example, the 42U server cabinet must be in place with dual powers. The cabinet comes with a network switch in the middle and each ethernet port must be provisioned. Once the server is connected to the central network, the NOC team assigns IP address, configure dynamic VLAN on the switches, and configures firewalls etc according to the connectivity requirement. The new server also needs to report to centralized monitoring solutions such as SolarWinds. In cloud operations, the scope and target of a landing zone is similar to those data center operations work, with networking being the core. The idea is that the landing zone ensures security, compliance and governance, so that applications (analogous to paratroopers) can focus on their primary responsibility.

Although Landing Zone is a general concept for any cloud service provider (CSP), each CSP has some prescriptive guidances on setting up landing zone in their particular environment. For example, an AWS prescriptive landing zone typically covers the following apsects:

Multi-account structure
Identity and Access Management
Governance (controls and guardrails)
Networking
Additional Security Services

The multi-account best practice is an important aspect in AWS as the OU structure dictates how effective Service Control Policies (SCPs) can govern the entire footprint. A landing zone orchestration solution should also apply guardrails and controls based on the organization’s security and compliance requirement. In addition, it is common expectation that landing zone orchestration solutions create required networking resources such that applications are ready to deploy securely.

This post is about Control Tower.

Intro to Control Tower

Control Tower is a landing zone orchestration solution available as an AWS service. I have three impressions over Control Tower. First, it is good with managing multi-account structure. It integrates closely with AWS Organizations and requires client to have a dedicated log archive account and a dedicated security tooling account, which also serves as audit account.

The second impression is Control Tower makes governance more straightforward. AWS seems to use the term control and guardrail interchangeably. Below is an illustration:

The detective controls only captures and reports violations. The preventive controls stops the violating API request. The proactive controls remediates the detected violations. Another way to look at the controls are the priority of the controls:

Mandatory controls: there are about 23 mandatory controls that are enforced on each account. As soon as a new workload account becomes part of Control Tower, these controls will come enforced.
Optional controls including strongly recommended controls and Elective controls.

There are many existing controls and AWS is still releasing new one (example). AWS Config service uses conformance pack to organize relevant controls. For example, there are sample conformance packs such as “Operational Best Practices for “NIST 800 53 rev4” and “Security Best Practices for EKS”. In addition, you can even build your own conformance pack. Once you deploy a conformance pack, AWS config deploys additional rules against the current environment.

The third impression is that Control Tower falls short with infrastructure automation. It provides a number of customization mechanism to leave infrastructure (mostly networking) automation with users. I will explain later.

Mandatory Accounts

The function of Control Tower does not depend on a specific account structure that aligns with the multi-account best practice. However it does require three mandatory accounts to set up automatically when creating control tower. The account where Control Tower is configured is the management account. The log archive account is the owner of S3 buckets that hosts buckets for loggings. The audit account is a restricted account that’s designed to give your security and compliance teams read and write access to all accounts in your landing zone. We also use this account as delegated administrator account for several security services at organization level. I’ll elaborate in the next section.

The management account is directly under root OU and there is no SCPs applied to the management account. Also Control Tower does not turn on AWS Config recorder and configure delivery channel on the management account itself.

The log archive account and audit account are under the Security OU (or otherwise named during the setup). At the end of Control Tower setup a set of mandatory controls will be applied at the OU level and thus effective to each account.

Apart from the mandatory accounts, oftentimes there is an Infrastructure OU. Typically we place shared services account under this OU. The account is for common services such as central networking, managed Active Directory, DevOps pipeline etc. Another optional OU is workload OU, where we can create new workload accounts or enrol existing workload accounts.

Each enrolled workload account have its CloudTrail configured with an organization-level trail, which sends log to the central bucket that log archive account owns. Each enrolled account also has AWS Config recorder configured with a delivery channel pointing to the configuration log bucket also owned by log archive account. Note that Control Tower does not turn on recorder on the management account itself.

Account Enrolment

Most likely a customer is already in use of AWS Organization. They can delegate one account as management account, and use Control Tower console to create a landing zone for the intended regions. During the creation, they will have to create two additional account, one as log archive account, and the other as security tooling or audit account. This is in alignment with multi-account best practices. After the creation of landing zone, the two new account are automatically enrolled in Control Tower.

Going forward, the client should create new accounts using Control Tower’s Account Factory to save a separate enrolment step. For existing accounts and OUs we’ll need to enroll them into Control Tower. At the beginning, the Control Tower Landing Zone (CTLZ) only has the mandatory controls, which are the bare minimum governance. When we enroll existing OUs and accounts, these controls (mostly detective and preventive) will extend to the newly enrolled OUs or accounts. Therefore, it is important to not introduce numerous controls prior to having all accounts and OUs enrolled. It is important to have a proper OU hierarchy upfront because both detective and preventive controls (Config Rules and SCPs) are typically applied at OU level and effective to all the children of the the OU.

To enroll an existing account, the account must meet some prerequisites. An IAM role with cross-account trust must be manually created. Since each account can have only one AWS Config configuration recorder and delivery channel, if an existing account already has one of them, they must be manually deleted using CLI.

Account Factory defines what happens when we create a new account and it is where we can bake in the customizations. When it comes to infrastructure automation, Control Tower leaves pretty much everything up to the account factory customization.

Delegated Administrator

Many AWS services are available as organization wide service. For example, CloudTrail, GuardDuty and even AWS Organization. Most of these organization level services allows you to specify one account as delegated administrator for the entire organization. For example, for System Manager and Service Catalog, the best practice is to designate an operation account or shared services account as the delegated administrator.

There are several security-related, organization-wide services, such as GuardDuty, Detective, Macie, Security Hub, Inspector, Audit Manager, and Firewall Manager. The best practices for these services is to designate the audit account as the delegated administrator. We think of the audit account as an aggregation point (or points for organizations that split the functionality across multiple accounts) for these AWS services. You can find this recommendation from the documentation of each service. However, currently Control Tower does not enforce this best practice. Even though Control Tower dashboard gives you visibility to security services such as GuardDuty and SecurityHub, you are still on your own to configure these services outside of Control Tower.

Whether you had never used these services, or you had previously designated another account as administrator for these services, it is good time to align with the best practice for delegated administrator right after Control Tower setup.

If you had previously designated other accounts, you need to take additional steps to revoke the old delegation and designate new delegated administrator account. These general steps can be performed from console or CLI and apply to all these services:

From the old administrator account, remove all the members
From the management account, revoke delegation
From the management account, delegate new administrator account (audit account)
From the new administrator account, invite all the members, and make sure to tell the service to automatically include new accounts going forward

Note that when you invite an account, that account must not have created, invited or associated with other accounts (as a member). In other words, we should never let a non-delegated administrator account create, invite or associate other account as member.

Control Tower Customization

Control Tower by itself does not play a big role in networking automation. You can create a VPC when provisioning a new account in Control Tower. That’s about what you can do. It does not address how the new VPC connects to existing networking space. It does not distinguish the existing VPC topology. To address this, Control Tower has a few customization options:

	AFC Account Factory Customization	AFT Account Factory Terraform	CfCT Customizations for Control Tower
Supported IaC language	Anything that Service Catalog Support (e.g. CloudFormation and Terraform HCL)	Terraform HCL	CloudFormation
Readiness	AFC is a native mechanism. You can specify the Service Catalog product during creation of an Account in Control Tower console.	Users need to first bootstrap the solution, which creates Terraform pipelines and other components.	Users need to first bootstrap the solution which creates a pipeline and other components.
How it works	Control Tower launches the specified Service Catalog product (aka blueprint) during account creation.	Quite complex. Explained in the section below	Control Tower posts a lifecycle events to Amazon EventBridge. A lambda function will process the event using pre-baked CloudFormation templates and Step Functions.
Prebuilt customizations	No. However, for certain products, there are some partner-maintained blueprints	Yes	No

AFC is fairly straightforward to use based on this post, as long as you know Service Catalog. It is a native capability of Control Tower. The other two ways for CT landing zone customization are not native capability and each require a separate bootstrapping process to deploy the pipeline-based solution.

Customizations for Control Tower

In CfCT, we first bootstrap the solution from management account, using this CloudFormation template on GitHub. We usually need to version control the custom configuration so we should select “AWS CodeCommit” as the value for the CodePipelineSource parameter. The bootstrapping (execution of this CloudFormation template) takes about 5 minutes.

The solution is based on Control Tower’s lifecycle event to trigger the pipeline with a source code repository (or S3 bucket). The source code repository (AWS CodeCommit) stores the custom configuration. A custom configuration consists of a manifest file in YAML format, made up with a number of resource sections. Each resource section references an artifact, either a CloudFormation template or a policy file to apply. The artifact can either be stored in the same repository, or in a remote S3 bucket.

Account Factory Terraform

Introduced in late 2021, the AFT solution is based on the terraform-aws-control_tower_account_factory repository. It is too complex as a solution in my opinion. At the beginning you need to have a designated OU and account for AFT and install the solution itself with Terraform. Note that, the solution does not address the state storage of the installation of the solution itself (chicken-or-egg). The installation also installed a few IAM roles required on the AFT management account. Having a landing zone management account and a AFT management account is not a neat setup.

Once installed, the solution consists of four repositories. Their purposes are as follows (copied from workshop instruction):

Account requests – handles placing or updating account requests. See example here.
AFT account provisioning customizations – manages non-Terraform customizations that are applied to all accounts. This stage runs before the global customizations stage. Examples available here
Global customizations – Global customizations – manages Terraform-based customizations that are applied to all accounts created by and managed with AFT. Examples available
Account customizations – Account customizations – manages Terraform-based customizations that are applied only to specific accounts created by and managed with AFT. Examples available

With this solution in place, when we create a new account via account request repo, quite a number of steps will happen after that, as the workshop instruction illustrated:

When I first looked into how this work I’m very concerned about the maintainability of this “solution”. The amount of services and pipelines involved in this solution makes it difficult to troubleshoot end to end. To improve that AWS even added an enhancement for request tracing. Nonetheless, I would not go with AFT just with how complex it looks like. For more details, check out the Control Tower workshop.

Summary

Control Tower helps you set up a Landing Zone without network infrastructure automation. It provides a couple of customization mechanisms, allowing you to bake in your own infrastructure automation. To build a landing zone, you can go with Control Tower in the following situations:

you have general regulatory requirement and need to enable governance capabilities
your networking stack isn’t centrally managed or isn’t large enough to warrant infrastructure automation
If you do need networking automation, then you will need the customization capability of Control Tower. In this case, your team should have good handle on infrastructure as code (CloudFormation or Terraform)

Out of the customization options, AFC seems the least complex to me. The other two, especially AFT, is too complex. We use pipelines to trail and error with other stacks, and the pipeline solution itself must be simple.

On the other hand, if you environment has strong regulatory requirement, or you are seeking a prescriptive network architecture, or your team does not have the capacity with CloudFormation or Terraform Templates, consider Landing Zone Accelerator on AWS.

Authentication to kube-apiserver via OIDC

Fri, 28 Jul 2023 09:20:00 -0400

Background

There are many benefits of using OIDC to authenticate to kube-api server, especially with multiple clusters that need consistent log-in experience. With the last post on how OIDC Authorization Code Flow works, now I will discuss options with authentication for kubectl to connect to kube API server.

To start, let’s look at the anatomy of kubeconfig file. The full schema is in the documentation. Looking at my kubeconfig file, there are three sections:

clusters: each entry specifies a cluster’s name, server address and certificate authority data (in base64 encoding or a file location).
users: each entry specifies a username. Some users are identified with client key and certificate. Some specify a command to provide client authentication. Refer to the authentication strategies.
contexts: each entry links a user to a cluster

Therefore, the key to use OIDC integration, is to use command to provide client authentication.

Vanilla Kubernetes

The documentation on authenticating has a diagram on how to use OpenID Connect tokens. The diagram does not give details on how access_token and id_token were obtained. So it could be any OIDC flow (Authorization Code Flow, Implicit Flow, etc) as we have discussed.

Although the instruction does not mandate which OIDC flow to use, we should use Authorization Code Flow in this architecture. The API server needs to trust the OIDC issuer, and the document covers how to configure API server.

In the diagram, step 2 and step 3 are required by kubectl itself does not perform these activities. All kubectl does is carry the JWT token in the Authorization Bearer. Vanilla Kubernetes does not provide a solution for OIDC integration. It only provides some instructions and we still need some helper scripts to glue all these instruction steps together.

There are many open-source project for this purpose. For example, Jetstack has kube-oidc-proxy and Int128 developed kubelogin. Other projects such as k8s-auth-client, k8s-oidc-helper, and gangway are no longer being updated. The kubelogin project remains influential. It has a clear diagram too:

From the diagram we can see kubelogin proposes authorization code flow.

Also, one design concern with Kubernetes control plane is the placement of endpoint. From this diagram we can see that even if the cluster endpoint is on private subnet, OIDC integration should still work. The control plane (specifically kube-apiserver) initiates outbound connection to OIDC Provider. There is no inbound connection to it from the OIDC provider.

In the kubelogin setup, the redirect URI is set to localhost:8080 because it stands up a server on the same host where browser is running. The browser can always resolve localhost. For a full configuration steps, Okta has this blog post on how to use kubelogin as helper, and Okta as Authorization Server to authenticate kubectl via OIDC.

Step-by-step with kubelogin

We’ll go through an example with int128/kubelogin because it works with any Kubernetes flavour, including managed Kubernetes services. It is also fairly simple. The instruction covers a few types of Authorization Servers (Google Identity Platform, KeyCloak, Dex with GitHub, Okta and Ping Identity). I’ll take KinD cluster as an example and use Azure AD as Authorization Server.

First, we’ll register an App in Azure Portal. Go to “App Registrations” and “New registration”. Give it a name “kubeoidc” and set Redirect URI to “Web” with URL “localhost:8000”. Click on Register.

The next page shows the details for this app. The Application (client) ID is important for next steps. Click on “Add a certificate or secret”, then “New client secret”, put in expiry and description. The secret value is generated and displayed on the next page, which is important for our next step.

We also need to find out the issuer URL. From the App page above, click on “Endpoints” and find out the URL from field “OpenID Connect metadata document”. My metadata document URL looks like:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0/.well-known/openid-configuration

The OIDC Issuer URL is the part before .well-known. In this case, it is:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0

Now we have collected what we need for the next few steps: ClientID, ClientSecret and OIDC Issuer URL. We can then create the KinD cluster, and reference ClientID and IssuerURL in the cluster configuration:

cat << EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
 kubeadmConfigPatches:
 - |
 kind: ClusterConfiguration
 apiServer:
 extraArgs:
 oidc-issuer-url: https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0
 oidc-client-id: 99999e88-e777-6666-c5c5-c444444d3d22
 - role: worker
 - role: worker
 - role: worker
EOF

kind create cluster --config=kind-config.yaml

The command will create cluster and configure kubeconfig file with a user named kind-kind as admin, which isn’t what we need. Now we use kubelogin helper command:

kubectl oidc-login setup \
 --oidc-issuer-url=ISSUER_URL \
 --oidc-client-id=YOUR_CLIENT_ID \
 --oidc-client-secret=YOUR_CLIENT_SECRET

This command will act as the client in the OIDC flow and prompts you to log in to Azure in a Browser. Once logged on, it gives you the next few commands to run. Since we’ve already created a cluster with the Issuer URL and Client ID, we can skip creating cluster, and run the steps to:

bind the Azure user to a ClusterRole
set up the kubeconfig locally with a user oidc, which needs to execute the oidc-login command

Lastly, we can test the oidc user with kubectl --user=oidc get nodes. We can also set the context to use oidc user by default. Voila.

From this example, we learned how to configure OIDC integration for any Kubernetes distros. The steps that we need to take are:

On the cluster side, we customize kube-api-server flag with OIDC provider info. Specifically, we provided Issuer URL and client ID in this example. In the OIDC Flow diagram, this step establish a trust from the Resource Server (K8s cluster) to the OIDC provider.
On the kubectl side, kubectl itself cannot fulfill all the duties of a client app in the OIDC Flow. It needs a helper script and we’ve made friend with int128/kubelogin.
On the Identity Store side, we expect it to be an OIDC-compliant Authorization Server. Otherwise, we consider using Dex as a broker in between.

When it comes to managed service, many allows us to customize the OIDC related flags for kube-api-server. Let’s look at how some managed services get this to work.

Azure Kubernetes Service

I find the kubectl authentication with AKS highly opinionated in its documentation. The recommendation is using Azure Active Directory as identity store but I don’t find it work with other OIDC providers.

To work with Azure Active Directory, you would configure the cluster and specify Azure role-based access control with the Group UUID in Azure AD. On the client side, you will need to install Azure’s kubelogin utility. Do not confuse it with int128’s kubelogin, which is for any cluster. This kubelogin is just for Azure. Once installed, you can use az-cli command to update your kubeconfig file, which call this utility from kubeconfig. To examine details about client configuration and AAD integration, check out the Terraform template in the azure directory of my cloudkube project.

The document only covers Azure AD integration and I tried to find if there’s a way to integrate with third-party OIDC providers. Unfortunately I have no luck. ChatGPT points me to a page about enabling OIDC provider but it is in the context of workload identity and it does not allow you to customize the issuer. So it’s completely irrelevant. Because you cannot customize OIDC issuer, etc, it simply won’t work with any third-party OIDC provider. Sure enough, most of Azure’s client use Azure AD anyways.

Azure makes it streamlined to configure OIDC integration of AKS with Azure AD, its own identity store. To my disappointment, it is currently not possible to integrate with third-party OIDC provider for authentication at cluster endpoint.

ROSA

RedHat OpenShift on AWS (ROSA) is a. However it reflects how OpenShift configures third-party identity provider.

Let’s first create a cluster off custom VPC and private endpoint. This is covered in my previous post about ROSA. After the cluster creation, we’re at the point where we can run oc command against cluster endpoint from Bastion host because it is a private cluster. However, being a private cluster is irrelevant to how we configure OIDC integration.

We’ll use Azure AD again as OIDC provider. So we need to register an app the same way as I did above in the kubelogin example. We need to have Client ID, Secret and OIDC issuer URL. For RedirectURI, go to your OpenShift console, and under Cluster, click on the cluster name → access control → identity providers → select OpenID. Note the page clearly states that this is Authorization Code Flow, and the OAuth Callback URL is provided. Use it to regiter App in Azure, and fill in the page with Client ID, Secret and OIDC issuer URL. Also fill in other fields accordingly and click on Add. Now you should have it configured!

Identity Providers for OpenShift cluster

The next step is trying to login. From Bastion host, run oc login and it will give me an URL to use. The URL contains the cluster endpoint, which resolvable from the Bastion Host itself. However I need a Browser session here, so I have to run Bastion host as SOCKS5 proxy and tell Chrome on my MacBook to use it:

~ open /Applications/Google\ Chrome.app --args --proxy-server="socks5://localhost:1080"

The browser session redirects to Azure AD for log in. Once completed the webpage will display a token that I can use with oc login command. Run this command with token from Bastion, I’m logged in:

oc login --token=sha256~3ZT5JGWELOcBzfohftEm9D2UwoOVFvATASuZk3_uxps --server=https://api.dhc.62q3.p1.openshiftapps.com:6443
oc whoami
oc get no

At this point, if I run oc whoami, I get the user name. However, this user cannot do anything. This is because it is not associated with a role yet. You grant more permission to this user: go back to OpenShift console, Clusters → ClusterName → Access Control → Cluster Roles and Access → Add user. Here you can map the user name to a role (let’s say ClusterAdmin). Then this user will have its priviledge:

The whole Flow works with private cluster, because the redirect URI is resolvable from the Bastion host. If you chose to expose cluster endpoint publicly (not recommended), you can perform the above steps directly from your MacBook or Laptop.

So the ROSA experience has been smooth. Unlike kubectl, the oc utility can act as the Client App in Authorization Code Flow. The other part of the configuration such as client secret and issuer URL are made in OpenShift console. Good job!

Elastic Kuberentes Service

EKS allows you to specify OIDC issuers from console or CLI to set up third-party OIDC configuration. There is a blog post from Okta on this, which works for private clusters. In the instruction, the author first manually created kubeconfig file with int128 kubelogin, and then bind ClusterRole with the user.

The blog post is very detailed. Instead of repeating it, I would like to discuss two SSO models available in AWS. I summarize them as below:

Both are SSO models for EKS. Model 1 (IAM Identity Center) is home grown as AWS using IAM Identity Center (formerly AWS SSO). Users start with an IAM principal (AWS construct) and use the aws-auth config map to tie them to Kubernetes groups. This AWS blog post and this support article are based on the IAM Identity Center model. On the other hand, Model 2 (Direct OIDC) is the vanilla Kubernetes approach. It takes group claim from OIDC identity token. The Okta blog post is based on this model.

The IAM identity center model works with SAML compliant identity providers, oftentimes Active Directory, although there seems to be a plan to support OIDC-compliant identity provider as well in the future. Even if it was supported today, I’d still prefer model 2 for working with any OIDC compliant identity provider because it’s simple. Why not leverage K8s’ native capability? For identity providers that do not support OIDC natively, or does not issue group claim (e.g. Google workspace), as we discussed, we can also consider alternatives such as Dex as identity broker. However, this model comes handy when a pipeline user with IAM role needs to authenticate into EKS.

One should always go for the direct OIDC model, if the upstream identity provider supports OIDC. The provider itself can even be AWS Cognito User pool. Cognito User Pool itself supports federated identity but again, I would directly connect EKS cluster to the OIDC compatible identity provider, instead of going through Cognito User Pool. As a result, the only use case where Cognito user pool is absolutely necessary, is when we need the Cognito user pool itself as the identity provider, as we have in CloudKube‘s eks directory.

Summary

Configuring OIDC provider for cluster endpoint can be confusing and we need to understand how OIDC flows work. I dived into OIDC in the previous post and in this post, I explained how to get it to work with vanilla Kubernetes. I summarized the three requirements and looked at the OIDC provider option with some managed services. Then I went through OIDC options for AKS, ROSA and EKS.

OAuth 2.0 and OIDC 2 of 2

Thu, 13 Jul 2023 21:24:01 -0400

I wrote a brief on this topic a while back. Now I need to configure OIDC in a few occasions I decide to dive deeper into the flows this time. As I stated in the last post, Nate Barbettini’s presentation from 2017 was awesome and I viewed it again. Slides are available here. Another great reference is this post from DeepNetwork Developer’s blog.

Back Channel and Front Channel

To understand why there are several different flows, it is important to understand the difference between back channel and front end channel.

In web development, traditional architecture involves a frontend (e.g. Browser, or any client-side app) and backend server. The web frontend is written in HTML, CSS, JavaScript, etc. There are also web frameworks such as Django, Angular, to save developers time. Backend (server-side) is responsible for storing and organizing data to ensure frontend can function. There might be multiple backend servers, such as session cache, data store, API server, etc.

In a nutshell, server-to-server communication is back channel, and browser-to-server communication is front channel. From security perspective, we regard front-channel as less secure, because we have less control of the location of the front-end and browser is easy to tamper with.

Back Channel and Front Channel (source https://www.okta.com/blog/2019/04/oauth-when-things-go-wrong/)

For traditional web applications with client-server architecture, when they communicate with third-party servers, they can initiate the communication from their backends, creating a back channel for better security posture. Single-page applications (SPAs) are applications without their own backends. When building an SPA, the front-end developer deals with frontend frameworks such as React, Angular or Next.js. When SPAs have to communicate with third-party APIs, they have to create a front channel. Also the API must support Cross-Origin Resource Sharing (CORS) for the browser to allow the cross-domain communication.

In the web development world, there is also Native App (aka Mobile App). Native App to Server communication is also considered back channel as we consider the client-side (Mobile App) secure. Nate’s talk makes the following recommendation for the flow (grant type) and I rephrase it as below:

Architecture	Recommendation
Traditional Web Application (client-server architecture)	There are both front and back channels. Use authorization code flow
Single Page Application (SPA, e.g. JavaScript) with API backend	Front channel only. Use implicit flow
Native mobile App	Use authorization code flow with PKCE
Microservices and APIs (Machine-to-Machine)	Back-channel only. Use Client credentials flow

The message, is that we should use Authorization Code Flow and use back channel, so long as the required component (backend) exists in the architecture. Note that the terminology for these types of applications may differ. For example, when you try to create an OIDC client for Amazon Cognito user pool, here’s how it categorizes client capability (app type):

Public client: A native, browser, or mobil-device app. Cognito API requests are made from user systems that are not trusted with a client secret.
Confidential client: A server-side application that can securely store a client secret. Cognito API requests are made from a central server.

Get used to different terms describing the same concept.

OAuth 2.0 and OIDC flows

The original problem that OAuth 2.0 (RFC6749) addresses is delegated authorization. In all OAuth flows, the authorization server issues an Access Token for the client to carry. The Access Token identifies the scope of resources that its carrier is authorized to access. However, third-party services do not always want to delegate authorization to the identity provider. They often just need identity information and want to perform authorization on their own. The OAuth 2.0 Access Token itself is all about permission and does not care about the identify of its carrier. It is not designed for authentication. OAuth 2.0 does not provide a standard way for Authorization Server to keep identity information of the principal. Many developers needs to address identity issue and they started to (mis)use the AccessToken to store identity information in custom fields, until OIDC came about.

OAuth: Access Token only

OIDC: Access + Identity Token

OIDC is a thin layer (5%) on top of OAuth 2.0 and one important addition is the ID token. The resource server can, in addition to being asked to allow access, now can understand the identity of the principal requesting to access resources from the ID token. The OIDC layer also uses standard set of scopes and proposes a userinfo endpoint for client to get more details about user information. The authorization code flows in OIDC and OAuth2.0 are roughly the same except for the additions.

In terms of the flows supported, the OAuth flows are defined in RFC6749, including the following grants:

Authorization Code grant
Implicit grant
Resource Owner Password Credentials grant
Client Credentials grant

The classic grant type is Authorization Code. After verifying with user, the Authorization Server fires a call-back to the client to pass the authorization code. The client, then takes the authorization code, along with its client ID and client secret, to fire a request to Authorization server in exchange for Access Token. The implicit grant skips the Authorization Code step and the client gets the Access Token in a one-stop shop via callback over front channel, which is less secure. The other two grants are less often used.

On the OIDC side, the specification document discusses three flows:

Authorization Code Flow (specification 3.1)
Implicit Flow (specification 3.2)
Hybrid Flow (specification 3.3)

The original OAuth2.0 flows should only be used in delegated authorization scenario. In most contexts, if we talk about ID token, and if our use case involves authentication, then we’re talking about OIDC not just OAuth. Since Authorization Code Flow is the classic one, out of all these flows, in the next section we take a closer look at the Authorization Code Flow in OIDC.

Authorization Code Flow in OIDC

We consider the Authorization Code Flow the baseline flow and others as variations of it owing to architectural limitations. When we mention OIDC we implicitly refers to the Authorization Code Flow unless the context suggests otherwise. Now let’s zoom in on it:

Here is the narrative from my own words:

The user launches client application, which detects that user has not logged in, and redirect to log in page.
The client app sends an HTTP request for authorization code to the /authorize endpoint of Authorization server. This request consists of the following fields:
- Response type: code, indicating it is requesting authorization code
- Scope: openid, standard for oidc
- RedirectURI: my.com/oidc-callback, Authorization Server will use this to call back with code.
The authorization server redirects the user to a prompt for log-in
The user completes authentication and consent
The Authorization server validates user information within its identity provider
The Authorization server fires an HTTP request call-back at the Redirect URI (on the backend), with Authorization Code.
The client app issues an HTTP request for ID Token and Access Token to the /token endpoint of Authorization server. This request consists of:
- Authorization Code (received from previous step)
- Client ID
- Client Secret
The Authorization server validates the information and process the request, and prepare the response with the following fields:
- ID Token: identifies the resource owner.
- Access Token: identifies what the client app can access
- Expiration
- (Optional) Refresh Token
The client app receives the tokens above in the HTTP response from the /token endpoint
With the Tokens, the client app issues API requests to the resource server
The resource server independently validates the token
The resource server send API response back the the client app.

There are also a few points of configurations. First, the Resource Server needs to trust the Authorization Server. The Authorization Server uses its private key to sign the JWT tokens and the Resource Server needs the public key to validates it. Second, the authorization server needs to know about the client app. We usually configure the Authorization Server upfront, to generate the client ID and secret. The Client app will keep them as part of its configuration.

It is important to note that, when I use the term client app (OIDC calls it client), the word “client” is relative to the Authorization server. The client app itself consists of both frontend (browser) and backend (aka relying party). In this flow, the authorization code is not exposed to browser.

Other OIDC Flows

Now we can discuss some flows for other architectures.

The Authorization Code flow has a close variation with the use of PKCE (Proof Key for Code Exchange). For native apps, postman and Okta recommend Authorization Code flow with PKCE. When client app first requests for authorization code, it also includes a challenge. After the callback, when it sends the authorization code back to authorization server in exchange for tokens, the request now adds a verifier. This way, even if the authorization code may not be securely saved, the authorization server can ensure it is the same client app that requests authorization code and that requests tokens.

An SPA or JavaScript app does not have a way to a. store Authorization code and b. listen on a call-back URI. As a result, it makes sense for the SPA to just fetch the Tokens directly. This make the implicit flow. The spec doc refers to it as simplified authorization code flow. The grant type is “implicit” because there is no intermediate credentials issued. In this flow, the Authorization server does not authenticate its client. The tokens may be exposed to resource owner or other applications with access to resource owner’s user-agent. This flow improve the responsiveness but we should be wary of the security implications.

As stated above, for SPAs (or in-browser JavaScript), we have to choice but the Implicit Flow because the client app is front-end only and we do not consider it able to securely store credentials. Client apps that can securely store client credentials may benefit from Hybrid Flow. In the hybrid flow, when the authorization server fires callback, the callback includes a single-use authorization code, along with ID token, access token, or both, depending on the provided response_type. Then the client app sends it back to authorization server, along with client credentials, in exchange for a second ID token and access token. The specification has a good table that compares the three flows:

Apart from the three flows, there are also some “unofficial” OIDC flows, that are not discussed in the specification. For example, Auth0 adopted some original OAuth2.0 grants in conformance to OIDC, including Client Credentials Flow with OIDC and Resource Owner Password Flow with OIDC. The client credentials flow is for machine-to-machine identity and is not concerned with user identity.

Identify the Flow

A challenge that I faced is to make sense of the Authorization Code Flow in real life. I realized that the components (Client App, Authorization Server and Resource Server) in Authorization Code Flow are conceptual. In real life we do not always find a counterpart that perfectly match their features. When we try to introduce OIDC for authentication, we often need to build our own solution, with additional tools.

Take the client app for example. We would need one or several components in real life to perform the followings in order to qualify as a Client App in the sense of Authorization Code Flow, it needs to:

know the authorization endpoint and construct the HTTP request for Authorization code;
stand up an HTTP service (relying party) to listen to call back, and parse the Authorization Code;
securely store Authorization code, and have access to client ID and client secret;
construct a request for tokens using client ID, secret and authorization Code received;
parse the tokens from the response from Token endpoint
to pass the tokens along

When developers builds an application with OIDC integration capability, they’d have to implement all these using the library of their programming language. In addition to application’s own server, the OIDC module will need its own backend capable of doing all the activities above. The alternative option is to introduce a OIDC capable client proxy service.

As for the Resource Server, it needs to have a trust on the Authorization Server, so that it can cryptographically validate the tokens that the Authorization Server has issued using the well-known public key.

On the Authorization Server side, as we discussed. It needs to provision client ID and client secret that itself can later recognize when client app connects to it. It also needs to have both authorization endpoint and token endpoint. Often times, the authorization server contains identity store and we’d like to call it the identity provider, but that is not always the case. A company may have a home grown identity store that does not support OIDC. In that case, to qualify as an OIDC Authorization server, they need a server proxy.

Open ID Connect Specifications

Despite the different implementation by different vendors, we often need to resort to the official standard documentation. This Open ID connect page lists all the specification if you expand “OpenID Connect specification” under Final Specifications. The most commonly used ones are:

OpenID Connect Core specification [OpenID.Core.Errata2], which covers the foundation and three login flows (Authorization Code, Implicit and Hybrid). This was developed early and the current version [OpenID.Core.Errata2] is from Dec 2023 but the two previous versions [OpenID.Core.Errata1] and [OpenID.Core.Final] had been around since 2014;
Open ID Connect Session Management [OpenID.Session], another core document that stipulates how to manage sessions, finalized in Sept 2022;
Open ID Connect Discovery 1.0 [OpenID.Discovery], which stipulates the hosting OIDC discovery document, finalized in Dec 2023;
Open ID RP-Initiated Logout [OpenID.RPInitiated], one of the logout flow specification, drafted in 2020 and finalized in Sep 2022;
Open ID Front-Channel Logout [OpenID.FrontChannel], one of the logout flow specification, drafted from March 2016 and finalized in Oct 2022;
Open ID Back-Channel Logout [OpenID.BackChannel], one of the logout flow specification, drafted in 2016 and finalized in Sep 2022;

Do read the specification when you’re configuring integration. It is worth noting that apart from the Core specification which has been finalized for a decade, most of the other specifications did not finalize until late 2022. Therefore, it is important for integrators to validate the compliance state of the components in the implementation.

OIDC Proxy

As a result, with regard to OIDC, there are two categories of proxies: OIDC client proxy and OIDC server proxy.

For example, in my previous post, I explained how to configure external authorization via OIDC in Istio. Looking at the diagram, it uses OAuth2 proxy to integrate with GCP as the authorization server. In this use case, GCP is natively OIDC capable, the the Hello Word App isn’t. Therefore, the OAuth2-proxy that we introduced is an OIDC client proxy.

For a corporate with Active Directory, the identity store only supports LDAP protocol. In order to quality the identity store as an OIDC Authorization Server, we would need a server-side proxy such as the LDAP connector in Dex, with the Active Directory as authentication source. The diagram of dex is a good summary of its role:

In some scenarios, we refer to this role as identity broker. Dex is an identity broker. Another important project to know is KeyCloak, which is sponsored by Red Hat and now a CNCF project. Although you can configure KeyCloak as an identity broker, it is much more than a broker. KeyCloadk is a full-fledged identity and access management solution on its own. It can act as the entire Authorization server. The diagram in this blog post summarizes its features well. It is for teams that wants to build their home grown identity store. Think of KeyCloak as a self-managed open-source alternative to IAM solutions such as Okta or Auth0.

The Amazon Cognito user pool plays a similar role. A user pool serves as an identity store to an app. The integration (no matter which flow and how Cognito calls them) is supposed to be OIDC compliant. However, as of date, the integration with Cognito user pool isn’t. For example, the logout endpoint requires client_id parameter where as the RP initiated logout specification has it optional. On the other hand, it can federate its own identity pool with a third party via standard protocol including OIDC.

In any use case where we need to bring OIDC integration, we need to start with the flow recommendation for each architecture, then we examine the existing component against the flow diagram. From there, we can identify the missing pieces and determine where and how we should configure the proxy.

Summary

The OIDC topic confuses me big time every time I need to configure identity store. With this post, I was hoping to elaborate on the Authorization Code Flow for OIDC. See OpenID certification for a list of providers. Further I discussed the two categories of proxies in the OIDC picture. Hopefully, when the OIDC topic comes back again, I will be able to quickly match which is which, and identify the missing piece to build a solution.

Kubernetes Platform as a Service and Red Hat OpenShift

Sun, 25 Jun 2023 11:10:15 -0400

The Three-layer model

Kubernetes is so complex that it becomes a buzz word itself. I categorize the related work into three layers: a cluster layer, a platform layer and an application layer, by their purposes. The three layers are illustrated as below:

Let’s examine each layer in this model and where the Kubernetes Platform as a Service fits in.

The Kubernetes Cluster Layer

At the bottom, the Kubernetes Cluster layer is the foundational layer. It focus on using self-hosted VMs or cloud resources to build a functional Kubernetes cluster and worker node groups. A functional cluster includes a highly available control plane, as well as scalable node groups that all communicate with the control plane. Cloud Service Providers like AWS and Azure provides managed Kubernetes service, which takes away the complexity (and flexibility as well) of managing control plane components such as etcd store and API server. The managed services also automatically provisions computing nodes and join them into the cluster. The cluster layer may also involve integration with of CNI and CSI, to ensure Pod-to-Pod communication and available storage classes.

Professionals working at this layer are infrastructure experts who understand networking, storage, as well as how to manage cloud resources or VMs, infrastructure as code. On a daily basis, they deal with VPCs/V-Nets, subnets, EBS/Azure Disk, File storage, EC2/Azure VMs, etc. When the team is doing a bad job at this layer, you might see symptoms like unresponsive cluster API, orphaned worker nodes, or kubectl failing to connect to cluster endpoint.

The tenants (applications) of the Kubernetes platform does not directly interact with this layer. If you decide to switch CSP vendor, this layer requires 100% re-engineering because the managed Kubernetes service by each CSP is different.

The Kubernetes Platform Layer

The Platform layer sits in the middle. When organization decides to adopt Kubernetes, they often underestimate the efforts required in this layer. This layer works on a functional cluster, without directly interacting with the underlying cloud resources. This layer involves any Kubernetes abstractions that do not creates tangible business value. Rather, this layer is an enabler. It allows the applications to deploy smoothly, evolve quickly, and more importantly, focus on the business.

Teams working on this layer needs to be Kubernetes experts. On a daily basis, they play with common CNCF toolings, such as Prometheus, ArgoCD, Istio, Cilium, Tekton, Open Policy Agent, etc. They are comfortable with Operators, Helm Charts, Ingress, etc. Inside of the Kubernetes cluster, they also manage the foundational services such as Event streaming (e.g. Kafka), PostgreSQL database (e.g. PostgreSQL), software-defined storage (e.g. Ceph), service mesh (e.g. Istio), Authentication (e.g. Keykloak) , etc. These services act as the infrastructure layer to the business workload. If the team is doing a bad job, you would see data loss with database, observability service not populating data, ingress does not process request, etc.

The tenants (application) share services in this layer. If you decide to switch CSP vendor. I estimate 80% of the work at this layer is portable, and 20% requires re-engineering. That is because each CSP offers different external resources, therefor the low level Kubernetes objects in this layer, such as storage classes, load balancers, supported CNIs are different. High level objects such as Kafka remains portable across platforms.

The Application Layer

The next layer at the top is application layer. Workloads in this layer are directly linked to the business value. The applications are very diverse. Most of the time, the release team is the main player at this layer. If the organization develops its own application, the software development team also work at this layer.

In terms of knowledge, the members of development team are experts in software engineering, and Software Development Life Cycle (SDLC), etc. On a daily basis, they deal with programming languages, product development, build and release. If they screw up their work, expect business errors, such as orders sent to wrong client, incorrect balance sheet, etc. This team has high visibility in the organization due to its direct link to business value.

This layer of work involves multiple tenants. Each tenant is isolated within their own namespace. When you switch CSP vendor, this layer should be readily portable with minimal effort.

It is also worth noting that, with solid platform and cluster layers, the team working at this layer do not write bespoke code for networking, observability, authentication and authorization, encryption and many other aspects not relevant to the core business. Once deployed, the application services are resilient, scale to demands, and cost efficient. This layer reaps the benefits of Kubernetes.

Kubernetes Platform as a Service

As the Kubernetes dust is still settling, a builder’s title may not always reflect which layer she or he focuses on. Today it is pretty common for infrastructure engineers to expand their role into the platform layer, or likewise, a software engineer to drill down to the platform layer.

The boundary between platform layer and cluster layer is clear. The cluster layer deals with underlying infrastructure, either in the cloud or on premise. They abstract away the complex infrastructure world from those working with the platform layer.

The boundary between platform layer and application layer is a little tricky to articulate. The application layer focuses on implementing the business logics. The platform layer takes care of the functions that are not part of business logic but essential to the business application. Take an HTTP request for example, application developer should not have to write code to terminate TLS (not part of business logic). They should only write the code to process the HTTP request (business logic). TLS termination is delegated to an Ingress, to be configured by platform builders.

The folks working at the Platform layer needs to interface with both sides. They provide Platform as a Service to the Application teams. However, their work appears mostly invisible in an organization. Their effort is oftentimes underestimated. There are several reasons for that. First, the platform layer does not directly create tangible business value. They are just someone else’s enabler. Second, their building blocks involve a lot of abstractions by Kubernetes API. Third, the idea of platform engineering is newly emerged. There hasn’t been a populous recognition of its value.

Red Hat OpenShift Container Platform

The platform team builds the platform with their choice of open-source tools. For clusters using OpenShift Kubernetes Engine, Red Hat introduces Open Shift container platform consisting of Red Hat’s opinionated (but validated) choice of toolings, for example:

OpenShift Service Mesh: Istio
OpenShift Streams: Apache Kafka
OpenShift GitOps: ArgoCD
OpenShift Container Platform Pipelines: Tekton
OpenShift Serverless: Knative
OpenShift Data Foundation: Ceph

Clients building their clusters with OpenShift Kubernetes Engine may build their own platform with the toolings in the OpenShift enterprise Kubernetes container platform. For more services, check out the documentation for OpenShift Container Platform. For customers with OpenShift Kubernetes Engine, their options to DIY platform are:

Entry-Level: Red Hat OpenShift Kubernetes Engine: Enterprise Kubernetes distribution on RHEL CoreOS
Mid-Level: Red Hat OpenShift Container Platform (RHOCP):
Plus-Level: Red Hat OpenShift Platform Plus: RHOCP + advanced cluster management, security, data management essentials, enterprise container registry

OpenShift runs the business model of Kubernetes PaaS.This is a unique business model that I do not find a matching competitor. Even if you choose to DIY your own platform, the Red Hat’s choices are still a great reference. The OpenShift enterprise Kubernetes container platform maps perfectly to the platform layer of the three-layer model, aiming to simplify the work in the platform layer.

Managed RedHat OpenShift

At first, the OpenShift container platform started as a value add-on to the Kubernetes Engine. Now it’s a separate product line in their business model. In the mean time, OpenShift partners with major CSPs, to develop the cloud service editions, including:

Red Hat OpenShift on AWS (ROSA)
Microsoft Azure Red Hat OpenShift (ARO)
Red Hat OpenShift Dedicated – on AWS and GCP
Red Hat OpenShift on IBM Cloud

These offerings are managed Kubernetes Platform as a Service in the cloud. Since RedHat is the only player in this model, we can refer to them as managed OpenShift services. In addition to an already-confusing world of Kubernetes platform portfolios, these offerings gives consumers even more options. On AWS for example, users have the following options:

Managed Platform: OpenShift Dedicated, managed by Red Hat
Managed Platform: Red Hat OpenShift Service on AWS (ROSA), managed by Red Hat and AWS
Self-built cluster: OpenShift Container Platform

This video discussed more details about these options, such as support model. It is also worth noting that these options tend to be much pricier than managed clusters such as EKS and AKS.

Since a Managed RedHat Platform makes it easy to deploy, let’s take ROSA as an example and create a cluster. To enable ROSA in AWS console, click on “Getting Started”. The next page ensures ROSA is enabled and checks other prerequisite such as meeting service quotas and creating ELB service-linked role, as show below:

Now, with an AWS account (and ROSA enabled), a RedHat account, and the rosa-cli utility, we can create a cluster with just a few commands. As a note, be wary of the cost and do not forget to delete the cluster afterwards.

Create a ROSA cluster

With the following set of commands, we can kick off cluster creation, using STS. We can bring our own VPC, so long as it meets certain prerequisites. I use the Terraform template in the vpc-base project, to create the underlying VPC. We’ll need the followings from this template:

The CIDR range of the VPC: as input with a default
The subnet Ids of the private subnet to place, printed in the output

The subnets are private subnets, because we want to provision the cluster with private node and private endpoint. When we use rosa CLI, we provide the CIDR and subnet IDs.

# start with AWS cli configured to the correct profile
rosa login # with redhat account and past token
rosa create account-roles --mode auto -y # this command creates the IAM roles ManagedOpenShift-*-Role, with RedHat account as trust entity
rosa verify permissions # optional
rosa verify quota # optional

export ROSA_CLUSTER_NAME="dhc" \
OPENSHIFT_VERSION=4.13.4 \
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \
AWS_DEFAULT_REGION=us-east-1

rosa create cluster --sts --private \
 --cluster-name $ROSA_CLUSTER_NAME \
 --multi-az \
 --private-link \
 --region $AWS_DEFAULT_REGION \
 --version $OPENSHIFT_VERSION \
 --enable-autoscaling \
 --min-replicas 3 \
 --max-replicas 3 \
 --compute-machine-type m5.xlarge \
 --machine-cidr 147.206.0.0/16 \
 --subnet-ids subnet-052852a1fb4d7d2ad,subnet-06d8d40ae39d55c47,subnet-0f67ce08bc588012c

The CLI will pick up the correct VPC by CIDR, and prompt you to confirm creation of private cluster. After the command kicks off, it will wait for OIDC provider creation, and role creation. Then it uses a Terraform template to create the related resources including VPC. Use this command to check status:

rosa list clusters
rosa describe cluster -c dhc

When the second command displays the state of waiting (Waiting for OIDC configuration), we can create OIDC provider:

rosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes
rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes

Throughout the process, we can monitor the install log (terraform output) with:

rosa logs install -c dhc --watch

In the log, you might see errors with terminals connecting to the terraform backend, which doesn’t necessarily indicate a cluster creation error. Always check the cluster state until it reports success.

Kick the tires

Eventually the describe cluster command will show ready state. We can now create an admin user:

rosa create admin -c $ROSA_CLUSTER_NAME

The command above prints an oc command (OpenShift CLI, equivalent to kubectl) with password to log in. Let’s examine the cluster with oc. Because it is a private cluster, the endpoint is not available publicly. However, it is accessible from the Bastion host. Use the SSM Session Manager technique from my previous post to SSH to the Bastion Host, which should have oc installed. To install oc yourself, use HomeBrew on Mac. On Linux or Windows, log on to OpenShift console, go to Downloads on the left pannel and find it out under CLI tools.

The oc command may report insecure TLS on the login URL. Wait for a few minutes for the certificate to come off as safe. Once you run the oc command with password, it should return “Login successful” and then we can connect to the cluster:

$ oc get node # or kubectl get node
NAME STATUS ROLES AGE VERSION
ip-147-206-135-41.ec2.internal Ready,SchedulingDisabled infra,worker 3m5s v1.26.5+7d22122
ip-147-206-155-141.ec2.internal Ready control-plane,master 25m v1.26.5+7d22122
ip-147-206-156-81.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-164-21.ec2.internal Ready infra,worker 3m3s v1.26.5+7d22122
ip-147-206-179-90.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-191-118.ec2.internal Ready,SchedulingDisabled control-plane,master 26m v1.26.5+7d22122
ip-147-206-192-232.ec2.internal Ready worker 19m v1.26.5+7d22122
ip-147-206-193-198.ec2.internal Ready infra,worker 3m20s v1.26.5+7d22122
ip-147-206-218-114.ec2.internal Ready control-plane,master 26m v1.26.5+7d22122

You can use oc the same way you’d use kubectl. Both works through SOCK5 proxy. In the meantime, log in to the RedHat console with your Red Hat credential, you can see the cluster in Ready state as well:

The rosa create admin command creates a htpasswd type (username-password) of identity provider (IdP) with a user named cluster-admin and a preset password. In real life however, we often configure third party IdP with OIDC integration. I’ll have to leave this to the next blog post.

We shall see the nodes as EC2 instances from AWS console as well. Note that there are three roles: control-plane, worker and infra. The infra nodes are for infrastructure services. These services (Ingress Controller, GitOps, Pipeliens) are the ones in the platform player as we discussed above.

There are many customizations you can make in this installation process and I’d have to defer to the ROSA documentation. To clean up, use the following ROSA command:

rosa remove cluster -c dhc

The output also gives you the command to delete operator roles and OIDC provider, for example:

rosa delete operator-roles -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes
rosa delete oidc-provider -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes

Then we can destroy the VPCs using terraform.

ROSA with HCP

Update Oct 2023:

The deployment above provisioned a few nodes for control plane, which add to the overall time to provision a cluster. In Aug 2023, there is a new option Hosted Control Plane (HCP) that came to allow users to provision a hosted control plane. This results in cost savings and shorter time to provision a cluster. Here is a table of comparison between the ROSA with HCP and ROSA classic.

Final words

In this post, I discussed the three-layer model and pointed out that platform layer isn’t as visible as the other two. I also experimented ROSA as a turn-key Kubernetes platform with its opinionated stack of services.

Some misinformed organizations even skip the entire platform layer in their estimate of effort. They build a cluster, ran a hello-world service and assumes they can start putting applications on the Kubernetes cluster. There are also customers who purchased the entire Managed OpenShift platform but only use it as a cluster. Yikes!

The concept of Kubernetes platform, or generally platform engineering is still spreading. The consulting team that I worked in full-time last year re-branded itself as platform engineering. Marketings are pushing it. Builders are doing it. We’ll keep an eye, on whether customers are buying it.

Connect kubectl to private Kubernetes cluster in EKS and AKS

Sat, 10 Jun 2023 19:31:00 -0400

Managed Kubernetes services give user a cluster endpoint and a number of worker nodes, with the choice. For each access, users have the choice of making them publicly available, or keeping them on private networking. In my opinion, any deployment beyond personal hobbies, should use Kubernetes private cluster, with both cluster endpoint and worker nodes on private subnet. There is no reason to expose computing nodes or Kubernetes management traffic publicly.

For worker nodes, it is fairly easy to put VMs on private network, but many companies still have the cluster endpoint exposed publicly. There are usually two reasons. First, their CI/CD agent is hosted somewhere else on the Internet (instead of on private network with private connectivity to Kubernetes cluster) and need to access Kubernetes cluster endpoint. Second, when the cluster needs to connect with third-party identity provider as OIDC provider, a two-way communication is needed.

There is a classic pattern of using a public bastion host (jump box), with a bastion host on the public subnet, routable to the private endpoint of managed Kubernetes service. Clients then connect to the bastion host via port 22 on a public IP address. The authentication is based on SSH key pair, or worse, password. The port forwarding (aka SSH tunnelling) capability enables all the magics.

Exposing a jump box in the public subnet with RSA key authentication is still not favourable. In this post, I’ll examine some secure patterns to connect to private endpoint with improved security posture.

AWS options

There are two problems. First, how to establish connectivity to the Bastion host in a private subnet. Second, how to use the Bastion host to proxy traffic to the cluster endpoint also in private subnet. To the first problem, there are two potential solutions: SSM Session Manager, and EC2 Instance Connect (EIC) with EIC endpoint (EICE).

SSM Session Manager was introduce in 2018. It runs an agent on the EC2, which initiates a connection to the SSM endpoint on the AWS side. This connection enables not only Session Manager, but also other Systems Managers (SSM) services such as Fleet Manager, Patch Manager and State Manager. The problem that session manager originally addresses is server management.

AWS launched EC2 Instance Connect (EIC) in 2019, and EIC Endpoint (EICE) in 2023. EIC addresses the problem with managing SSH key pairs at scale. It dynamically generates an SSH key pair for server access, based on IAM permission. However, it still requires an instance to have its SSH port publicly accessible. With EICE, it is no longer a requirement. In the diagram, EICE is placed in a private subnet, allowing EICE service to reach private instances at their SSH port.

Here is a comparison of the two:

	EC2 Instance Connect (EIC) with EIC Endpoint	SSM Session Manager
Location of Bastion host	Private Subnet.	Private Subnet
Need Ingress Port	Yes. Port 22 must open to the endpoint.	No. SSM agent initiate outbound connection from the instance
Traffic Path	AWS CLI → AWS EIC ES → EICE→EC2 Inst	AWS CLI → AWS SSM ES → SSM ← EC2 Inst
Authentication	AWS IAM and ephemeral SSH key when using AWS CLI directly AWS IAM and long-term SSH key when using SSH proxy command	AWS IAM and long-term SSH key when using AWS CLI directly or SSH proxy command
Work with OpenSSH	Yes	Yes
Cost	There is no additional cost for using EIC.	No additional cost, unless private SSM Endpoint.

Let’s take a look at each option.

EC2 Instance Connect

To use EIC, pick an AMI that has it pre-installed and ensure instance profile has correct policy, as the document states here. AWC CLI will make use of local OpenSSL client. So make sure there connection at port 22 is open. To make it work with EC2 instance on a private subnet, create an EC2 Instance Connect Endpoint on the VPC, and ensure that the security group of EC2 allows port 22 from the Endpoint. Run this command:

$ aws ec2-instance-connect ssh --instance-id i-00ea30a6e02db33fe

The command above simply generates a key pair internally, add the public key to the server side, and connect with SSH from the client side. The command takes you to an SSH session. Checking ps -ef | grep ssh on the client machine, you can see the full parameter of SSH, including the location of the ephemeral private key.

However, if you use AWS CLI open-tunnel as proxy command to ssh, then you’d still have to use the key pair used to create the EC2 instance. As suggested at the bottom of this blog post, the command is:

$ ssh ec2-user@[INSTANCE] -i [SSH-KEY] -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

This is a bummer, because with native SSH tool you do not get the primary benefit of EIC – ephemeral key pair.

SSM Session Manager

Now let’s look at SSM session manager. Similarly, it needs an agent installed and IAM role configured. You can connect to from web console but more importantly, from AWS CLI:

$ aws ssm start-session --target i-0531b19bec8ad022d

This command takes you to an SSH session with user ssm-user, without starting an OpenSSH client process locally. User do not have to manage key pair. There is also a document about using this command as proxy command, which uses an SSM document. I have one of the SSH config entry as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

This allows me to directly SSH to client using OpenSSL client (e.g. ssh i-0531b19bec8ad022d) by Instance ID. With this, I also need to specify my own OS user and matching private key.

I know I will use the OpenSSH client a lot from pipelines because it is very powerful. In both options, I have to live with managing key pairs myself. With SSM session manager’s proxy command, the instance does not need port 22 to open, which is a great advantage, in terms of security and operation. SSM Session Manager is a winner.

SOCKS5 proxy for kubectl

Either SSM Session Manager or EIC with EICE enables an SSH tunnel with key encryption between client (a local computer or a pipeline agent). On top of the SSH tunnel, we can build a SOCKS5 proxy. Kubernetes document has a good page on how to do this. I managed to get this to work with a few gotchas.

To put this in practice, I first created a VPC stack with a bastion host using terraform template from my vpc-base project. The terraform output will give the next set of commands to run to create a private cluster, using a manifest rendered from the file private-cluster.yaml.tmpl:

# cd aws_vpc
# terraform init
# terraform plan
# terraform apply
# ... run the given command ...
# envsubst < private-cluster.yaml.tmpl | tee | eksctl create cluster -f -

# Run this from a remote host without access to cluster endpoint.
# Run terraform apply and terraform output contains the variables needed for the next steps

# the command below may take 15 minutes to create a private cluster

eksctl create cluster -f private-cluster.yaml 

aws eks update-kubeconfig --name private-cluster

At this point, the kubeconfig file has been updated, but kubectl (from Internet or on-prem) is unable to connect to cluster endpoint (on private network). In order to

BASTION_SECURITY_GROUP_ID=$(terraform output -raw bastion_sg_id)
CLUSTER_SECURITY_GROUP_ID=$(aws eks describe-cluster --name private-cluster --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)

# In Cluster Endpoint's security group, open up port 443 to Bastion host
aws ec2 authorize-security-group-ingress --group-id $CLUSTER_SECURITY_GROUP_ID --source-group $BASTION_SECURITY_GROUP_ID --protocol tcp --port 443

# Test with connecting to Bastion host with ssh i-0750643179667a5b6, assuming .ssh/config file is configured as above. From the bastion host, you can test:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api 
# if you get an HTTP response, even an error code 403, the bastion host has TCP connectivity to cluster endpoint

# then we can start an SSH session as a SOCKS5 proxy on the remote host
ssh -D 1080 -q -N i-0750643179667a5b6

# add > /dev/null 2>&1 & to push it to background, or use ctrl+z after running the command
# to validate that the SOCKS5 proxy is working, you can run the same curl command with a proxy parameter:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api --proxy socks5://localhost:1080

# you can instruct kubectl to use the SOCKS5 proxy with the following environment variable
export HTTPS_PROXY=socks5://localhost:1080

kubectl get node

# alternatively, add "proxy-url: socks5://localhost:1080" below server attribute in ~/.kube/config file.

There are some pitfalls to watch for. On the remote host both ssh command and kubectl command implicitly uses AWS CLI. Therefore, make sure the profile and IAM role are correctly configured. For example, if SSM agent requires one IAM role, and kubectl is created with another IAM role, then make sure AWS CLI assumes the correct IAM role using environment variables, and use “aws sts get-caller-identity” to validate the IAM identity being used.

What about AKS in Azure

I touched on this in my Azure notes in 2021 and did a research again. Unfortunately, options are still fairly limited.

The first option is to use a managed service called “Azure Bastion”, which requires public IP and a dedicated subnet with the exact name of AzureBastionSubnet, as well as some additional requirement. I’m not impressed with these requirement because it is meant to be a managed service. The other option, is essentially to DIY a JumpBox. The idea is the same: put the jumpbox in a public subnet, which is routable to private subnets. When you need to connect to private VMs, get to the jumpbox first.

Apart from having to put the bastion VM on a public subnet, the pattern that we discussed above involving SOCKS5 proxy still works. Exposing a bastion host isn’t ideal but it still reduces attack surface significantly, comparing to exposing the cluster endpoints of all Kubernetes API servers.

Summary

Many immature Kubernetes configurations exposes private endpoint publicly. Having cluster endpoint in private subnet greatly improves security posture. In my opinion, there are very few situations where cluster endpoint must exposed publicly. Having private endpoint should be mandatory for all Kubernetes cluster. In the next post, I also cover how to create a ROSA cluster with private endpoint.

Kubernetes with Multiple CPU Architectures 2 of 2 – Node and Workload

Sat, 20 May 2023 01:34:00 -0400

The most common server CPU architectures today are amd64 (aka x86_64) and arm64. Although AMD developed the former first, Intel names it as x86_64 (or x64 for short). In terms of compatibility, they are the same. In general, arm64 architecture consumes less power and therefore mobile systems first favour it. Its power efficiency now drives a trend towards computing infrastructure. For example, Apple’s MacBook moved to M1 processor in 2020. Since 2018, Amazon’s Graviton processor has entered the third generation. In 2022, Azure also brought Ampere Altra processor, and GCP introduced ARM based VMs. Less power consumption ultimately leads to less computing cost.

I can only see more workloads gradually move to servers with ARM architecture. With Kubernetes, we will most likely have a fleet of computing node consisting of hybrid CPU architectures. We can take a look at what the arm64 adoption entails for workloads on Kubernetes.

ARM64 architecture

Graviton processor is on 64-bit Arm Neoverse cores, targeting for optimizing cloud-native workloads. Currently at AWS, the majority of arm64 instances use Graviton2 processor. This AWS blog posted the news about Graviton3-based general purpose (m7g) and memory-optimized (r7g) EC2 instances. At the bottom, there is a chart that compares the performance of Graviton3 with Graviton2, x86 and M6g instances. We can expect that in a few months the services that supports Graviton2 processor to start supporting Graviton3 processor.

In the serverless landscape, you can specify CPU architecture for Lambda function. If your runtime supports arm64 architecture, you enjoy up to 34% price performance improvement according to this post. In late 2021, AWS Fargate for ECS also started to support Graviton2 Processor with arm64 workload. As to Fargate for EKS, it has not supported Graviton2 processor as of yet, but is on track.

As to Kubernetes, I’ve discussed how to get container registries to support platform-specific images. So we can assume image registries all support OCI format image index(aka fat manifest), which points platform-specific images for arm64 and amd64. In this post, I’ll focus on the node and workload, with EKS as an example. Since control plane is a managed service, we will focus on the worker node, where the Pods are running.

Worker Node

The cloudkube project uses Terraform to build our test EKS cluster. One of the node groups consists of the new m7g.large instance (Graviton3 processor). For this new node group, the AMI type must be AL2_ARM_64, so it picks up an EKS optimized AMI for arm64 during node provisioning. The IAM role of each node has SSM policy so we can use session manager and pre-installed SSM agents to connect to each node. One the m7g node, I would like to check a few things:

The node CPU
The containerd package
The kubelet executable.

They all should be for the right CPU architecture, as the following commands clarifies:

$ lscpu | grep -i arch
Architecture: aarch64
$ yum list | grep containerd
containerd.aarch64 1.6.6-1.amzn2.0.2 @amzn2extra-docker
containerd-stress.aarch64 1.6.8-1.amzn2.0.1 amzn2extra-docker
$ file -b $(which kubelet)
ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=5c7a059f13f8bece4ce30f3357d57631c28bdde2, for GNU/Linux 3.7.0, stripped

We can also check image pulling with containerd. Let’s check what is the correct image first, by examining the image index with manifest-tool:

$ manifest-tool inspect digihunch/colorapp:v0.1

The index in the response tells us that:

the digest of the whole image index starts with 0fa335;
the manifest digest for the arm64 variant starts with 7479df;
the manifest digest for the amd64 variant start with 1bd198;

Now if we use ctr utility to pull image (“sudo ctl pull image”), we can see the correct digest for arm64:

The default behaviour of “ctr image pull” is to pull a platform-specific image, overridable with --platform or --all-platforms. On an amd64 node, I get the corresponding results as well.

Workload

Let’s ensure all workloads in the Namespaces are using correct images. We’ll deploy the colorapp, and then examine that along with some system Pods.

For example, DaemonSet aws-node has one pod per node. To verify the distribution, we can get to pods’ command shell and check CPU architecture with uname command:

$ kubectl -n kube-system get po -l app.kubernetes.io/name=aws-node -o name | xargs -I{} kubectl -n kube-system -c aws-node exec {} -- uname -m

The command above verifies that pods scheduled to arm64 nodes correctly. It does not however, proof that the arm64-specific image is being used. I find it pretty tricky to validate a container is using intended image on arm64 node. I have not found a working kubectl command. There is a plausible tag named imageID under container status. For kube-proxy Pod in kube-system namespace, there are two values. However, for colorapp pods, there is only one value with different format, even though they are scheduled to nodes of both architectures.

$ kubectl -n kube-system get po -l k8s-app=kube-proxy -o yaml | grep 'imageID:' | sort | uniq
 imageID: sha256:04beb3b811d345722d689a70a30bafa27e0edd412613bee76c3648b024b25744
 imageID: sha256:b9b6705d4ad6be861f0e98b7325e5106715ef21a82692f7e8a005a280f159518
$ kubectl -n default get po -l app=color -o yaml | grep 'imageID:' | sort | uniq
 imageID: docker.io/digihunch/colorapp@sha256:0fa335fdbcc3b644d57c8debe075775b19011985b6342adfb430e7011456d12e

This issue reports such inconsistency and the issue unfortunately did not get attention. The reporter also asks to have sha256 of the actual image. However, the Kubernetes developers regard this as an CRI issue. Currently we cannot tell exactly which image is used.

I figured out a workaround, by getting on the node and dump the image on the node:

$ sudo ctr -n k8s.io image list
$ sudo ctr -n k8s.io image export /tmp/x.tar docker.io/digihunch/colorapp@sha256:0fa335fdbcc3b644d57c8debe075775b19011985b6342adfb430e7011456d12e

In the export tar file review the manifest.json file which contains layer digests. We should find these layer digests match those of the platform-specific image’s.

Utilities

Since we can ensure that Pod running on a node can always pull the correct platform-specific image, we do not need to worry about Helm chart. We just need to make sure our container registry references an index digest that points to images of multiple architecture. For the same reason, we do not need to worry about pod autoscaling.

When it comes to node autoscaling, all node should support have kubernetes.io/arch and kubernetes.io/os labels (e.g. Karpenter). However, we generally prefer to expand the arm64 node group since it is cheaper. With cluster autoscaler, we can use priority based expander. With Karpenter, we can set weight so that the provisioner for arm64 node group carries higher weight.

Scheduling

With multi-arch image, the container runtime will pick up the right version of image. From deployment perspective, we do not worry about the difference between nodes in CPU architectures. However, in some use cases, we still want to schedule certain Pods to nodes with one CPU architecture over the other. I call these platform-specific workload.

We mainly needs to control scheduling behaviour. There are two mechanisms, node affinity, and taints & tolerations.

Node Affinity is based on node labeling. From the well-known labels, annotations and taints, all Kubernetes distribution should label their nodes with the kubernetes.io/arch and kubernetes.io/os labels. The value for arch is either arm64 or amd64. When we add a node affinity of requiredDuringSchedulingIgnoredDuringExecution type to Pods, scheduler takes matchExpressions under nodeSelectorTerms into consideration, when placing Pods to Nodes.

When a Pod has lots of nodeSelectorTerms, it can be brain twisting to sort through the logic. In that case we can use Taints and Tolerations. The idea is that once we taint a node, the scheduler will not schedule any Pod to the Node, unless the Pod has a matching Toleration.

In this post, the author customized the bootstrap script so the node provisioning process automatically taints arm64 nodes with arch=arm64:NoSchedule. Otherwise, we can manually taint a node:

$ kubectl get no -o wide # and check KERNEL-VERSION column, taint the ones with aarch64
$ kubectl taint nodes ip-147-207-3-164.us-west-2.compute.internal arch=arm64:NoSchedule

This can be a very useful technique when you’re not sure if every workload image are capable of multi-arch, and you want to avoid scheduling any Pods without tolerations on the arm64 nodes. A Pod cannot get scheduled on those nodes until you confirm their container images, and add corresponding tolerations.

Summary

Given the power efficiency, a lot of workload will gradually migrate to arm64 architecture. However, software will take a while to get ready. For example, hyperkit has not supported M1 processor and I still cannot use it on newer MacOS for Minikube. Hybrid architecture is here to stay and we need to have an end-to-end examination of our supply chain.

Kubernetes with Multiple CPU Architectures 1 of 2 – Container Image

Sat, 15 Apr 2023 12:17:00 -0400

While working on a multi-arch Kubernetes cluster, I came across quite a few issues with image logistics and decided to put these issues in a separate post here. The goal is to supply images with multi-arch support in a standard way. Therefore it is not advisable to rely only on arbitrary image tags to distinguish CPU architecture. If that was the case, each team may use different tag keys and values. The image consumers would have to build custom logics to pull the right image. The standard way is to use Manifest List (Docker’s term), or Image Index (OCI’s term) on the image registry, to announce the multi-platform support of the images in the repository, and to ensure that a container runtime can pick the appropriate index entry that matches the local node’s architecture and platform.

Introducing Image Index

For image index/manifest, there have been three standards, with two currently active:

Docker Image Manifest V2, Schema 1: deprecated in 2019
Docker Image Manifest V2, Schema 2: supports attribute for platform specific image
OCI Image Specification: supports attribute for platform specific image

In 2016, Docker contributed the Docker V2 Image specification as the basis of the OCI image specification. Therefore, the two current active formats (V2.2 and OCI) still look alike today. They are both JSON document and shares many attribute names. However, I take OCI as the standard since it is adopted in CNCF (e.g. containerd, quay.io). Most of the registries (e.g. ECR, ACR and even Docker Hub) support both formats, and you can tell the format by requesting manifest document. When working with Kubernetes we should expect OCI format index even though the Docker Manifest List v2.2 will most likely be compatible.

When examining an image manifest we should look for the following structure:

Manifest list and Manifest, source: https://www.opensourcerers.org/2020/11/16/container-images-multi-architecture-manifests-ids-digests-whats-behind/

In the Docker lingo, a “fat manifest” is simply a list of manifests, equivalent to image index in OCI terms. A manifest document consists of several attributes in sha256 digest, as the diagram shows. There are four types of digests and they should not be mixed:

index-sha256: for the index or fat manifest
manifest-sha256: for the manifest of a single container image
config-sha256: for the config section
layer-sha256: for the image layer files

If two images share a layer, then from their respective manifests, we should be able to find a layer with the same sha256 digest. If an index (fat manifest) references other manifest, we should also find that from the sha256 digest. When you reference an image, you should use the sha256 digest for the entire index or manifest list.

To view image index, “docker inspect” is insufficient. It is executed within Docker daemon, against the image pulled based on the CPU architecture of Docker daemon’s server. So let’s look at some tools to check index from remote registry.

Tools to view image index

The most popular tool is Docker’s experimental manifest inspect command. However, it is still not mature. For example, we have to use Docker CLI version 23.0.0 or later with OCI compatibility issue fixed. Prior to v23.0.0 (Feb 2023), Docker CLI were not able to correctly display a list of OCI-format manifests, and it simply says “no such manifest”. Even after 23.0.0, I still find it clunky. For example, it does not display the sha256 digest of the OCI image index itself (the digest on the far left of the diagram above).

To troubleshoot the reason, I borrowed some idea from this post, and have my shell script as below, to check the index for my colorapp image:

#!/bin/sh

ref="${1:-digihunch/colorapp:v0.1}"
sha="${ref#*@}"
if [ "$sha" = "$ref" ]; then
 sha=""
fi
wosha="${ref%%@*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
 tag="latest"
fi
apio="application/vnd.oci.image.index.v1+json"
apiol="application/vnd.oci.image.manifest.v1+json"
apid="application/vnd.docker.distribution.manifest.v2+json"
apidl="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
 | jq -r '.token')
curl -H "Accept: ${apio}" -H "Accept: ${apiol}" -H "Accept: ${apid}" -H "Accept: ${apidl}" \
 -H "Authorization: Bearer $token" \
 -w '\nResponseCode:%{http_code}\nResponseHeader:\n%{header_json}\n' \
 -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}"

It appears that the sha256 digest of the OCI image index itself is provided in the response header, instead of response payload. So docker manifest tool misses the header!

Luckily, there are some alternatives, such as skopeo or manifest-tool. I wasn’t able to get the former to work with OCI index. The latter displays my OCI index in a pretty format and I was able to view a few other indexes that I wasn’t able to with Docker manifest. So I have a good impression of it. So in summary, for the four tools to view image index, my recommendations are:

Docker manifest: still glitch as of 2023 but pretty widespread
manifest-tool: pretty solid, the output is pretty format, requires manual install
skopeo: not straightfoward to use. easy to install
Self-scripting in bash: only for troubleshooting

For the rest of this post, I’ll however continue to use docker manifest, given its popularity and availability. When it fails to display an index, I’ll try manifest-tool.

Inspecting Image Index

With docker manifest command (v23.0.1), let’s take a peak at three images:

docker manifest inspect --verbose osimis/orthanc:22.12.2
docker manifest inspect --verbose public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0
docker manifest inspect --berbose ubuntu:23.04

In the JSON document return by the first command (osimis/orthanc image), we see the following structure:

{
 "Ref": "docker.io/osimis/orthanc:22.12.2",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:20413096878fb56bf8d09af08cf4055993dbcf507526f0561b26fc4d0ed7affc",
 "size": 11227,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "schemaVersion": 2,
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "digest": "sha256:9de20d8a006c6377b85dba9f817d47048982bd0f15fac7daacb64f42060d4b6d",
 "size": 16518
 },
 "layers":[
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "digest": "sha256:025c56f98b679f70b7a54241917e56da7b59ab9d2defecc6ebdb0bf2750484bb",
 "size": 31412852
 },
 ......
 ......
 ......
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "digest": "sha256:a24cf4c71e6977b692bbf517eba7bf6f454d41ceab24c1b2694c3303bc718a1c",
 "size": 174739
 }
 ]
 }
}

We can see that the return is a single manifest. The Descriptor key suggests that it is built for amd64 architecture. The mediaType, along with SchemaV2Manifest and schemaVersion, suggest that it is a Docker Manifest (v2.2) format. Other attributes are summarized here. In the response from the second command(public.ecr.aws/amazonlinux/amazonlinux), we see the structure below:

[
 {
 "Ref": "public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d",
 "size": 529,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "size": 1478,
 "digest": "sha256:d27c2e45784db13b0b2bc89a52be6661aa1d53bd25c070b41626768c9c563c3d"
 },
 "layers": [
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "size": 62386320,
 "digest": "sha256:d78505e615251c4f4af6eaa9507b67917d263d23551dcc5a1eed3c012d32a54d"
 }
 ]
 }
 },
 {
 "Ref": "public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed",
 "size": 529,
 "platform": {
 "architecture": "arm64",
 "os": "linux",
 "variant": "v8"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "size": 1493,
 "digest": "sha256:a1ea533a0632c6501d7848c7ed481e8fb0398c3277c0d9fddf0b0fdcd5731c09"
 },
 "layers": [
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "size": 64003805,
 "digest": "sha256:71343c2791199c6e2c19c308cff6493497a02f57e225c11405e1934dc7428b3c"
 }
 ]
 }
 }
]

Comparing this structure with the first return, we noticed that the return is a list of two manifests, each with its own platform architecture. So the amazonlinux image has multi-architecture capability. It can serve as base image for custom images for both architectures. On the “image tags” tab of registry page, we can see a list of published tags. Each is tied to either a single image manifest, or a manifest list:

Each image tag is associated with a single manifest or a manifest list (aka “fat manifest”)

When you click on “image manifest”, you can see both “Image manifest media type” and “Artifact media type” values. When you click on “manifest list”, you see the “Image manifest media type” value, because the “fat manifest” does not point to a single artifact.

Now, let’s review the third command (ubuntu:23.04) response:

[
 {
 "Ref": "docker.io/library/ubuntu:23.04@sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0",
 "Descriptor": {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0",
 "size": 424,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "OCIManifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "config": {
 "mediaType": "application/vnd.oci.image.config.v1+json",
 "size": 2299,
 "digest": "sha256:beb2152822b716b4deac2996f16bc84db0a14b7cbc549579635590438f9c0e1d"
 },
 "layers": [
 {
 "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
 "size": 26638886,
 "digest": "sha256:db781b8aed497363312ef32499cbfac28821e0494db7f0cadc4e716853e02a12"
 }
 ]
 }
 },
 {
 "Ref": "docker.io/library/ubuntu:23.04@sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c",
 "Descriptor": {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c",
 "size": 424,
 "platform": {
 "architecture": "arm64",
 "os": "linux",
 "variant": "v8"
 }
 },
 "Raw": "......",
 "OCIManifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "config": {
 "mediaType": "application/vnd.oci.image.config.v1+json",
 "size": 2316,
 "digest": "sha256:eb2d2fb228861107934403e776544a3f516bc7123a1275d52f1992bada8e94d6"
 },
 "layers": [
 {
 "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
 "size": 25802344,
 "digest": "sha256:29d183ded65aecf549f39ef891c21feb9034b5b10f341533b4af297bb5c60bb8"
 }
 ]
 }
 },
 ......,
 ......,
 ......
]

I readapted the response document for simplicity. Notice that the document is also a “fat manifest” except that the mediaType, OCIManifest and schemaVersion keys suggest that it is an OCI format. The image supports more platforms (combination of OS and CPU architectures). The attributes for OCI index is available here.

Single-platform image Build

At the beginning of the supply chain, we build platform-specific image with CI/CD jobs. Traditionally, the docker build process can only produce images for the platform where the build command run. For the process to work in multiple platforms, we’d need multiple build agents of different platforms. Each agent runs build process and pushes artifact to the registry (with “docker push”). At the end we create a fat manifest that combines the images for all platforms, with “docker manifest create” command. A blog post on Docker from April 2020 refers to this as the “hard way”.

Since then, Docker introduced a new client Docker Buildx, a CLI plugin that extends the docker command with the full support of the features provided by BuildKit builder toolkit. One such feature is the ability to produce multi-platform images in one command run. There is also a push switch that helps you generate manifest behind the scene. In my own testing, I am able to build image for both amd64 and arm64 on MacOS (M1).

Multi-platform image build

Take the colorapp Python application as a simple example, I followed this guide to generate corss-CPU-architecture build. First, we can examine if we already have a build instance that supports our desired platform and whether that is already selected:

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
vibrant_hypatia * docker-container
 vibrant_hypatia0 unix:///var/run/docker.sock running v0.11.3 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default docker
 default default running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux docker
 desktop-linux desktop-linux running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

In the example above, I’ve already got one named vibrant_hypatia that supports linux/arm64 and linux/amd64 and it is selected. If that’s not the crease, we can create a new build instance and tell Docker to use it. Then

$ docker buildx create --use

Then we can run the actual build command. In the build command we specify the platforms and tags. We also specify –push so we can push the entire binary at the same time.

$ docker buildx build --platform linux/amd64,linux/arm64 --push -t digihunch/colorapp:v0.2 ./colorapp

Note that in this single command, the building for both platforms is completed and pushed with the correct OCI-compliant manifest. There is no need to build them separately and work separately on the manifest. We can use the manifest command from last section to verify the image manifest.

The behaviour of defaulting to OCI-compliant image index is a change in the buildx version 0.10 in Jan 2023. With —provenance=false, we can control manifest format with oci-mediatypes=true. In buildx 0.10, the default value for provenance changed from false to true, which always makes OCI the image manifest format. This change of default, along with the glitch of “docker manifest” prior to v23.0.0, had pretty big impact and raised confusions (such as this bug report from Ubuntu, and the issue in this post). However, since I take OCI-compliant format as the standard, I do not have a problem with this change. We can verify the image manifest with:

$ docker manifest inspect --verbose digihunch/colorapp:v0.1. ## docker CLI version > 23

Platform-specific image

Our Dockefile looks like this, with a base image.

FROM public.ecr.aws/amazonlinux/amazonlinux:2
RUN yum update -y && yum install -y python3 && yum clean all && rm -rf /var/cache/yum
COPY serve.py ./
RUN chmod +x ./serve.py
CMD ["python3", "-u", "./serve.py"]

To build platform-specific images, we need the base image(amazonlinux) to support multi-arc too, so that the build process picked the correct platform specific image as base. We can verify this is the case by looking at the sha256 digest of the first layer of each colorapp variant. They are d78505 for the amd64 image, and 71343c for the arm64 image. We can find the same layer digest from the amazonlinux manifest.

We now have mutli-architecture images, as well as the index. Docker Hub shows both digests with their architecture, although unlike ECR it does not show whether the tag is a manifest list or a manifest:

In real life, our supply chain usually has multiple levels of base images or multiple base images. It is important to ensure platform-specific image are available by examining their manifests.

In the Dockerfile, we consider it a best practice to reference base image by digest instead of by tag. We just need to make sure the digest actually points to an image index (manifest list), with each manifest points to the platform-specific image.

Summary

Now we’ve build the first part of our supply chain with arm64 capability. We have build platform specific images that can use their own platform-specific base images. We’ve also created an OCI-compliant image manifest to announce the image supports multi-architecture. Next, we will examine how containerd on different CPU architectures consumes the platform-specific image.

Landing Zone in Azure – Introduction

Sat, 25 Mar 2023 01:30:00 -0400

I recently renewed my associate administrator certification, and feel it’s a good opportunity to brush up on Azure landing zone.

The lame part of this is the semantics. I found many similar terms across cloud service provider (CSPs). In the context of Azure, it makes sense to clarify the terms again for Cloud Adoption Framework (CAF) and Cloud Operating Models.

Cloud Adoption Framework (CAF)

Similar to AWS Cloud Adoption Framework (CAF), Azure also has the concept of CAF and it means the same thing. This part may feel lofty, but it’s in fact foundational. To get started on the cloud there are thousand ways to configure the foundation (right or wrong). The adopter needs CAF to navigate through the offerings and define what they can achieve.

The CAF documentation is good although length. The most “beefy” part is Ready section.

Cloud Operating Models

Every cloud company has some narrative about cloud operating model. For example, Here’s Hashicorp’s definition, and here’s AWS‘ white paper on it. In the context of Azure, the CAF document gives some guidance on developing your own operating model in alignment with the CAF. In addition, it also gives a few example cloud operating models:

Decentralized operations
Centralized operations
Enterprise operations
Distributed operations

There is a comparison table that highlights their differences as well as an accountability chart proposing team divisions. Another insightful table is the one that list out implementation starting point and typical path of iterations for each operating model. The table also suggests that Azure Landing Zone includes two implementation options: starting small and CAF enterprise-scale.

Landing Zone at High Level

Followed by Cloud Operating Model is the design and implementation of Azure Landing Zone. There are currently eight design areas:

Billing and Active Directory tenant: including Azure AD tenant
Identity and Access Management: including hybrid identity
Network Topology and Connectivity
Resource Organization: different levels of resource containers
Security
Management
Governance
Platform automation and DevOps

Out of the many design areas, I fell short off IAM and Network so I’ll try to discuss them in more details below in the next section. As for resource organization, apart from Resource Group and Subscription, it is also important to understand management group.

Most cloud engineers work with subscription and resource group. That is where a lot are going on. For enterprises however, Azure has to address the requirement for the capability of top-down enforcement. Management Group provides a governance scope above subscriptions, provided that all subscriptions trust a single Azure AD account. Management groups may form a hierarchy of up to six levels to help you configure policies and access, so that the all the subscriptions under each management group have unified policy and access configuration. At the very top is root management group. Any assignment of user access or policy on the root management group applies to all resources within the directory. Because of this, all customers should evaluate the need to have items defined on this scope.

We can apply policy guardrails (e.g Azure Policy) at management group level so that the policies are effect across subscriptions. Azure Policy can also address operational compliance considerations by monitoring configuration drift.

Identity and Access Management

First, we really need to distinguish AD DS on Windows Server, Azure AD and Azure AD DS. In an old post, I discussed what is a Windows domain, the key role of a domain controller (to manage user identity, as well computer identity), and the fact that Active Directory is a complete redesign of Windows Domain system since Windows 2000. So we can start with AD DS on Windows Server:

AD DS on Windows Server: In the good old days, some common network administrative activities were to configure Active Directory (including the X.500 compatible database, the OUs, domains and forests) on Windows Servers, joining computers to the company’s domain, configure group policy, configure LDAP and Kerberos, upgrading Domain controllers, etc. Over the years, Microsoft moved these activities to the cloud and offer them as a managed service, known as Azure AD DS.
Azure Active Directory Domain Service (Azure AD DS): allows you to use managed domain services (e.g. Windows Domain Join, group policy, LDAP, Kerberos authentication) without having to deploy, manage or patch domain controllers. It is a SaaS offering to manage your domain controllers in the cloud, with a pay-as-you-go model. The counterpart in AWS is “AWS Directory Service” which lets you run Microsoft Active Directory (AD) as an AWS managed service.

In summary, both AD DS on Windows Server (self-hosted) and Azure AD DS (managed service) are identity stores that operates on Windows domains. Even though the latter is a managed service, it supports LDAP or Kerberos as integration protocol for third party applications (usually on-premis) to use. Both LDAP and Kerberos came around prior to the cloud era and they are not optimized for cloud connectivity. For example, insecure bind (on port 389) in LDAP is still prevalent. Kerberos is fairly complex to configure. However, they are not phased out right away because of their established presence as well as the domain’s awareness to authenticate devices.

Many organizations have to keep their domain service and when they move to cloud so they still have to use Active Directory as identity store. For this, Azure has Azure AD connect. On the AWS side, there is also an AD connector tool to allow on-prem users to log into AWS applications and services. With AD connector you can also join EC2 instance to existing AD domain.

Now let’s examine Azure AD.

Azure AD: is an IAM solution. It contains an identity store (with users and groups in a flat directory structure) but more importantly it integrates with external identity stores (including Domain Service, self-hosted or SaaS managed), which gives it hybrid-identity capability. A company can even sync their own on-prem identity store to Azure AD using Azure AD Connect. As an IAM solution, Azure AD also allows a company to tie their identity store to applications using modern protocols such as SAML and OAuth. Azure AD treats applications as objects, and they can represents either Microsoft Applications (Office 365, Dynamics 365, Azure) or third-party ones (Slack, Salesforce) as long as they use the supported protocol for SSO. The closest AWS counterpart of Azure AD is Amazon Cognito (arguably), even though their capabilities are not identical in every aspect.

Compared to Domain Service, Azure AD alone doesn’t have the concept of domain. Therefore you cannot join a server or PC to a domain and configure group policy. Azure AD’s native identity store is a flat directory structure without OUs or forests. Azure AD is NOT a replacement of domain service, either self-hosted or managed.

Now coming back to the Azure landing zone literature, the document lays out the key decision to make about identity:

A critical design decision for enterprise organizations adopting Azure is whether to extend current on-premises identity domains into Azure or to create new identity domains.
Azure Active Directory (Azure AD) and hybrid identity

The document even includes a comprehensive identity decision guide. After this decision, we’ll know what identity store to use. Then we can address the problem of platform access vs workload access. In other words, IAM of management traffic vs business traffic, which opens up topics such as RBAC, service principle and managed identities.

Networking

Back in 2017, Azure published a white paper about V-Net and it focuses on mesh network and hub-and-spoke. Back then Azure customers run multiple lines of business (LOB) on different V-Nets. The V-Net peering feature allows early cloud adopters to organize all their V-Nets in a mesh topology, ensuring all peers have access to all other peers, or a hub-and-spoke topology to aggregate shared resources in hubs so they can be shared by the spokes in the network.

When setting up a landing zone, network topology is a big decision. In the landing zone document today, clients need to consider the followings:

Traditional Azure networking topologies, including:
- large flat V-Net
- multiple V-Nets connected with multiple Azure ExpressRoute circuits/connections
- hub-and-spoke
- full mesh
- hybrid
Microsoft managed networking topology (on top of Virtual WAN)

From the 2017 white paper, most organizations at that time solve their need for network isolation and connectivity by creating a mesh architecture among various V-Nets. All nodes in the network are interconnected so network traffic is fast and can be easily redirected. However, mesh topology has significant disadvantages because it requires too many connections as the footprint expands, making it very costly to operate and quick to reach limit of number of peering links. It is not scalable. The white paper is to advocate the use of hub-and-spoke topology, which I will discuss in the next section.

It is worth-noting that, today (Jan 2023) one can create both topologies with Azure Virtual Network Manager. It is currently a preview service but I can foresee it will eventually get integrated with landing zone.

No matter which topology, another issue to address is connectivity to on-prem network, and to Azure PaaS services. If the traffic is light, we can use VPN gateway to configure IPSec tunnel that goes over public internet encrypted. It is simple to configure with a good aggregate bandwidth. This connection requires a VPN device on premise as well. A faster alternative is Azure ExpressRoute, which runs a private connection with a third-party connectivity provider. ExpressRoute is more complex and expensive to set up, but it supports much higher bandwidth with direct access and better SLA. In reality, many clients configures ExpressRoute with VPN failover for connectivity to on-prem network. For connectivity to PaaS services, options are service endpoint and private link endpoint.

Hub-and-spoke topology

In the traditional topologies, hub-and-spoke network topology is popular as the hub network provides a central point of management. Also it overcomes subscription limits and institutes a separation of concerns. The Azure documentation recommends hub-and-spoke architecture for larger cloud adoption efforts. If the footprint is even massive, we can even extend the model to a cluster of hubs and spokes.

A cluster of multiple hub-and-spoke

We can connect multiple hubs using:

V-Net peering
Azure ExpressRoute
Azure Virtual WAN
Site-to-site VPN

Within a single hub-and-spoke model, the Hub V-Net hosts shared services and acts as central point of connectivity (to many spoke V-Nets). Often in the Hub V-Net are Azure Bastion, Azure Firewall and VPN Gateway or ExpressRoute gateway. The spoke V-Nets (in same or different subscriptions) isolates and manage workloads in prod, non-prod, etc. Since a single V-Net cannot traverse subscription boundaries, you have to use V-Net peering (preferred), ExpressRoute circuit, or VPN Gateways. V-Net peering works across regions, and across Azure AD tenants. It is low-latency but isn’t transitive.

In some cases we also configure perimeter networks (aka DMZs) in the hub-and-spoke architecture, to handle external traffic. Perimeter networks host services such as External Load balancer, Azure Firewall, Azure Application Firewall on Azure Application Gateway or on Azure FrontDoor) , network virtual appliances (NVAs), IDS, IPS, and other security appliances. Incoming packets flow through the security appliances before reaching back-end servers. Internet-bound packets from workloads must also flow through security appliances in the perimeter network before they can leave the network. The document gives an example of a DMZ hub V-Net with two perimeter networks.

Virtual WANs

This page discusses what is WAN and SD-WAN. WAN connects multiple LANs in different geographic areas and is common with companies with multiple offices in different regions. WAN infrastructure may be privately owned or leased as a service from a third-party service provider (hybrid WAN). Companies may use IPSec VPN, SSL VPN or direct connection to build their WANs. Software-defined WAN (SD-WAN) leverages virtualization technologies, network overlays, on-site SD-WAN devices and software platforms to build hybrid WANs.

Azure Virtual WAN (similar to AWS cloud WAN) is a managed service to build a virtual WAN with a single operational interface that brings many networking, security and routing functionalities together. It simplifies end-to-end network connectivity (within Azure, between Azure and on-prem) by creating a hub-and-spoke architecture.

Virtual WAN

Virtual WAN is essentially an integrated connectivity solutions (in hub and spoke), with a global transit network architecture. The configurations, including spoke setup) is automated and troubleshooting is more intuitive. Global transit network configures multiple virtual WAN hubs with hub-to-hub connectivity, which ultimately enables any-to-any connectivity, with different paths discussed here.

The landing zone document recommends Virtual WAN for new large or global network deployments in Azure where you need global transit connectivity across Azure regions and on-premises locations.

Summary

Landing Zone configuration involves many components and there is no way to discuss everything thoroughly. In this post I put down my notes reading Azure landing zone documentation. Overall, working on landing zones requires learning a variety of services by the CSP.

A taste of IoT device tracking

Fri, 03 Mar 2023 10:53:00 -0400

Last fall I worked on a demo project for IoT device tracking on AWS so I had some reading.

Overview

From a strategic level, as servers move away to the cloud, AWS envisions that the footprint left on premise will mainly be IoT devices. The role of the cloud therefore becomes a central point of management for IoT devices. As a result, most of the AWS services for IoT are managed services. The best resources are two white papers: IoT Lens – AWS Well-Architected Framework and Securing Internet of Things (IoT) with AWS. The most important services are:

IoT Analytics: Makes it easy to run sophisticated analytics on volumes on IoT data. It connects to the underlying IoT data store and allow you to build your own analytical queries and Jupiter notebooks.
IoT Events
IoT Core: Core features for IoT.
IoT SiteWise
IoT Device Management
IoT Greengrass
FreeRTOS: IoT devices usually need to be small and power efficient. The software and OS running on the device is important. FreeRTOS is a real-time operating system for micro-controllers supported by AWS. FreeRTOS provides kernel, OS and libraries to securely connect your edge device to the cloud in no time.

In this post I will explore IoT architecture at high level. In real life, you program your device with AWS IoT Device SDK and AWS IoT API in different languages. In this post, I use a script to simulate GPS data, and push it to AWS IoT using SDK. Then I render the location using sample AWS code for Amazon Location.

IoT Architecture

When creating IoT services we consider registration and telemetry capturing flow. The Well architected white paper proposes registration flow as such:

Registration Flow

Device Registry to keep track of devices (aka Things). You can find where your devices are, and filter by a common feature (e.g. ModelX device only). Registration flow usually involves a testing of communication between device and server. The authentication must be 2-way where server needs to validate device identity, and device needs to validate server identity. You can use a unique X.509 certificate per device to adhere to security best practices on AWS. This way, if one device gets hacked, the entire fleet of devices is not affected by one certificate being compromised. An alternative authentication method is Cognito. With Cognito you can sign your users into a mobile application, so you use IAM policies to authenticate them into viewing different dashboards or viewing the data that pertains to them specifically. IoT Core policies can help manage authorization.

The white paper also proposes a few options for capturing telemetry:

Options for capturing telemetry

These options presents a common pub-sub pattern, where the device streams message by topic to the Message Broker in IoT core. The IoT core also involves policy and rules. A rule may involve a subscriber to consume the messages. A more comprehensive architecture from AWS IoT device connectivity workshop looks like this:

The message path from device to IoT core remains the same. When building an IoT solution, we first address the messaging path.

IoT Protocols

In IoT core, Device Gateway is the entry point for IoT devices connecting to AWS.It supports MQTT, WebSockets and HTTP 1.1 protocols, on top of TLS. Registration flow uses HTTP/REST protocol for provisioning, and with MQTT protocol for a message test. For pushing telemetry, we can use both but prefer MQTT (topic based) because of its advantages in IoT messaging. Here is a good article on the differences.

AWS has a white paper on designing MQTT topics, with a few communication patterns and best practices. The SDK documentation also explained the communication protocols, including:

HTTPS: publish only
MQTT: publish and subscribe
MQTT over WebSocket: publish and subscribe. Device Gateway will maintain long lived, bi-directional connections, enabling devices to send and receive messages at any time with low latency.

Pay attention to the authentication mechanism. As to what protocol is used in a communication, they can be dynamically negotiated using the ALPN protocol. ALPN (Application-Layer Protocol Negotiation) is a TLS protocol extension that allows the application layer to negotiate which protocol should be performed over a secure connection in a manner that avoids additional round trips and which is independent of the application-layer protocols.

GPS data simulator

I don’t have a GPS chip. To get sample GPS data, I used geojson.io website, use a pen to paint the points and collect the result in JSON format. The data looks like this:

{
 "type": "FeatureCollection",
 "features": [
 {
 "type": "Feature",
 "properties": {},
 "geometry": {
 "coordinates": [
 [
 -119.4966304331144,
 49.88901098598916
 ],
 [
 -119.4966304331144,
 49.889903059931726
 ],
 [
 -119.49658503509582,
 49.89066350338888
 ],
 [
 -119.49658503509582,
 49.891365440561145
 ],
 [
 -119.4966304331144,
 49.89219897769374
 ],
 [
 -119.4966304331144,
 49.88735841200943
 ],
 [
 -119.49660773410511,
 49.8880603972496
 ],
 [
 -119.4966304331144,
 49.88885011844141
 ]
 ],
 "type": "LineString"
 }
 }
 ]
}

Save this file as map.geojson to later feed it to device simulation script.

Rendering location data

In AWS location samples project, the sample project maplibre-js-react-iot-asset-tracking is a good demo of IT. The readme document contains a walk through, using AWS amplify services. The steps includes creating certificates, configuring lambda function to add location data to tracker. The project directory also includes the device simulation script, as index.js. I slightly modified the content to this:

const awsIot = require("aws-iot-device-sdk");

// Replace with your AWS IoT endpoint
const THING_ENDPOINT = "safdsa-ats.iot.us-east-1.amazonaws.com"; // get from console
const CLIENT_ID = "trackThing01";
const IOT_TOPIC = "iot/trackedAssets";
const DEVICE_ID = "thing123";
const GEOJSON_FILEPATH="geojson/map.geojson";
const fs = require('fs')
const file_raw = fs.readFileSync(GEOJSON_FILEPATH).toString();
const positions_raw = JSON.parse(file_raw).features[0].geometry.coordinates
const POINTS_ON_MAP=[]
for (var i=0;i<positions_raw.length;i++){
 POINTS_ON_MAP.push({lat:positions_raw[i][1],long:positions_raw[i][0]})
}

const device = awsIot.device({
 host: THING_ENDPOINT,
 keyPath: `${__dirname}/certs/private.pem.key`,
 certPath: `${__dirname}/certs/certificate.pem.crt`,
 caPath: `${__dirname}/certs/root-CA.pem`,
 clientId: CLIENT_ID,
 keepalive: 60000,
});

console.log("Connecting to %s with client ID %s", THING_ENDPOINT, CLIENT_ID);

device.on("connect", async function () {
 console.log("Connected to device %s", CLIENT_ID);

 for (const point of POINTS_ON_MAP) {
 const message = {
 payload: {
 deviceId: DEVICE_ID,
 timestamp: new Date().getTime(),
 location: point,
 },
 };
 console.log(
 "Publishing message to topic %s: %s",
 IOT_TOPIC,
 JSON.stringify(message)
 );
 device.publish(IOT_TOPIC, JSON.stringify(message), { qos: 1 });

 // Set timeout to sleep
 await new Promise((resolve) => setTimeout(resolve, 10000));
 }

 device.end();
});

We can obtain certificate ID from AWS console or by CLI command:

aws iot list-certificates --output text --query 'reverse(sort_by(certificates,&creationDate))[:1].[certificateId]' | cat

When running the script, it pushes data to IoT core service. The AWS Amplify project creates Lambda function that is subscribed to the topic and trigger actions. The data are used to render points on a map, which is available on the front end.

Summary

This is an overly simplified use case but it covers the basics. IoT solution will use a lot managed service and familiar technologies (e.g. TLS, certificate). Creating an IoT solution is mostly about address the onboarding services and make use of the MQTT based workflow. AWS managed services makes these easier.

More IoT workshops are available on AWS workshops.

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

GitHub Action Gotchas

Fri, 27 Jan 2023 01:15:02 -0400

I started with GitHub Actions a couple years ago. Recently I came across a few interesting use cases while I was trying to setup Terraform workflow with GitHub actions. These use cases prompted me to make use some new features in GitHub Action. So I put them in a post here.

Runners can assume IAM Role in AWS

In many scenarios we want to execute AWS CLI command from GitHub action. Also, executables such as terraform inherits credential from AWS CLI. The credential should be a temporary role-based credential instead of an IAM user based on access keys.

There is a GitHub Action called configure-aws-credentials-for-github-actions that can help configure GitHub runner using OIDC identity provider (since Nov 2021 v1.6.0). With the action, the GitHub runner can assume an IAM role as an IAM user (with access key), or using a web identity.

For a GitHub runner to have a web identity thereby assume an IAM role, we should configure OIDC provider in AWS. We can do that from AWS console (i.e. under IAM), or using CloudFormation code. Below is a snippet as an example:

Resources:
 GitHubOIDC:
 Type: AWS::IAM::OIDCProvider
 Properties:
 Url: https://token.actions.githubusercontent.com
 ClientIdList: 
 - sts.amazonaws.com
 ThumbprintList:
 - 6938fd4d98bab03faadb97b34396831e3780aea1

Then from the configured OIDC provider, we can obtain a thumbprint. GitHub action gives the thumbprint here. In AWS, we configure an IAM role whose AssumeRolePolicyDocument will reference the thumbprint. Here is an example. In the condition section of AssumeRolePolicyDocument, we can also specify a specific GitHub repository so that only Actions from that repository can assume the IAM role with their web identities.

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1-node16
 with:
 role-to-assume: ${{ vars.IAM_ROLE_ARN }}
 aws-region: ${{ vars.AWS_REGION }}

This way, we map a GitHub runner’s web identity to an IAM role with a step using the Action above. We also filter what GitHub org and what repo can trigger actions that assumes the role, by the condition clause in the role statement. If the step fails, we can look at CloudTrail on the AWS side for causes. Look for entries with AssumeRoleWithWebIdentity as Event Name.

Reusable workflows

For better reusability of Action steps, GitHub introduced reusable workflows (generally available since Nov 2021). It is particularly helpful when we need to run a workflow for different environments. The reusable workflows files (YML) can be placed in separate repositories, and be reference as such. This allows enterprises to centralize the management of reusable workflows.

I have been using the act project to emulate GitHub action locally on MacBook. As of Jan 2023, act does not support reusable workflow.

With the split between caller and reusable workflows, we have a new challenge of passing secrets and variables between them. It is not straightforward and GitHub documentation needs improvement to get the documentation clear. Also because the word “environment” is used in different contexts, it is ambiguous and therefore difficult to Google relevant information.

Passing variables

First, there are several types of variables in GitHub action:

Environment variable: declared under env keyword in a workflow. To use environment variable, use the env context. For example: ${{ env.MY_VARIABLE }}
Configuration variable: introduced in Jan 2023, configuration variables are defined at repository, environment and organization levels. To use configuration variable, use vars context, and ensure the workflow job specifies a value for environment attribute.
Secrets: GitHub also calls it Environment secret when defined at environment level. It works the same way as a configuration variable because it is also specific to an environment. The content is not viewable once set.

The reason GitHub action makes this so confusing, is that on one page, its documentation distinguishes between environment variable and configuration variable:

On another page, the document refers to configuration variables at environment level as environment variable:

It seems that “configuration variable” is too new for GitHub to refine its documentation as of January. This semantical confusion gave me a hard time investigating how to pass “Environment variable” to reusable workflows. I will stick to the meaning on the first page to distinguish environment variable and configuration variable at environment level.

Passing environment variable isn’t straightforward. In this discussion thread, people discussed how inconvenient it is. I used the workaround in this comment, where I had to create a job for the sake of storing variable values to output.

Pass secret is easier. This is an insightful blog post (Dec 2021) about passing secret to reusable workflow. The attempt 3 in the post works for me. First, we pass the value of environment to the reusable workflow as an input, then at job level specify the environment with the value. Then in the jobs we can reference secrets as ${{ secrets.NAME }}. The job will pick up the secret based on the correct environment.

It appears that since May 2022, GitHub introduced secrets: inherit keyword to address this. However, the method above still works for configuration variable.

Authentication of GitHub Actions

By default, a GitHub action can access the code repository that triggers the action and no other repositories (with GITHUB_TOKEN). However, in many cases we need to access external repositories. For example, terraform init command from a GitHub action implicitly calls git clone to pull module code from external repositories. A GitHub workflow may also reference a workflow file from external repositories.

The question is how to authenticate GitHub workflow to access external repo. This post has a thorough discussion. We may create a Personal Access Token and pass it to set-git-credentials action. We are essentially sharing a personal credential (and repo access) with a GitHub action, which is not a good practice.

The proper way to solve this problem, is to create a separate GitHub App and grant the access only the repo that the workflow needs to access. The GitHub App will generate a private key. Then we supply the private key to workflow-application-token-action so the workflow can act as the GitHub App, thereby access the external repos. The post has more details in the GitHub App section. Suppose we have terraform get command to clone external repo, the actions may look like this:

 - name: HashiCorp - Setup Terraform
 uses: hashicorp/setup-terraform@v2

 - name: Get RepoReader App Token
 id: get_repo_reader_token
 uses: peter-murray/workflow-application-token-action@v2
 with:
 application_id: ${{ vars.REPO_READER_APPLICATION_ID }}
 application_private_key: ${{ secrets.REPO_READER_PRIVATE_KEY }}

 - name: Cache Git Creds
 uses: de-vri-es/setup-git-credentials@v2
 with:
 credentials: https://x-access-token:${{ steps.get_repo_reader_token.outputs.token }}@github.com/

 # Terraform Get implicitly calls git clone which uses the credential cached as above
 - name: Terraform Get 
 run: terraform get

Another benefit of using GitHub App is that the token is a short-lived credential that expires as the job is finished, whereas a PAT will expire on a preset date. In this use case we can think of GitHub App as a service account with minimized privilege to read a short list of repos.

Final words

I came across this reddit post against pipeline use and there are some legit points. For example, the user is frustrated with the limitation with variable passing and unclear documentation. The post wasn’t specific to GitHub action but I do share some frustration while working with GitHub Actions.

On a positive note, since I first used GitHub actions, it has evolved quite a bit with new features, although the documentation is somewhat lagging. It is still very helpful as all of these are free to personal use. I look forward to more interesting features.

Optimize CPU and Memory for Kubernetes Pod

Fri, 13 Jan 2023 11:47:00 -0400

When optimizing workload performance, it is important to understand how on earth operating system allocates CPU and memory to processes. This helps understand how to set resource limit Kubernetes Pod in an optimal way.

CPU resource assignment

The OS distributes CPU resource to processes by the unit of time share of CPU time. Most of the time, many processes with CPU instructions (machine code) are waiting in the Job queue, for their share of CPU time in order to execute their instructions. As soon as CPU becomes idle, the CPU scheduler selects a process from the ready queue to run next:

Ideally, OS should schedule CPU in a way that it should not waste any CPU cycle. It should also minimizes waiting time and response time of processes. At a high level, there are two types of CPU scheduling:

Preemptive: OS allocate CPU resources to a process for only a limited period of time and then takes those resources back. It could interrupt a running process to execute a higher priority process.
Non-preemptive: New processes are executed only after the current executing process has completed its execution.

Here is more information about preemptive and non-preemptive scheduling.

CPU is compressible resource in Linux

In the Linux world, all scheduling is preemptive. We also call it kernel preemption. As the wikipedia entry states: the scheduler is permitted to forcibly perform a context switch (on behalf of a runnable and higher-priority process) on a driver or other part of the kernel during its execution, rather than co-operatively waiting for the driver or kernel function (such as a system call) to complete its execution and return control of the processor to the scheduler when done.

The Linux scheduler implements a number of scheduling policies, which determine when and for how long a thread runs on a particular CPU core. The scheduling policies in RHEL include real time policies such as SCHED_FIFO and SCHED_RR where processes have a sched_priority value in the range of 1 (low) to 99 (high); and normal policies such as SCHED_OTHER, SCHED_BATCH and SCHED_IDLE, where sched_priority (specified as 0) is not used in scheduling decisions.

It is important to understand preemptive CPU scheduling on Linux. When OS allocate CPU resource to a process for one time slot, it is not committed to the same process for the next time slot. The OS reserves the ability to revoke the next CPU use and re-assign it for processes of higher priority.

Because of this, we regard CPU as a compressible resource. The compressible characteristic impacts how we optimize CPU utilization for a process, including setting CPU request and limit for Kubernetes workload.

Memory is non-compressible resource

A few years ago, I discussed how to calculate memory usage. A process requests memory from OS using memory allocation functions (the malloc family), and return memory to OS using free functions. The design of Linux OS knows that processes have a tendency to request more memory than they use, which causes under-utilization. In combat against under-utilization, the Linux OS supports memory overcommitment (on by default), allowing processes to request more memory than what is available. The processes have access to virtual memory space and the OS may swap some pages out to disks. The overcommitment mechanism also prevents processes from crashing due to insufficient memory assignment. The kernel can also OOM kill a process when the entire system is in a crisis.

Memory is non-compressible resource. When OS assigns memory pages to a process, the process has to right to keep those pages, until the OS takes them away. Unlike assigning CPU cycles, the assignment of memory pages to processes does not have an expiry time. This is the non-compressible characteristic of memory assignment.

CPU limit and requests for Kubernetes workload

It was considered best practice to set request and limit for memory and CPU. However, knowing CPU is compressible resource and memory isn’t, we should re-consider this practice. In short, for CPU, we should set request only, without setting limit. For memory, we should set limit to exactly the same as request.

A process has different level of demands for CPU at different times. Depending on the activity in the process, the level of demand can even be spiky. If there is a lot of iowait, it may not need a lot of CPU. But when there are lots of computing-bound activities, the program is CPU-thirsty as it is programmed to to more. The last thing we want is to throttle the CPU use for a process in such legit situations. When throttling happens, the process does not get sufficient time share of CPU time. At the platform level, we can’t control when the Pod (process) gets busy. The best thing it can do, is trying to fit more CPU time shares to this process when it becomes CPU thirsty. When we apply a limit of CPU in workload setting, we are potentially throttling the CPU use for a process at the times it needs more CPU time shares, which is counter-productive.

We should still configure CPU request, so that kube-scheduler factors it in when scheduling multiple Pods to a Node. The CPU request alone ensures the number of Pods are not excessive. This is the only thing we can do about controlling CPU assignment for Pods. We should also monitor CPU throttling.

Memory limit and request for Kubernetes workload

Memory is not compressible, therefore we should set both limit and request to the same value. We set memory request so that kube-scheduler has an idea assigning Pods. We set the limit so that no single Pod takes more memory than its fair share. Unlike CPU, once a Pod takes more memory than its fair share, the platform will have to be aggressive to reclaim it back, which may impacts the running of the Pod (process). In contrast, CPU scheduler never guarantees the assignment of CPU time share to a Pod beyond the end of the current CPU cycle.

When we’re setting memory limit and request with different values, we’re sending a confusing signal. We’re inviting Pods to use more memory than they requested. This increases the chance of memory shortage at the node level, and hence the need to OOM kill a Pod.

Horizontal autoscaling and Cluster Autoscaling

The native HPA is metrics-based. As I previously discussed, neither CPU nor memory metrics are good indicators of time to scale. A process or a Pod may have a temporary high demand of CPU purely due to how programmers write the code. Even if we followed the best practices as above, I would still not regard CPU and memory metrics as a reliable indicator to drive auto scaling. If a service is a potential point of congestion, we should use a queue in front and the queue size is almost always a much better indicator of the timing to scale.

As to cluster autoscaler, on it FAQ, it says flat out that you should NOT use a CPU usage based scaling mechanism. I guess this is for a similar reason (compressibility). As discussed, when a Pod is pending for schedule for too long, it emits and event that drives the cluster autoscaler.

Summary

When I first worked on Kubernetes workload I did not give this much thought and proposed the use of CPU limit. As of January 2023 I still find static code analysis tools that requires CPU limit for Pods in the check (e.g. CKV_K8S_11 on Checkov), which leads me to investigate the issue further, and noticed more voices advocating the correct use of resource limit (such as this post) in 2022. For existing deployments, it is worth a review the resource limit configuration.

EKS impression

Fri, 23 Dec 2022 18:18:19 -0400

I’ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.

Similar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.

In the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.

Computing Nodes in EKS

There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.

With self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.

If you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.

Fargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here‘s an example) at the time that they’re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring).

The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn’t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere.

To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them.

Node AutoScaling

For workloads that don’t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in “Autoscaling on Kubernetes Platform“. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it’s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the “scaling cooldown“. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we’d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions.

Because the Cluster Autoscaler doesn’t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.

Karpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.

Karpenter’s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.

Identity Management for EKS

For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic.

Traffic type	AWS	Azure
I. Cloud Service Endpoint (Management Traffic for Cloud Service)	AWS IAM identity	Azure RBAC
II. Kubernetes API (Management Traffic for K8s Cluster)	IAM mapping or OIDC	Azure RBAC (implementation of OIDC)
III. Business traffic	Up to Kubernetes Ingress	Up to Kubernetes Ingress

For business traffic (type III), it is all up to the Ingress. I’ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.

For access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as “Cluster Authentication“. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:

When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.

This special scenario (I call it the “implicit master user“) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration.

The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.

AppMesh

AppMesh is AWS’ Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh’s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:

On the EKS cluster, install AppMesh Controller using Helm
Associate the cluster with IAM OIDC provider
Create an IAM role for the appmesh-controller service account

After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.

AppMesh uses a different set of CRDs than Istio. Key CRDs are:

Mesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6)
VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type).
GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target.
VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService.
VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target.
VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represent a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group.

You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.

EKS cluster using Terraform

Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.

I also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.

Below is the diagram of the processes.

Create EKS cluster with Terraform module

The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity.

Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:

The IAM user that Terraform uses has no permission other than assuming a “PowerUser” role
The PowerUser role trusts the IAM user. It also has the permission to assume the “EKS-Manager” role
The EKS-Manager role trusts PowerUser’s role session.

However, the role chaining scenario above is not currently supported in Terraform.

I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.

Summary

I first came across this article about EKS and its awfulness and then decided to check out EKS. I’m not sure all points are still valid but it’s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.

Landing Zone in AWS – An Introduction

Thu, 08 Dec 2022 22:37:00 -0400

Cloud adoption has gone through phases. Hashicorp’s CTO Armon Dadgar has a great stream on Hashcorp’s narrative of the three Phases of Cloud Adoption:

Phase 1, with main focus on agility, app teams in wild west, account sprawl, inconsistent configuration, security & compliance challenge
Phase 2. consistent platform layer providing an opinionated way of configurations, security and compliance control, introduction of platform team, whose customers are application team, scaling challenges with platform team capacity not keeping up with application team’s demand
Phase 3. self-service platform at scale for many application teams. CICD, infra-as-code

New greenfield clients today should aim at phase 2 or phase 3 depending on their target operation size. Either way, they need a landing zone for consistency. This post focuses on the landing zone options on AWS and the key constructs.

Overview

AWS Prescriptive Guidance defines landing zone as a well-architected (secure, scalable, compliant, etc), multi-account AWS environment that is a secure baseline from which you can deploy workloads and applications. A landing zone may consists of:

Multi-account architecture
Identity and Access management solution
Governance, compliance, logging and auditing solutions
Security and networking design

Landing zone reflects an enterprise opinion on how to configure networking and IAM. Establishing a landing zone can be a very involving process. In early days of cloud operation people used build landing zone by “clickops” on AWS console, which cannot keep up with the growth of landing zone and associated security services. Compared to SMB clients, some sectors such healthcare and national security have much more regulations and compliance frameworks.

In fact, landing zone is such a buzzword that I have learned to be very sensitive to the context. If a solution has words “landing zone” in its name, given the complexity and loose use of the words, there is a good chance that the solution only delivers some of the aspects above. Cloud consultants are still to address the gaps. Therefore I decided to write this post about what I learned about landing zone in AWS.

Multi-account and Organization

Since 2017, AWS has been officially advocating the use of multiple accounts as a best practice and security strategy. They encourage clients to view an account as a resource container, just like Resource Group in Azure (here is an article on the account structure between AWS and Azure) Client builds an account for security boundary and financial container. Many blog post came along (such as this one) on the implementation details. From tooling perspective, AWS launched AWS Organization in 2017 to facilitate multi-account management. This presentation from re:Inforce 2019 is a good material to understand multi-account environment with AWS Organizations. In 2021, AWS published a new white paper on best practices with multiple accounts. My previous coworkers authored two blog posts here and here to reflect multi-account setup as of late 2020.

AWS Organization is an account management service that provides a vehicle to centrally manage AWS accounts by groups, which brings many benefits, such as centralized logging, compliance management, consolidated billing, etc. As an administrator, you can create accounts in your organization and invite existing accounts to join the organization.

Typical account structure involves:

A root organization named Root
A organization hierarchy with one or more OUs under root, with each OU having one or more child OUs.
Each OU can have multiple accounts, with each account having one email address.

You can also configure service control policies (SCPs), a type of organization-level policy that you can use to manage permissions in your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization. SCPs are a means of implementing guardrails in your AWS organization.

To vend multiple account automatically, AWS introduced the Account Factory (which became part of Control Tower later in 2019) and let users create new account from AWS console and specify which OU it belongs to. It also allows users to implement customization after account creation with Service Catalog products. Some legacy orchestration solution (e.g. Augmented Account Factory) were based on this mechanism. Another orchestration solution prior to the launch of Control Tower was the AWS Landing Zone solution (ALZ, introduced in 2018), which uses AWS CodePipeline to provision accounts and deploy resources.

From Landing Zone Solution (ALZ) to Control Tower

The launch of Control Tower was a game changer in 2019 to bring users to multi-account best practices. Control Tower, as an AWS product (instead of a solution by some service teams at AWS) matured over several years. It gradually deprecated the previous generation orchestration solutions. Newer orchestration solutions all have to support Control Tower. According to this page, we can customize Control Tower based Landing Zone in these ways:

AWS Control Tower console:
- Instead of creating OU in AWS organization, do it under Control Tower in the console, under “create required OUs”.
Outside of AWS Control Tower console
- using Account Factory for Terraform (AFT): Terraform-based account provisioning pipeline, for heavy Terraform shops.
- using Customizations for AWS Control Tower (CfCT) solution. The LZ created remains in sync with Control Tower.
  - You first launch a standardized CF stack to set up the mechanisms for customization.
  - Then you create a custom package to define the customization. This includes a manifest file.

On the other hand, the original ALZ solution is currently in long-term support and will not receive any additional features. It is deprecated. AWS advises its customers migrate to AWS Control Tower based landing zone. The ALZ page is redirected to a page about customizing Control Tower landing zone.

The Control Tower based landing zone configures OUs, accounts, SSO and guardrails. It however does not entail networking design except for the guardrails.

Security Reference Architecture (SRA)

SRA is just a reference architecture and accompanying recommendations on AWS security services, and how they work together in a multi-account environment to host a single-page application. It comes with a repository to demonstrate how to configure a secure multi-account environment with Control Tower, CfCT, as well as security services. Alternatively, you can deploy it with CloudFormation StackSets. For fully automated deployment of this architecture, check out Landing Zone Accelerator down below.

One important feature that reflects the notion of centralized security in a multi-VPC topology, is the use of a dedicated VPC for centralized interface endpoints. This pattern is seen in all generations of security reference architectures.

AWS Security Environment Accelerator (ASEA)

Regulated customers often find they need to add additional controls and capabilities to be defined and setup outside of Control Tower. ASEA, as an orchestration solution, aims to remove the complexity of having to develop and maintain a separate codebase to manage the additional customizations, by providing a tool to help deploy and operate secure multi-account, multi-region AWS environments on an ongoing basis.

Overtime, as Control Tower introduces new capabilities to support the customizations required in heavily regulated environments, the capabilities will be removed from ASEA and enabled directly within the Control Tower managed service, further reducing operational burden. Read this for its relationship with ALZ and Control Tower.

ASEA was first released in late 2020. It covers more on networking design and has a fairly comprehensive installation process. ASEA primarily cater to government of Canada‘s PBMM Security Configuration Profile with an opinionated configuration. The first few revisions were referred to as PMBB architecture. While it provides a great reference architecture for highly regulated landing zone, it has not gotten much traction elsewhere. The recommendation going forward, is to use Landing Zone Accelerator, which incorporates the features and lessons learned from ASEA.

Landing Zone Accelerator (LZA)

First released in May 2022, the Landing Zone Accelerator on AWS solution deploys a cloud foundation that is architected to align with AWS best practices and multiple global compliance frameworks. LZA operates on top of Control Tower managed landing zone. This page in the documentation brings a good explanation of how it works. Basically it employs CodeBuild as an orchestration engine, and leverages CDK to drive resource deployment. There are two repositories: the GitHub repository for Landing Zone Accelerator itself, and a CodeCommit configuration repository provisioned during preparation.

As document states, LZA is a fully automated implementation of the architecture guidelines documented in the SRA. LZA also incorporates features and lessons learned from ASEA and Compliant Framework for Federal and DoD Workloads in GovCloud (US), neither of which are recommended for new deployment. In addition, LZA aims to enable iterations and extensions of a secure environment over time. The vision is to eventually replace AFT, CfCT, and ASEA. As AWS releases newer versions of LZA, client should be able to upgrade it in a pipeline run.

Baseline architecture for LZA

The Landing Zone Accelerator project also provides a samples configurations in each regulated frameworks. For example, the healthcare best practice come up in Oct 2022 for healthcare industry. The healthcare best practice sample incorporates healthcare specific configurations, such as the detective guardrails defined in the Operational Best Practices for HIPAA Security conformance pack. To deploy the best practices, modify the configuration in config repo, and run the pipeline again.

Landing Zone Orchestration Options

To summarize, we have the following landing zone options in AWS:

Orchestration Layer	Lifecycle	Summary
Solutions based on account factory with Service Catalog (e.g. ALZ, Augmented Account Factory)	All solutions pre-dates the launch of control tower have been or will soon be deprecated.	Pre-Control Tower solutions should migrate to current alternatives.
AWS Control Tower	GA in June 2019	To customize the landing zone, use AWS console, or alternatively, one of the following three options: – CfCT: Customization for Control Tower – AFT: Account Factory Terraform – AFC: Account Factory Customizations For example, CfCT can be used to deploy Security Reference Architecture (SRA)
AWS Secure Environment Accelerator (ASEA)	Released 2020	Canadian Centre for Cyber Security (CCCS) Medium Cloud Control Profile, formerly known as PBMM. This approach will be replaced by LZA
AWS Landing Zone Accelerator (LZA)	Released in 2022	A low-code deployment option. Samples provided in support of reference architectures that align with industry best practices or compliance frameworks. Examples for industry best practices: – general best practice – healthcare – finance and tax – education Example for compliance frameworks: – US state local government – FedRAMP for US Federal and Department of Defence (DoD) – CCCS Medium for Canadian government – AWS Trusted Secure Enclave (TSE) Sensitive Edition (SE), which also aligns with other medium level security profiles such as NIST 800-53, ITSG-33, FedRAMP moderate, CCCS-Medium, IRAP, etc

If your organization has a compliance framework that LZA supports, it makes sense to start with LZA. Otherwise, it is sufficient to use Control Tower for multi-account setup.

Networking constructs

Landing zone involves multiple VPCs so it’s important to understand VPC peering and Transit Gateway. VPC peering can only be setup between two VPCs and transitive peering relationship is not supported. For 6 VPCs to talk to all each other, we’d need 15 peering setups, which is not sustainable. We have two options:

We can connect many VPCs to a single Transit Gateway and the VPCs will be able to talk to each other. We can also connect Transit Gateway to site-to-site VPN or Direct Connect. This is a good explanation.
We can still use peering, but dedicate one VPC as “Transit VPC” in a hub and spoke model.

When you attach a VPC to a transit gateway, you must specify one subnet from each AZ to be used by the transit gateway to route traffic. Specifying one subnet from an AZ enables traffic to reach resources in every subnet in that AZ.

Here is a comparison table. Read the white-paper “Building a Scalable and Secure Multi-VPC AWS Network Infrastructure” for more about network design. For example, Some clients need to inspect traffic. When deploying multiple VPCs, we also need to ensure DNS resolution works across VPCs, and between on-prem networks, by configuring Amazon DNS server.

Our VPCs also need to connect to managed AWS services. For services like S3 or DynamoDB (of the same or different account), workload in VPC can access them via public DNS. The traffic goes through Internet Gateway of VPC and then public Internet. It is neither secure nor economical. We often want all network traffic to stay on the global AWS backbone. There are three types of Endpoint under VPC to help us.

	Gateway VPC Endpoint or Gateway Endpoint	Interface VPC Endpoint or Interface Endpoint	Gateway Load Balancer Endpoint
Purpose	VPC access native AWS services	VPC access native AWS services	VPC access your own service (aka Endpoint Service)
Enabled by Private Link	No	Yes	Yes
Traffic remain on AWS network	Yes	Yes	Yes
Mechanism	Use the public IP address of the service along with configuration in routing table to access target resource. The routing table acts as gateway. You can use the public DNS name of the service.	Use private IP address from the VPC to access the target service. Require endpoint-specific DNS name for the target service. Incur extra charge. S3 as example. More secure because there is ENI in the VPC controlled by security group.	Your Endpoint Services is hosted in front of a fleet of network virtual appliances. You can select endpoint type as you create it in console under VPC
Access from clients on premise or from other region	Does NOT allow access from clients on premise or in VPC from other region	Allows access from clients on premise or clients in VPC from other region	Allowed

Private Link integrates with a subset of AWS services. To check the list of interface endpoint, use:

aws ec2 describe-vpc-endpoint-services --query "ServiceDetails[?ServiceType[0].ServiceType=='Interface'].ServiceName"

Given the inter-VPC connectivity, most deployment centralizes interface endpoint into a dedicated VPC.

Centralized Interface Endpoints

There are several benefits to use a single VPC as dedicated provider of interface endpoint in a multi-VPC topology. First, the interface endpoints incurs a standing charge and it makes financial sense to consolidate them in one VPC. Second, this setup centralizes the configuration and security aspects as well.

In most cases, interface endpoint services follow the format of com.amazonaws.<region>.<endpoint>, with the dns name looking like: <endpoint>.<region>.amazonaws.com. For example:

Endpoint Service Name: com.amazonaws.us-east-1.ssm
Private Hosted Zone: ssm.us-east-1.amazonaws.com
Zone Record (A alias): ssm.us-east-1.amazonaws.com

However, there are a few exceptions to that which can make it tricky to implement interface endpoint with infrastructure as code.

Exception 1. Private DNS name suffix is api.aws instead of amazonaws.com:

Endpoint Service Name: com.amazonaws.us-east-1.eks-auth
Private Hosted Zone: eks-auth.us-east-1.api.aws
Zone Record (A alias): eks-auth.us-east-1.api.aws

Exception 2. Endpoint Service Name doesn’t start with com.amazonaws

Endpoint Service Name: aws.sagemaker.us-east-1.notebook
Private Hosted Zone: notebook.us-east-1.sagemaker.aws
Zone Record (A alias): notebook.us-east-1.sagemaker.aws

Exception 3. Two A-records are required under the same PHZ

Endpoint Service Name: com.amazonaws.us-east-1.ecr.dkr
Private Hosted Zone: dkr.ecr.us-east-1.amazonaws.com
Zone Record (A alias): dkr.ecr.us-east-1.amazonaws.com and *.dkr.ecr.us-east-1.amazonaws.com

Exception 4. Two PHZs are required:

Endpoint Service Name: com.amazonaws.us-east-1.ec2
Private Hosted Zone: ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws
Zone Record (A alias): ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws

The infrastructure code that handles interface endpoints should be able to take care of the exceptions.

Security Constructs

AWS Network firewall is a configuration under VPC. You associate a Network Firewall with one or more subnets in the VPC. You also associate the Network Firewall with Firewall Policies. Each policy consists of rule groups. Each rule blocks or filters traffic. The log can be published to CloudWatch or S3 via Kinesis. Usecases of AWS Network firewall include:

inspect VPC-to-VPC traffic;
filter outbound traffic;
prevent inbound internet traffic;
secure AWS Direct Connect and VPN traffic

When you associate a firewall to your VPC, you must provide a subnet for each Availability Zone where you want to place a firewall endpoint to filter traffic

There are many points of configuration for firewall rules in AWS. AWS Firewall Manager is a place for central management. It connects with other services such as:

AWS WAF (L7)
AWS Network Firewall (L4)
AWS Shield (DDos)
Amazon Route 53 Resolver DNS Firewall
Security Groups
Third party firewall support

IAM Identity Center (previously AWS SSO) is for logging into AWS portal, giving each identity from an external identity store (such as AD) an identity on an AWS account. It addresses several challenges as a result of having multiple AWS accounts.

First, a user from AD needs to access multiple accounts. Second, in each account with access, s/he needs to have an IAM role. These IAM roles are based on attributes of his/her external identity, and can be different per AWS account. With Permission Set IAM Identity Center:

Allow you to login on different AWS account using the same credential
Allow you to federate with external identity store( e.g. using SAML)
Manage permission set for each account.

This video has a good walk-through of how to configure Azure AD as identity store, and tie it to specific AWS accounts and specify permission set.

Note that IAM Identity Center has a different purpose than Cognito. Cognito is to connect your app with an IdP. Your app can be an EC2-hosted application, serverless application on API gateway, or container-based application on Kubernetes, as long as the support open identity standards (e.g. OAuth 2.0, SAML 2.0 and OIDC). Cognito has two pools:

User pool for authentication. With a user pool, your app
Identity pool for authorization

So Cognito is for AuthN & AuthZ to your own app’s endpoint (business traffic) and it supports a number of standards. IAM Identity Center is for AuthN & AuthZ to your AWS account (management traffic). If you use it for your own application, your application user will have direct access your AWS resources. It mainly supports SAML (and OIDC but less used) and is commonly connected with Active Directory (Microsoft or AWS).

One can use AWS Directory Service to host a compatible and managed directory service. AWS Directory Service supports four modes:

AWS managed Microsoft AD: actual Microsoft Active Directory in AWS Cloud
Simple AD, powered by Linux-Samba Active Directory-compatible server
AD Connector: a proxy for redirecting directory request to your existing Microsoft AD without caching any information in the cloud
Cognito user pools

If you combine two AD domains, you will need domain trust.

Other security services

In addition to the core services above, LZA involves the following four services as well.

AWS Macie is a fully managed data security and privacy service based on ML and pattern matching. It continually evaluates your S3 environments to discovery PII and act on them. It also reports alerts on unencrypted buckets, publicly accessible buckets, etc.

AWS GuardDuty continuously analyze S3, container & instance workloads, user and accounts for potential threads, across account, based on ML for intelligent threat detection. In addition, GuardDuty also acts on findings.

AWS Config is an essential service that traces resource inventory, their changes and monitors for compliance:

When you turn on AWS Config, it first discovers the supported AWS resources in your account and generates a configuration item (poin-in-time view of attributes) for each resource. AWS Config also generates configuration items when the configuration of a resource changes, and it maintains historical records of the configuration items of your resources from the time you start the configuration recorder.
On an ongoing basis, AWS Config keeps track of all changes to your resources, whether or not it is initiated by the API
If you are using AWS Config rules, AWS Config continuously evaluates your AWS resource configurations for desired settings. You can deploy several related rules in a pre-built compliance pack.

As you may have noticed, there are many AWS services related to firewall, security and compliance. AWS Security Hub aims to be a a consolidated view of your security status in AWS. You can automate security checks, manage security findings, and identify the highest priority security issues across accounts in client environment. It can also:

consolidate security findings from GuardDuty, Inspector (vulnerability scanner and management), Macie, Config, Systems Manager, Firewall Manager, IAM Access Analyzer and other Integrated APN solutions
Check findings against best practices
Client can take action: i.e. investigate findings or take response and remediation actions

Summary

Landing zone deployment can take numerous iterations to finalize the requirement. It is important to have a vision of the client’s cloud operating model, which is usually discussed prior to implementation. At the implementation phase, I discuss the topic on two more separate posts on control tower and landing zone accelerator.

Key mapping for external PC keyboard on Mac

Wed, 23 Nov 2022 10:38:00 -0400

This post is to document my steps to use external Windows keyboard on Mac with custom key mapping.

Background

Apple’s magic keyboard does not support multi-device so I have to repurpose my Logitech K810 keyboard with MacBook. Logitech K810 is and old model with Windows key layout even though it also supports MacOS.

The bottom row has the same number of keys as magic keyboard but with different keys. For example, magic keyboard has cmd key (aka GUI keys) besides space bar but K810 has Alt. Magic keyboard has Option key next to cmd key and K810 has Windows key on the left and Ctrl key on the right. The position of left Ctrl and FN is different on magic keyboard too. Since I rely on shortcuts on Magic keyboard, I therefore decide configure key mapping on the Windows key, Alt keys and right ctrl key to match magic keyboard.

K810 Layout

In a nutshell, I need to remap some keys on an external Windows keyboard connected to MacOS (Monterey).

What is not working

In MacOS, you could modify keys from keyboard preference as below. However, in my case I need to keep left ctrl key and overwrite the right ctrl key. MacOS does not distinguish them unfortunately. The native method is not flexible so I did not bother.

Use MacOS keyboard preference to modify keys

In the mean time, I hate to install a third-party application (such as Karabiner-Elements) just for the purpose of key mapping. Even though I appreciate the efforts by the community, I just don’t find it a neat solution for the additional dependency.

Challenges with key mapping

I then came across a developer’s post on using a utility called hidutil to map keys. This is promising after I tested using this command to swap key a/A and b/B.

hidutil property --set '{"UserKeyMapping":[{"HIDKeyboardModifierMappingSrc": 0x700000004,"HIDKeyboardModifierMappingDst": 0x700000005},{"HIDKeyboardModifierMappingSrc": 0x700000005,"HIDKeyboardModifierMappingDst": 0x700000004}]}'

The command above takes effect immediately, on all keyboards. In my case I only need mapping on an external keyboard, not the built-in keyboard. Luckily hidutil has –matching switch where I can specify ProductID and VendorID. I can find both of them from About This Mac -> System Report -> Hardware -> Bluetooth:

Vendor ID and Product ID

From this technical note, we can find out the usage IDs of the keys in the table at the bottom of the page. Then use the Usage ID as the last two bytes of the source and destination values of the mapping expression above. For example, 0x700000004 stands for key A/a, and 0x700000005 stands for key B/b. In my case, after going through this table, my key mapping is:

Source Key on K810	Source Code on K810	Destination Key	Destination Code
Left Alt	0x7000000E2	Left GUI (cmd)	0x7000000E3
Right Alt	0x7000000E6	Right GUI (cmd)	0x7000000E7
Left Win	0x7000000E3	Left Alt (option)	0x7000000E2
Right Ctrl	0x7000000E4	Right Alt (option)	0x7000000E6
Fn+F12	0x700000045	Globe	0xFF00000003

There is even a website that helps you generate the usage ID. I put together a command, which is working, on the external keyboard.

hidutil property --matching '{"ProductID":0xB319,"VendorID":0x046D}' --set '{"UserKeyMapping":[{"HIDKeyboardModifierMappingSrc":0x7000000E2,"HIDKeyboardModifierMappingDst":0x7000000E3},{"HIDKeyboardModifierMappingSrc":0x7000000E6,"HIDKeyboardModifierMappingDst":0x7000000E7},
{"HIDKeyboardModifierMappingSrc":0x7000000E3,"HIDKeyboardModifierMappingDst":0x7000000E2},
{"HIDKeyboardModifierMappingSrc":0x7000000E4,"HIDKeyboardModifierMappingDst":0x7000000E6},
{"HIDKeyboardModifierMappingSrc":0x700000045,"HIDKeyboardModifierMappingDst":0xFF00000003}
]}'

The other way to confirm that key mapping is working, is to look up key mapping with the following command:

hidutil property --matching '{"ProductID":0xB319,"VendorID":0x046D}' --get "UserKeyMapping"

To wipe out the mapping, we can use the following command:

hidutil property --matching '{"ProductID":0xB319,"VendorID":0x046D}' --set '{"UserKeyMapping":[]}'

Note that we have to use the same matching expression. However, I noticed that the command only works while the bluetooth keyboard is connected. When I reboot the MacBook the mapping goes away as well.

The solution to key mapping

Even though I could run this command after reboot using a plist file in ~/Library/LaunchAgents/, as the reference blog suggests, it will not work because I cannot guarantee the bluetooth keyboard is connected before MacOS executes it. Also, I might disconnect and re-connect the keyboard and I want to ensure that the system runs that command whenever the keyboard gets connected.

In the Karabiner Element community, some developers touched on this. Basically we can customize the plist file to tell MacOS when to fire the hidutil command, by using “LaunchEvents” key in the XML. There is an example for USB keyboard. For myself, I edit the plist file to be:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>local.hidutilKeyMapping</string>
 <key>LaunchEvents</key>
 <dict>
 <key>com.apple.iokit.matching</key>
 <dict>
 <key>com.apple.bluetooth.hostController</key>
 <dict>
 <key>IOProviderClass</key>
 <string>IOBluetoothHCIController</string>
 <key>idProduct</key>
 <integer>B319</integer>
 <key>idVendor</key>
 <integer>046D</integer>
 <key>IOMatchLaunchStream</key>
 <true/>
 </dict>
 </dict>
 </dict> 
 <key>ProgramArguments</key>
 <array>
 <string>/usr/bin/hidutil</string>
 <string>property</string>
 <string>--matching</string>
 <string>{"ProductID":0xB319,"VendorID":0x046D}</string>
 <string>--set</string>
 <string>{
 "UserKeyMapping": [
 { 
 "HIDKeyboardModifierMappingSrc":0x7000000E2,
 "HIDKeyboardModifierMappingDst":0x7000000E3
 },
 { 
 "HIDKeyboardModifierMappingSrc":0x7000000E6,
 "HIDKeyboardModifierMappingDst":0x7000000E7
 },
 { 
 "HIDKeyboardModifierMappingSrc":0x7000000E3,
 "HIDKeyboardModifierMappingDst":0x7000000E2
 },
 { 
 "HIDKeyboardModifierMappingSrc":0x7000000E4,
 "HIDKeyboardModifierMappingDst":0x7000000E6
 },
 {
 "HIDKeyboardModifierMappingSrc":0x700000045,
 "HIDKeyboardModifierMappingDst":0xFF00000003
 }
 ]
 }</string>
 </array>

 <key>RunAtLoad</key>
 <true/>
</dict>
</plist>

Store the file above as ~/Library/LaunchAgents/com.local.hidutilKeyMapping.plist (create LaunchAgents directory if it doesn’t exist) and then run the following to load it:

launchctl load ~/Library/LaunchAgents/com.local.hidutilKeyMapping.plist

This works perfectly. The dict section (line 7 to line 23) of the file is based on this document as the GitHub comment suggests. The LaunchEvents section narrows down the type of events that will trigger the program with arguments. It triggers whenever a bluetooth device with specific ProductID and VendorID connects to Mac.

We can still use hidutil command from the previous section to ensure that the custom key mapping remain in effect:

Note that the output above has usage values converted to decimal from hexadecimal, with destination on top and source at the bottom.

I like this solution as it is light-weight, targeting a specific event, highly customizable and resolves an issue in a very specific circumstance.

Final words

K380 layout

After all this customization effort, I came across the Logitech K380 model (current alternative to K810), whose layout is exactly the way I customized K810 to be. So I ordered the new K380 model for just $29.99 during the week of black Friday. Remembering that I had my K810 for $94.98 back in 2015, what’s the point of all this….

AWS serverless services and developer tools

Wed, 09 Nov 2022 12:19:00 -0400

As discussed, serverless simply means cloud services that delegate autoscaling management to cloud platform. In my mind, the word “serverless” translates into “managed autoscaling”. As long as a service’s capacity is managed automatically, we can consider it as serverless. Given that capacity scaling accounts for a good amount of work in IT operation, moving to serverless significantly reduces operation overhead.

Unlike containers, the serverless ecosystem lacks standard. Since AWS is leading the charge in this field, let’s take a look at its offerings.

Lambda Function

Lambda functions have triggers. We can configure a trigger from an AWS service, or even a non-AWS service which supports AWS event bridge, to invoke Lambda function. Based on the trigger (e.g. SQS, S3, DynamoDB), you then specify event source mapping. For example, if trigger is SQS, you need to specify the queue name, batch size and batch window in event source mapping. If trigger is S3, the event source mapping needs to specify bucket, S3 action, etc. Event source mappings vary significantly among trigger types.

Depending on the trigger, a Lambda function may be invoked in one of the two ways:

	Synchronous invocation	Asynchronous invocation
Summary	Requestor fires request and waits until it receives response before closing connection.	Lambda places triggering event in a queue and immediately returns a success code (202). Then a separate process reads events off the queue and sends them to your Lambda function. When the function returns a success response or exits without throwing an error, Lambda sends a record of the invocation to an EventBridge event bus.
Pros	Requestor get invocation result as soon as function run is complete	The queue decouples the request and invocation.
Cons	When load is high the function requires higher capacity for concurrency	Involves more parties at play and can be quite complex to troubleshoot.

Here is a chart with the triggers that support each invocation mode.

Real-life applications often need to include libraries and dependencies that requires language-specific steps. For example, NodeJS applications need bundling (often using Webpack or esbuild). We can use Lambda layers to simplify the management. The additional libraries often need to be packaged in S3 bucket or as container image.

Lambda functions are subject to cold start to cope with. When a lot of invocations occur about the same time, the order of executions might be different than the order of upstream events that invokes the function.

API Gateway

AWS API Gateway is an API gateway implementation for REST and WebSocket APIs. It couples with Lambda in the classic multi-tier serverless architecture pattern.

In this classic pattern Lambda function often need to work with a database (e.g. DynamoDB etc), to perform CRUD operations and other custom business logics. The CRUD operations are so commonplace that it makes sense to use mapping template to configure CRUD operation instead of writing similar set of functions for each new data model. API Gateway supports such a mapping template called Velocity Template Language (VTL), a technology from Apache Velocity Project.

API Gateway integrate with many other AWS services. Here are the available integration types:

Integration Type	Description	Integration Mode	How it works
AWS integration	connects gateway to an AWS service action as the end point.	AWS_PROXY	this mode only supports only one action with one service: the function invoking action for Lambda service. Therefore it is available only for Lambda integration and no other AWS services. For that reason, it is also known as Lambda proxy integration. This mode is recommended for Lambda integration and is the default mode for LambdaIntegrationOption CDK construct. It connects a method (PUT, GET, etc) to a Lambda function and pass along the request on the way in, and the response on the way out. You do not set integration request or integration response. Even if you do, there’s no effect. This “pass-along” mode is easier to understand and configure.
AWS integration	connects gateway to an AWS service action as the end point.	AWS	this mode connects an API method to a broad range of supported AWS service action. Function invoking for Lambda service is a common example but not the only service action supported in this mode. When used in Lambda integration, it is also referred to as “Lambda custom integration”, or “normal (request/response mapping) integration”. This mode is good for advanced use cases (e.g. header modification) but involves more management effort. You have to control the mapping between method request and integration request, and between integration response and method response.
HTTP integration	for generic HTTP service endpoint	HTTP_PROXY	the pass-along mode for upstream HTTP endpoint
HTTP integration	for generic HTTP service endpoint	HTTP	the request/response mapping mode for upstream HTTP endpoint
MOCK integration	the API gateway itself serves as the endpoint.	MOCK	the API gateway itself acts endpoint without an upstream. One example use case is to return CORS-related headers upon a pre-flight OPTIONS query.

The table above is a summary of API Gateway integration types as covered here. In the AWS context, proxy mode suggests that the request is not being morphed (transformed). In non-proxy mode, request or response may be modified on their ways in or out.

For API gateway to invoke lambda function, synchronous invocation is used by default. You can also configure API gateway to invoke Lambda function asynchronously by using headers.

As with other API gateway implementations, AWS API Gateway can also connects to an authorizer (either another Lambda function or Cognito service) in order to authorize the incoming request.

AppSync

In a previous post, I discussed GraphQL as a modern and efficient alternative to REST API. AppSync to GraphQL is the same as API Gateway to REST API. One of the advantages that GraphQL has over REST API is more information in the response. Oftentimes, we use a proxy that supports GraphQL in front of REST API service. AppSync can act as such proxy. I think of it as a managed Apollo since they play the the role in the architecture.

Another benefit of GraphQL is the support of subscription, obviating WebSocket configuration. AppSync supports pub/sub API for real-time experience. Client application can get near real-time update as the data on the server is changed.

We can configure AppSync to connect to different data sources to formulate GraphQL response. The data source can be an HTTP endpoint, a Lambda function, a database (Relational or DynamoDB), or OpenSearch.

App Sync

Between data source and the request, AppSync uses resolver to convert GraphQL payload to the underlying protocols and executes if the caller is authorized to invoke it. Resolvers are comprised of request and response mapping templates, which contain transformation and execution logic. AppSync also uses VTL as the mapping template for resolvers.

Messaging Services

Messaging services have three patterns: queues, pub/sub, and event buses. In AWS, the corresponding services are SQS, SNS and EventBridge. Here is a good post about how to choose among them.

Queues are temporary storage to decouple the source and destination systems. The expectation is that the actions in response to the message can be delayed. If that is not the case and the response needs to be immediate, that is by definition an event-driven pattern. A common solution is to have the AWS service invoke Lambda function. If that is not supported, we can use SNS as an intermediary.

SNS is for simple event-driven pattern. In a more complicated event-driven architecture, we often need an event bus to route events in certain ways. We also need to support various event sources by different software providers. Moreover, we want the capability to register our own event schema. These are the scenarios where EventBridge can help.

Developer tools

In application development with serverless stack, the line between application and infrastructure is somewhat blurred. Developers often find themselves making repeated configuration on cloud resources while writing application code. For example, to test Python code (application), developer has to upload the code to S3, create Lambda function referencing the code, etc. This would require too much work with AWS CLI, or CloudFormation. The AWS SAM (AWS Serverless Application Model) is a better utility for serverless development workflow.

SAM comes with its own CLI and developers can feed it with template that interacts with a number of serverless resources (e.g. API, SimpleTable, Function, etc). I view SAM as one layer of abstraction on top of CloudFormation that handles some resources in serverless stack. It saves developers from re-writing resources in CloudFormation templates and keeping them consistent, which would have been tedious. Once you deployed your application using SAM, it will appear as a Lambda Application in the console.

SAM templates are also declarative. It is simple but limited in feature. AWS CDK is also a powerful utility that works with general purpose programming language for IaC and interacts with all AWS resources. It is very powerful for building serverless applications.

AWS Amplify

We have CDK, SAM and CloudFormation but we still have to ensure integration between those resources and our applications. For example, when creating a S3 buckets, they have to get the endpoint and reference it in the application code. So is Cognito. As a result, developers still have to spend time on resource integration. They still can’t focus on business logic. We need a tool that can make opinionated configuration of cloud resources and automatically reference them from the application code. This is the purpose of AWS Amplify.

Amplify not only help application developers create backend resources with opinionated configurations. It also provides libraries for the application to use and connect to those resources seamlessly. Developers will need to include Amplify libraries in the application code. To create cloud resources, they can use Amplify CLI or Studio (a web portal from AWS console).

Amplify natively supports a number of serverless resources such as API (using API gateway or AppSync), Storage (S3 and CloudFront), Lambda function, Cognito, etc. Developer may use Amplify CLI command to create such supported resources. Amplify will prompt some guiding questions in order to configure them correctly. In addition to the natively supported resources, developers can also create custom resources. They have to declare those custom resources with CloudFormation or CDK. I came across this project as a good illustration of how Amplify works.

The Amplify Studio can save developers from using Amplify CLI commands. It also can integrate with Figma, providing developers with a framework for frontend development. However, in my experience, it is still glitchy. For now I stick to Amplify CLI.

Amplify also supports hosting capability, providing users with opinionated and customizable CI/CD pipeline configuration.

How about “clientless”

Lambda function can also run client-side logics. System administrators have to create lots of client-side scripting. While Lambda functions can encapsulate those logics, we’d still need a script orchestrator to invoke those functions. AWS Step Function comes to rescue. It was even regarded as the most under-utilized service. As the author states, a state machine (as design pattern) is simply a flow of actions with decision making logics. AWS Step function helps you groom the logic flow of existing actions with Amazon States Language. There are standard and express workflows. Each step is a state. A state can be of several different types, such as Choice, Task, Succeed, Fail, End, Map, Wait, Parallel. The task can be a Lambda function, and even AWS API calls.

AWS Step function integrate with Lambda functions. It is typically used for patterns with long process and the need to orchestrate the execution of several Lambda functions. For specific use cases, look at these sample projects. For example, a network professional needs to run a lot of connectivity testing, reusing the same Python script but run it from different subnets. We need to create a Lambda function for commands like “nc -vz” then invoke the lambda function from VPC, multiple times from different VPCs. We should use step function to drive this. It works like a Makefile on Linux, without requiring your own computer to run.

AWS Step function also integrates with other AWS services, such as SNS, SQS, Dynamo, Batch, Glue, EMR, EKS, API gateway, event bridge.

Whenever we need to create a script, involving custom actions, or AWS API calls, we should consider using AWS step function to organize the actions. The benefits are: it saves you a laptop or bastion host (“client-less”), many ways to invoke them (not just cron”).

Summary

In this post, we started with some serverless services in AWS. For server-side application, we can use API Gateway to invoke Lambda function. For client-side, we can use step function to invoke Lambda function. We then discussed some tools to speed up application development in serverless pattern.

AWS categorizes both Amplify and AppSync under Front-end Mobile. Amplify does not represent any computing resources in AWS cloud. It is a library and CLI tools to enhance developer experience. AppSync on the other hand is a cloud computing resource acting as GraphQL API for mobile or web application.

Computing services: from PaaS to Serverless

Fri, 21 Oct 2022 19:31:00 -0400

Silicon Valley startups in mid-2000s likely do not run their own IT operations (i.e. renting their own data centre spaces, purchasing their own rack-mounted servers). Since the launch of EC2, AWS has been renting extra computing capacity to those startups, in the IaaS model. The leased infrastructure requires maintenance work, and AWS realized that many of these customers cannot afford specialized database admins, network admins, storage admins, or even server admins. As a result, they created a handful of managed services aiming to cut out admin overhead and let their customer focus on coding. This is how Platform-as-a-service (PaaS) came about.

Let’s take a look at what are exactly operation activities.

Ops activities

IT operation team manages server provisioning, installation of operating system, tuning performance, configuring auto scaling and load balancing, configure networking and storage systems, etc. Networking can be so complex that many infrastructure teams have a dedicated Network Operation Center (NOC), who along with security team, manages key aspects of networking, such as segmentation, router configuration, load balancing, firewall configuration.

For client-server application, the client-side code will make outgoing connections, utilizing the TCP/IP stack on the host through an ephemeral port. The server-side code has to be wrapped as a service. A daemon ensures the process running this service stays up and listens to a TCP port in order to respond to request by invoking the functions. Application team usually assumes these activities.

If database is involved, then the patching, upgrade, replication, data protection are all Ops problems. If storage is involved, then Ops has to manage mass data accumulated over years, the integration between storage and database and applications, performance, replication, etc. Some larger organizations have full-time database administrator and storage administrators.

Then comes container. Containers have their benefits but it increases the operation overhead by an order of magnitude. Running container application at scale warrants its own platform, most likely a Kubernetes platform, to address all of the problems above again at the cluster level. Some organization created platform team to manage container and VM platforms.

It is the Ops, that turns functional code into a running business. It is also the Ops, that becomes a pain point as a startup scales. With IaaS and PaaS models, AWS managed to convince many small businesses to delegate their IT operations to AWS. This is the humble start of cloud computing.

Elastic Beanstalk

At first, I wasn’t too impressed with Elastic Beanstalk, since it abstracts away too many details. However, I later realized that it has been surprisingly popular in the developer community, especially with individual developers and SMBs. It simplifies deployment to the point that their users don’t need to know other AWS services, allowing them to focus on coding application logic.

You configure Applications and Environments (one application may have multiple environments). In the Environment layer, you can specify code platform (e.g. Python 3.8 on 64bit Amazon Linux 2, Java, Go, PHP, Ruby) and even container platform (Docker on EC2 or ECS). Behind the scene, Elastic Beanstalk configures EC2 instances, Elastic Load Balancers, etc on the selected VPC and integrate with logging and monitoring services. In the console, Elastic Beanstalk exposes a list of configurations options (e.g. AMI, instance type). This centralized configuration page is dummied down for those who don’t want to deal with Ops.

The downside of Elastic Beanstalk is it takes away a lot of flexibility. Many developers find Elastic Beanstalk limit their choices of deployment, as their applications scale. Elastic Beanstalk does not suit for applications that demand extensive operation efforts. Its niche market is individual developers and SMB. Few enterprise applications run on Elastic Beanstalk.

Containerization with ECS and EKS

When we containerize an application, we build container images. Then we run these images with container runtimes, is a core feature of container platform. Container platform also provides orchestration engine since we frequently take containers up and down. In addition, container platform provides mechanisms for container networking and storage.

AWS has a couple options for container platform. ECS (Elastic Container Service) came out earlier. It organizes a group of EC2 instances as a cluster. You can manage autoscaling, networking, and persistent storage (EFS, FSx etc) on ECS. EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. Just like AKS, it provides a managed control plane along with computing nodes.

I see ECS as a proprietary and simplified container platform, and Kubernetes as an open-source standard for full-fledged container platform with an entire ecosystem. EKS includes an upstream-certified Kubernetes distribution with a set of tools specific to AWS. Since Kubernetes is the de-facto standard container platform, I prefer EKS by default, unless I can justify the use of ECS. In fact, ECS and Kubernetes have many concepts in common. For example, a “Task” in ECS is equivalent to a Pod in Kubernetes.

Whether it is ECS or EKS, right-sizing the computing node group is always challenging especially when the application traffic load is irregular. AWS Fargate is a technology that provides on-demand, right-sized compute capacities. It works with ECS and EKS. When integrated with EKS, we delegate the node management (e.g. scaling) to Fargate and forget about sizing the node pool.

Using ECS and Fargate involves quite a bit of configurations. To simplify that, we can use App Runner, which builds ECS cluster and uses Fargate to execute the container behind the scenes. App Runner helps client in a way similar to Elastic Beanstalk, but concentrate on Container workload.

Serverless with Lambda and API Gateway

The services above have their limitations when it comes to scaling capability. First, they cannot scale to zero. You still pay for idling resources. Also, it is not easy to find the optimal autoscaling setting. Lambda and API Gateway together solves these challenges. AWS refers to it as serverless, which has since become a buzzword. To understand what it is, let’s examine two concepts:

Function as a Service: service with the ability to execute code on demand. Users only pay for code execution time and do not care where the underlying runtime is
Backend as a Service: service with the ability to listen to a port and respond to web request

Lambda itself is a function as a service. Triggered by events, it only incurs a charge when it’s invoked. It does not stay up and listening to a TCP port for incoming web request, as does a backend service. In order to act as a backend, Lambda needs to pair up with API gateway. In this configuration, API gateway listens to a web request, and it fires an event to trigger the execution of Lambda function. Lambda and API gateway together makes a backend as a service. In AWS, the coupling of API gateway and Lambda function ensures an idle service does not incur computing cost.

Since Lambda supports many types of events as trigger, it is also used in event-driven architecture, either standalone or from a VPC. Under the hood, Lambda runs code in a container (with a quick startup time relative to a VM).

Developers can release Lambda code by uploading zip package to S3 bucket, or just packaging code into container image. For deployment, apart from AWS console and CLI, one can leverage CloudFormation, SAM (serverless application model), or CDK.

Lambda vs Fargate

Both Lambda and Fargate are serverless capabilities, at least from a marketing perspective. Both can be used to back web service but there are differences. They provision computing resource at different granularity.

In a web service, the execution duration of a Lambda function is the response duration to an API request, in terms of seconds. While the server is waiting for a request, there is no usage of the computing resource so you’re not paying for waiting for a request. However, this also creates the delay of cold-start, especially when the code size is large. There are several ways to optimize the cold start (e.g. SnapStart for Java), but none of those can completely get rid of the cold-start delay with a once-after-a-while request. A light GET call could take 5 seconds with cold start.

On the Fargate side, the resource provisioning is based on container lifecycle, instead of request lifecycle. As a result, you’re still paying for wait time, and it is not per-request billing. Since the container remains up, your request is not going to experience the cold-start if it’s been idle for a while. Although, Fargate saves you from the effort to right-sizing the computing nodes for container execution, it is not quite the idea of “scale-to-zero when idle” by itself.

Serverless Architecture

In the white paper AWS Serverless Multi-Tier Architectures with Amazon API Gateway and AWS Lambda, AWS advocates the serverless architecture as a modern alternative to the traditional widely adopted three-tier architecture (presentation, logic and data tiers). In the three tier architecture, the scalability of three tier are managed separately. The modern serverless architecture that AWS whitepaper proposes uses API Gateway and Lambda function in place of Load Balancer and EC2 instances (e.g. in an Auto Scaling Group), as illustrated below:

Both API Gateway and Lambda scale automatically to support the need of application workload. It assumes the role of logic tier in three-tier architecture but requires minimal maintenance work. For presentation tier, AWS has serverless alternatives such as CloudFront, S3. For data tier, AWS has serverless alternatives such as Amazon Aurora for relational database and DynamoDB for NoSQL. However, the “no request, no pay” model for Lambda does not apply to the data tier in serverless architecture.

Whilst this paradigm benefits small shop IT who wants to minimize infrastructure cost, it has downsides. There is no ability for infrastructure optimization. Since you do not manage where the code runs, client may have concerns over security (e.g. multi-tenant runtime). As business grows, keep using Lambda can result in technology lock-in. Also, a less used application usually requires warm-up time. A code start (downloading the code and preparing the environment behind the scene) can take 100ms to over a second.

Conclusion

PaaS attempts to help startups simplify the “grunt work” of IT operation. Serverless takes it even further. The semantics of serverless computing is confusing and the Wikipedia page acknowledges it as a misnomer. The nature of serverless model, is the cloud users delegate server capacity management to cloud platforms. The users don’t need to manage servers, VMs, instances, containers, etc on their own. In a previous post, I discussed the ability to scale to zero, which is just one of the many enabling technologies of serverless. Also, “no request, no pay” is neither an inherent nature of serverless model. Serverless service may involve storage (e.g. data service, S3, Aurora serverless) which incurs storage cost. There is a whitepaper on choosing the right AWS service to deploy your website or web application, with a decision tree.

AWS pioneered serverless with Lambda release in 2014 but competitor follows. In the Azure landscape, there is an entire suite of computing services from virtual machine to serverless (also with a decision tree in documentation). Azure’s counterpart for serverless architecture is Azure Function (released in 2016 for GA) with API Management. As for GCP, the serverless suite includes the event-driven Cloud Function (introduced in 2017) and Knative-based FaaS Cloud Run (introduced in 2019).

There are voices in advocacy of standardization of serverless model, and CNCF had since made minuscule efforts such as CloudEvents. The status quo, unfortunately, is anything but standardized.

Lastly, here is a table that summarizes the pros and cons of each computing service model.

Computing Service Model	Pro	Con
EC2	– Most straightforward and widespread legacy model – Legacy	– Ops tasks can be heavy (e.g. patch and vulnerability management of OS) – Utilization can be low
ECS	– Container orchestration is managed – Convenient to scale – Well integrated with other AWS services	– Limited advanced features – Vendor lock-in
EKS	– Highly scalable and flexible – Advanced, platform-neutral deployment tools available (e.g. Helm, ArgoCD, etc) – Custom configurations (e.g. operators)	– Significant operation overhead – Steep learning curve (especially for teams)
Lambda	– automatically scale – low ops overhead – pay per use	– limited choices of runtime – subject to latency due to cold start; yet warm start incurs cost – not suitable for long running tasks (batch processing jobs etc)

In summary, the Lambda-based serverless model is good for stateless server-side workload with short response time (<15s), tolerance of cold-start, no need for portability across platforms, and no complex package dependency.

GraphQL and gRPC

Fri, 07 Oct 2022 14:49:00 -0400

Big Picture

For inter-process communication at a high level, the two styles are asynchronous and synchronous styles:

Asynchronous event-driven style: involving an event broker as a middle man.
Synchronous request-response style: including several families of technologies:
- RPC (Remote Procedure Call):
  - CORBA (Common Object Request Broker Architecture)
  - Java RMI (Remote Method Invocation)
- SOAP (Simple Object Access Protocol)
- REST (Representational State Transfer)
- gRPC
- GraphQL
- Apache Thrift

RPCs, built on top of TCP/UDP, are usually complex to implement. SOAP improved it and can operate on HTTP. Many large companies today still used SOAP for message exchange. However, it has a limitation with complex format and specifications for XML messaging, giving rise to REST. REST is not a standard, but rather a loosely defined architectural style. Its payload can be in any format (XML, JSON, etc) specified in the header. As long as the API conforms to certain guidelines (criteria outlined in this article), we can consider the API RESTful. REST has been steadily replacing SOAP in the past few years. In this post, I start with REST, then dive into gRPC and GraphQL.

REST

The de facto method of building microservices using REST architectural style is use HTTP protocol with JSON format payload. JSON format is human readable, but not optimized for machine-to-machine communication. So there is some room for compression. To emulate a web request in REST, one can use curl, a common utility to emulate any HTTP client.

REST has its shortcomings. For example, the interface between REST client and server is not strongly typed. You can choose to use OpenAPI/Swagger specification to define types but it is still not tightly integrated. There is no enforcement on the format of the payload either. RESTful services are quite bulky, inefficient, and error-prone. gRPC and GraphQL emerged to address different challenges with REST. GraphQL operates on HTTP and we can view it as a layer on top of REST in a broad sense. gRPC on the other hand, operates on HTTP /2, and thereby inherits many advantages from it.

HTTP /2

HTTP/2 is the second major version of HTTP. It overcomes some issues with HTTP/1.1 on security, speed, etc. This post is a good rundown of the difference between HTTP /2 and HTTP /1.1, which account for many of the advantages of gRPC. A thorough discussion on the differences between the two HTTP versions is beyond what this post can cover. One of the important difference with HTTP/2, is that all communication between a client and server is performed over a single TCP connection that can carry any number of bidirectional flows of bytes. This makes gRPC a high-performance RPC framework. In HTTP/2, the key concepts to understand are:

Stream: a bidirectional flow of bytes within an established connection. A stream may carry one or more messages;
Frame: the smallest unit of communication in HTTP/2. Each frame contains a frame header, which at a minimum identifies the stream to which the frame belongs.
Message: a complete sequence of frames that map to a logical HTTP message that consists of one or more frames.

The request message is always triggered by the client. During the interaction, the client and server break down the message into frames, interleave them, and then reassemble them on the other side. In this way HTTP /2 multiplex the messages, and enables the following communication patterns:

Simple RPC: a single request and a single response in the communication;
Server Streaming RPC: a single request and message followed by multiple response messages;
Client streaming RPC: client sends multiple messages and the server replies with one response message;
Bi-directional RPC: client setups connection by sending header frames. Once connection is established, both client and server send messages simultaneously without waiting for the other to finish;

The streaming communication patterns fundamentally improves performance, enabling the duplex streaming capability for gRPC.

gRPC

At the protocol level, gRPC has the following advantages over REST:

well-defined service interface and schema
strongly typed data
duplex streaming (thanks to HTTP2)
built-in commodity features (e.g. authentication, encryption, resiliency, service discovery, etc)

I previously touched on gRPC protocol in the context of Envoy and etcd. Envoy makes use of gRPC for its control plane, where it fetches configuration from management server(s) and in filters, such as for rate limiting or authorization checks. Etcd store implements gRPC protocol for client utility to communicate with.

Since the typical use case of gRPC is internal communication, I have not been able to find a playground online. To get a taste of gRPC client-server interaction, just play with etcd on MacOS. gRPC is language neutral. To start developing, refer to the tutorial in different languages (e.g. Golang, Python). When developing a gRPC application, the first thing to do is define a service interface in IDL (interface definition language). gRPC uses protocol buffers as the IDL to define the service interface. Protocol buffers are a language-agnostic, platform-neutral, extensible mechanism to serializing structured data. Using that service interface definition, we can generate the server-side code known as a server skeleton. Also you can generate the client-side code, known as a client stub. The methods that you specify in the service interface definition can be remotely invoked by the client side as easily as making a local function invocation.

gRPC also has some disadvantages. Currently, the ecosystem is still small. When we have a service interface, we have to maintain the interface across versions.

GraphQL

In most of the use cases of gRPC and GraphQL, GraphQL works for external-facing services/APIs while internal services backing the APIs are implemented using gRPC. Honeypot created good documentary for GraphQL available here.

We use GraphQL for external facing services because it gives API client the ability to query. An SQL query allows client to filter requested data based on conditions, and define the interested columns in the data return. This capability is missing in the REST style guideline. One may choose to implement their RESTful service to support their client’s query requirement, GraphQL standardizes this capability with typed data, thereby prevents unnecessary network round trips, over- and under-fetching of data.

The official site is a good reference for learning. One needs to know concepts around queries and mutations, schema and types (scalars, variable, fragment, interfaces, unions) to understand how query works.

This page lists a number of publicly available services in GraphQL. For example, country information service is available here. On the web page you can put in a query like this:

query myCountry {
 countries(filter:{code:{in:"CA"}}) {
 code,
 name,
 capital
 }
}

The query (with the name of myCountry) above asks to return entries with countries as type. Then it filters the result by the condition that the country code must include “CA”. The return should include code, name and capital columns. The web page looks like this:

GraphQL is on HTTP, so I can emulate the call with curl and get the same result:

curl --request POST https://countries.trevorblades.com/ \
--header 'Content-Type: application/json' \
--data-raw '{
 "query" : "query myCountry { countries (filter: { code: { in: \"CA\" } }) {code name capital} }"
}'

The return prints the same result:

The document has a page on library options for different languages, such as Graphene and Ariadne for Python. You can use Ariadne and Flask to build GraphQL API as this post suggests.

The example on this page has a better example that highlights how GraphQL allows client to ask multiple questions at the same time, and how the response includes different pieces of answers. GraphQL backend often needs to connect to different downstream system. This requires a proxy service to connect to disparate data sources. Examples of such services with such capabilities include Apollo, AWS AppSync, or ApiGee.

Summary

For API protocols, we often compare among REST, gRPC and GraphQL. REST is loosely defined and widely adopted for the past few years. The use case of REST diverged into two areas: external facing API and internal service-to-service communication. gRPC and GraphQL are relatively new and they each are good for one of the use cases.

Microservice pattern based on gRPC and GraphQL (source: “gRPC up & running” by Indrasiri & Kuruppu)

The gRPC protocol on HTTP2 is often used in internal communication between microservices. GraphQL offers query capability and is therefore often used to face external client.

Build and Manage Kubernetes Clusters

Fri, 23 Sep 2022 11:50:00 -0400

There are numerous options to build a Kubernetes cluster. If your company has a multi-cloud strategy, most likely you will have to deal with cluster creation on multiple cloud platform or on virtual machines on premise.

Most likely, the chosen cloud platform already make it simple for us. However, it is still important to understand what it really takes to build a Kubernetes cluster. In general, we need to figure out these tasks:

Decide where to host the computing infrastructure (i.e. Node) : on premise or public cloud;
Choose a Kubernetes release: either the vanilla release or one of the third-party distributions;
Install Kubernetes to the computing environment, and integrate it with the cloud platform;
Determine required add-ons (e.g. Istio or Linkerd for Service Mesh, dashboard utility, etc);
Deploy application workload to Kubernetes platform;

A public cloud platform provider usually can assist you with task 1 through 3, and partially 4, depending on the provider. If your Kubernetes resides on private cloud or on-prem environment, you can use a Platform solution such as VMware Tanzu or Openshift, which usually covers task 1, 3 and 4. There is no standard about what task these platform solution must address. Therefore it is important to have this list of tasks in mind in order to make a good comparison. I will discuss each of the tasks in this post.

Hosting environment

Nodes are the building blocks of a Kubernetes cluster. We need master nodes as well as worker nodes. In addition, a working cluster also requires storage, and networking infrastructure.

Public cloud platforms typically provides control plane as a service, obviating administrator’s effort to provision master nodes. For example, the control plane of Azure AKS has two levels of uptime commitment: a free tier of 99.5% SLO and a paid tier with an SLA of 99.95% (using AZs) and 99.9% (without using AZs). This uptime commitment applies to control plane only and do not apply to worker nodes. The management of etcd store is also a responsibility of the cloud provider, which frees up the cluster administrator from managing etcd store. However, they cannot access etcd store either. This is not very convenient because as the size of the cluster grows it is a common requirement to connect to etcd store for troubleshooting purpose.

The deployment APIs for public cloud allow the cluster administrator to define the instance size, count and availability zone for the worker nodes. They also automatically register the worker nodes to control plane so that the cluster administrators do not have to do so by themselves. As to storage, the public cloud usually provide some default storage classes based on their storage as service. For networking device, the cluster provision process automatically configures the cloud API so the cluster can manage cloud resources such as network load balancer.

With private cloud or data centre, we usually use virtual machines, or bare-metal servers. Cluster administrators will need to make their own control plane with master nodes. and install worker nodes and register them to the master nodes. The Kubernetes Installation section below will discuss this.

Kubernetes release

If you have to install Kubernetes, you have to think about the Kubernetes release being used. You can use the binary from official Github repository. For example, the release note of version 1.24.3 points to the change log file for download links to server binaries, node binaries. This is the vanilla Kubernetes release.

Apart from the vanilla release, many developers build their own distributions, based off forks of the Kubernetes project. CNCF has a page to keep track of certified Kubernetes distributions. Some of the distributions are open source and can be used for on-prem infrastructure. Here is a list of top players:

Distribution Name	Repo	Description
EKS Distro	Link	Used in EKS managed service or EKS Anywhere for on-prem infrastructure
AKS Engine	Link	Used in Azure Stack for on-prem infrastructure.
Google Kubernetes Engine	N/A	Used in GKE managed service only.
OpenShift Kubernetes Engine	Link	Community distribution (OKD, or OpenShift Kubernetes Distribution) is the open-source upstream.
Rancher Kubernetes Engine (RKE)	Link	still using Docker as container runtime. Supported CNI include: Canal, Flannel, Calico and Weave
K3s	Link	Lightweight distro without small resource requirement. Great for Edge, IoT, ARM etc
RKE2	Link	Originally named RKE government. Supports deployment via Cluster API. Supports containerd as container runtime. Supported CNI include: Cillium, Calico, Canal and Multus. Lightweight
VMware Tanzu	Link	VMWare Tanzu Grid and VMWare Tanzu Community

Above is just a very incomplete list of Kubernetes distributions. There are many more distributions that are not on this list, such as CoreOS Tectonic, Docker Kubernetes, Heptio, Mesosphere, Mirantis, Platform9, Stackube, Telekube. For full details of how each distribution is different, you will need to go over their documents.

With the selected distribution, we still need to deploy the binaries to the nodes. We can do this with a cluster management platform, or standalone installers. Cluster management platform can also help us with baseline configuration (e.g. IAM integration, CNI plugin), in addition to the binary installation.

Cluster Management Platform

These platforms are also sometimes referred to as container management platform.

For example, OpenShift container platform is a self-managed platform based on OpenShift Kubernetes Engine and can run on a variety of hosting environment, public cloud, or private cloud. The installation steps varies depending on the hosting environment. When running on public cloud such as AWS (aka ROSA), the public cloud only provides computing nodes and associated infrastructure. Many corporate with multi-cluster strategy use this option on public cloud to keep their Kubernetes cluster fleet consistent across cloud vendors.

The Openshift container platform also packages some useful open-source add-ons with corporate support, for example:

Red Hat’s strategy is to pick the most renowned open-source project in each domain and add enterprise support to it. However, for management portal, Red Hat developed its own Advanced Cluster Management tool for Kubernetes, and open-sourced it in 2020 in the upstream project Open Cluster Management.

Similar to OpenShift, VMware Tanzu also attempts to cover the domains, with a smaller product portfolio:

Service Mesh: compatible with Istio
Mission Control: management portal
Observability

Google Anthos is also a container platform. Their product line include, but not limited to:

Anthos Config Management (ACM)
Anthos Service Mesh (ASM, an Istio distribution)

SUSE, the developer of RKE, RKE2, and K3s) offers Rancher as multi-cluster management platform. Apart from the engines, SUSE also offers Lonhorn as a storage solution. However, they do not have offerings for service mesh or GitOps. So there is no doubt that Red Hat OpenShift has the most complete portfolio for Kubernetes.

There are also companies that only offers management platforms without their own Kubernetes distribution. For example:

Product capabilities in this category vary a lot and you should refer to their specific documentation to understand. You will probably see a stack chart from each of the platform provider (e.g. SUSE Enterprise Container, OpenShift, Tanzu, Anthos, Rafay) with all technology integrations.

Cluster Installation Tools

As we saw in the installation steps for OpenShift, they are highly dependent on platform. With public cloud, the provisioning process also applies only to a specific platform. Since Kubernetes Installation process is tedious, some tools emerged to help, for example: kubespray, kubeadm, kops and Cluster API. These are governed by SIG cluster lifecycle special interest group.

Here are some traditional options to install a Kubernetes clusters:

kube-up: the first tool to build cluster from 2015. It has been deprecated.
Kubeadm: a tool built to provide best-practice “fast paths” for creating Kubernetes clusters that are minimum viable, and secure. Kubeadm’s scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools. It is first released in Sep 2016. The high level configuration steps goes through initialization (kubeadm init), control plane (kubeadm join control plane), and node (kubeadm join node). Kubeadm does not integrate with cloud providers and it does not install addons (auth, monitoring, CNI, storage class)
Kubespray: runs on bare metal or VMs using Ansible for provisioning and orchestration. The first release was in Oct 2015. Since v2.3 (Oct 2017) kubespray started to use kubeadm internally. In addition to kubeadm, kubespray configures CNI, storage class, other CRI. It supports cloud providers and air-gap environment. However it does not support infrastructure management.

The options above are official options. You may use kubeadm and kubespray to quickly (i.e. in an hour) spin up clusters for education purposes. However, with their limitations, it typically requires a lot of efforts to build a production-grade cluster with the needed addons and integrations.

Apart from the official options, there are also unofficial tools such as kubicorn, which was first introduced in 2018 as a cluster management framework with modular support for cloud providers. However it appears to be short-lived.

In the next two sections, we introduce kops and cluster API, two most recent projects to install cluster.

Kops

The kops utility directly perform the provisioning and orchestration via API to the cloud deployment engine. Kops, with first release in Oct 2016, is tightly integrated with the unique features of the cloud providers (e.g. AWS: ASG, ELB, EBS, KMS, S3, IAM). However, kops is only CLI without controller-style reconciliation. It does not support baremetal or vsphere. It also bundles addons with fixed version.

When picking a tool to install cluster, we need to strike a balance between how much simplification the tool brings, and how many different platform the installer can work with. Kops appears to be such a good compromise. It works with a number of cloud platforms using different set of APIs, although most are in alpha and beta stages today. Here is how to install cluster on AWS.

Both kops and Cluster API have good momentum but they work differently. Cluster API was first released in Mar 2019, and is currently less mature than kops. However, it is declarative and may reflect the direction of where cluster lifecycle management is heading.

Cluster API

Cluster API focuses on following areas:

Manage cluster lifecycle declaratively
Infrastructure abstraction (e.g. computing, storage, networking, security, etc)
Utilizing existing tools (e.g. kubeadm, cloud-init)
Modular and pluggable: to be adaptable to different infrastructure providers.

It involves a number of CRs as illustrated in its diagram. We should be clear on the providers for Bootstrap, Infrastructure and Control Plane.

The biggest benefit is the controller pattern to manage the entire lifecycle of a cluster. This allows managing clusters with GitOps, and rolling upgrade of the cluster. It also allows for declarative node scaling, self healing and multi-cluster management.

The client utility for is clusterctl, and with that along with the manifest, we can create a cluster in a few commands. A lot of workflows are still in development but we can take a look at its quick start guide to get a taste of how it works. The installation steps vary a lot based on the environment and the cluster. Also it introduces the separation of management cluster and workload cluster.

Workload cluster is the target cluster being created, as per the manifests.
Management cluster is where you keep track of the workload cluster being managed. You can manage multiple workload clusters from a single management cluster. Note that this management cluster will store credentials about workload clusters, and may become a single point of failure.

Although Cluster API reflects a great initiative to standardize the provisioning of Kubernetes cluster, whether it will succeed has to do with the level of complexity. In the next section, we will get a taste of how it looks to deploy a Kubernetes cluster in a lab.

Management cluster vs workload cluster

In the lab, I use my MacBook to create a management cluster with KinD. Then we configure a workload cluster in AWS from the management cluster.

Cluster API Lab

Note that the steps here are based on the quick start guide on Cluster API document. Also, there is a bug with the AWS provider so the end of the lab will report a warning. The main purpose of this lab is to demonstrate how Cluster API is supposed to work, even though it still has yet to mature.

To start, I install clusterctl (the cluster API client utility), clusterawsadm (the utility specific for AWS) on MacBook, then start a simple KinD cluster.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.0/clusterctl-darwin-amd64 -o clusterctl
chmod +x ./clusterctl
sudo mv ./clusterctl /usr/local/bin/clusterctl
clusterctl version

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.4.1/clusterawsadm-darwin-amd64 -o clusterawsadm
chmod +x clusterawsadm
sudo mv clusterawsadm /usr/local/bin
clusterawsadm version

kind create cluster

So far, I installed the required utility and a KinD cluster on MacBook. Then I use clusterawsadm to create InstanceProfile, ManagedPolicy and IAM Roles required for cluster creation. The AWS region and access are configured as environment variables:

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=J8ByduiofpwuisDjDoijOISDs
clusterawsadm bootstrap iam create-cloudformation-stack

This runs a CloudFormation stack to create the permission related resources:

Then I initialize the management cluster with the clusterctl utility, specifying AWS as a provider. I also need to assign the environment variable AWS_B64ENCODED_CREDENTIALS with proper value:

export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
clusterctl init --infrastructure aws

Now I use clusterctl to generate the manifest for the workload cluster. In environment variables, I specify cluster and node sizes, SSH key name, control plane machine type and node machine type:

export AWS_SSH_KEY_NAME=cskey
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large
clusterctl generate cluster myekscluster --kubernetes-version 1.24.3 --control-plane-machine-count=3 --worker-machine-count=3 > capi-quickstart.yaml

kubectl apply -f capi-quickstart.yaml

At the end I tell the management cluster to create a workload cluster as per the manifest, by simply declaring the CRs. It will take some time for the cluster to create, and there are a number of ways to monitor the progress. You can monitor the log on the controller pods in their respect namespaces. You can also check the cluster status with:

kubectl get kubeadmcontrolplane
clusterctl describe cluster myekscluster

Currently there is a bug and the commands at the end will report as below:

Hopefully the bug will be fixed shortly. To delete the cluster, simply delete the resources in the manifest with kubectl delete -f capi-quickstart.yaml

Summary

There are numerous ways to build a Kubernetes cluster. Before deciding on the approach, I recommend having a full understanding of the hosting environment. This is because installation approach and hosting environment are still tightly coupled. This is the status quo and is not going to change in the near future. Both kops and cluster API reflects initiative to decouple the two but both are still in early stage and already facing growing complexity. Cluster API manages complexity with CRDs to abstract system resources and infrastructure, as illustrated here:

CRDs and providers to abstract system resources and infrastructure

The diagram is from the “Cluster API and declarative Kubernetes Management” white paper. Here is a stream with more about the same topic.

MinIO for S3-compatible Object Storage

Fri, 09 Sep 2022 09:00:00 -0400

I reviewed some storage technologies on Kubernetes but they are all for block and file storage. In this post, I will discuss the current available options for container workload to use object storage. I will also touch on MinIO as an object storage solution.

Object storage

Block and file system are more native to operating system because they present themselves to the OS as a block device or file system attached to the OS. In other words, application processes running on the OS will be able to access the storage by address expressed as a POSIX-compatible path. On the contrary, object storage is a REST API service, operating at the application layer in the TCP/IP stack. Therefore, we can think of object storage as “storage as a web service”.

Object storage can be made very cheap. However, the application protocol may vary depending on the object storage provider. Amazon S3 is a forerunner in object storage market and its protocol has emerged as the de-facto standard for object storage. When building an application and if there is one object storage protocol to support, it should be S3. For non-S3 object storage services, we can front them with an S3 interface, if the provider itself does not have one. For example Ceph storage has its Gateway S3 API.

Container Object Storage Interface

If we use S3 as the universal object storage protocol, does that also address object storage access with container workload on Kubernetes? Absolutely. Nonetheless, for a number of reasons using REST API from containers are not the best option. From platform’s perspective, it is the platform that should define how to access object storage, instead of leaving it with an application-layer protocol.

When a pattern (for storage, or networking, etc) turns out very common, the platform layer should incorporate it as an infrastructure service, manage it with its own standard, and provide it to application so that developer can focus on business features. With that vision, the community brought up the Container Object Storage Interface (COSI) initiative. It is currently in very early stage, but the idea is to commoditize object storage in Kubernetes platform with a unified interface. For more background about this initiative, refer to the post “Beyond block and file – COSI enables object storage in Kubernetes“.

COSI is the ultimate cloud native solution but it is still in pre-alpha phase as of mid 2022. Unfortunately, it is not a recommended solution to any real-life project in 2022, and we are stuck with the unified API approach until COSI matures.. The unified API approach is by no means cloud native, but has come to maturity for adoption. S3 Rest API is our friend, regardless of whether the client process is in a container or not.

Update: on Sept 2, 2022, Kubernetes introduced COSI as alpha feature.

MinIO Introduction

In order to use S3 protocol without using Amazon S3 storage, we can use MinIO to build our own object storage service serve client via a S3-compatible REST API interface. The main developer of the MinIO project is MinIO Inc, a startup from 2014. Having learned the lessons from GlusterFS, the founders and developers make MinIO very simple. MinIO operates in two modes: gateway mode (soon to be legacy) and server mode.

In the Gateway mode, MinIO as a gateway between client and destination storage, and does not persist data to itself. In the past, the destination storage can be Azure Blob and Google Cloud Storage (GCS) and HDFS as backend. However, these supports are deprecated now. The current release (July 2022) only supports S3 and NAS backend. According to MinIO’s blog post from February 2022, the entire MinIO Gateway feature will be removed in August, leaving server mode the only option for MinIO.

In the Server mode, the MinIO service will persist data to itself in a file system (or volume). You can specify that file system (or volume) as you launch the server. As one of the quick-start guides shows, we can host MinIO server using a single executable. For administrative tasks, MinIO has a web console and a client utility called mc.

MinIO Deployment Options

For storage service, there are a number of deployment options:

SNSD (single-node, single-drive): single MinIO server with a single storage volume or folder.
SNMD (signle-node, multi-drive): single MinIO server with four or more storage volumes.
MNMD (multi-node, multi-drive, aka distributed): multiple MinIO servers with at least four drives across all servers. This should be considered for production grade configuration.

The deployment options above describes the node and volume topology. No matter which topology option, there are also a number of ways to host the MinIO service process: on Linux OS, Windows OS, MacOS, Docker Container, and on Kubernetes platform.

In addition, MinIO Inc ships the software under different business models. For example, there are fully managed applications in Azure Marketplace, AWS Marketplace, and GCP Marketplace all hosted on virtual machines with extra charges. Clients not willing to pay can host MinIO storage all on their own, either on virtual machines, or on managed Kubernetes environment provided by each cloud provider.

MinIO Hosting solutions

MinIO lists these hosting solutions under multi-cloud products. These hosting solutions (or “products” in MinIO’s term) vary in terms of where peripheral services and data tiers are hosted. Here is the list of the supported platforms:

To illustrate how these solutions are different, I put some details on a few options together for an incomplete comparison below:

	Kubernetes	EKS	AKS	GKE
Hot Storage	Direct PV (NVMe)	EKS EBS CSI	Azure CSI	GKE Standard SSD
Warm Storage	Direct PV (HDD)	S3 IA	Azure BlobStore	GCS
Cold Storage	Public Cloud storage	Glacier	Azure Cool Blob	GCS for Data Archiving
Encryption	HashiCorp Vault	KMS	Azure Key Vault	Cloud Key Management
Observability	Elastic Stack and Grafana	Managed ElasticSearch Prometheus	Azure Monitor	Stack Driver
Identity Provider	KeyCloak	LDAP, SSO	Azure Active Directory	GCP Cloud Identity
LB and Cert Mgmt	Nginx, Let’s Entrypt	AWS Cert Mgr, ELB	Azure Load Balancer, JetStack, Let’s Encrypt	GCP Cloud LB and Managed Cert

Note that all of these hosting solutions are based on some flavour of Kubernetes. The hot tier is usually based on storage options available to the platform. MinIO service access this hot tier via Kubernetes persistent volume. The warm and cold tiers are backed by different object storage service. Between MinIO and storage client, it always use the same S3 compatible Rest API.

MinIO also has tiering capability. While the hot storage destination has to be either a file system or Kubernetes persistent volume, remote tiers can be S3 , Azure Blob, or GCS. MinIO supports encryption at rest (SSE-KMS, SSE-S3, SSE-C) and in transit (TLS) for security, as well as many other useful features such as object replication, versioning, locking, events, Prometheus metrics, lifecycle management etc.

Connect to MinIO server with S3 client

To validate that the client is compatible, we use MinIO’s client utility (mc) to connect to an AWS S3 bucket. Then we use AWS CLI to connect to a MinIO server, similar to this instruction. To do so, we first install client and server utilities:

brew install minio/stable/minio
brew install minio/stable/mc
minio --version
mc --version

Then, we start MinIO server and store an object using AWS CLI’s S3 tool. In our working directory, we create a new directory called minio_data and launch MinIO server with it:

mkdir minio_data
minio server minio_data --console-address :9090

Once the server is up, the screen should display the details, including the portal URL and the default username and password will be used as Access Key ID and Secret Key:

Note that the MinIO service does NOT have TLS enabled by default, on the console or API service. At this point, we can browse to the console web page using the given credential. Then, we can configure AWS CLI with a new profile just to act as a client to communicate with the MinIO server:

$ aws configure --profile minio-cli
AWS Access Key ID [None]: minioadmin
AWS Secret Access Key [None]: minioadmin
Default region name [None]: us-east-1
Default output format [None]: json
$ aws configure set default.s3.signature_version s3v4 --profile minio-cli

At this point, the AWS CLI is configured to communicate with MinIO server. Then, we can create bucket, list object in the bucket, copy an object to the bucket, etc

$ aws --endpoint-url http://127.0.0.1:9000 s3 ls --profile minio-cli # list all bucket, should return empty
$ aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://hehebucket --profile minio-cli # create new bucket
make_bucket: hehebucket
$ aws --endpoint-url http://127.0.0.1:9000 s3 cp README.md s3://hehebucket --profile minio-cli # copy a file to bucket as a new object
upload: ./README.md to s3://hehebucket/README.md
$ aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://hehebucket --profile minio-cli # list objects in the bucket
2022-07-09 00:30:23 631 README.md

The created bucket and object are also visible in MinIO web console, under “Bucket”:

The steps above validate that AWS CLI can talk to MinIO server. Because of that, MinIO server can emulate an S3 service in any development environment so users do not always have to use S3 from AWS. This makes sense for both cost and security reasons for the organization.

Connect to S3 with MinIO client

In this lab, we create an S3 bucket and use mc utility to store an object to it. In order to consistently create S3 bucket and associated permissions, I use the CloudFormation template in this repo. The output of the CloudFormation stack returns the Access Key ID and Secret Key required for the client to access the bucket. Once we cloned the repo, let’s enter the obj-store-helper directory, and run aws cli command to launch the CloudFormation template, assuming it has been configured:

BUCKET_NAME=c0sas2dsadigihunch
S3_STACK_NAME=$BUCKET_NAME-stack
aws cloudformation create-stack --template-body file://aws-s3-stack.yaml --stack-name $S3_STACK_NAME --parameters ParameterKey=S3BucketName,ParameterValue=$BUCKET_NAME --capabilities CAPABILITY_IAM

# to delete stack after test, run: aws cloudformation delete-stack --stack-name $S3_STACK_NAME

In the AWS console, we should see the configuration information as below:

Supposed the bucket name is vna-tst-c0sas2dsadigihunch as shown above, this allows us to configure the client utility MC as below:

mc alias set awss3 https://s3.amazonaws.com # Fill in access key ID and Secret key at the prompt
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, should return empty
mc cp README.md awss3/vna-tst-c0sas2dsadigihunch/README.md # upload and object to bucket
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, the uploaded object should be there
mc rm awss3/vna-tst-c0sas2dsadigihunch/README.md # delete the object
mc alias remove awss3 # remove awss3 alias

Once we emptied the bucket, we can delete the CloudFormation stack. This test only needs client utility mc to verify that MinIO client is able to talk to AWS S3 server.

Erasure Coding

For scalable production use, we should deploy MinIO in distributed mode. When MinIO is configured in distributed deployment (MNMD, or multi-node, multi-drive), it implicitly enables an important feature called erasure coding. This erasure coding feature further unlocks a number of other MinIO features:

Erasure coding is MinIO’s data redundancy and availability feature that allows MinIO deployments to automatically reconstruct objects on-the-fly despite the loss of multiple drives or nodes in the cluster. Erasure coding provides object-level handling with less overhead than adjacent technologies such as RAID. The key concept is Erasure Set, a set of drives in a MinIO deployment that supports Erasure Coding. MinIO evenly distributes object data and parity blocks among the drives in the Erasure Set.

Two important variables are M and N: for a given erasure set of size M, MinIO splits objects into N parity blocks, and M-N data blocks. MinIO uses the EC:N notation to refer to the number of parity blocks (N) in the deployment. To determine optimal erasure set size for the cluster, use MinIO’s Erasure Coding Calculator tool.

To help client to specify per-object parity with Erasure Coding, MinIO uses storage classes. Note that the storage class concept in MinIO is distinct from AWS S3 storage class or Kubernetes storage class. In MinIO, a storage class defines parity settings per object. The STANDARD storage class (default) defines EC:N based on M, which can be overridden. In addition, there is REDUCED_REDUNDANCY storage class, whose parity must be less than or equal to that of STANDARD storage class.

The erasure coded backend also protects the storage against Bit Rot with HighwayHash algorithm.

More Features

Authentication and authorization between MinIO client and MinIO server have a number of options. MinIO client may use the built-in standalone identity management in MinIO server. This is the default mode. In addition, one may delegate IAM to external service. To Active Directory via LDAP, or any Identity provider that supports OIDC (JWT with Authorization Code Flow).

As to Object Lifecycle Management (OLM), MinIO allows you to define a remote tier storage for each local target (bucket). The remote tier can be Amazon S3, Google Cloud Storage or Azure Blob storage. We can use mc utility to administer the remote tier and OLM. Configuration steps (e.g. Azure Blob, AWS S3) usually include:

Configure required permissions on the MinIO bucket, create user account for OLM activities.
Configure the Remote Storage Tier
Create and Apply an ILM Transition Rule. The rule can be expressed in a json document.
Validate the creation of ILM transition rule
Validate the effect of transition rule.

As for encryption, MinIO can support encryption at rest. It can also work with etcd store to store encrypted IAM assets if KMS is configured.

Conclusion

Even though we watch for the progress of COSI initiative, we still use Rest API to access object storage from container, which is no different than from a virtual machine. If we develop an application, then we should make it support S3 protocol, a de-facto standard protocol for object storage. As for the storage backend, if we want to be vendor neutral, the feature-rich MinIO is the best bet. We can use MinIO to build our own Object storage as a service compatible with S3. We can also lifecycle our object to remote object storage tier backed by Azure, GCP or S3. In this post we validated the S3 compatibility, and discussed some advanced MinIO features.

Kubernetes Storage on Azure 3 of 3 – Ceph by Rook

Fri, 26 Aug 2022 19:43:00 -0400

In the last two posts, I covered the native storage options on Azure Kubernetes Service, as well as Portworx as an example of a proprietary Software Defined Storage (SDS) solution. There are also a number of open-source alternative SDS solutions. Ceph has nearly a decade of history from prior to containerization, and is the most widely adopted storage platform. In this post, we continue to explore Ceph as an open-source storage solution on Azure Kubernetes.

Ceph by Rook

Ceph is an open-source SDS platform for distributed storage on a cluster and provides object, block and file storage. Installation of Ceph SDS can be complex, especially on Kubernetes platform. Rook is a graduated CNCF project to orchestrate storage platform. Rook by itself is not SDS and it supports:

Ceph: configure a Ceph cluster. Think of this as the equivalent of cephadm on Kubernetes platform.
NFS: configure an NFS server. Think of this as the equivalent of nfsd daemon on Kubernetes platform.
Cassandra: an operator to configure a Cassandra database cluster. It is now deprecated.

We play with Rook Ceph. I also refer to it as Ceph by Rook. The contribution of Rook project is it simplifies the installation as a matter of declaring custom resources using CRDs. Here are some high-level CRDs to know:

CephCluster: creates a Ceph storage cluster
CephBlockPool: represents a block pool
CephFilesystem: represents a file system
CephObjectStore: represents an object store
CephNFS: spins up a NFS Ganesha server to export NFS shares of a CephFilesystem or CephObjectStore.

As with typical Kubernetes resources in controller pattern, Ceph by Rook needs an operator along with custom resources. We can use YAML manifest for both of them, and the manifests are usually very tediously long. We can also use Helm to install both of them, by providing a value file. Now we will install Ceph on AKS.

Install Ceph Operator on AKS

The steps are influenced by two relevant posts (here and here). However, I’ve incorporated the cluster configuration in the Azure directory of the cloudkube project, a modular Terraform template to configure AKS cluster and facilitate storage configuration. The node group and instance sizes are selected to be just enough to run a ceph POC cluster with minimum cost. One of the node groups is tainted with storage-node, as if the following command were run:

kubectl taint nodes my-node-pool-node-name storage-node=true:NoSchedule

You will only need to taint the nodes with the command above if you choose not to use the cloudkube template. The taint ensures that only Pods with corresponding toleration and effect can be scheduled to those nodes.

We use Helm to install Rook Operator. We need a value file (e.g. rook-ceph-operator-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/operator-chart.md
crds:
 enabled: true
csi:
 provisionerTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
 pluginTolerations:
 - effect: NoSchedule
 key: storage-node
 operator: Exists
agent:
 # AKS: https://rook.github.io/docs/rook/v1.7/flexvolume.html#azure-aks
 flexVolumeDirPath: "/etc/kubernetes/volumeplugins"

Then we install the operator with Helm:

helm install rook-ceph-operator rook-ceph --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-operator-values.yaml
kubectl -n rook-ceph get po -l app=rook-ceph-operator

After installing the operator, we check the Pod status to make sure it is running. Then we can install the actual Ceph Cluster in one of the two ways. We can declare a CephClusterCRD ourself, or we can use Helm again to declare the CRD. Helm Chart gives us a lot of useful default values and saves us from editing a large body of YAML manifest.

Install Ceph CR on AKS

We use Helm to install CephCluster CRD. We create a value file (e.g. rook-ceph-cluster-values.yaml) with content as below:

# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/ceph-cluster-chart.md
operatorNamespace: rook-ceph
toolbox:
 enabled: true
cephObjectStores: [] # by default a cephObjectStore will be created. Setting this to null disables it
#cephBlockPools: # by default a cephBlockPool will also be created with default values
#cephFileSystems: # by default a cephFileSystem will also be created with default values

cephClusterSpec:
 mon:
 count: 3
 volumeClaimTemplate:
 spec:
 storageClassName: managed-premium
 resources:
 requests:
 storage: 10Gi
 resources:
 limits:
 cpu: "500m"
 memory: "1Gi"
 requests:
 cpu: "100m"
 memory: "500Mi"
 dashboard:
 enabled: true
 storage:
 storageClassDeviceSets:
 - name: set1
 # The number of OSDs to create from this device set
 count: 3
 # IMPORTANT: If volumes specified by the storageClassName are not portable across nodes
 # this needs to be set to false. For example, if using the local storage provisioner
 # this should be false.
 portable: false
 # Since the OSDs could end up on any node, an effort needs to be made to spread the OSDs
 # across nodes as much as possible. Unfortunately the pod anti-affinity breaks down
 # as soon as you have more than one OSD per node. The topology spread constraints will
 # give us an even spread on K8s 1.18 or newer.
 placement:
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: kubernetes.io/hostname
 whenUnsatisfiable: ScheduleAnyway
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd
 tolerations:
 - key: storage-node
 operator: Exists
 preparePlacement:
 tolerations:
 - key: storage-node
 operator: Exists
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: agentpool
 operator: In
 values:
 - storagenp
 topologySpreadConstraints:
 - maxSkew: 1
 # IMPORTANT: If you don't have zone labels, change this to another key such as kubernetes.io/hostname
 topologyKey: topology.kubernetes.io/zone
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - rook-ceph-osd-prepare
 resources:
 limits:
 cpu: "500m"
 memory: "4Gi"
 requests:
 cpu: "500m"
 memory: "2Gi"
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 resources:
 requests:
 storage: 100Gi
 storageClassName: managed-premium
 volumeMode: Block
 accessModes:
 - ReadWriteOnce

During the cluster provisioning, there will be a number of preparing Pods. We want those Pods to run on nodes with label agentpool=storagenp. In real life, we need to orchestrate where to run each workload, by restricting the nodes to schedule certain types of workload.

Then we can install the cluster using Helm:

helm install rook-ceph-cluster rook-ceph-cluster --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-cluster-values.yaml

After running the Helm install, it may take as long as 15 minutes for all resources to settle. Watch the Pod status in rook-ceph namespace. At the end, make sure that the cluster is created successfully:

kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get CephCluster
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL
rook-ceph /var/lib/rook 3 15m Ready Cluster created successfully HEALTH_OK
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephBlockPools
NAME PHASE
ceph-blockpool Ready
kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephFileSystems
NAME ACTIVEMDS AGE PHASE
ceph-filesystem 1 20m Ready

In my case it took 15 minutes before the cluster comes up as created successfully. You should notice that two storage classes were also created as a part of the install. It however did not create a storage class or CRD for object storage, because we explicitly disabled it in the Helm value file by setting cephObjectStores value to null.

Dashboard

We enabled dashboard. To configure the dashboard view properly, we would need an ingress. For a quick view here, we can play port forwarding tricks. First we fetch the admin password for use in the next step. Then expose the dashboard to the bastion host:

$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d
$ kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443

Since I don’t have UI on the bastion host, I use the port forwarding trick again from my own MacBook. Start a new terminal and SSH to the bastion host with port-forwarding switch:

$ ssh -L 8443:localhost:8443 kubeadmin@20.116.132.8

The command above suppose the public IP of the bastion host is 20.116.132.8. Then from my MacBook I can browse to localhost:8443 (with Safari browser which gives me the option to bypass certificate error). At the web portal, provide username (admin) and password (as retrieved above):

Ceph console for Kubernetes

Apart from the dashboard, we can also use ceph admin tool from a toolbox pod, following this instruction. For monitoring, Ceph by Rook can expose metrics for Prometheus to scrape.

Performance

With default ceph configuration on AKS, I ran quick performance test using kube-str . The result is as follows:

	read_iops	write_iops	read_bw	write_bw
ceph-block	IOPS=464.507294 BW(KiB/s)=1874	IOPS=243.296143 BW(KiB/s)=989	IOPS=509.928162 BW(KiB/s)=65797	IOPS=248.530762 BW(KiB/s)=32338
ceph-filesystem	IOPS=438.701324 BW(KiB/s)=1770	IOPS=226.270660 BW(KiB/s)=920	IOPS=405.936340 BW(KiB/s)=52456	IOPS=208.869293 BW(KiB/s)=27229

The metrics reflects performance under default configuration. It should not be considered as the best performance that Ceph can deliver on Azure Kubernetes.

Summary

I discussed three storage options for Azure Kubernetes but the idea applies to other Kubernetes platform hosted on a CSP. The native storage has significant limitation. NFS has latency. Block storage does not address high availability at the storage layer. Portworx and LINSTOR fill that gap as a commercial solution. Ceph is based on Object storage.

Kubernetes Storage on Azure 2 of 3 – Portworx

Fri, 12 Aug 2022 15:23:00 -0400

In the previous post, we have discussed built-in storage classes on Azure Kubernetes. Further to that, we will examine some third-party software defined storage (SDS) options that are compatible with Azure Kubernetes Service in this post. Then we take Portworx on Azure as an example. Although, these options are specific to Azure, most of the players also have solutions for other managed Kubernetes platforms. Also, the methodology to study storage options remain the same regardless of cloud service provider.

In fact, I touched on software defined storage (SDS) in the context of general non-containerized workload in a separate post. In short, storage providers decouple the SDS appliance from the full storage solution in order to lower the cost and increase flexibility. To the storage consumer (e.g. a process running on Linux OS), SDS can present a block disk or file system. There are also SDS solution that can host your own object storage and we will discuss that later. In fact, SDS has gained significant popularity in recent years. For example, the report “Validating Software-Defined Storage Operating Models for the Enterprise” by archiectingit divided the evolution into four phases and cited that Gartner predicts the SDS revolution to reach 50% of the storage market by 2024, from 15% in 2020.

This post discusses SDS in the context of container storage. Then we will install Portworx on Azure Kubernetes.

SDS for Kubernetes

Many SDS appliances also developed the capability to present storage volumes to containerized workload. I put together a list of SDS products that works on Azure with their supported access modes:

Solution	Licence	Development and Support	Access Mode
Portworx	Closed source. Free Essential tier. Enterprise features on License.	Commercially supported by PureStorage	RWO, RWX
Ceph by Rook	Open source. Rook is a graduated CNCF project.	Developed and commercially supported by Red Hat, Canonical and SoftIron	RWO, ROX, RWX
OpenEBS	Open source. Sandbox CNCF project.	Developed and commercially supported by MayaData et al.	RWO
Longhorn	Open source. Incubating CNCF project.	Originally developed by Rancher, and commercially supported by SUSE	RWO
StorageOS	Closed source. License required.	Commercial support by Ondat.	RWO
LINBIT	Open-source with enterprise plans	Enterprise support

RWO is the most commonly supported mode. The report “Performance Benchmarking Cloud Native Storage Solutions for Kubernetes” makes a comparison of performance among some of the options in early 2021. Another potentially opinionated comparison list is by LINBIT.

Portworx is a leading player with commercial SDS solution and I will test its free Essential tier in the rest of this post. Ceph is one of the most mature leading open-source offering and I will test it in the next post.

For Portworx, we can use the terraform template cloudkube for Azure. The template assigns the kubelet’s managed identity as contributor of the node resource group. The template also creates a bastion host with direct SSH access to the nodes.

Portworx Operator on Azure

We use Portworx Operator to configure storage cluster. Portworx has an instruction for AKS but it is not tailored to specific identity model. For simplicity, use my cloudkube Terraform template to create the AKS cluster, and skip the “Prepare Your AKS Platform” page. Instead, follow the “Deploy Portworx using Azure managed identity on new AKS cluster” page starting at step 7.

At that step, we need to create a secret with the client ID of the managed identity for node agent. The terraform template outputs the BYO identity’s client ID. After cluster creation, we simply SSH to the bastion host and create the secret using the output.

To install Porworx using operator, we can follow a wizard in PX-central. If this is the first time, we need to create an account and log in to the portal. If this is not the first time and you have previously created a cluster, you need to detach that cluster by going to Profile from bottom left corner on the portal page.

Follow the guide in “Install Portworx on AKS using the Operator“. In the wizard, click on “Portworx Essentials” for free tier, or “Portworx Enterprise” for the 30-day trial. Then select operator with latest version. In the rest of the wizard steps, select options applicable to Azure environment. The last step will present two kubectl commands to install operator and install the CR. Run the command to install operator and verify result:

kubectl apply -f 'https://install.portworx.com/2.9?comp=pxoperator'
kubectl -n kube-system get deployment portworx-operator

Portworx Custom Resource

To install the CR, we need to customize the given manifest in order to use our managed identity. We can download the YAML manifest (portworx_essentials.yml) and modify it in text editor. As the page “Deploy Portworx using Azure managed identity on new AKS cluster” suggest at step 9: in the env section, remove the AZURE_CLIENT_SECRET and AZURE_TENANT_ID sections but keep the AZURE_CLIENT_ID section. My CRD declaration looks like this:

# SOURCE: https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
 name: my-very-long-px-cluster-id
 namespace: kube-system
 annotations:
 portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true"
 portworx.io/is-aks: "true"
 portworx.io/misc-args: "--oem esse"
spec:
 image: portworx/oci-monitor:2.10.2
 imagePullPolicy: Always
 kvdb:
 internal: true
 cloudStorage:
 deviceSpecs:
 - type=Premium_LRS,size=150
 kvdbDeviceSpec: type=Premium_LRS,size=150
 secretsProvider: k8s
 stork:
 enabled: true
 args:
 webhook-controller: "true"
 autopilot:
 enabled: true
 monitoring:
 prometheus:
 enabled: true
 exportMetrics: true
 featureGates:
 CSI: "true"
 env:
 - name: AZURE_CLIENT_ID
 valueFrom:
 secretKeyRef:
 name: px-azure
 key: AZURE_CLIENT_ID
---
apiVersion: v1
kind: Secret
metadata:
 name: px-essential
 namespace: kube-system
data:
 px-essen-user-id: aaaabbbbccccddddmyverylongpxessenuserid
 px-osb-endpoint: ssssssyyyyyyyzzzzzzmyverylongpxosbendpoint

In this custom resource manifest, I specify to get the cluster to create disk from Azure, even though it has the capability of using an existing disk. Then apply the CRD manifest:

 kubectl apply -f portworx_essentials.yml

This will take up to 10 minutes to create several related resources. There are several check points to ensure the cluster is created successfully. Here are some useful commands:

kubectl -n kube-system get po # all Pods related to portworx should be ready and running, especially the portworx-api ones
kubectl -n kube-system get storagecluster # the status should report online
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') 
# get the name of one of the portworx Pod for storage cluster
kubectl -n kube-system exec -c portworx -it $PX_POD --tty --stdin -- /opt/pwx/bin/pxctl status 
# should report "Status: PX is operational" with valid license loaded
kubectl get sc # portworx related storage classes are available

The pre-built storage classes are not CSI based. However, CSI should be automatically enabled in recent operator versions. We should create our own CSI storage classes and PVCs using our own storage classes. Here are two examples:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-database
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "high"
 io_profile: "db"
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-artifact
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "medium"
 io_profile: "sequential"

That is a bare minimum Portworx installation. Check out their documentation for the full features. With this minimum install we can go to the section “Validate Persistent Storage” from the previous post to validate the persistent volume.

For troubleshooting purpose, pxctl is the utility and it is available on Portworx Pods.

Performance

We care not only the functionality, but also the performance. So I ran a quick performance test using kube-str, using all default configuration. The result is as follows:

	read_iops	write_iops	read_bw	write_bw
px-csi-database	IOPS=969.614136 BW(KiB/s)=3894	IOPS=729.698059 BW(KiB/s)=2935	IOPS=1172.772827 BW(KiB/s)=150639	IOPS=691.626526 BW(KiB/s)=89053
px-csi-artifact	IOPS=780.681946 BW(KiB/s)=3139	IOPS=682.522766 BW(KiB/s)=2746	IOPS=773.548584 BW(KiB/s)=99549	IOPS=659.015320 BW(KiB/s)=84890

Note that those numbers reflect performance under default configuration, and they should not be considered as the best performance that Portworx can deliver on Azure Kubernetes. Before moving to production, it is important to establish your own test parameters that best represents the container workload, and then iterate through different parameters for the storage class based on the requirement and performance output.

Kubernetes Storage on Azure 1 of 3 – built-in storage and NFS

Sun, 31 Jul 2022 15:22:00 -0400

In the previous post, we understand that to host stateful workload, we need to manage persistent storage to the Kubernetes platform. In this post, I will explore the different storage options. These options are specific to Azure Kubernetes service. However, the principals apply to any Kubernetes platform regardless of cloud vendor.

In another old post, I discussed in-tree and CSI storage classes, and from a developer’s perspective, how to mount volumes statically and dynamically once the storage class is available. Here in this post we are concerned with how to make storage classes available, from a platform specialist perspective.

Storage in Azure Kubernetes

As soon as the Azure Kubernetes cluster is launched, a number of built-in storage classes are available. Unlike third-party storage classes, they do not require kubelet identity to be contributor for node resource group. As discussed we shall use CSI based storage classes.

managed-csi
managed-csi-premium
azurefile-csi
azurefile-csi-premium

The main difference between them is the backing technology. However, form Kubernetes workload’s perspective, the Pods as storage consumers are concerned with the access mode instead of backing technology. Here are supported access modes:

ReadWriteOnce: read-write by a single node
ReadOnlyMany: read only by many nodes
ReadWriteMany: read-write by many nodes
ReadWriteOncePod: new in Kubernetes 1.22 to restrict volume access to a single Pod

The storage classes managed-csi and managed-csi-premium support ReadWriteOnce. The storage classes azurefile-csi and azurefile-csi-premium support ReadWriteMany.

Apart from these built-in options, Azure also suggests a few more options based on other types of Azure resources. For example, AKS can integrate with HPC cache and it for HPC. We can also self-manage a virtual machine configured as NFS server, and use the NFS subdir external provisioner to configure storage class. Despite of the overhead with managing a VM, you have more configurability. My previous client reports that they gain better performance than the built-in options.

Another alternative is Azure NetApp Files. However, being a full enterprise grade solution (similar to FSx ONTAP), Azure NetApp Files costs an arm and a leg. In this comparison, it cost 60 to 100 times as the cost by built-in options.

Another option is Azure Ultra Disk, which needs to be enabled at cluster level. You can provision performance target (DiskIOPSReadWrite and DiskMBpsReadWrite) in the storage class. Ultra Disk is a good middle ground between the pricey NetApp files and the less performant built-in options.

Terraform Template for AKS

To explore the storage options, I use my own terraform template to create an AKS cluster. The template is in the azure directory of the cloudkube repo. The template consists a few configurations with Azure Kubernetes to facilitate storage configuration. First, it configures an SSH key pair to use between the bastion host and the node. Users can SSH to Kubernetes nodes from bastion host as soon as terraform apply is completed. Second, the third party storage options installed after the cluster creation need their Pod to instruct Azure to create Azure disks. This requires that a Kubernetes node agent have the permission to provision resources in the node resource group. This is important to understand because there are a couple of managed identities at play (refer to this post) when building an AKS cluster.

In Azure Kubernetes, it is the managed identity of kubelet, that needs to have contributor permission over the resource group for the nodes (not the one for the AKS cluster itself). A managed identity is expressed by a client ID, an object ID (aka principal ID), and the identity ID. We can find them out with an AZ CLI command as below:

We can also tell that the kubelet managed identity represents node agent, by connecting to a node and looking at the argument (kubernetes.azure.com/kubelet-identity-client-id) of kubelet process:

In the template, I also chose to designate the same BYO identity for both the cluster and for kubelet (node agent), in order to minimize my requirement on permission. If I had left it with a system assigned identity for node agent, I would have to assign that identity as a contributor for the node resource group, either as a user, or via Terraform’s identity. Either way, it is beyond what a Contributor is allowed to do.

Benchmarking with kubestr

I used fio utility for storage benchmarking from virtual machines. However, fio utility is not for container. For fio testing on Kubernetes, I’d have to use a Docker image, and test with target volume attached. Fortunately, the Kasten team shared their initiative in the open source project Kubestr. The kubestr release is available as an executable on common platforms. It connects to the cluster the same way as kubectl and here is a demo. To begin with, download the utility to bastion host, and run it without any argument, which prints the storage classes and volume snapshot classes:

curl -L -o kubestr.tar.gz https://github.com/kastenhq/kubestr/releases/download/v0.4.31/kubestr_0.4.31_Linux_amd64.tar.gz
tar -xvf kubestr.tar.gz && rm kubestr.tar.gz && chmod +x kubestr

./kubestr # if kubectl is configured, this command will print out the details of storage classes and volume snapshot classes

In addition to outputting details, it is also very simple to perform storage benchmarking with kubestr. All we need to do is giving it the storage class name and it will run four tests by default with common global options (ioengine=libaio verify=0 direct=1 gtod_reduce=1). The four tests are:

JobName	block_size	filesize	iodepth	rw
read_iops	4k	2G	64	randread
write_iops	4k	2G	64	randwrite
read_bw	128k	2G	64	randread
write_bw	128k	2G	64	randwrite

During each test, it measures and reports IOPS and bandwidth (throughput). If your I/O profile falls out of the four jobs, you can even customize your test by supplying a fio config file. For example, you need a longer test duration, or you need a larger total size for the test. Before the test, kubestr automatically mount their test volumes using the storage class being tested.

./kubestr fio -s my-storage-class # benchmarking a storage class

For read_iops and write_iops, we mainly look at the IOPS. For read_bw and write_bw, we mainly look at the bandwidth. The iops and bw based on samples are reported as first line of result, followed by min, max and average.

Metrics

With kubestr I ran a performance test amongst the native storage classes with Azure Kubernetes Service, with results as below:

	read_iops	write_iops	read_bw	write_bw
managed-csi	IOPS=314.729797 BW(KiB/s)=1275	IOPS=297.071136 BW(KiB/s)=1204	IOPS=315.311188 BW(KiB/s)=40887	IOPS=261.048645 BW(KiB/s)=33941
managed-csi-premium	IOPS=493.395844 BW(KiB/s)=1990	IOPS=426.731812 BW(KiB/s)=1723	IOPS=455.950348 BW(KiB/s)=58894	IOPS=422.888855 BW(KiB/s)=54662
azurefile-csi	IOPS=259.333282 BW(KiB/s)=1053	IOPS=283.985779 BW(KiB/s)=1152	IOPS=240.447403 BW(KiB/s)=31298	IOPS=230.689804 BW(KiB/s)=30048
azurefile-csi-premium	IOPS=394.044739 BW(KiB/s)=1586	IOPS=371.181793 BW(KiB/s)=1494	IOPS=380.360535 BW(KiB/s)=49018	IOPS=491.313446 BW(KiB/s)=63310

These numbers are based on default test parameters (e.g. 100Gi PVC size). The numbers indicate that block storage generally performs better in default setting. So we should use managed disk instead of azure file unless multiple pods needs to access the same volume.

Validate Persistent Storage

We can use mysql as a quick and dirty test of storage classes. We can deploy the following manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: mysql-pvc
spec:
 storageClassName: managed-csi-premium # the storage class being tested
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi
---
apiVersion: v1
kind: Secret
metadata:
 name: mysql
data:
 password: eHl6  # base64 code of xyz
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
 labels:
 app: mysql
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:5.6
 name: mysql
 env:
 - name: MYSQL_ROOT_PASSWORD
 valueFrom:
 secretKeyRef:
 name: mysql
 key: password
 ports:
 - containerPort: 3306
 name: mysql
 volumeMounts:
 - name: mysql-persistent-storage
 mountPath: /var/lib/mysql
 volumes:
 - name: mysql-persistent-storage
 persistentVolumeClaim:
 claimName: mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
 name: mysql-service
 labels:
 service: mysql
spec:
 selector:
 app: mysql
 ports:
 - name: tcp-mysql
 protocol: TCP
 port: 3306
 targetPort: 3306

Once the Pod has been created, then we use a throw-away Pod to connect to mysql service and build some data:

$ kubectl run mysql-cli --rm -i --tty --image imega/mysql-client -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 9 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]>

From the mysql shell, we create a test database with dummy data populated:

CREATE DATABASE `testdb`; 
USE testdb; 
CREATE TABLE IF NOT EXISTS tasks ( 
 task_id INT AUTO_INCREMENT PRIMARY KEY, 
 title VARCHAR(255) NOT NULL, 
 description TEXT, 
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP 
) ENGINE=INNODB; 
INSERT INTO tasks (title, description) VALUES ('Job A','Morning Standup');
INSERT INTO tasks (title, description) VALUES ('Job B','Latte with two shots of espresso');
INSERT INTO tasks (title, description) VALUES ('Job C','Coding coding and coding');
INSERT INTO tasks (title, description) VALUES ('Job D','git commit');

Each SQL command should return with number of rows affected and then we can exit the MySQL shell and Pod shell. Once we exit out of the Pod shell, the Pod is deleted. We can re-connect to validate the data are still present:

$ kubectl run mysql-tester --rm -i --tty --image imega/mysql-client -- mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz --database testdb --execute='SELECT * FROM tasks;'
If you don't see a command prompt, try pressing enter.
Got error: Access denied for user 'root'@'147.206.3.15' (using password: NO)
+---------+-------+----------------------------------+---------------------+
| task_id | title | description | created_at |
+---------+-------+----------------------------------+---------------------+
| 1 | Job A | Morning Standup | 2022-06-22 20:29:12 |
| 2 | Job B | Latte with two shots of espresso | 2022-06-22 20:29:12 |
| 3 | Job C | Coding coding and coding | 2022-06-22 20:29:12 |
| 4 | Job D | git commit | 2022-06-22 20:29:13 |
+---------+-------+----------------------------------+---------------------+
Session ended, resume using 'kubectl attach mysql-tester -c mysql-tester -i -t' command when the pod is running
pod "mysql-tester" deleted

The output validates the persistent storage of the data. When testing a different storage class, simply start over with a different storage class specified in the PVC part of the manifest.

Summary

This post focuses on storage options for persistent volumes on Azure Kubernetes service. I use my own Terraform template with some custom configuration. I also covered kubestr as benchmarking tool and ran it against the built-in storage classes.

In some situations, the built-in options do not suit your needs. For example, you might want your application to use persistent volume in a consistent way across multiple cloud vendors. We will have to resort to third-party software defined storage layer. In the next post, I will explore a couple of SDS-based options, namely Portworx and Ceph. The custom configurations in the Terraform template will be helpful when we configure Portworx.

Intro to Ceph storage

Thu, 21 Jul 2022 18:55:00 -0400

Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. In this post, I will introduce Ceph and explain how it stands out from traditional enterprise storage technology.

Software defined storage

In the realm of enterprise storage, I discussed PowerScale (Isilon) from Dell EMC, and touched on ONTAP by NetApp as an alternative. These solutions usually include both enterprise grade hardware, and the software layer that manages those expensive hardware. As the competition with cloud storage arises, those vendors start to decouple the software layer from the hardware to sell them separately. As a result, clients have the options to use commodity hardware. On the other hand, the software layer is built to be more accommodative to different hardware options. Eventually, the software layer evolves into Software Defined Storage (SDS) with the purpose of supporting cheaper storage hardware.

This table shows the full solution offering and SDS offering from NetApp and Dell EMC:

	Full solution offering	SDS offering
NetApp	ONTAP	ONTAP Select
EMC	PowerScale	PowerFlex

It is not easy to make a proprietary SDS appliance support commodity hardware. For example, PowerFlex currently supports (and bundles with) DELL’s commodity hardware only. It is most likely an involuntary move. Then, why would these commercial providers even be motivated to support a broader range of hardware by moving to SDS? It is because they face fierce competition from open-source SDS technologies, which were born to support commodity hardware. In this family of technologies, Ceph is a rising star. This family also includes other technologies such as Gluster and HDFS.

Note that the performance of a storage based on SDS still has to do with the underlying hardware. Therefore, comparing Ceph storage with PowerScale is apple to orange, without identical storage hardware. Now that we decoupled SDS and hardware, let’s take a look at two important aspects of SDS: the distributed technology to manage hardware, and the interface it provides to storage clients.

Distributed storage

The reason to use an SDS layer to manage hardware in a distributed architecture is for better scalability and high availability. The soul of this SDS layer is the ability to manage distributed system. However, a distributed storage introduces problems of its own, such as coordinating consistency. Different storage technologies have their own way to tackle these problems. For example, with PowerScale, OneFS has its own Group Management Protocol. Ceph uses CRUSH for data distribution. GlusterFS uses DHT(Distributed Hash Table) Translator.

Storage architects usually do not need to know these technologies in detail. It is not the intention of this post to cover the details of any distributed technology in any of the storage options above. However, storage architects needs to know supported API very well.

Access API

The supported access API of a storage system determines its compatibility with client systems. One good example is NFS for file storage, which defines the protocol for file share without defining the underlying implementation. Most GNU/Linux distributions come with nfsd (NFS server) which exports directories on XFS or ext4 FS as a file share with NFS protocol. In order to transfer data over network, NFS uses RPC, a request-response protocol. With object storage, S3 is a widespread protocol. Below is a list of storage implementations and their supported access API:

Ceph supports librados, S3, Swift and FUSE
GlusterFS supports SMB, NFS, FUSE,
PowerScale supports NFS, SMB/CIFS, HDFS, Object, POSIX

CephFS is distributed file system built on top of Ceph RADOS. It is also a client-server architecture. A Ceph Client, via librados, interacts directly with OSDs to store and retrieve data. In order to interact with OSDs, the client app must invoke librados and connect to a Ceph Monitor. For compatibility, CephFS namespaces can be export over NFS protocol using NFS-Ganesha NFS server.

Ceph Architecture

Ceph is a high-performance, distributed storage platform. It provides object storage, block storage and distributed file system, all backed by a single, reliable storage cluster running on commodity server hardware. A Ceph Storage Cluster consists of Ceph Nodes on a network. A Ceph Storage cluster requires at least one Ceph monitor (ceph-mon), Ceph Manager (ceph-mgr) and Ceph OSDs (ceph-osd). For file system clients, it also requires Ceph Metadata Server (MDS, ceph-mds) to allow user to execute basic commands on POSIX file system (e.g. ls, find)

Under the hood, Ceph stores data as objects within logical storage pools. Using the CRUSH algorithm, Ceph calculates which placement group (PG) should contain the object, and which OSD should store the placement group. The CRUSH algorithm enables the Ceph Storage Cluster to scale, rebalance, and recover dynamically.

Ceph is based on RADOS (reliable autonomic distributed object store), a self-healing system that distributes and replicates data across nodes. It then layers CephFS (a distributed file system), block storage service (RADOS Block Device or RBD), and s3-compatible object storage (RADOS Gateway or RGW) on top of RADOS. For a better description, refer to this page. The chart above shows how Ceph interacts with different kinds of client. For CephFS, the client can interact with the file system via metadata daemon, as illustrated below. This diagram looks similar to the diagram for NFS.

In a RADOS cluster, each server runs some daemons (i.e. OSD, MON or MDS).

When an I/O request occurs, it needs to be mapped to the specific OSD that keeps the storage units. Here is an illustration of the mapping:

As typically observed in distributed system, there is quite some communication overhead to serve a file.

Ceph Cluster Installation

Installing a VM-based Ceph cluster is no trivial effort and there are several methods. The recommended method is Cephadm. Here is a good instruction, where you will notice a lot of steps on each nodes, such as configuring NTP, installing docker, configuring hostname, Linux user and SSH, etc. You may also check this video for how involving it is. Red Hat adopts Ceph project as a product and has an installation guide on its documentation.

Previously, there was a legacy tool ceph-ansible to help administrators with server configuration. It is similar to the way kubespray helps administrators configure Kubernetes cluster. However, the document suggests that ceph-ansible is not integrated with new orchestrator APIs and therefore is not a viable option anymore. Also I did not find a way to install a single-node ceph cluster just for a quick demo. It involves tweaking the CRUSH map configuration.

If we deploy Ceph on Kubernetes for Kubernetes workload, we use Rook, an orchestrator running on Kubernetes, to integrate storage to a cluster.

Cloud Native Storage

Moving to cloud native storage, instead of presenting storage to operating system, we need to configure storage classes for Pods to use persistent volumes dynamically, using storage provisioners. Ceph also shows good presence in cloud native storage ecosystem. In a self-managed Kubernetes cluster, Ceph gives us the capability to configure storage classes to access connected storage. In public cloud, Ceph allows us to configure storage classes connecting to disks attached to the Nodes, an alternative to the cloud vendor provided native storage classes with high availability across availability zones. This layer enables the organization to normalize how their application connects to persistent volumes, a capability particularly helpful in the multi-cloud strategy of the cluster.

Rook is a CNCF project to orchestrate storage system on Kubernetes. It automates storage administrative tasks such as deployment, bootstrapping, configuring, provisioning and monitoring, using declarative templates. It supports Ceph and a number of other storage backends such as Cassandra, NFS, MinIO.

Kick the tires on ArgoCD

Sun, 10 Jul 2022 00:10:00 -0400

Background

In January, I wrote about FluxCD, and adopted it in the Korthweb project. I like the simple design of FluxCD, and I am comfortable with commands without using a web UI. Half a year later, I am re-visiting this choice, with ArgoCD in mind.

After reading numerous recent posts that compare the two (such as this one from the new stack), I started to give more thoughts on ArgoCD, for a couple reasons. First, ArgoCD has more contributing companies and public references. Red Hat adopted Argo CD as the underlying technology for OpenShift GitOps (and Tekton for pipeline). Second, ArgoCD has a mature UI. In corporate collaboration, especially with those not well-versed with command line, a UI is extremely helpful. Bundled with the UI, Argo CD also uses its own RBAC independent of Kubernetes RBAC, making it a great choice for continuous deployment for enterprise applications.

User Interface

You can choose to install just the core components, without UI, SSO and multi-cluster features. To me, it does not make sense because those features are exactly the reason to choose Argo CD. Argo CD also has an eponymous CLI tool, similar to flux for Flux CD. Since Argo CD uses its own RBAC independent of Kubernetes RBAC, Argo CD will have its own credential. The default username is admin and password needs to be retrieved from Secrets. In the tutorial we use this user’s credential to interact with the target cluster. This is different from Flux where the client simply uses kubectl configuration.

In the “Getting Started” guide, we create a namespace “argocd” and install it with manifests. Alternatively, we can use CLI tool to install it. With Argo CD, we can consider hosting it on a different port than the workload’s port, such as 8443. This port serves as “GitOps management port” separated from the application port (e.g. 443) for business workload. Just like any application on Kubernetes, we usually need to configure an Ingress on the management port.

In the argocd namespace, we can see a few services. The application set controller acts as the main controller that reconciles between actual state and desired state. There is also a redis server, for storing data in Argo CD. Another service to note is the dex server, indicating that the SSO capability is provided by the Dex project.

Features

In FluxCD, they use a CRD Kustomization with kustomize.toolkit.fluxcd.io/v1beta2 as version. This name is confusing. Luckily, in ArgoCD, the CRDs are Application (with argoproj.io/v1alpha1 version) and ApplicationSet (with argoproj.io/v1alpha1 version). The controllers monitors CRs created with these CRDs. ArgoCD defines Application as a group of Kubernetes resources as defined by a manifest. Argo CD Application resource deploys resources from a single Git repository to a single destination cluster/namespace. On the other hand, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once. ApplicationSet controllers monitors ApplicationSet resources. We can define a template within the declaration of an ApplicationSet and it allows for parameter substitution.

Another key capability for a GitOps utility is the support of multiple templating tools. FluxCD supports Helm and Kustomize. So do Argo CD. In addition, ArgoCD also supports jsonnet (ksonnet is not supported anymore). Jsonnet is a templating tool with many useful operators. Kubectl can directly consume Jsonnet’s JSON output as if it were YAML. In FluxCD, there are different CRDs for Helm and Kustomize (HelmRelease and Kustomization). In ArgoCD, the Application CRD covers Helm, Kustomize, and Jsonnet by embedding them as attributes in the declaration. This makes it even simpler than Flux CD.

Secret Management

It’s a big no-no to put secret in a code repository. Therefore any GitOps solution must solve the problem with secret logistics. What baffles me is ArgoCD remains un-opinionated on this matter. There appears to be some context for this stance. As a result, ArgoCD only points to a few third-party secret management solutions.

Secret logistics is a tricky problem with GitOps workflow. One of the techniques is sealed secret. In the repo we store the secret as encrypted by public key. The controller keeps the private key which decrypts the secret at the time of deployment. A few engineers don’t find it feasible for operation, as explained in this post and this post.

An alternative is to use external secret operator. Such operators will be able to sync a secret from external secret store such as Hashicorp Vault, AWS secret manager, Azure Key Vault, etc. This is much neater because we only store reference to secret in the repository. However, when the cluster assesses external secret store, it still requires either a secret, or a permission.

Another technique that drawing attentions is using sops for secret and kustomize-sops to integrate with ArgoCD. Mozilla’s sops (SecretOPerationS) project releases a binary utility called sops to encrypt and decrypt YAML manifest. It can encrypt only the values in YAML manifest and not the attributes (keys). We use sops to encrypt YAML files in order to store them in Git repo. We also use sops to decrypt YAML right before we deploy it with kubectl. The key pair for encryption and decryption can be stored in Azure Key Vault, AWS KMS, GCP KMS, and even age and pgp. Here is a simple tutorial.

Example

In a few command we can configure a minimal example. I first use KinD or Minikube to create a cluster following the steps in real-quicK-cluster repo. Then we configure an application using YAML manifest.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
$ kubectl port-forward svc/argocd-server -n argocd 8443:443

From MacOS, I can now browse to https://localhost:8443/ for ArgoCD UI, and login with admin user and the password as printed above. We can then use the UI to create an Application. Alternatively, we can configure an application using CLI utility:

$ brew install argocd
$ argocd login localhost:8443
$ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
$ argocd app sync guestbook

On the GUI we should see the application is sync’ed.

On the left panel, click on the gear icon for settings. Under Clusters, we can see the cluster URL, which we use as dest-server value above. We can add more clusters here.

Pick the right open-source project

My experience with ArgoCD and FluxCD raised an issue to reflect on. For the same problem, there are multiple choices of open-source project. What would be a good methodology to choose the right open-source technology? I would first look at the following factors:

History
Community size
License model
Enterprise support
Governing model

The history and community size give a good idea of the technical maturity level. The License model has legal implications as to where you can use the technology. License model can change even after the project has been launched and even become not open-source any more. One example is Elasticsearch license change in Jan 2021, from the very open APLv2 to under dual license of both Elastic License (by Elastic) and Server Side Public License (SSPL, introduced by MongoDB). Neither license is certified by OSI as compliant with open source definition. For enterprises, apart from licensing, I’d also consider if any organization provides commercial support for the open-source product.

What is often neglected is the governing model. The governing model determines how product decisions are made that will shape the future of the project. For example, CNCF is a vendor neutral foundation for cloud computing and if an organization contributes a project to CNCF, then the member projects have to align with CNCF’s governing practices. Google also introduced an OUC (Open Usage Common) governing model.

Another good way is to just follow Red Hat’s choice. Red Hat built its business model around open-source technologies. They have to pick open-source project to provide enterprise level support so presumably they have a rigour selection process that one can piggyback off. Products supported by Red Hat are composed of open source components often vetted by multiple upstream communities, and changes made to these components are pushed to their respective upstream projects, often before they land in a supported product from Red Hat. Here are more insights in this business model.

Had I gone with the Red Hat rule, I would have picked Argo CD in the first place because that is the base of OpenShift GitOps!

Verdict

ArgoCD and FluxCD are tools for GitOps. However, just using a GitOps tool does not guarantee that all workload are deployed continuously. First, these GitOps controllers cannot replace Operators, which has domain knowledge about how to orchestrate their workloads. Second, resources not directly managed by these operators (such as those installed by Helm) are not being monitored continuously. For continuous deployment over all workloads, I recommend use operator to install third party applications, and use Argo CD or Flux CD to manage the operators.

GitOps makes use of the controller pattern to manage deployment in a continuous matter. Controller works closely with CRDs. ArgoCD uses similar set of CRDs to manage continuous deployment. What sets it apart is the user friendly web UI, the IAM integration and multi-cluster capabilities. These capabilities make ArgoCD well adapted in enterprise IT eco-system.

Chaos Mesh – Cloud Native Chaos Engineering

Thu, 30 Jun 2022 21:49:00 -0400

In this post, we discuss the resilience test problem and why chaos mesh emerged. Then we go over a lab of chaos mesh with a few experiments.

The Problem

I used to support a server application installed in customer’s data centre. The server application receives data from client application, cleanse the data and put them on customer-managed storage. Unfortunately, the customer-managed storage system hang up from time to time. When this happens, the application does not get an I/O failure to begin with so it continues to serve client traffic with many threads pending for I/O completion. All the threads will fail eventually after a long, unresponsive pause.

What is really bad, is that the server application first acknowledges the client application of the recipient of their data, and then asynchronously archive them to the customer’s storage. In this case, the client applications think they securely send out the data upon receiving acknowledgement. However, the server application lost the data afterwards because it does not interact with storage system in a fail-safe fashion. As a result, I often find myself in the business of restoring client data from the cache directory on the servers, and grumbling about how the quality process allowed this issue slipped into production.

To be fair, this is not an easy catch in the quality assurance process. It is caused by a hardware issue, which is technically the customer’s own problem. However, it is the vendor’s responsibility to design fault-tolerant software. Further, if we put on the site reliability engineering goggle, we see vendor’s software and customer’s hardware as a whole, instead of two disparate silos. The problem is however, how we emulate this kind of fault in house?

Chaos Engineering

A software engineer could have unmounted the storage while the software is active during test. This approach is faulty because it has a couple of issues. First, unmounting the storage does not produce exactly the same symptom. Unmounting the storage in most cases gives a clear failure upfront and the application would have caught it. What we need is a delay in storage long enough to cause a time out. Second, we’d have to perform this activity with an operating system command with sufficient privileges. The software being tested usually do not have such privileges. We need to systematically inject failures of particular kind.

Chaos are interruptions along with the cascading effects, that happen in production but are impossible or extremely difficult to emulate in lower environments through automated quality process. To improve robustness, we need the capability to emulate various failure scenarios in a systematic manner. Chaos Engineering aims to improve such capability.

Chaos Engineering emerged as a discipline to improve software’s ability to tolerate failures while still delivery adequate quality of service. This is known as the resiliency of the software. Suppose the storage has 99.9999% SLA, or 31 seconds of downtime every year. Chaos Engineer requires the functional parts of our software do what they can to protect the data, during the 31 seconds of turbulence each year, instead of just falling apart. In regulated industries, this is critical. Resilience testing is difficult in quality assurance process, but we can’t avoid it.

By definition, Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. It is particularly useful in distributed system, where network faults are normal. The one-page chaos engineering website has a good summary of such intent. At Netflix, they built Chaos Monkey, a tool to randomly cause failures on certain computing instances in the cloud. It is one of the first Chaos Engineering tools and it is a big step forward. Chaos Monkey can perform one type of experiment (faulty server) and requires writing custom code.

Chaos Mesh

Moving to Kubernetes platform, where everything operates on network, it is even more important to have a suite of resilience testing tools. Thankfully, we have Chaos Mesh, a cloud-native Chaos Engineering platform that helps platform engineers and software engineers simulate faults in a variety of scenarios. There are other chaos engineering tools (such as Litmus, also a CNCF project) but I will just focus on Chaos Mesh in this post because it has longer development history.

With Chaos Mesh, we can run experiments to simulate certain types of faults, including faults on Pod, Network, File I/O, DNS, Time, JVM, Linux Kernel, HTTP. We can also emulate incidents on cloud platforms (Azure, AWS, GCP) and emulate stress. We can even orchestrate multiple experiments into a workflow. Experiments can start ad hoc or on schedule. In addition to Kubernetes platform, the Chaos Mesh team also provides a tool Chaosd to emulate faults on a physical nodes. In the rest of this post, we will install Chaos Mesh and perform four simple experiments using the bookinfo sample application from Istio’s installer.

Demo Platform

In this lab, we install chaos mesh on Minikube on my Mac environment. I have install the platform in a particular way so the experiments will all be successful. I use this script in my real-quick-cluster project to create minikube cluster. This script configures a 3-node minikube cluster, with containerd as CRI, and enables plugins metallb and metrics server. Metallb provides a load balancer for the cluster. Metrics server allows scaling based on metrics, a test scenario for stressor in Chaos Mesh.

I will also install Istio in this lab, because I use it for Ingress and service routing. I also use a sample application called bookinfo which comes with Istio code repository. It is highly recommended to have an ingress as it is used in production, although you may use your own choice of ingress and service routing tool such as ingress-nginx. To install istio, we run this simple script:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
if istioctl x precheck; then
 echo ready to install istio and label namespace for istio-injection
 istioctl install -f istio-operator.yaml -y --verify
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

Below is the content of istio-operator.yaml

apiVersion: install.istio.io/v1alpha2
kind: IstioOperator
metadata:
 name: istio-operator
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.2
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 overlays:
 - kind: Deployment
 name: istiod
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 ingressGateways:
 - name: istio-ingressgateway
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 5
 minReplicas: 1
 service:
 ports:
 - name: http
 port: 80
 targetPort: 20080
 protocol: TCP
 - name: http-dashboard
 port: 8080
 targetPort: 28080
 protocol: TCP
 overlays:
 - kind: Deployment
 name: istio-ingressgateway
 patches:
 - path: spec.template.metadata.labels.version
 value: 1.13.2
 - path: spec.template.spec.containers[name:istio-proxy].lifecycle
 value:
 preStop:
 exec:
 command: ["sh", "-c", "sleep 5"]

For more about istio installation, refer to my previous post. Next, we’ll create the namespaces, install bookinfo and install chaos mesh using Helm:

kubectl create ns chaos-testing
kubectl create ns bookinfo

kubectl -n bookinfo apply -f istio/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl -n bookinfo apply -f istio/samples/bookinfo/networking/bookinfo-gateway.yaml

helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-testing --version 2.2.0 --set dashboard.service.type=ClusterIP --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock

As soon as we install bookinfo, we should be able to access it at http://192.168.64.16/productpage, suppose the IP address is 192.168.64.16. We will simply use HTTP because certificate is not the point of this demo. Note that when we install Chaos Mesh we have to specify the runtime (containerd). Otherwise, we might run into issues when running experiments (e.g. “unable to flush ip sets” for network faults). Before fault injections, we’ll start a curl command and observe the output:

while true; do
 curl -I http://192.168.64.16/productpage --connect-timeout 4 --max-time 5
 sleep 5
done

We can watch the output of the command above as we inject each of the following faults later on.

Chaos Mesh UI

Since we installed chaos mesh as cluster IP service, we need to expose it using Istio Gateway and Virtual Service with the following manifest. If you have your own choice of Ingress, use a corresponding ingress resource.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: chaos-dashboard-gateway
 namespace: chaos-testing
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 8080
 name: http
 protocol: HTTP
 hosts:
 - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: chaos-dashboard-vs
 namespace: chaos-testing
spec:
 hosts:
 - "*"
 gateways:
 - chaos-dashboard-gateway
 http:
 - route:
 - destination:
 port:
 number: 2333
 host: chaos-dashboard

Once we configured Virtual Service and Gateways, and waited all Pods in chaos-testing namespace to come up, then we should be able to access the web portal of chaos mesh at 192.168.64.16:8080. Follow the on-screen instruction to fetch the token for access at Cluster level.

Chaos Mesh Web Portal

Now, we can create experiments, either using web portal, or using their CRDs. We will use CRDs in this demo.

Network Latency Injection

In this section we artificially introduce a network latency of 3 seconds for a period of 45 seconds. It takes effect as soon as we create the NetworkChaos CRD as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: network-delay
spec:
 action: delay # the specific chaos action to inject
 mode: one # the mode to run chaos action; supported modes are one/all/fixed/fixed-percent/random-max-percent
 selector: # pods where to inject chaos actions
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 delay:
 latency: '3s'
 duration: '45s'

Watch for the curl output, which normally returns 200 code every 5 seconds, now fails for about 45 second window.

After the window of latency injection, the curl test returns normal. During the experiment, we can check the status of the fault injection by describing the NetworkChaos object. Alternatively, we can look at Experiments and Events in Chaos Mesh web portal. When completed, we can delete the NetworkChaos object. Alternatively, we can archive the experiment from web portal.

HTTP Failure Injection

The NetworkChaos API injects faults at Pod’s network layer. HTTPChaos API introduces HTTP error. We can configure an HTTPChaos object following the instruction:

apiVersion: chaos-mesh.org/v1alpha1
kind: HTTPChaos
metadata:
 name: test-http-chaos
spec:
 mode: all
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 target: Request
 port: 9080
 #method: GET
 path: '*'
 abort: true
 duration: 2m

This object intercepts HTTP calls to the Pods as labelled and aborts the request. If we monitor the curl output at the same time, we can see that it starts to receive 503 error as soon as we injected the fault.

Once we remove the HTTPChaos object, the return code becomes 200 again.

Pod Failure Injection

We can emulate a fault that kills a Pod with the following manifest:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
 name: pod-failure-example
spec:
 action: pod-kill
 mode: one
 duration: '30s'
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage

In the mean time, watch for the Pods in the bookinfo namespace, we’ll notice the old Pod being terminated.

Because the Pod is managed by a replica set, the replica set brings up a new Pod to ensure the desired number of replicas.

Stressor

In this experiment, we add an HPA (Horizontal Pod Autoscaler) to the productpage deployment, emulate a memory stress to the Pod and watch it scale up. Before the testing, we need to first configure a resource request for memory for the deployment. To configure memory request, edit the deployment productpage-v1 by adding memory request:

Then we add an HPA as below:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: productpage-deploy
 namespace: bookinfo
spec:
 maxReplicas: 4
 minReplicas: 1
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: productpage-v1
 metrics:
 - type: Resource
 resource:
 name: memory
 target:
 type: Utilization
 averageUtilization: 70

Now if we describe the HPA, we shall see that HPA is active:

Then we create a StressChaos object as below:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
 name: memory-stress-example
spec:
 mode: one
 selector:
 namespaces:
 - bookinfo
 labelSelectors:
 app: productpage
 stressors:
 memory:
 workers: 2
 size: 128MiB

It will take some time for the HPA to detect the memory stress and scale up. We can tweak this with scaling policy. After scaling activity, the HPA status will report with the new desired number of replica:

We can also confirm that by checking the deployment size. If we want the size to reduce after we remove the stressor, we need to configure HPA accordingly with scale down triggers.

Chaos Mesh uses stress-ng utility to emulate memory or cpu stress on a Pod.

Summary

Chaos mesh is a cloud native resilience testing tool. It supports many fault experiments not only on Kubernetes, but also on virtual machines. I wish I came across this tool sooner in my professional life. Had I had chaosd in the past, I could have incorporate it in our quality assurance process on Linux platform. Without Chaos Mesh, we’d have to use different utilities for different types of faults (e.g. stress or stress-ng for cpu and memory stress). In the domain of platform engineering on Kubernetes, it helps not only software developers, but also platform engineers with correct configuration for workloads on Kubernetes. In this post we explored four types of faults. For more about all fault types and their attributes, go to their documentation for more details.

Etcd – the key-value store for Kubernetes

Tue, 14 Jun 2022 00:10:00 -0400

Etcd in Kubernetes

In Kubernetes architecture, etcd is the data store. It stores the desired state of Kubernetes object. API server is the only client that connects to etcd (via gRPC protocol). Cluster builder specifies the endpoint of etcd as a parameter to the kube-api-server process. Other Kubernetes components, whether in the control plane or from the nodes, connect to API server. API server translates their request into etcd query, and then translates etcd query result into what its clients ask for. For this reason, communication with etcd accounts for a lot of network traffic in a Kubernetes cluster.

The etcd store is a CNCF project for “a distributed, reliable key-value store for critical data in a distributed system”, developed by CoreOS team. So it is essentially a distributed key-value store for any distributed application. If an application runs on Kubernetes, it can leverage etcd store, by keeping their configurations in ConfigMap and Secret objects. One key feature is to watch for specific keys or directories for changes, and react to the changes. Voila! This is the underlying mechanism for controller!

A Kubernetes cluster may have stacked etcd deployment or connect to an external etcd store.

stacked etcd architecture

external etcd architecture

In managed Kubernetes services such as EKS in AWS and AKS in Azure, users usually do not directly access etcd store. However, it is still a very important component to understand. Its use case includes:

Configuration sharing
Service discovery
Consistency
Watching mechanism
Expiry and extension of key

The consistency use case is based on Raft protocol for distributed consensus.

Raft protocol

I am not an expert in distributed consensus protocols and nor do I intent to cover it in depth. At a high level, I have heard of three of them so far:

Etcd uses Raft protocol
Zookeeper uses ZAB protocol
Cassandra uses paxos protocol

Here is a good intro to the three protocols. Instead of getting into the fine details, I would like to discuss why we need such a consensus protocol (or consensus mechanism) in distributed systems, which are also decentralized systems.

Centralized, Decentralized, Distributed systems

The reason a distributed system needs consensus protocol, is that a distributed system lacks a single source of truth as centralized systems do. Different parts of the distributed system may receive different signals but they must come to agreement of a single plan to act. Lamport studies this with an analogy of Byzantine Generals problem, and first proposed Paxos protocol. Paxos has been an important foundation to modern distributed systems. In Paxos, consensus is achieved in two phases, which creates the problem of livelocks. Raft is an alternative to Paxos, and is widely adopted today. Here is a link to an animated illustration for Raft protocol. The Raft protocol is also used in Redis. It has three roles: Leader, Candidate, and follower. ZAB protocol is similar to Raft, where it needs to select a leader.

Etcd Lab

In troubleshooting, if we suspect that the response from API server is inconsistent with etcd store, we want to directly connect to it.

Managed Kubernetes services do not expose their etcd store. We can use KinD or Minikube. There are two types of jump box to access etcd store: using etcd Pod, or SSH to a Node. To connect to etcd, we also need the X509 key, certificate and CA’s certificate, in addition to the endpoint, usually an IP with port 2389. When I connect to Pod shell, I find the command shell not easy to use. They might miss basic command such as ls, or do not support auto completion.

Take KinD for example, we first create a secret, then we can connect to the node with docker CLI command:

kubectl create ns myns
kubectl -n myns create secret generic mysecret --from-literal key1=value1
kubectl -n myns get secret mysecret -o jsonpath='{.data.key1}' | base64 -d
docker exec -it control /bin/bash

From the node,

apt update && apt install etcd-client
etcdctl version
nc -vz localhost 2379
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

export ETCDCTL_API=3
export ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379'
etcdctl member list write out=table

Now we can see the secret object directly with etcd store:

etcdctl get /registry/secrets/myns/mysecret

With get query, when using –prefix, we can use –keys-only switch to list keys without values:

etcdctl get --prefix /registry/api --keys-only
etcdctl get --prefix /registry/namespace -wjson

We can write key-value with put command:

etcdctl put myloc 0
etcdctl get myloc -wjson

In Kubernetes, all the key names start with / which makes the key looks like a POSIX path. Every Kubernetes object is stored in etcd with a unique key following a self-explanatory naming pattern. To display the path, we can also use debug log that records the call to API server:

kubectl get ns myns -v9

Look for curl command such as:

I0523 22:51:43.517728 32347 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.23.6 (darwin/amd64) kubernetes/ad33385" 'https://127.0.0.1:64081/api/v1/namespaces/myns'

From there we can see the etcd query as the URI is namespaces/myns, which we use in etcdctl query path:

etcdctl get /registry/namespaces/myns

Every type of Kubernetes object has a storage.go file in their implementation that defines how api server should write object. Here is an example for Pod object.

Etcd also supports watch command to watch for changes. For example:

etcdctl watch --prefix /registry/namespace # watch output k create ns newns

Now we create a namespace with kubectl:

kubectl create ns myns

The output from etcdctl will reflect the change. The communication between etcdctl and etcd is gRPC protocol. The output is based on stream, as we can see from the watch result.

Etcd Maintenance

Like any distributed store, etcd needs maintenance and operation work. For example, we can check endpoint status with endpoint command:

etcdctl endpoint status

We can also backup and restore etcd store with etcdctl command:

etcdctl snapshot save /tmp/backup.db

This was an question in CKA exam. In real life, when the workload scales up, the etcd store may come across many pitfalls, such as degraded performance, unresponsiveness, some etcd member going down, network partition on etcd store causing split brain. It is important to ensure efficient communication between API server and etcd store. The etcdctl provides defrag and compact commands for common maintenance activities.

Hosting database on Kubernetes

Sun, 29 May 2022 11:01:00 -0400

Background

“We want to host Postgres database on Kubernetes. Can you help us?”. The client appears assertive and reluctant to resort to managed services. So I did some homework and went through this tutorial. My thought: it’s doable, but don’t do it unless operating database as a service is your main business.

I believed that was the client’s best interest, until I came across this the blog post A Case for Databases on Kubernetes from a Former Skeptic. The author explained his journey from being a skeptic, to grudging acceptance, and eventually to an evangelist on running database on Kubernetes. The same voice came from the author of the upcoming book Managing Cloud Native Data on Kubernetes, who also advocates hosting database on Kubernetes. While the points in the chapters are valid, the book also includes a good amount of technical details which might lead reader to believe the opposite view.

Just a few years back, Kubernetes was not mature to host database. This is changing in 2022. Nowadays, for clients with their own Kubernetes platform, technological maturity is no longer the main reason that keeps them from hosting database on Kubernetes, it is the operational cost. The operational cost has to do with whether the client has in-house expertise in database and Kubernetes. If they do, the hard path makes economical sense.

In this post, we discuss what we need to be aware of in order to host database on Kubernetes.

Benefit with Kubernetes

The first few versions of Kubernetes only supported stateless workload (reference documentary). That is what Kubernetes was born to solve. Built-in objects such as replicaSet, deployment, horizontalPodAutoscaler are abstractions of operations particular to stateless workload. Pods for stateless workload are ephemeral: they crash and get replaced at any time. Because they don’t carry persistent data themselves, they are expendable.

Kubernetes’ orchestration capability are driven by controllers. As discussed, the controller pattern is adopted in all controller implementations. They are the engines of the platform that works tirelessly in a control loop to ensure desired states matches their declared states. This is a key feature of Kubernetes as container platform. Let’s examine a web service that requires 5 instances behind load balancer. With traditional hosting model on Linux servers, you’d have it installed on all five VMs. If the process on one of the VMs dies, the VM has to be removed from the load balancer’s target pool.

One may wrap the process with process monitor and control utility such as supervisord, and re-install the application using automation utility (e.g. Ansible). However, each server is unaware of the status of its peer. Without a central “Control Plane”, there is no coordination between the activities of each VMs. Kubernetes controller solved all these operational problems.

Control Loop

Kubernetes comes with a set of build-in controllers that run inside the kube-controller-manager. Here is a good page about how controllers work. Controller is what is missing in many automation tools other that Kubernetes. Even though Red Hat now brands Ansible as Automation Controller, it does not involve a control loop or controller pattern. If there’s one thing that sets Kubernetes apart from other hosting platforms and automation platforms, it is the implementation of controller pattern.

Stateful workload

Does the controller pattern also benefit stateful workload? Yes. How to orchestrate Pods for stateful workload is usually more tricky. CRD can define a custom object type for controller to consume. In this case, an operator is an implementation of the controller pattern. This pattern is also known as the operator pattern. In a replicaSet, Pod names have extensions of randomly generated numbers. A statefulSet names its the Pods by sequential numbers.

For Postgres database, Bitnami built a good Helm Chart to install the database automatically. However, it does not have a control loop. If someone changes the workload after initial installation, the change is not monitored or controlled by any controller. This is a disadvantage of Helm chart as compared with operators. For PostgreSQL, there are a number of operators, the most notable being PGO (Postgres Operator) from Crunchy Data.

To install an instance of PostgreSQL database, we need to install the operator, and then declare a Custom Resource using the PostgresCluster CRD. The operator will set up the cluster according to the declaration made in the PostgresCluster CR. I used the quick start guide to bring Postgres up real quick on an Azure Kubernetes cluster. The operator (v5) supports common cloud Kubernetes platforms (GKE, EKS, AKS), VMware Tanzu, Openshift, Rancher, Kubernetes. It does not explicitly indicate whether PGO supports Minikube or kind.

So far, I’ve discussed the pros of running PostgreSQL on Kubernetes using Postgres Operator. We can describe the database deployment in a CR and the controller (operator) will monitor the resource incessantly to ensure the actual state matches the state defined in the CR. Not only is it doable to host database in Kubernetes, it makes our lives even easier.

Persistent storage

Database is not only a stateful workload, it also has special requirement on storage. It needs to persist data, support ACID transaction, and make optimal use of disks. When we operate everything on premise, we use fibre cable with a SAN as the storage media for database file. The operating system allows the database process to interact with blocks on the storage volume via device mapper.

In Kubernetes, we need to give Pods persistent volumes. There are a few APIs: Storage Class, Volume Storage Class, Persistent Volume and Persistent Volume Claims. Storage Class represents how Pod can connect to a storage. Pods will need PVCs in order to read and write on PVs. However, since Pods are ephemeral – a Pod may crash any time, even if it is in the middle of writing to a PV, during an ACID transaction. The scheduler may reschedule the crashed Pod to a different node. Then it will need to pick up the PV from where it left off, on the new Node.

Take Azure Kubernetes Service for example, a few storage classes are available by default, backed by Azure managed disk (managed-csi) or Azure file storage (azurefile-csi):

	StorageClass	Azure storage service
in-tree	default	Managed Disk using Azure StandardSSD
	managed-premium	Managed Disk using Azure Premium Storage
	azurefile	Azure File Share using Azure Standard Storage
	azurefile-premium	Azure File Share using Azure Premium Storage
csi	managed-csi	Managed Disk using Azure StandardSSD
	managed-csi-premium	Managed Disk using Azure Premium Storage
	azurefile-csi	Azure File Share using Azure Standard Storage
	azurefile-csi-premium	Azure File Share using Azure Premium Storage

If we use storage class based on Azure disks to create a PV, only one Pod can use the PV. If we use storage class based on Azure files to create a PV, then the storage is mounted as NFS (Linux) or SMB (Windows) share. File storage is not a valid use case for database workload and it can significantly degrade database performance. When I tried to use a file-storage based CSI with PGO, the Pod reports an error and will not start properly. We should use Azure disk based CSI storage classes. That leaves us with two options: managed-csi and managed-csi-premium.

High Availability

Even with these to options left, we still have to investigate how database Pods interact with persistent volume for high availability, in order to determine whether any of the options are suitable. The two storage classes differ by disk performance but both have its own limitation with multi-AZ support on Azure managed disks. When the cluster operates across zones, the Kubernetes scheduler may reschedule a Pod crashed in one zone to a Node in a different availability zone (a different data centre). Even though the managed disks, when attached to VMs, can be configured as zone-redundant, when they are used as Kubernetes volume, they are NOT zone-redundant. So the node in a different zone will not be able to attach PV to the new Pod.

There are SDS (software-defined storage) solution such as Portworx that solves the limitation of Azure disk for cross-region storage volume. The SDS layer brings managed disks from multiple availability zones into a pool. This storage pool acts as a highly available, cross-zone storage tier presented to AKS as persistent volumes. We can install Portworx as the SDS layer using Portworx operator. To do so, we first have to configure grant the cluster the permission to provision resources in Azure, because the Portworx operator will use node’s identity (kubelet identity) to provision Azure resources on behalf of the nodes. Portworx will provision Azure disks and acts as the intermediary layer.

Apart from cross-zone high availability enabled by PX-Store, Portworx can also help with cross-region replication of persistent volumes. The PX-DR component can perform asynchronous replication across Azure regions. The destination region needs to have its own cluster because a single AKS cluster cannot span across regions.

Storage Class

Once we have portworx installed, the following storage classes are available by default:

px-db
px-db-cloud-snapshot
px-db-cloud-snapshot-encrypted
px-db-encrypted
px-db-local-snapshot
px-db-local-snapshot-encrypted
px-replicated
px-replicated-encrypted

The steps for installing porworx on AKS are documented here. This blog post has more details in the installation process on a different platform. We can also built CSI based storage classes with different IO priority and replication factors.

In summary, Kubernetes operator pattern makes it easier to manage stateful workload. However, database performance depends largely on storage. To host database on Kubernetes, one will have to also manage the storage volumes on their own. There has not been a study on the impact to performance by moving database to Kubernetes platform. However, I only expect a degraded performance due to the layers introduced.

Example

In this section we configure a (minimally viable) PostgreSQL cluster using Crunchy Data pgo to demonstrate the idea. The steps are based on its tutorial but it works on a local KinD cluster. As discussed in a previous post, I use KinD for testing workload requiring persistent storage because Minikube has this open issue with permissions on PVs with multiple nodes.

To prepare the cluster, we can use kind-config.yaml file from my real-quicK-cluster repo:

kind create cluster --config=kind-config.yaml
# to delete cluster after testing: kind delete cluster --name kind

We use Helm to install the operator. Since the Helm chart is not hosted in a public repo, we’d have to download the directory of the Helm Chart.

git clone https://github.com/CrunchyData/postgres-operator-examples
cd postgres-operator-examples
helm install -n postgres-operator --create-namespace crunchy-pgo helm/install
kubectl -n postgres-operator get po --watch
kubectl explain postgresclusters

Now we can create a YAML file for the Custom Resource and let’s call it test-cluster.yaml with the following content:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
 name: hippo
 namespace: postgres-operator
spec:
 backups:
 pgbackrest:
 image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.38-1
 repos:
 - name: repo1
 volume:
 volumeClaimSpec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi
 storageClassName: "standard"
 image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.3-0
 instances:
 - dataVolumeClaimSpec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi
 storageClassName: "standard"
 name: instance1
 replicas: 3
 minAvailable: 2
 postgresVersion: 14

In the manifest, we specified a cluster, using storageclass named “standard”, with 3 replicas and requiring 2 available. We assume a storage class named “standard” already exists and optimized for database workload. In the manifest, we also configured a backup job. We can apply the manifest and watch for the Pods to come up in a few minutes.

kubectl apply -f test-cluster.yaml
kubectl -n postgres-operator get po --watch
kubectl -n postgres-operator describe postgresclusters hippo

The Pods in the postgres-operator namespace should report something like this:

NAME READY STATUS RESTARTS AGE
hippo-backup-mwpm-ps8wk 0/1 Completed 0 21s
hippo-instance1-6mls-0 4/4 Running 0 3m35s
hippo-instance1-hjp6-0 4/4 Running 0 3m35s
hippo-instance1-k4qf-0 4/4 Running 0 3m35s
hippo-repo-host-0 2/2 Running 0 3m35s
pgo-548d5f48bc-9w4z4 1/1 Running 0 8m41s
pgo-upgrade-566b9cc98f-d7gkr 1/1 Running 0 8m41s

Three Pods for PostgreSQL are all up. The first backup run has completed already. We can connect to the cluster using psql following the quick start guide. We can also configure an application. A good example application that uses PostgreSQL database is KeyCloak. We briefly mentioned it in OIDC discussion. Currently the keycloak example on Crunchy pgo’s quick start guide is outdated. Instead, use the following content as keycloak.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: keycloak
 namespace: postgres-operator
 labels:
 app.kubernetes.io/name: keycloak
spec:
 selector:
 matchLabels:
 app.kubernetes.io/name: keycloak
 template:
 metadata:
 labels:
 app.kubernetes.io/name: keycloak
 spec:
 containers:
 - image: quay.io/keycloak/keycloak:latest
 name: keycloak
 args: ["start-dev"]
 env:
 - name: DB_VENDOR
 value: "postgres"
 - name: DB_ADDR
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: host } }
 - name: DB_PORT
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: port } }
 - name: DB_DATABASE
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: dbname } }
 - name: DB_USER
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: user } }
 - name: DB_PASSWORD
 valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: password } }
 - name: KEYCLOAK_USER
 value: "admin"
 - name: KEYCLOAK_PASSWORD
 value: "admin"
 - name: PROXY_ADDRESS_FORWARDING
 value: "true"
 ports:
 - name: http
 containerPort: 8080
 - name: https
 containerPort: 8443
 readinessProbe:
 httpGet:
 path: /realms/master # https://stackoverflow.com/questions/70577004/keycloak-could-not-find-resource-for-full-path
 port: 8080
 initialDelaySeconds: 30
 restartPolicy: Always

Once we apply keycloak.yaml, in a minute we should see and be able to port-forward web traffic:

$ kubectl apply -f keycloak.yaml
$ kubectl -n postgres-operator get po -l app.kubernetes.io/name=keycloak
NAME READY STATUS RESTARTS AGE
keycloak-7995d78d7c-zjp4d 1/1 Running 0 4m29s
$ kubectl port-forward deploy/keycloak -n postgres-operator 8080:8080

After using the port-forward command, we can browse to web portal on my MacBook by http://localhost:8080 and configure an initial password, as shown here:

In real life system we would need a proper Ingress. After testing, delete the cluster with kind command and specify the cluster name (kind).

Operation Cost

Operation cost is an important consideration. Troubleshooting on Kubernetes platform is in general more complicated than just on a Unix system. Hosting database on Kubernetes requires skills not only on the Kubernetes platform, but also on database. There used to be database administrator positions where someone has to maintain the upgrade, the storage, the replication, the multi-tenancy and the performance optimization of database. With a database hosted on Kubernetes, the database administrator will have to perform all these activities on a containerized platform. This is not an easy undertaking, and in many occasions warrants a full-time position on its own. Therefore, don’t host your database on Kubernetes, unless that is your main business. It is not the technology that shots down this option. It is the operation cost, such as complexity of configuration, and staff skillset, that makes this option not worth it.

FSx ONTAP – Enterprise storage on AWS

Sat, 14 May 2022 22:28:00 -0400

Even though object storage has gained a lot of popularity, file storage is still prevalent. AWS has Elastic File System but the performance is insufficient for enterprise workload. The FSx product line has enterprise storage options and on Sept 2, 2021, AWS launched FSx ONTAP.

This post is my impression about FSx ONTAP. As previously discussed, FSx ONTAP is a managed NetApp storage service by AWS. Essentially, AWS installs NetApp arrays in their data centres, so that users can provision ONTAP volumes from AWS console, or using AWS CLI.

FSx ONTAP

ONTAP (or Data OONTAP) has been a very successful operating system to manage storage arrays. It was so successful that NetApp uses ONTAP as brandname for their storage arrays. This is similar to Isilon, a name of BSD based operating system to manage storage and later becomes brandname of EMC’s storage products. ONTAP has been in competition with other enterprise storage players such as EMC Isilon, HP 3PAR, etc and AWS landed on ONTAP as their partner for enterprise storage. It appears that NetApp still owns their ONTAP storage technology. AWS operates the data centre and provides capability via the CLI layer and console.

Sometimes, people confuses FSx ONTAP with NetApps offering Cloud volumes ONTAP. The two are fundamentally different. Cloud volumes ONTAP works with Cloud Manager (available as self-hosted or SaaS) as the management UI. It manages volumes provisioned from cloud vendor such as AWS, Azure, etc. Clients often configure these “Cloud volumes” as extension to an existing on-premise ONTAP storage deployment. When their on-prem ONTAP volumes fall short of space, the Cloud Manage is aware of a remotely available, cloud backed volume to move cold data off to.

Client Tool

FSx ONTAP is essentially an ONTAP storage cluster sitting in AWS data centre. Users have the options of using AWS CLI or ONTAP CLI to manage the cluster. Users with storage administrator background are likely to prefer the latter. In my professional service experience, I have taken some iterations to come to best practice to use the right tool to interact with ONTAP resources. In a nut shell, it depends on the level of resource that we are interacting with. I categorize those resources into two classes:

Category	Example	Identification
AWS-level resource	FSx ONTAP file system, Storage Virtual Machine	These resources are from ONTAP but they are identified as AWS resources (with ARN). They are also exposed to AWS SDK and can be managed by AWS CLI.
ONTAP-native resource	Volume, Snapshot policy, Schedule, Snapmirror relationship, Vserver	These resources come from ONTAP and can only be managed using ONTAP CLI. The AWS CLI cannot manage _these resources simply because they are not exposed to AWS SDK. Some type of resource such as Volume, may be managed by AWS CLI with very limited options. So we still prefer ONTAP CLI to manage resources.

The AWS CLI (v 2.2.37) has very limited options when creating volumes. For example, the create-volume documentation states that the OntapVolumeType section of output can display types of RW, DP (data protection), or LS. However, it doesn’t allow users to create a volume other than the default RW type. When we configure SnapMirror destination, we need a volume of DP type. We had to use ONTAP CLI to achieve that.

Our practice works out to be: use AWS CLI to create a file system and storage virtual machine. Then we use ONTAP CLI to create everything else. Even though AWS CLI intend to support volume creation, we prefer ONTAP CLI for full functionality support, and alignment with the ONTAP literature.

Administration Tasks

You should configure most of the administrative tasks with ONTAP CLI. We use the documentation by ONTAP as reference. For example, when a volume runs out of Inode, the AWS CLI reports that there is no space left. We need to increase the inode limit and this is, again, not something that AWS CLI can manage. We’d have to use the volume modify command from ONTAP CLI.

NFS version

The AWS document states that FSx ONTAP supports NFSv3.0, v4.0 and v4.1. However, FSx ONTAP is currently backed by NetApp ONTAP 9.10.0, which partially supports NFSv4.2, with basic protocol and Labled NFS feature. NetApp’s ONTAP Best Practices and Implementation Guide suggests a method for clients to mount as NFSv4.2. In the POC we mount as NFS v4.2 in all of our testings.

Clarity of terminology

The ONTAP storage system has been around for a while and many of its concepts are well known in the storage community. For example, A “Snapshot copy” is a read-only, point-in-time image of a volume. (ref: ONTAP 9 documentation -> ONTAP concepts -> Replication -> Snapshot copies) This concept becomes “Snapshot” in AWS literature. It took us some research to come to realize that “Snapshot” in AWS document, essentially maps to “Snapshot copy” in ONTAP literature.

This creates confusion, because we use ONTAP documentation for operation guidance because we can’t get enough help from AWS documentation. The terminology in AWS documentation should align with ONTAP.

The other example is the difference between “backup” and “snapshot” in AWS documentation. It is my understanding that they both use the same underlying Snapshot technology on the ONTAP side. I’m not exactly sure what their difference is.

ONTAP can create a Snapshot copy nearly instantaneously. However, when taking a snapshot using web console in AWS, it takes up to 10 minutes to update the status. This is confusing because it creates a perception that it takes 10 minutes to complete snapshot.

Cross Region Replication

There is a document page on AWS about using SnapMirror at a very high level. It points to two documents: using NetApp Cloud Manager and ONTAP CLI.

The former is not a viable option as we started natively on FSx ONTAP and do not have NetApp Cloud Manager. As to the latter, we managed to configure cross-region replication with ONTAP CLI following the document and identified some gaps in the documentation. Specifically, it would be helpful if AWS documentation calls out that:

inter-cluster network connectivity is a prerequisite (e.g. via VPC peering, transit gateway)
Port 10000, 11104-11105 must be added to security group for inter cluster communication.
The ONTAP CLI command to validate connectivity between clusters (using the ping command from ONTAP CLI).

With AWS CLI alone it is not possible to configure cross-region replication.

Final words

As someone who lived with enterprise storage for more than a decade, I’m glad to see that cloud vendors brings enterprise storage into their data centre, acknowledging that consumer grade file storage are just insufficient for heavy storage use cases such as medical imaging. FSx ONTAP seems to be in early maturity level. However, since AWS exposes ONTAP CLI access to users, ONTAP professionals are able to leverage its full potential.

Knative Eventing Introduction

Fri, 29 Apr 2022 10:16:00 -0400

In the previous post, I mentioned that Knative Serving and Knative Eventing should be seen as two different projects. The former is supposed to be widely used as a serving layer for microservices, whereas the latter has a narrower customer base. There are a dozen companies who need to build Platform as a Service, and will benefit from event-driven architecture. They are the niche customer for Knative Eventing. In this post, we go through two demos for Eventing.

Event Driven Architecture

Before jumping into Knative eventing, let’s first understand the value of event-driven architecture. As covered in Knative presentation in 2020, a growing micro-service architecture involves lots of service-to-service communications that looks like a spider web in a diagram. Two services communicating with each other are tightly coupled: changing one requires a change of the other. Event-driven architecture evolves to decouple those interdependent services. Knative eventing tries to standardize the event data with Cloud Event and also proposes a framework to drive this architecture.

The documentation uses different words to express the same concepts. So let’s take a note first: source = producer which emits events, and sink = subscriber = consumer which is the destination of events. Knative Eventing supports two models of in even-driven architecture.

Brokers and Triggers provides an “event mesh”. Event producers delivers events to a Broker. Triggers then distribute them uniformly by consumers.

Broker is a “hub” for events. It is a central location to receive and send events for delivery. We can choose from the following broker types:

MT (multi-tenant) channel-based broker (which a channel implementation, such as Kaka channel or an in-memory channel)
GCP broker
Apache Kafka broker
RabbitMQ broker

Triggers represents a desire to subscribe to events from a specific broker. It can act as a filter

channel-subscriber model

Channels and Subscriptions provide a “event pipe” model which transforms and routes events between Channels using Subscriptions. This model makes more sense when events out of one system needs to be transformed and then routed to another process.

A channel can be either a generic channel object, or a custom channel implementation (such as Kafka channel or in-memory channel)

The subscription consists of a Subscription object, which specifies the Channel and the Sink (aka the Subscriber) to deliver events to. You can also specify some Sink-specific options, such as how to handle failures.

The key difference is Broker-Trigger model allows you to filter events, where as Channel-Subscription model allows you to transform events.

Note that a Knative Service can act as both a Source and a Sink for events, and for good reason. You may want to consume events from the Broker and send modified events back to the Broker, as you would in any pipeline use-case. Between these components, Knative Eventing uses CloudEvents to send information back and forth.

Knative Eventing Lab

We can create a lab cluster using Kind as per the instruction in real-quicK-cluster. In docker-desktop settings, I configured 6 CPU and 8GB memory as the lab involves a number of nodes. The creation takes a couple minutes. This Knative eventing lab has four paths one can take, based on different model and different implementations:

Lab A demonstrates the configuration of broker-trigger model using Kafka based broker.
Lab B demonstrates the configuration of channel-subscription model, using in-memory channel.

The two labs share some common steps. We first install Knative Eventing using the provided manifests:

$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-crds.yaml
$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-core.yaml
$ kubectl get pods -n knative-eventing

The Pods should come to READY fairly quickly. In the mean time, we can configure the consumer as well using the manifests below:

apiVersion: v1
kind: Namespace
metadata:
 name: event-example
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: hello-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: hello-display
 namespace: event-example
spec:
 selector:
 app: hello-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 replicas: 1
 selector:
 matchLabels: &labels
 app: goodbye-display
 template:
 metadata:
 labels: *labels
 spec:
 containers:
 - name: event-display
 image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
---
kind: Service
apiVersion: v1
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 selector:
 app: goodbye-display
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080

The consumers (aka sinks) will be able to process events at the validation step at the end.

Lab A – Install Kafka and configure Kafka-based Broker

This lab is for broker-trigger model. We first configure a Kafka Broker, which requires a Kafka cluster. The document uses Strimzi operator to quickly install Kafka cluster. Instead, here we use Helm to install the Kafka cluster:

helm install my-cluster-kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3

At the end of installation, the Helm notes indicate that the bootstrap server is my-cluster-kafka.kafka.svc.cluster.local:9092. All Kafka pods should come ready in about 5 minutes. After cluster install, we need to install Kafka controller and the Broker-layer data plane which allow you to map a Kafka instance to Broker CRD later:

kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-controller.yaml
kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-broker.yaml
kubectl get pods -n knative-eventing
kubectl -n knative-eventing edit cm kafka-broker-config

As the command above suggests, once data plane is installed, we edit the default broker config by updating the bootstrap.servers with the bootstrap server address noted above. We also need to make sure the replication factor is no greater than number of partitions, which should not be a problem in our lab as the replica count was set to 3.

Now we can create a Broker in the same namespace with consumer:

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
 name: default
 namespace: event-example
 annotations:
 eventing.knative.dev/broker.class: Kafka
spec:
 config:
 apiVersion: v1
 kind: ConfigMap
 name: kafka-broker-config
 namespace: knative-eventing

Once the manifest above is applied, we should see the status of broker (named default) to be READY. We can also validate broker creation with kn CLI command:

 $ kubectl -n event-example get broker default
NAME URL AGE READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m True
$ kn broker list -n event-example
NAME URL AGE CONDITIONS READY REASON
default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m 7 OK / 7 True

Then we create Triggers that connects both to the broker with filters, and to the consumers by referencing service name. The manifest to create triggers is provided below:

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: hello-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 type: greeting
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: goodbye-display
 namespace: event-example
spec:
 broker: default
 filter:
 attributes:
 source: sendoff
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm the setups above by monitoring the resources (like we did for broker), or use knative CLI:

$ kn trigger list -n event-example
NAME BROKER SINK AGE CONDITIONS READY REASON
goodbye-display default service:goodbye-display 3m24s 6 OK / 6 True
hello-display default service:hello-display 3m24s 6 OK / 6 True

Now we can go to the validation step to send event and validate result.

Lab B – Install in-memory channel and configure Channel

This lab is for channel-subscription model. we first install an in-memory channel:

kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/in-memory-channel.yaml

Then we can map this channel to a Channel resource in Eventing. We can do so by applying the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-1
 namespace: event-example
---
apiVersion: messaging.knative.dev/v1
kind: Channel
metadata:
 name: demo-channel-2
 namespace: event-example

The creation of Channel resources can be verified as below

$ kn -n event-example channel list
NAME TYPE URL AGE READY REASON
demo-channel-1 InMemoryChannel http://demo-channel-1-kn-channel.event-example.svc.cluster.local 6m34s True
demo-channel-2 InMemoryChannel http://demo-channel-2-kn-channel.event-example.svc.cluster.local 6m34s True

Now we need to create Subscription which connects Channels to Consumers. Subscriptions can be created with the manifest below:

apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-1
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-1
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: hello-display
---
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
 name: demo-subscription-2
 namespace: event-example
spec:
 channel:
 apiVersion: messaging.knative.dev/v1
 kind: Channel
 name: demo-channel-2
 subscriber:
 ref:
 apiVersion: v1
 kind: Service
 name: goodbye-display

We can confirm subscription:

$ kn -n event-example subscription list
NAME CHANNEL SUBSCRIBER REPLY DEAD LETTER SINK READY REASON
demo-subscription-1 Channel:demo-channel-1 service:hello-display True
demo-subscription-2 Channel:demo-channel-2 service:goodbye-display True

Now we can go to validation step.

Validation

For validation, we need to first configure an event source. The source in our lab is a throw-away Pod with curl utility. I use my favourite nicolaka netshoot to fire off the events using. To launch the Pod and get to command shell:

$ kubectl run tmp-shell -n event-example --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.1#

We can POST message to the corresponding endpoint URL to fire events. The endpoint is different for each lab:

Lab	URL
A (Kafka-based broker)	http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default
B (channel)	http://demo-channel-1-kn-channel.event-example.svc.cluster.local http://demo-channel-2-kn-channel.event-example.svc.cluster.local

With the correct URL, we can build a payload for POST method. Below is three curl commands with different payload POST to the same endpoint (taking lab A as an example):

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: not-sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: not-greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Goodbye Knative!"}'

# curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default" \
-X POST \
-H "Ce-Id: say-hello-goodbye" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Type: greeting" \
-H "Ce-Source: sendoff" \
-H "Content-Type: application/json" \
-d '{"msg":"Hello Knative! Goodbye Knative!"}'

All the requests should receive 202 return code. As highlighted above, we specified type and source for each event fired to the broker. On the consumer side, we can simply check the pod log to verify the events received. Below is an example of events received as a result of the events sent above (take broker-trigger model as example):

$ kubectl -n event-example logs -l app=hello-display --tail=100
2022/03/09 17:05:59 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: not-sendoff
 id: say-hello
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:19.196589801Z
Data,
 {
 "msg": "Hello Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }
$ kubectl -n event-example logs -l app=goodbye-display --tail=100
2022/03/09 17:05:58 Failed to read tracing config, using the no-op default: empty json tracing config
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: not-greeting
 source: sendoff
 id: say-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:27.430702588Z
Data,
 {
 "msg": "Goodbye Knative!"
 }
☁️ cloudevents.Event
Context Attributes,
 specversion: 1.0
 type: greeting
 source: sendoff
 id: say-hello-goodbye
 datacontenttype: application/json
Extensions,
 knativearrivaltime: 2022-03-09T17:14:42.977803608Z
Data,
 {
 "msg": "Hello Knative! Goodbye Knative!"
 }

This is the gist of Knative evening. Note that the format of HTTP payload when we fire off events needs to conform to CloudEvents format. The container image (event_display) used in the container is also created for the purpose of processing events received and display them in stdout.

Conclusion

Persistent storage allows for stateful applications on Kubernetes. It also enables architectural patterns that requires persistent storage. Knative eventing is a framework that enables event driven architecture. It supports many messaging channels (including NATs, a cloud-native messaging system) and broker types (MT channel based, GCP, RabbitMQ).

Knative Serving Introduction

Sun, 17 Apr 2022 11:33:00 -0400

Background

As per IBM‘s definition, Knative enables serverless workloads to run on Kubernetes clusters, and makes building and orchestrating containers with Kubernetes faster and easier. It has drawn a lot of attention recently. It released version 1.0 in November 2021, and was accepted as a CNCF incubating project in March 2022.

Glories aside, the value it delivers is the ability to go serverless on a Kubernetes platform. Originally, Knative was built with three components: build, serving and eventing. The build component was deprecated in favour of the Tekton project. Tekton is cloud native CI/CD pipeline. It connects to source code repo and build artifacts. It can also deploy applications with pipeline as code. Tekton’s documentation includes a quality tutorial here linked to interactive terminal. On the other hand, we’ll focus on Knative Serving in this post, and Knative Eventing in the next.

Knative can be seen as a serverless framework with a number of open-source technologies as building blocks, such as Istio for ingress gateway, Kafka or Google Pub/Sub as event-streaming engine, and Prometheus for observability to name a few. Despite of using the same CLI tool (named kn), Knative Serving and Eventing are considered separate capabilities using different groups of CRDs. They are installed separately, using their respective YAML files, or operator.

Serving vs Eventing ?

After reading the article “Did we market Knative wrong” from Ahmet Balkan, my impression is the two are not related. They could have been two separate projects without sharing a common name. As the author puts, these two shared some core logics. But beyond that, they don’t have anything in common.

Serving is lightweight and requires Istio. It provides the capability to scale from N to 0 when the system is idle, and from 0 to N when requests come in. It is the missing serving layer for running microservices on Kubernetes and Ahmet position it as the first thing that people installed after creating a cluster.

The target user of Eventing is much smaller. Eventing is for event-driven architecture and is more complex than Serving. Ahmet admits that they over-estimated how many people on the planet want to build a Heroku-like PaaS layer on top of Knative. There are a couple of dozen companies who would work through the complexity and build their own Kubernetes-based internal PaaS or even public-facing FaaS using Knative. They are the niche audience of Knative Eventing.

For most platform builders who just need to run micro-service, Knative Serving is all they need.

A Demo on Serving

The definition of term “serverless” is very loose. Instead, Knative documentation promotes the ability to scale to zero and refer to it as “some people call this Serverless”. This suggests that “the ability to scale to zero” should be the better term to describe this capability. To deploy it, we need the Service object with apiVersion serving.knative.dev/v1. Below is a guide for a quick hands-on, with some steps modified from the Knative Serving tutorial and installation guide, with optional steps (extensions, DNS) skipped.

Follow the first part of this post or this guide in real-quicK-cluster project, to install minikube with metal LB and configure istio (using istioctl as per instruction). Then Knative serving can be installed with two YAML manifests:

$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-crds.yaml
$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-core.yaml

Then we can install Knative istio controller to integrate them so we can see the knative gateways:

$ kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.2.0/net-istio.yaml
$ kubectl -n knative-serving get gateway

Now we can install the dummy service, either using kn CLI tool or using YAML manifest. The manifest is given below:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
 name: hello
 namespace: default
spec:
 template:
 metadata:
 name: hello-world
 annotations:
 autoscaling.knative.dev/class: "kpa.autoscaling.knative.dev"
 autoscaling.knative.dev/metric: "rps"
 autoscaling.knative.dev/target: "50"
 autoscaling.knative.dev/min-scale: "0"
 autoscaling.knative.dev/max-scale: "10"
 spec:
 containerConcurrency: 0
 containers:
 - env:
 - name: TARGET
 value: World
 image: gcr.io/knative-samples/helloworld-go
 name: user-container
 ports:
 - containerPort: 8080
 protocol: TCP

We can notice that in the annotations there are some settings to overwrite the default configuration. They are explained in the autoscaling section of the Knative serving documentation. Here we use request per second (rps) as metric, and have 50 as target. The scale range is between 3 and 10 replicas. Once the resource is created, we can validate it with one of the two commands:

$ kubectl get services.serving.knative.dev
$ kn service describe hello

The creation of knative service also created a number of Kubernetes resources, such as deployment and services:

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello ExternalName <none> knative-local-gateway.istio-system.svc.cluster.local 80/TCP 52s
hello-world ClusterIP 10.110.127.231 <none> 80/TCP 108s
hello-world-private ClusterIP 10.111.27.73 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 108s
kubernetes ClusterIP 10.96.0.1 <none>
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 1/1 1 1 71s

Before testing, let’s mock the DNS entry to resolve to the Istio ingress service IP like below in /etc/hosts:

192.168.64.16 hello.default.example.com

Now we can confirm that the service returns the expected result (after a brief pause):

$ curl hello.default.example.com
Hello World!

Wait for a short period without curl or connection from browser, the service should scale down to zero:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s

From a separate terminal, emulate HTTP request with hey command, so the service receives more than 50 rps per pod:

$ hey -n 200 -c 20 -z 20s -m GET http://hello.default.example.com

We can watch the deployment get scaled up to 10. Once the hey command completes its task, the deployment will scale back down:

$ kubectl get deploy -l serving.knative.dev/service=hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world-deployment 0/0 0 0 78s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 0 0 3m54s
hello-world-deployment 0/1 1 0 3m54s
hello-world-deployment 1/1 1 1 4m2s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 1 1 4m4s
hello-world-deployment 1/10 10 1 4m5s
hello-world-deployment 2/10 10 2 4m9s
hello-world-deployment 3/10 10 3 4m10s
hello-world-deployment 4/10 10 4 4m10s
hello-world-deployment 7/10 10 7 4m10s
hello-world-deployment 9/10 10 9 4m10s
hello-world-deployment 10/10 10 10 4m11s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 10/5 10 10 5m18s
hello-world-deployment 5/5 5 5 5m18s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 5/2 5 5 5m20s
hello-world-deployment 2/2 2 2 5m20s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 2/1 2 2 5m22s
hello-world-deployment 1/1 1 1 5m22s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 1/0 1 1 5m52s
hello-world-deployment 0/0 0 0 5m52s

Autoscaling in Knative serving is backed by KPA (Knative Pod Autoscaler, used in the example above) or HPA. Also note that the trigger to scale up and down is client connection (RPS). This is different than KEDA. KEDA also supports the capability to scale to zero. However, with KEDA, it is an event that scales a deployment up from zero. In Knative serving, it is the connection to the Service itself that wakes up the service.

Cost of scale-to-zero

As discussed, both KEDA and Knative serving supports the ability to scale to zero and wake up from zero. The trigger to wake up from zero is different. With KEDA, workload wakes up by an event. With Knative Serving, workload has to wake up upon receiving a connection to the service. In the design to solve this problem, Knative has four sub-components :

Activator – When a service is scaled to zero, its request are routed to the activator which waits for a Pod to become ready (wake up) and proxies the traffic while editing the underlying Endpoints to route the traffic directly to the Pod(s)
Autoscaler – KPA
Controller – The main component responsible for watching API objects for Knative CRDs (KService, Configuration, Route, Revision) and manage their lifecycle, create the underlying Kubernetes resources and garbage-collect old objects.
Webhook – a Kubernetes Admission Webhook acting both as validating admission controller and mutating admission controller.

The key player is activator, which receives requests when a service is scaled to zero. The wake-up process is summarized as follows:

After receiving request, activator will buffer (hold onto) the request
Then, it will look at request’s hostname to find which KService it is for.
Then, it will scale up the Kubernetes Deployment and wait for a Pod to become ready.
In the meanwhile, it updates Kubernetes Service to point to Pod IP addresses (so that activator gets out of the network path if the KService is awake).
Finally, activator proxies the request to the started pod

So the “plumbing” of the service connection managed by the Activator pod is the cost of scale to zero. This is also called Load Balancing, and the behaviour can be tweaked using two parameters: activator capacity and target burst capacity:

Target burst capacity: once the target deploy has one more more Pod, service requests may still route via activator or bypassing the activator. Target burst capacity determines at what point service request should start to bypass activator.
Activator capacity: determines how many requests can activator hold on to. Considering the service-to-service connection also routes via Service CRD, there can be a lot of connections via Activator so its capacity needs to be adjusted.

The additional overhead of managing and optimizing the “Load Balancer” is also part of operational cost for the ability to scale to zero.

In comparison, KEDA now has an HTTP add-on still at beta but allows connection-based wake-up. The design is a little different. As discussed in this previous post.

Conclusion

Knative consists of two disparate components: Serving and Eventing. Serving uses KPA to provide scaling based on service request, and the ability to scale to zero. It is often compared with KEDA, a single-purpose lightweight tool for autoscaling. Here is a blurb on their difference by KEDA. The takeaways is that KEDA is more focused on scalability, whereas Knative serving covers more aspects. For example, the Service object of Knative also supports Traffic Splitting. Knative serving can integrate with istio.

Kubernetes Operator

Thu, 07 Apr 2022 09:39:00 -0400

Kubernetes has a number of tools to automate the deployment of a single workload. In previous posts, we had covered Helm and Kustomize. What are left unresolved is how to maintain the status of workload after deployment is completed. In this post, I will give an introduction to Kubernetes Operator. Compared with Helm (templating approach) and Kustomize (patching approach), Kubernetes Operator follows the operator pattern. Operators are usually provided by the developer of the application.

Operator Pattern

In Kubernetes, we know that a controller takes care of routine tasks to ensure that desired state expressed by Kubernetes resource types matches the current state. One example is that the Deployment controller ensures the number of pods running matches the amount specified in the replica field. Controller is the key to ensure that resources can be managed by declarative manifests for Kubernetes resources.

Kubernetes makes use of controller pattern throughout its own design. One of its key component, Controller Manager, is a collection of many controllers. Each controller is in charge of a control loop, responsible for listening the object it manages. Another component, Kube-scheduler, is also a special type of Controller. The kube-scheduler monitors unscheduled Pod and health of nodes and determines the best Node to schedule the new Pod to. Then it writes the decision to etcd store for kubelet to execute.

This controller pattern is fairly successful in what it does and we can extend the use of it. Beyond the built-in resource types, we can create our own custom resource definitions (CRDs), and create controllers that watches for the manifest that declares custom resources (CRs). The controller ensures that the resource status matches their specifications. This is also known as reconciliation, which is implemented as a control loop. Operator pattern can be illustrated in the diagram below:

Operator Pattern

Technically, there is no difference between a controller and an operator. What makes an Operator (used to install workload) different than a native Kubernetes controller, are two things. First, an Operator usually needs CRDs because the built-in resource types are insufficient. Second, the operator reflects the domain knowledge to keep the target workload running. For example, stateful workloads such as database needs their operational steps executed in certain orders.

On Operator Pattern, CNCF published a whitepaper with a deeper review. This white paper is the best reference for a good understanding of the Operator Pattern.

Custom Resource Definition

The built-in controllers work with built-in objects (pre-defined APIs). Custom operators usually need their own APIs to function. To extend Kubernetes API, we define the schema of these APIs in the form of CRDs (custom resource definitions) using OpenAPIv3 standard. Then, we can declare Custom Resources (CRs) in compliance with the schema. The OpenAPIv3 schema in the CRD resource tells validating web hook (admission control) how to validate the schema when we send an CR in to API server.

When we work with third-party operators, they usually provide CRDs along with the operator implementation. For example, in my operator example project, we have a minimalist CRD WordPress with one property: sqlRootPassword and we can declare a CR as in this example. For a more realistic use case, we can take a look at Kiali CRD. In the next section, we’ll use it along with Kiali operator to install Kiali.

Operator Usage

Like Artifact Hub to Helm, OperatorHub is a public registry of most used Kubernetes Operators. In this section, we will take an example of using Operators. We will install Kiali as an add-on to Istio using Kiali CR and operator, which also depends on Prometheus to be installed using Prometheus Operator first. Note that the Kiali installation outlined in this section is not the the quick-start install manifests from Istio’s sample directory. For Kiali on production system we have to customize the installation.

Suppose we have installed Istio, we can then install Prometheus operator using Helm. The Prometheus operator will install Prometheus. Then we use Helm again to install Kiali operator. The Kiali operator will watch for creation of Kiali CRD, to deploy services:

$ helm install -f prometheus-values.yaml --namespace istio-system --repo https://prometheus-community.github.io/helm-charts --version 13.6.0 istio-prometheus prometheus --insecure-skip-tls-verify
$ helm install -f kiali-operator-values.yaml --namespace kiali-operator --repo https://kiali.org/helm-charts --version 1.45.0 kiali-op kiali-operator --create-namespace
$ kubectl apply -f kiali-cr.yaml

I include example content for each file in the commands above on Github gist (prometheus-values.yalm, kiali-operator-values.yaml and kiali-cr.yaml). For more options for installing Kiali, refer to their documentation.

I use this example to install Kiali and it includes two Operators, the Prometheus Operator and the Kiali Operator. The Prometheus Operator is one of the first ever written Kubernetes Operator. As soon as the operator is deployed, it starts to deploy the operator service. For the Kiali operator, we need to deploy Kiali CR after the Kiali Operator has been deployed. Both are valid patterns.

Operator Development

Operator is powerful. However, authoring an Operator is not a trivial effort. One usually start with a framework. A framework creates a body of boiler plate code that has the pattern implemented and allows developers to enrich the functions following the pattern. The white paper introduced three frameworks:

CNCF Operator Framework – aims at Operator Developers with an SDK, a scaffolding tool and a test harness. It currently supports three project types: Golang, Helm and Ansible. CNCF Operator framework consists of SDK and OLM.
Kopf (Kubernetes Operator Pythonic Framework) – an easy-to-use framework in Python that abstracts away most of the low-level Kubernetes API communications hassle.
kubebuilder – helps build a Manager similar to the native kube-controller-manager. For difference with OperatorSDK, read here.
Metacontroller: lightweight Kubernetes Controller as a Service

In CNCF Operator Framework, the Operator SDK supports development using Ansible, Helm and Golang. The author of this post makes a general comparison as follows:

Type	Best use case	Underlying technology	Amt of Effort
Helm	Stateless workload	Helm Charts	Med
Ansible	Stateless workload	Ansible Roles and Playbooks	Med
Golang	Stateful workload	Code developed in Golang	High

The aforementioned Kiali operator is an example of Operator developed in Ansible. The prometheus operator, is developed in Golang as the workload can be stateful depending on configuration. One needs to know how to develop operator in Golang in order to tackle the most complicated situations. This is requires some serious development effort. The documentation with a quick start section is available here. Even that is not very straightforward. RedHat, the maintainer of the CNCF Operator framework has a good blog post on how to develop an Operator in Golang.

The example requires some development knowledge to go through. On my MacOS (Intel) I have to configure the following prerequisites:

Install gcc, using command: xcode-select –install
Install the right version of golang. You can find the version here. The MacOS has a version of golang installed already so I had to install version 1.17 and link to it: brew install go@1.17 && brew link –force go@1.17
Install operator-sdk with home brew: brew install operator-sdk
When you run “operator-sdk version”, ensure the result shows a golang version that matches your installation.
If you need to push docker image, also connect to docker registry by running: docker login

Then we can create our working directory, initialize the repository and create boilerplate code (scaffolding) with these commands:

$ mkdir wordpress-operator && cd wordpress-operator
$ operator-sdk init --domain digihunch.com --repo github.com/digihunch/wordpress-operator
$ operator-sdk create api --group wordpress --version v1 --kind WordPress --resource --controller

With the repo initialized, we can go to the section “Defining the API” and “Implementing the Controller”. The blog post does not cover every code editing needed to bring up wordpress. You are supposed to go to the author’s repository to fit the changes into your own repo. The author’s repo has a few more controllers such as common.go and mysql.go.

At the end of the lab, you should be able to run the controller and bring up wordpress. I used my own repository for this lab and have made the code changes for this lap in a couple commits. To test locally with the code:

$ git clone git@github.com:digihunch/wordpress-operator.git
$ cd wordpress-operator
$ make install run

Then we can validate wordpress install from a new terminal as the instruction shows:

$ kubectl create -f config/samples/wordpress_v1_wordpress.yaml
$ minikube service wordpress --url

For Developers that requires more details, RedHat has an eBook for Kubernetes Operators, in supplement to the documentation. As DevOps professional, I’m mainly concerned with understanding how Operator works and using Operators correctly.

Too many Tools?

Now we seem to have too many choice of tools when it comes to deploying workload on Kubernetes. Kustomize and Helm can deploy simple workloads. Operator can deploy stateful workloads, as well as keep the workload status in check. Further, we have FluxCD and ArgoCD based on GitOps workflow.

When assessing a tool, we should think about the complexity of the workload deployed. If it is a single stateless workload, Kustomize or Helm should be sufficient. If it is not very simple but still stateless, we can consider using Helm charts developed by the community. For multiple workloads, we can build our own top-level chart to combine existing sub-charts created by the community.

Helm is essentially a package manager. It does not follow controller pattern and therefore will not monitor the current status of deployment. Helm has other limitations compared to Operator. For example, as a templating scheme, it reaches limitation when dealing with complex logic, even with the help of its helper functions. It is also hard to reason through the template code when we have to troubleshoot a deployment. Refer to this blog post for the author’s experience with Helm.

If we want our deployment to be fully declarative and continuous, then we will follow the Operator pattern by using a Kubernetes Operator. When we have many workloads of different levels of complexity, we can combine them with GitOps tool. Operator is one of the underlying technologies behind GitOps.

Workload profile	Just Installation	Installation and Maintain Status
Single stateless workload	Helm or Kustomize	Operator (using Ansible or Helm)
Single stateful workload	Helm or Kustomize	Operator (using Golang)
Multiple workloads	Helm (e.g. build parent chart)	GitOps in combination with Operator, Helm and Kustomize

The table above helps refine deployment requirement. It’s not a recommendation, but rather a model of analyzing deployment requirement.

Autoscaling on Kubernetes Platform

Mon, 28 Mar 2022 13:14:00 -0400

Introduction

The concept of autoscaling on Kubernetes platform dates from the era where virtualization first became widespread and the overhead of provisioning a new server became lightweight through the use of cloud-init. With public cloud, customers operate on usage-based billing. Autoscaling allows workload to scale down during idle times to reduce cost, and scale up during peak time to meet the demand of business traffic. Vertical autoscaling replaces a VM with one of higher capacity, which is usually interruptive. Horizontal autoscaling adds or removes VMs to adjust capacity, and works in conjunction with load balancing mechanism to assign load to a specific target in the group. Unless otherwise specified, we simply refer to horizontal autoscaling as autoscaling. Depending on what triggers autoscaling, it can be metrics based or event driven.

Metric-based autoscaling is based on VM (or Pod) metrics in the autoscaling group. The metrics are mostly about CPU usage, memory, IOPS, number of connections, etc. For example, when the average CPU utilization across all VMs in the last five minutes hits 70% threshold, then the scaler introduces a new VM into the autoscaling group. The trigger can factor in a variety of metrics. Advanced autoscaling APIs can also support lifecycle hooks, i.e. custom activities upon creation of new VMs during scale-up, or upon deletion of existing VMs during scale-down. Other aspects of custom behaviours include a cool-off period, i.e a no-activity window after the previous scaling activity. Since scaling activities are re-active.

Using real-time metrics as a trigger of scaling is not always a good idea. For example, a buggy order processing program may consume 100% of CPU due to an infinite loop, or 99% of memory due to memory leak. Metric-based scaling may fire off even though there is currently no order pending in the queue. Event-driven approach is more flexible. Event can fire from any type of source. For example, in Kubernetes, when scheduler fails to schedule a Pod due to constraints, it is an event This event can trigger scaling. In some case, a metric hitting a threshold fires an event. For example, scale up when size of order queue reaches 20. In this sense, metric-based autoscaling is a special case of event-driven autoscaling.

With Kubernetes, let’s examine node group autoscaling (aka cluster autoscaling) and workload autoscaling (Pod autoscaling).

Node Group Autoscaling

Cluster autoscaler is the mechanism to auto-scale node groups for Kubernetes. As per its documentation, any metric-based cluster/node group autoscalers are NOT compatible with CA. They are also not particularly suitable for Kubernetes in general. Take AKS for example, the events to trigger scale-up and scale-down are as below:

The cluster autoscaler component can watch for pods in your cluster that can’t be scheduled because of resource constraints. The cluster then automatically increases the number of nodes.
The cluster autoscaler decreases the number of nodes when there has been unused capacity for a period of time. Pods on a node to be removed by the cluster autoscaler are safely scheduled elsewhere in the cluster.

Both are in essence event driven. The behaviours can be fine-tuned with a number of parameters as below:

scan-interval
scale-down-delay-after-add
scale-down-delay-after-delete
scale-down-delay-after-failure
scale-down-unneeded-time
scale-down-unready-time
scale-down-utilization-threshold
max-graceful-termination-sec
balance-similar-node-groups
expander: random, most-pods, least-waste, priority
skip-nodes-with-local-storage
skip-nodes-with-system-pods
max-empty-bulk-delete
new-pod-scale-up-delay
max-total-unready-percentage
max-node-provision-time
ok-total-unready-count

The parameters above constitute the autoscaler profiler, and are effective if cluster autoscaler is enabled. For many implementations, cluster autoscaler can be enabled and disabled even after the cluster has been created, and the parameters can be changed. The overhead of provisioning a new node should not be overlooked, because that is usually the window that a Pod needs to wait to get scheduled. As stated in CA’s FAQ, the main purpose of CA is to get pending pods a place to run, instead of pre-emptively accommodating to increasing workload.

The delay in pod scheduling while adding a new node can be controlled to a certain degree with one of the two workarounds below:

With HPA or KEDA, set lower threshold so the workload level scaling acts more aggressive than the increase of demand. This buys some buffer time
Use a tool to puff up utilization, such as cluster overprovisioner, which deploys pods that
- request enough resources to reserve virtually all resources for a node
- consume no actual resources
- use a priority class that causes them to be evicted as soon as any other Pod needs it.

In practice the cluster autoscaler setup should be conservative and keep node size as stable as it can. For example, a 20 minutes idle-window (low utilization) on a node is not worth the overhead to remove a node and add it back in 20 minute later.

When the cluster do need to scale down by removing a node, one common symptom is failing to scale down because some Pods have nowhere else to schedule to. Here is a list of possible causes as troubleshooting tips.

If node scaling should be triggered sparsely, then pod scaling is by design very dynamic. Cloud native applications should assume that pod scaling occurs very frequently.

In late 2021, AWS released the open-source project Karpenter for cluster autoscaler. Karpenter addresses some challenges with native Cluster Autoscaler on EKS. Karpenter is gaining momentum and now adding support for other cloud service providers including Azure.

Workload Autoscaling

Stateless workload are controlled by a Deployment object, which is associated with a replicaSet object. For stateless workload we can use HorizontalPodAutoscaler, or HPA. There is a VerticalPodAutoscaler (VPA) which is much less common. HPA is metrics based with flexible options such as specifying an object, depending on what metrics are available via metrics API. There are two versions of HorizontalPodAutoscaler: autoscaling/v1 and autoscaling/v2. The latter supports scaling policies, such as adjusting downscale stabilization window, and limiting scale down rate. No matter which API version, the metric-based triggers in HPA are fairly limited.

We already know that metrics are not always the best indicator to trigger scaling. We need an option to trigger scaling based on the status of other components such as queue size. KEDA (Kubernetes Event Driven Autoscaling) is a great option to consider for horizontal workload scaling. KEDA works with HPA, and significantly enriches trigger options. Apart from metrics, KEDA can use a number of external mechanisms as triggers, for example:

RabbitMQ/Kafka/SQS: scale based on queue size
Azure Log Analytics: scale based on a kusto query result against Azure Log Analytics
AWS CloudWatch, Azure Monitor: scale based on metrics from Azure Monitor/AWS CloudWatch
Azure Pipelines: scale based on agent pool queues of Azure Pipeline
Elasticsearch: scale based on elasticsearch query result
Kubernetes Workload: scale based on the count of running pods of a specified workload
MSSQL, MySQL, Postgres, Cassandra: scale based on a query result
Prometheus: scale based on prometheus query result

KEDA is a single-purpose and lightweight component. With KEDA, we don’t need to explicitly define HPA. It allows us to select from a longer list of triggering mechanisms for our auto scaler. We shall not underestimate the work needed to select the most suitable trigger because having an incorrect trigger (e.g. bad metrics) is costly. Let’s take Java applications as an example. Java workload operates in a JVM inside of the container. JVM request the entire heap size from operating system. The garbage collection activities also consumes a good portion of CPU cycles. This pattern makes CPU and memory metrics inaccurate as an indicator for scaling activity. Because of this we need to find out what is the best scaler for Java application, based on understanding of how the entire solution stack works as a whole.

The other aspect that KEDA beats HPA is its ability to scale to 0. This can be helpful when a service is idle most of the time but cannot shut down.

KEDA lab

let’s use Kafka as an example to configure KEDA for a dummy workload. We create a mock cluster using Kind with a simple configuration file. Then, let’s start with the following dummy workload with replica count set to 1:

apiVersion: v1
kind: Namespace
metadata:
 name: workload
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: aks-helloworld-one
 namespace: workload
spec:
 replicas: 1
 selector:
 matchLabels:
 app: aks-helloworld-one
 template:
 metadata:
 labels:
 app: aks-helloworld-one
 spec:
 containers:
 - name: aks-helloworld-one
 image: neilpeterson/aks-helloworld:v1
 ports:
 - containerPort: 80
 env:
 - name: TITLE
 value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
 name: aks-helloworld-one
 namespace: workload
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
 app: aks-helloworld-one

We need to install KEDA and Kafka using Helm:

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda -n keda --create-namespace
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3

Watch for all Pods to come up. Also read the notes from Kafa installation and confirm the Kafka service address. Now, we will apply KEDA scaled object, defined as below:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: kafka-scaledobject
 namespace: workload
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: aks-helloworld-one
 pollingInterval: 10
 cooldownPeriod: 30
 idleReplicaCount: 0
 minReplicaCount: 2
 maxReplicaCount: 5
 fallback:
 failureThreshold: 3
 replicas: 1
 advanced:
 restoreToOriginalReplicaCount: true
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 300
 policies:
 - type: Percent
 value: 100
 periodSeconds: 15
 triggers:
 - type: kafka
 metadata:
 bootstrapServers: kafka-0.kafka-headless.kafka.svc.cluster.local:9092
 consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics
 topic: test
 # Optional
 lagThreshold: "5"
 offsetResetPolicy: latest

The field definitions are on KEDA deploy documentation. Items under Kafka trigger are on the trigger documentation. In this lab we set the idelReplicaCount to 0. It will scale up with average lag of all partitions reaching 5. In the next few steps, we’ll mock up some messages posted to the Kafka topic named “test” for consumer group my-group. We can watch deployment size grow along with the growth of lags.

To emulate Kafka client activity, we can spin up a Kafka test Pod:

kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.8.1-debian-10-r73 --namespace kafka --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace kafka -- bash

From within the Pod, we can leverage the client-side scripts located in /opt/bitnami/kafka/bin/. For example, to post message to a topic (e.g. named test):

kafka-console-producer.sh --topic test --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092

The Helm installer also gives the command with broker list. To consume messages from a topic (e.g. test) to a given consumer group:

kafka-console-consumer.sh --topic test --bootstrap-server kafka.kafka.svc.cluster.local:9092 --group my-group

We can have two command terminals, post test messages on one terminal and watch it consumed nearly immediately on the other terminal.

The Kafka trigger documentation suggests that the number of replicas will not exceed the number of partitions on a topic when a topic is specified. To make this lab work, we need to have set 5 partitions:

kafka-topics.sh --alter --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test --partitions 5
kafka-topics.sh --describe --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test

Once we confirm five partitions, we can spin up two command terminals, one to produce message and the other to consume messages. If working, we can stop the consumer and use the command below to watch for the lag for each partition.

kafka-consumer-groups.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --describe --group my-group

Now we can artificially trigger scaling by increasing average lag. We keep posting messages on the producer (each carriage return posts a message), and we can check the lag after posting:

Growth of average lags

The size of deployment starts with 0 as defined in the scaled object. As the average exceeds 5, we can see deployment size growing.

Growth of deployment size

This lab is an oversimplified scenario to illustrate the idea of scaling. Kafka is a typical queue construct and other queue configuration such as RabbitMQ or AWS SQS works in very similar ways. Real life use case involves more aspects to consider, such as multiple topics, and authentication.

Connection triggered wake-up

KEDA uses Events to scale workload from zero to one (wake up). There is no way to scale (wake up) based on an incoming web request. In many cases, such as serverless configuration, we need to scale the deployment size from zero to N once the service receives incoming web request. This is not supported by KEDA. By definition KEDA uses Events to wake up.

There is an HTTP-add-on for KEDA still at beta but it is trying to address this problem. This page shows the design. Suppose a service has scaled down to zero, the followings will happen to wake it up:

The incoming request is routed to an interceptor behind the service
interceptor keeps track of number of pending HTTP request
The scaler periodically watches for the size of the pending queue on the interceptor
Based on the queue size, the scaler reports scaling metrics as appropriate to KEDA
As the queue size increases, the scaler instructs KEDA to scale up as appropriate

The periodical check activity is the key to make it work and also what makes it a pseudo-trigger.

Summary

Fine-tuning autoscaling is important to the performance of workload on Kubernetes. At node level, we briefed on cluster autoscaler and suggest that we only use it sparsely. At pod level, we introduced native HPA as well as KEDA, with an example. We also discussed KEDA has limitations and the HTTP-add-on. In the next post, we’ll explore Knative’s autoscaling capability.

Posts on Digi Hunch

Training and Inference in SageMaker AI

Frameworks and Data

Training and Inference

SageMaker SDK Abstractions

Batch Transform

Inference Endpoint

Summary

SageMaker AI and Networking

Naming Shenanigans

Workload Categories

Studio Notebooks

Inference Endpoint

Pipeline Jobs

Summary

The Leanest Web and Email Hosting

WordPress Hosting

WordPress Migration

CloudFlare Security

CloudFlare Cache Rules

SMTP service

Sending Reputation

Summary

Firewall Deployment Patterns

Inspection Requirements

Inspection Architecture

Firewall deployment patterns

Rules

Summary

Cloud Certifications for Learning?

Dosage of Marketing

Stability of Topics

Breadth and Depth

Upkeep

More ways of learning

Debating between count and for_each in Terraform

Problem with count loop

Embrace for_each loop

A real-life example with classic pattern

Implementation using count

Implementation with for_each: first attempt

Is for_each a bad idea?

Implementation with for_each: second attempt

One more shot with for_each

Conclusion

Test Open ID Connect Flows Locally

Architecture

Choice of Tools

Configuration

Summary

IAM Roles for any workload

Background

IAM Role Introduction

The IMDS service

IAM Role for EC2 Workload

IAM Role Anywhere Concept

IAM Role Anywhere Lab

Summary

Managing EC2 instances across accounts with Ansible

Prerequisites

Configure AWS CLI

Configure AWS CLI Profiles

Configure SSH to EC2 via SSM

Configure Ansible Inventory

Conclusion

Public Key Infrastructure 3 of 3 – PKI Implementation

Root CA

AWS Private CA as intermediate CA

Step CA

Cert Manager on Kubernetes

Summary

Public Key Infrastructure 2 of 3 – Certificate Automation

Validation

Certificate Automation

The ACME Protocol

Renewal and Revocation

Summary

Public Key Infrastructure 1 of 3 – Basics

Concepts

Implementation Options

Problem with `count` loop

Embrace `for_each` loop

Implementation using `count`

Implementation with `for_each`: first attempt

Is `for_each` a bad idea?

Implementation with `for_each`: second attempt

One more shot with `for_each`