Docker Images on Digi Hunch

Kubernetes with Multiple CPU Architectures 1 of 2 – Container Image

Sat, 15 Apr 2023 12:17:00 -0400

While working on a multi-arch Kubernetes cluster, I came across quite a few issues with image logistics and decided to put these issues in a separate post here. The goal is to supply images with multi-arch support in a standard way. Therefore it is not advisable to rely only on arbitrary image tags to distinguish CPU architecture. If that was the case, each team may use different tag keys and values. The image consumers would have to build custom logics to pull the right image. The standard way is to use Manifest List (Docker’s term), or Image Index (OCI’s term) on the image registry, to announce the multi-platform support of the images in the repository, and to ensure that a container runtime can pick the appropriate index entry that matches the local node’s architecture and platform.

Introducing Image Index

For image index/manifest, there have been three standards, with two currently active:

Docker Image Manifest V2, Schema 1: deprecated in 2019
Docker Image Manifest V2, Schema 2: supports attribute for platform specific image
OCI Image Specification: supports attribute for platform specific image

In 2016, Docker contributed the Docker V2 Image specification as the basis of the OCI image specification. Therefore, the two current active formats (V2.2 and OCI) still look alike today. They are both JSON document and shares many attribute names. However, I take OCI as the standard since it is adopted in CNCF (e.g. containerd, quay.io). Most of the registries (e.g. ECR, ACR and even Docker Hub) support both formats, and you can tell the format by requesting manifest document. When working with Kubernetes we should expect OCI format index even though the Docker Manifest List v2.2 will most likely be compatible.

When examining an image manifest we should look for the following structure:

Manifest list and Manifest, source: https://www.opensourcerers.org/2020/11/16/container-images-multi-architecture-manifests-ids-digests-whats-behind/

In the Docker lingo, a “fat manifest” is simply a list of manifests, equivalent to image index in OCI terms. A manifest document consists of several attributes in sha256 digest, as the diagram shows. There are four types of digests and they should not be mixed:

index-sha256: for the index or fat manifest
manifest-sha256: for the manifest of a single container image
config-sha256: for the config section
layer-sha256: for the image layer files

If two images share a layer, then from their respective manifests, we should be able to find a layer with the same sha256 digest. If an index (fat manifest) references other manifest, we should also find that from the sha256 digest. When you reference an image, you should use the sha256 digest for the entire index or manifest list.

To view image index, “docker inspect” is insufficient. It is executed within Docker daemon, against the image pulled based on the CPU architecture of Docker daemon’s server. So let’s look at some tools to check index from remote registry.

Tools to view image index

The most popular tool is Docker’s experimental manifest inspect command. However, it is still not mature. For example, we have to use Docker CLI version 23.0.0 or later with OCI compatibility issue fixed. Prior to v23.0.0 (Feb 2023), Docker CLI were not able to correctly display a list of OCI-format manifests, and it simply says “no such manifest”. Even after 23.0.0, I still find it clunky. For example, it does not display the sha256 digest of the OCI image index itself (the digest on the far left of the diagram above).

To troubleshoot the reason, I borrowed some idea from this post, and have my shell script as below, to check the index for my colorapp image:

#!/bin/sh

ref="${1:-digihunch/colorapp:v0.1}"
sha="${ref#*@}"
if [ "$sha" = "$ref" ]; then
 sha=""
fi
wosha="${ref%%@*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
 tag="latest"
fi
apio="application/vnd.oci.image.index.v1+json"
apiol="application/vnd.oci.image.manifest.v1+json"
apid="application/vnd.docker.distribution.manifest.v2+json"
apidl="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
 | jq -r '.token')
curl -H "Accept: ${apio}" -H "Accept: ${apiol}" -H "Accept: ${apid}" -H "Accept: ${apidl}" \
 -H "Authorization: Bearer $token" \
 -w '\nResponseCode:%{http_code}\nResponseHeader:\n%{header_json}\n' \
 -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}"

It appears that the sha256 digest of the OCI image index itself is provided in the response header, instead of response payload. So docker manifest tool misses the header!

Luckily, there are some alternatives, such as skopeo or manifest-tool. I wasn’t able to get the former to work with OCI index. The latter displays my OCI index in a pretty format and I was able to view a few other indexes that I wasn’t able to with Docker manifest. So I have a good impression of it. So in summary, for the four tools to view image index, my recommendations are:

Docker manifest: still glitch as of 2023 but pretty widespread
manifest-tool: pretty solid, the output is pretty format, requires manual install
skopeo: not straightfoward to use. easy to install
Self-scripting in bash: only for troubleshooting

For the rest of this post, I’ll however continue to use docker manifest, given its popularity and availability. When it fails to display an index, I’ll try manifest-tool.

Inspecting Image Index

With docker manifest command (v23.0.1), let’s take a peak at three images:

docker manifest inspect --verbose osimis/orthanc:22.12.2
docker manifest inspect --verbose public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0
docker manifest inspect --berbose ubuntu:23.04

In the JSON document return by the first command (osimis/orthanc image), we see the following structure:

{
 "Ref": "docker.io/osimis/orthanc:22.12.2",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:20413096878fb56bf8d09af08cf4055993dbcf507526f0561b26fc4d0ed7affc",
 "size": 11227,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "schemaVersion": 2,
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "digest": "sha256:9de20d8a006c6377b85dba9f817d47048982bd0f15fac7daacb64f42060d4b6d",
 "size": 16518
 },
 "layers":[
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "digest": "sha256:025c56f98b679f70b7a54241917e56da7b59ab9d2defecc6ebdb0bf2750484bb",
 "size": 31412852
 },
 ......
 ......
 ......
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "digest": "sha256:a24cf4c71e6977b692bbf517eba7bf6f454d41ceab24c1b2694c3303bc718a1c",
 "size": 174739
 }
 ]
 }
}

We can see that the return is a single manifest. The Descriptor key suggests that it is built for amd64 architecture. The mediaType, along with SchemaV2Manifest and schemaVersion, suggest that it is a Docker Manifest (v2.2) format. Other attributes are summarized here. In the response from the second command(public.ecr.aws/amazonlinux/amazonlinux), we see the structure below:

[
 {
 "Ref": "public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d",
 "size": 529,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "size": 1478,
 "digest": "sha256:d27c2e45784db13b0b2bc89a52be6661aa1d53bd25c070b41626768c9c563c3d"
 },
 "layers": [
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "size": 62386320,
 "digest": "sha256:d78505e615251c4f4af6eaa9507b67917d263d23551dcc5a1eed3c012d32a54d"
 }
 ]
 }
 },
 {
 "Ref": "public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed",
 "size": 529,
 "platform": {
 "architecture": "arm64",
 "os": "linux",
 "variant": "v8"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "size": 1493,
 "digest": "sha256:a1ea533a0632c6501d7848c7ed481e8fb0398c3277c0d9fddf0b0fdcd5731c09"
 },
 "layers": [
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "size": 64003805,
 "digest": "sha256:71343c2791199c6e2c19c308cff6493497a02f57e225c11405e1934dc7428b3c"
 }
 ]
 }
 }
]

Comparing this structure with the first return, we noticed that the return is a list of two manifests, each with its own platform architecture. So the amazonlinux image has multi-architecture capability. It can serve as base image for custom images for both architectures. On the “image tags” tab of registry page, we can see a list of published tags. Each is tied to either a single image manifest, or a manifest list:

Each image tag is associated with a single manifest or a manifest list (aka “fat manifest”)

When you click on “image manifest”, you can see both “Image manifest media type” and “Artifact media type” values. When you click on “manifest list”, you see the “Image manifest media type” value, because the “fat manifest” does not point to a single artifact.

Now, let’s review the third command (ubuntu:23.04) response:

[
 {
 "Ref": "docker.io/library/ubuntu:23.04@sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0",
 "Descriptor": {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0",
 "size": 424,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "OCIManifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "config": {
 "mediaType": "application/vnd.oci.image.config.v1+json",
 "size": 2299,
 "digest": "sha256:beb2152822b716b4deac2996f16bc84db0a14b7cbc549579635590438f9c0e1d"
 },
 "layers": [
 {
 "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
 "size": 26638886,
 "digest": "sha256:db781b8aed497363312ef32499cbfac28821e0494db7f0cadc4e716853e02a12"
 }
 ]
 }
 },
 {
 "Ref": "docker.io/library/ubuntu:23.04@sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c",
 "Descriptor": {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c",
 "size": 424,
 "platform": {
 "architecture": "arm64",
 "os": "linux",
 "variant": "v8"
 }
 },
 "Raw": "......",
 "OCIManifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "config": {
 "mediaType": "application/vnd.oci.image.config.v1+json",
 "size": 2316,
 "digest": "sha256:eb2d2fb228861107934403e776544a3f516bc7123a1275d52f1992bada8e94d6"
 },
 "layers": [
 {
 "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
 "size": 25802344,
 "digest": "sha256:29d183ded65aecf549f39ef891c21feb9034b5b10f341533b4af297bb5c60bb8"
 }
 ]
 }
 },
 ......,
 ......,
 ......
]

I readapted the response document for simplicity. Notice that the document is also a “fat manifest” except that the mediaType, OCIManifest and schemaVersion keys suggest that it is an OCI format. The image supports more platforms (combination of OS and CPU architectures). The attributes for OCI index is available here.

Single-platform image Build

At the beginning of the supply chain, we build platform-specific image with CI/CD jobs. Traditionally, the docker build process can only produce images for the platform where the build command run. For the process to work in multiple platforms, we’d need multiple build agents of different platforms. Each agent runs build process and pushes artifact to the registry (with “docker push”). At the end we create a fat manifest that combines the images for all platforms, with “docker manifest create” command. A blog post on Docker from April 2020 refers to this as the “hard way”.

Since then, Docker introduced a new client Docker Buildx, a CLI plugin that extends the docker command with the full support of the features provided by BuildKit builder toolkit. One such feature is the ability to produce multi-platform images in one command run. There is also a push switch that helps you generate manifest behind the scene. In my own testing, I am able to build image for both amd64 and arm64 on MacOS (M1).

Multi-platform image build

Take the colorapp Python application as a simple example, I followed this guide to generate corss-CPU-architecture build. First, we can examine if we already have a build instance that supports our desired platform and whether that is already selected:

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
vibrant_hypatia * docker-container
 vibrant_hypatia0 unix:///var/run/docker.sock running v0.11.3 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default docker
 default default running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux docker
 desktop-linux desktop-linux running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

In the example above, I’ve already got one named vibrant_hypatia that supports linux/arm64 and linux/amd64 and it is selected. If that’s not the crease, we can create a new build instance and tell Docker to use it. Then

$ docker buildx create --use

Then we can run the actual build command. In the build command we specify the platforms and tags. We also specify –push so we can push the entire binary at the same time.

$ docker buildx build --platform linux/amd64,linux/arm64 --push -t digihunch/colorapp:v0.2 ./colorapp

Note that in this single command, the building for both platforms is completed and pushed with the correct OCI-compliant manifest. There is no need to build them separately and work separately on the manifest. We can use the manifest command from last section to verify the image manifest.

The behaviour of defaulting to OCI-compliant image index is a change in the buildx version 0.10 in Jan 2023. With —provenance=false, we can control manifest format with oci-mediatypes=true. In buildx 0.10, the default value for provenance changed from false to true, which always makes OCI the image manifest format. This change of default, along with the glitch of “docker manifest” prior to v23.0.0, had pretty big impact and raised confusions (such as this bug report from Ubuntu, and the issue in this post). However, since I take OCI-compliant format as the standard, I do not have a problem with this change. We can verify the image manifest with:

$ docker manifest inspect --verbose digihunch/colorapp:v0.1. ## docker CLI version > 23

Platform-specific image

Our Dockefile looks like this, with a base image.

FROM public.ecr.aws/amazonlinux/amazonlinux:2
RUN yum update -y && yum install -y python3 && yum clean all && rm -rf /var/cache/yum
COPY serve.py ./
RUN chmod +x ./serve.py
CMD ["python3", "-u", "./serve.py"]

To build platform-specific images, we need the base image(amazonlinux) to support multi-arc too, so that the build process picked the correct platform specific image as base. We can verify this is the case by looking at the sha256 digest of the first layer of each colorapp variant. They are d78505 for the amd64 image, and 71343c for the arm64 image. We can find the same layer digest from the amazonlinux manifest.

We now have mutli-architecture images, as well as the index. Docker Hub shows both digests with their architecture, although unlike ECR it does not show whether the tag is a manifest list or a manifest:

In real life, our supply chain usually has multiple levels of base images or multiple base images. It is important to ensure platform-specific image are available by examining their manifests.

In the Dockerfile, we consider it a best practice to reference base image by digest instead of by tag. We just need to make sure the digest actually points to an image index (manifest list), with each manifest points to the platform-specific image.

Summary

Now we’ve build the first part of our supply chain with arm64 capability. We have build platform specific images that can use their own platform-specific base images. We’ve also created an OCI-compliant image manifest to announce the image supports multi-architecture. Next, we will examine how containerd on different CPU architectures consumes the platform-specific image.

AKS Lessons Learned 2 of 2

Sat, 18 Dec 2021 01:18:00 -0400

Even though Azure Kubernetes Service (AKS) is a managed service, building a cluster is not trivial. For help resources, I would start with the webinar “Configure Your AKS cluster with Confidence” from April 2021, which focuses on a set of working best practices (convention over configuration) but obviously not every recommendation suits every use case. For a deeper technical tour, the John Savill’s Technical Training channel has good videos (from 2020) on AKS overview, high availability and networking. Lastly, there is also an AKS checklist to remind you of the implementation details to consider.

All the references aside, I need to write down some gotchas from my implementation experience in the last two month.

Identity and Access Management

AKS is a special type of Azure resource in the sense that it manages other Azure services on user’s behalf. Therefore the access management needs to consider several aspects:

Access Type	Mechanisms involved	Example
User access Kubernetes API	Azure AD, Azure RBAC and Kubernetes RBAC. – Azure AD is for authentication – Azure RBAC for Kubernetes – Kubernetes RBAC	A user connects to Kube-API server using kubectl
AKS access other Azure resource	There are several identities that represents different components of AKS. For example, the AKS cluster, the node agent pool, and each add-on. The AKS cluster can be represented as a service principal, or managed identity (system assigned or user assigned). The node agent pool can be represented as a managed identity	AKS cluster connects to a VNet in a different resource group. (requiring cluster’s identity to have network contributor role on the target network resource group) AKS node agent pulls images from ACR (requiring the node agent pool’s identity to have ArcPull role on the target ACR)
Pod access other Azure resource	AAD-Pod Managed Identity	Business workload connects to managed database service such as PostgreSQL on Azure.
Pod access Kubernetes API	Access Kubernetes API using Service Account. This issue is solved completely by Kubernetes native mechanisms. Roles and ClusterRoles defines permissions. RoleBindings and ClusterRoleBindings associates Service Accounts with permissions	Workload access ConfigMap, Secret etc.

In the first access type, for RBAC with user to access Kubernetes API, there is an overlap between Azure RBAC and Kubnernetes RBAC. Azure RBAC has four built-in roles and three of them (reader, writer, admin) are namespaced. When you use Azure CLI to manage to assign one of those roles, the rolebinding and cluster rolebinding record stored in etcd will be updated accordingly.

RBAC mechansim	Use case
Azure RBAC for Kubernetes	Manage RBAC programmatically using Azure CLI, or infrastructure as code
Kubernetes RBAC	Manage RBAC declaratively with more granularity for all types of Kubernetes resources including CRD

For ease of operation it is advised to use Kubernetes RBAC whenever possible. Azure RBAC is still used for RBAC at the level of Azure resource but not at the level of Kubernetes resource.

In the the second access type, AKS cluster may use managed identity or service principal. Azure’s recommendation is managed identity over service principal. Managed Identity is a wrapper around Service Principal with less overhead. Managed Identity can be system assigned (created at the time of cluster creation), or user assigned (can be created ahead of time by Azure administrator and imported to the cluster’s context).

The second access type can be further broken down because there are several components in AKS that uses their own identities. I list the related managed identities as below:

Name	Purpose	BYO identity with Terraform
cluster identity	This identity represents the cluster	Specify in identity block in kubernetes_cluster resource
agent pool identity	This identity represents kubelet running in the agent pool	Specify in kubelet_identity block.
addon: azurepolicy	This identity represents azure policy addon to access the policy	N/A
addon: omsagent	This identity represents OMS agent to access monitoring etc	Specify in oms_agent_identity block
addon: secret	This identity represents to the secret addon, to access AKV	Specify in secret_identity block
addon: ingress gateway	This identity represents the ingress application gateway	ingress_application_gateway_identity block

By default, the system creates a new managed identity for each of the required identity above. For simplicity with identity management, we may create a managed identity and use it for all the occasions where an identity is needed and user assigned (BYO) identity is supported.

In the “az aks show” command return (a JSON document), the identity section (root level) reports the cluster identity, the identityProfile section (root level) reports the agent pool (kubelet) identity. Other identities such as omsagent, are reported in their own child document.

Node Networking

In Azure, a subnet can span across multiple availability zones. Therefore an AKS cluster can put its nodes on a single subnet with nodes evenly distributed across three AZs for high availability. The AZ of each node is indicated in the node label, and can be displayed with kubectl command.

Within a single AZ, a good practice to minimize latency between nodes is to place the nodes in a proximity placement group (PPG). However, only a single PPG can be associated with a node group. You can’t have three PPGs, one in each AZ, for a single subnet.

Pod Networking

The default Pod networking model is kubenet, which involves overlay network. Pod-to-Pod traffic across nodes requires Network Address Translation (NAT). To overcome this performance tax, Azure introduces Azure CNI which gives each Pod an routable IP address from the VNet’s CIDR. This requires advanced IP planning to prevent IP exhaustion. A risk introduced in Azure CNI is that all Pods are exposed on the V-net, which needs to be protected by Network Security Group and/or outbound firewall.

DNS

On the DNS side, when AKS cluster integrate with an external node network, it may create weird issues that are hard to troubleshoot. Another example is with DNS. If the V-Net uses an external DNS server (which is common for enterprises with hybrid network to use an on-premise DNS server), then the cluster creation failed with time-out with misleading error messages (for example, this comment). This is because the DNS name of the newly created cluster is not resolvable within the V-NET, which points to the on-prem DNS server. The fix to that is:

Use a BYO DNS zone (in Azure) for AKS cluster creation;
The AKS cluster will publish the A-record to the zone. To allow this to happen, the AKS cluster’s managed identity needs to have DNS contributor permission for the zone;
Configure the on-prem DNS for conditional forwarding to the DNS zone

This fix will allow AKS to resolve its name and therefore confirm its own creation. Here is a good blog about the DNS zone options.

Another potential issue introduced with the use of on-prem DNS server, is the resolution of single-label hostname of the nodes. This is not just an issue in the context of AKS. It is a generic issue with VMs running on a V-Net pointing to on-prem DNS, as explained in detail here.

In this situation, we should use the fully qualified hostname instead of single-label hostname. The fully qualified hostname with DNS suffix can help the on-prem server to configure conditional forwarding. For example, when the DNS suffix is *.internal.cloudapp.net, then forward it to Azure’s virtual internal DNS server 168.63.129.16 which can resolve the hostname.

If only the Pods need to resolve those FQDNs, then we can configure Core-DNS with conditional forwarding, which will take effect only at the cluster level without the need for changing the on-prem DNS. The Core-DNS configuration looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
 name: coredns-custom # this is the name of the configmap you can overwrite with your changes
 namespace: kube-system
data:
 cloudapp.override: | # you may select any name here, but it must end with the .override file extension
 log
 rewrite continue {
 name regex ^(.*[0-9]{7}-vmss[0-9]{6})$ {1}.internal.cloudapp.net
 answer name ^(.*)\.internal\.cloudapp\.net$ {1}
 }
 forward internal.cloudapp.net 168.63.129.16
 cloudapp.server: |
 internal.cloudapp.net:53 {
 errors
 log
 cache 10
 forward . 168.63.129.16
 }

Alternatively, use Pod DNS policy so that the Pod can use an external DNS server.

Initial Service Account

When a cluster is created, an Azure AD user or group can be assigned as cluster administrator. For a CI/CD pipeline to interact with the newly created cluster, a service account in Kubernetes is needed. Suppose we use Terraform to create the AKS cluster, we can create such service account automatically with the Kubernetes provider. This requires that the Terraform execution environment to have network access to the cluster. If the AKS cluster is located in a private network, then the agent where Terraform CLI runs should also be on the network. Alternatively, use Terraform Enterprise hosted in an environment with access to the cluster’s network.

Integration with Azure KeyVault

Azure Key Vault can store several types of secrets, key value pair, X509 keys and certificate. When AKV is integrated with an AKS cluster, the Kubernetes workload can access the secrets as mounted volumes, using CRD named SecretProviderClass. Further, they can be presented as Kubernetes Secret, using a Pod to sync between mounted content and Secret. AKV has three types of entries: key, certificate and secret (key-value). The certificate entry requires both key and certificate are stored, with optional certificate chain. In my opinion this is an over design. Unless we need Azure to manage the certificate (e.g. rotation) I would simply use secret to store my own X509 key and certificate.

Docker storage

Tue, 03 Nov 2020 20:22:00 -0400

Microservices are all about stateless and ephemeral workloads, and containers are great microservices. This may suggest that that Docker is all about ephemeral storage. In fact, Docker supports both non-persistent and persistent storage, such as database, kafka, etc.

Non-persistent storage is automatically created, alongside the container and is tied to the lifecycle of the container. On Linux system, it is /var/lib/docker/ as part of container. This is referred to as local storage.

Docker has a concept of volume, which is essentially a file or a directory. Volumes are for persistent data. they are de-coupled from containers and are not tied to the lifecycle of any container. Volume allows process in docker container to bypass the default uionFS, and stores file or directory on host machine. It also allows different containers to share data. You may mount a volume to a container. even if container is deleted, volume persists.

By default, Docker creates new volumes with the built-in local driver. Local volumes are only available to containers on the node they’re created on. There are also third-party drivers as plugins that provides advanced options to integrate external storage system with Docker. (NAS, SAN, etc)

There are more than 25 volume plugins that you can specify with -d switch, to cover all three categories of storage

Block storage tends to be high performance and good for small-block random access workloads.
File storage is high performance, shared amongs multiple containers with NFS or SMB protocols.
Object storage is good for long term storage of large data blobs that do not change frequently. It is often content addressable and relatively low performance.

Note that if you share volume with multiple containers, the application needs to worry about data collision.

You may use docker volume create command to create volume. Note that there is no quota management within docker so the partition needs to be managed at operating system level.

Implementation of Volume

Remember that Docker image is built on multi-layer file system. When we run a container, Docker places a read-write layer on top of the image, such that the active files in running container are all placed in this read-write layer. When container is deleted, so are the files. The file system in Docker is a pseudo file system implemented in unionFS. Volumes bypasses the uionFS and directly accesses the host file system.

When we create a Docker volume, Docker places the volume data to /var/lib/docker/volumes and under each directory named after volume, creates a directory _data, which is attached to the corresponding container.

You can even mount an NFS volume to container. Reference here.

We mentioned UnionFS a couple times so far. UnionFS is a light-weight, layered file system. It can mount the contents of multiple directories to the same directory, to form a single file system. User can use unionFS like a directory. It is the foundation of Docker image and container and enables saving of spaces.

There are three common types of union FS: AUFS, DeviceMapper, and OverlayFS.

AUFS file system

AUFS is the earliest driver that Docker uses for file system, most common in Ubuntu and Debian. To check if the system support AUFS, check out the documentation here.

AUFS is recommended in Ubuntu or Debian. For CentOS and Redhat, it needs to be installed and make sure the command above returns aufs. To configure AUFS, create file /etc/docker/daemon.json and add:

{
 "storage-driver":"aufs"
}

Then restart docker service. Run “docker info” and examine the Storage Driver section, as documented here.

AUFS layers multiple directories on a single Linux host and presents them as a single directory. These directories are called branches in AUFS terminology, and layers in Docker terminology. The unification process is referred to as a union mount.

Layers of a Ubuntu container

This section describes how the layers work and this section describes how it reads and writes files (Copy-on-Write (CoW) strategy to maximize storage efficiency and minimize overhead). CoW characterized AUFS.

AUFS has not been adopted in the Linux kernel mainline for lack of maintainability. So for CentOS, the recommended file system driver is devicemapper.

Devicemapper file system

Devicemapper is a technical framework to map physical block device to virtual block device, introduced since kernel 2.6.9. So it’s essentially different from AUFS. The Logical Volume Manager (LVM) in Linux is also implemented based on devicemapper.

The three critical components in devicemapper are:

mapped device: a virtual device that devicemapper provides to client
target device: the underlying physical device or a section of it.
map table: keeps track of the offset, range, etc between mapped and target devices.

Devicemapper uses target driver to block, filter, and forward I/O requests (e.g. Raid, encryption, think provisioning, etc). In thin provisioning, storage driver only assigns spaces that are needed. Docker uses snapshot technology in thin provisioning. This part of the documentation provides further details as to how device mapper works.

Ubuntu and busybox image layers

Devicemapper has to modes:

loop-lvm: in dev and test environment
direct-lvm: recommended in production

Here is the performance best practice. To configure devicemapper, create /etc/docker/daemon.json file and add:

{
 "storage-driver":"devicemapper"
 "storage-opts":[
 "dm.directlvm_device=/dev/xdf",
 "dm.thinp_percent=95",
 "dm.thinp_metapercent=1",
 "dm.thinp_autoextend_threshold=80",
 "dm.thinp_autoextend_percent=20",
 "dm.directlvm_device_force=false"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure direct-lvm mode is on.

Since devicemapper uses block device to store files, it is faster than directly operate on file system. It is adopted as default driver as unionFS for a long time, ensuring stable performance under Red Hat and CentOS.

OverlayFS file system

Earlier versions of OverlayFS (known as overlay driver) is not stable. Later version is known as overlay2, which is very stable and recommended in overlay2. It requires:

Docker version higher than 17.06.02;
Kernel version higher than 3.10.0-514 for CentOS and RHEL; or higher than 4.0 for other distributions of Linux;
Using with xfs file system with d_type turned on

In production environment, it is recommended to moutn /var/lib/docker to separate disk or partition, to prevent the directory getting full from impacting the host OS. The option pquota is recommended for mounting options in /etc/fstab.

To configure storage driver, create file /etc/docker/daemon.json, with the following content:

{
 "storage-driver":"overlay2",
 "storage-opts":[
 "overlay2.size=20G",
 "overlay2.override_kernel_check=true"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure storage driver is overlay2 and d_type is true.

The way overlay2 works is similar to AUFS, involving union mount process, with lowerdir, upperdir and merged. More details are here, including how overlay2 works with file read and file write (e.g. CopyOnWrite).

Today, overlay2 driver is officially recommended by Docker for its stability and performance, it should be used if all the conditions are met.

Host legacy application in Docker 1 of 2

Fri, 04 Sep 2020 16:24:00 -0400

This is my notes from containerizing a legacy application with Docker compose. We have to run multiple instances of our application because we’re unable to secure additional VMs for this single-VM education environment. The application is target of containerization, because it requires mass reconfiguration (around TCP port) to run multiple instances of the application. We want to use the same application configuration file for multiple containers, and map the TCP port to different groups of ports on the host, leveraging port mapping in Docker. On the other hand, the auxiliary services are not being containerized, such as Cassandra database and ElasticSearch because they can be shared for multiple application instances. In other words, we use Docker to isolate processes of the same application.

Prepare environment

The CentOS server needs to have docker-ce (through YUM) as well as docker-compose (direct download). They can be installed this way:

$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum install docker-ce docker-ce-cli containerd.io

$ curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo systemctl start docker

Our Docker registry is not publicly available. So we need to port the Docker image we need to remote server and load it into the local registry. We first examine the registry locally:

$ curl -XGET https://admin:password@docker.digihunch.com/v2/dhunch/tags/list | python -m json.tool

Once we identify the image, we export it to a tar file:

$ docker save docker.digihunch.com/dhunch > dhunch_image.tar

SCP the file to remote server and load it locally:

$ docker load -i /home/dhunch/dhunch_image.tar
$ docker image ls

We need to distinguish these commands:

docker save: saves an (non-running) image with all layers to file
docker export: saves a running or paused container to file
docker import: import the contents from a tarball to create a filesystem image, most used with docker export
docker load: load an image from a tar archive or STDIN, most used with docker save

Build docker-compose file

I need to cater to the customer environment with a newly create docker-compose file. The customer environment includes specific storage and networking configurations. Docker compose’s official documentation is here. We repeat the following commands for our troubleshooting:

$ docker-compose up -d
$ docker-compose exec -it dhunch1 bash
$ docker container ls

Once we start the container, the status might go unhealthy after it starts. The documentation explains two reasons you’re seeing an unhealthy container:

a single run of the command takes longer than the specified timeout
health check fails; the health check command will retry a number of times before it declares the container as unhealthy.

In our case, It is most likely because it does not pass a built-in health check mechanism. We need to understand where the health check was defined. There are four ways to enable health check:

Dockerfile instruction when building the image
Docker run command
Docker-compose or docker stack yaml file
Docker service

With #1, unfortunately, you can’t reverse engineer an image and view the Dockerfile that were used to built it and review the health check statement. What you can do is check docker events, or inspect the container, and go to the log files as specified under logPath section in the inspection result and look for HealthCheck section. We determined it is the case, then we can disable, or override the built-in healthcheck command from image, with a statement in docker compose.

For network interface, docker compose also allows us to specify MAC address for each container with mac_address keyword (for license key). MAC address generator are available on the internet.

EntryPoint vs CMD

The difference between EntryPoint and CMD is very important when launching container. Some literature also mentions RUN, which is only used when building a new layer of images so it is not relevant here (in the context of launching a container from image). EntryPoint and CMD has similar functionalities both allowing you to specify a command to run. The difference is whether they can be overwritten by command line arguments that user provide to docker-compose or docker run in an ad-hoc manner. As their names suggests, EntryPOINT means what is specified under it must be executed as it launches into the container, regardless of any adhoc commands. On the other hand, CMD is just an entry to save users from typing in a command every time they run docker compose or docker run. Should user prefer a different command, it can be provided as an explicit argument and it will be respected overwriting the pre-defined CMDentry in Dockerfile or command entry in docker-compose.yml.

Both CMD and EntryPoint supports shell and exec forms. More details here.

Choice of Networking

With single-host deployment, the containerized application needs to communicate with other existing, non-containerized service on host, such as database or elastic search. If docker uses host network, the container shares interface with the host and it does not have its own IP address. Host network removes isolation between container and host. This allows container to run the application that was licensed to the host based on MAC address. There is also no port mapping from container to host network. Container simply uses port on host, and is subject to the availability of TCP/UDP port on host.

We will have to use bridge network here. We can force MAC address the app container, and pre-generate license. For container to communicate with a service on host, through bridge network, there are two problems to address:

Container knows the IP of the host (layer-3 connectivity, ping);
Making host service available to container (layer-4 connectivity, telnet);

Docker creates its own interface for bridge network. If it’s an unnamed network, i.e. not explicitly declared under networks section in docker compose, then interface docker0 is used. If it’s a named network, then an interface name starting with br- is used.

The first problem is easier to address, we simply needs to IP address of the host on the interface. We can validate by pinging from container to host. Docker can also use host.docker.internal to reference the host. Unfortunately, this stopped working for linux since 18.09.3.

It is reportedly to be fixed in 20.04 and until it is available, we may add it to manual dns. The following command outputs the entry to add to /etc/hosts in container.

# ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}'

To do this automatically in docker compose, we need some tricks:

Store the Host IP in host environment variable ( use an export command)
Use compose to pass host environment variable to container environment variable
Have the container write its environment variable to /etc/hosts

The compose file will contain a line like this:

services:
 myenv1:
 image: alpine
 command: >
 sh -c "apk update &&
 echo $$HostDNSLine >> /etc/hosts &&
 bash"
 #network_mode: bridge
 environment:
 - HostDNSLine=${HOSTDNSREC}

Note ampersand might be mistakenly displayed as & in the above. Then we run it with the following:

# export HOSTDNSREC=$(echo 1.2.3.4 host.docker.internal) && docker-compose up

The second problem is harder to address because the service on host may not bind to docker’s interface. Some services such as ssh bind to all interfaces on host and you can telnet to port 22 with any IP address the host is associated with. This is however not the case for most other services, such as Cassandra or Elastic Search. They typically only bind to main interface, such as ens192, or eth0, and not to the docker interface. In order to make the service available to container, we either need to bind these services to the docker interface, or use iptables rules as an alternative.

Suppose it is a named network and Docker’s interface name is br-90ae024d5324, and the service on host listens to port 9042, we will need the following two commands from host:

# sysctl -w net.ipv4.conf.br-90ae024d5324.route_localnet=1

# iptables -t nat -A PREROUTING -p tcp -i br-90ae024d5324 --dport 9042 -j DNAT --to-destination 127.0.0.1:9042

Note that docker compose can configure to run sysctl in container but not from host. If there are multiple ports, we can turn this into a shell script:

#!/bin/bash
tcp_port_list="9200 9042 8302 8303 8304 8305 8306"

if_name=$(basename -a /sys/class/net/* | grep ^br- | head -1)
echo enable route localnet on interface $if_name
sysctl -w net.ipv4.conf.$if_name.route_localnet=1
for tcp_port in $tcp_port_list; do
 echo open host tcp port $tcp_port to interface $if_name
 iptables -t nat -A PREROUTING -p tcp -i $if_name --dport $tcp_port -j DNAT --to-destination 127.0.0.1:$tcp_port
done

echo $(ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}')

On the other hand, binding service to multiple interfaces usually require some re-configuration on the service itself. For example, if it is Elastic Search, we need to update [network.host] entry in elasticsearch.yml to include multiple IP addresses. For Cassandra, we need to update rpc_address to 0.0.0.0 or set rpc_interface in cassandra.yml.

Integration with storage

The application in the container need to store files to storage available to host, whether it is an NFS share or a block disk. We can use volume mapping with Docker compose, to map a path in container to a path presented to host as persistent volume. At this step, we might run into permission issues.

By default, containers initializes as root (uid=1) within the container, and the entrypoint script launches application as root. When application writes to persistent volume, files are written as root user. In the legacy non-container setup, we expect the application to write file as dhunch user. Moreover, NFS volume will not allow writing files as root (if the server has root squash configured). To address this, there are two approaches:

launch container as a regular user
launch container as root user, then have the entrypoint script launch application as regular user (dhunch)

For approach 1, we need to tell Docker to launch container as a regular user by specify the uid and gid for container to run application. We can specify the following envrionment variable in the compose yaml:

user: ${CURRENT_UID}

Then we assign the environment variable before running docker-compose:

# export CURRENT_UID=$(id -u dhunch):$(id -g dhunch) &amp;&amp; docker-compose up

This allows container to initialize as the regular user. However, if the entry point script needs to perform activities that requires root permission within the container, it will fail. For example, a regular user in container will not be able to update /etc/hosts;

With approach 2, we do not specify user in docker compose so container initializes as root. Then the entry point script launches application as regular user. For example, use su command before launch Java:

su dhunch -c "
exec java \
 -Xms512M -Xmx8192M \
 -Djava.io.tmpdir=$APP_HOME/var/tmp \
 -server \
 -XX:CompileCommandFile=$APP_HOME/etc/hotspot_compiler \
 -jar $APP_HOME/lib/jar/jruby-complete-*.jar \
 --1.9 \
 $APP_HOME/lib/rubybin/runapp.rb
"

Before doing this, we need to first create user dhunch within container, and the uid and gid must match those of the host. So that when container picks up dhunch user, it converts it to the correct uid.

groupadd -g 1011 dhunch
useradd -m -c 'regular user' -u 1011 -g 1011 dhunch

To further understand how uid and gid work, here are two posts with more information.

This user ownership setup will also work for NFS. To configure NFS, we need some extra client-side configurations in the container, as well as a special volume driver for NFS. Refer to this post.