ubuntu on Digi Hunch

AWS CDK example in Python – provision Kubernetes Nodes

Thu, 03 Dec 2020 21:14:00 -0400

There are two mechanisms to initialize instances in AWS. Cloud init and CloudFormation Init. Both are widely used and we discuss each of them in this posting. Then we will give an example of using AWS CDK in Python.

Cloud-Init

Cloud-Init is a service originally built for Ubuntu, as a bootstrapping utility to customize a Linux VM as it boots for the first time. It has evolved to be an industry standard multi-distribution method for cross-platform (public or private) cloud instance initialization, or even bare-metal installation. In cloud-init you can install packages and write files, or configure users and security. Because cloud-init is called during the initial boot process, there are no additional steps or required agents to apply your configuration.

Cloud-Init uses UserData, which is part of instance metadata. With AWS, you can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.

The cloud-config files are text files encoded in base64, with more details covered in the documentation here. cloud-init also works across distributions. For example, you don’t use apt-get install or yum install to install a package. Instead you can define a list of packages to install. cloud-init automatically uses the native package management tool for the distro you select.

CloudFormation Init

The cloudformation init mechanism does not only initialize instance, it also provides a mechanism for the resource being created to communicate with other resources. It allows an instance to emit signal to a different resource (via cfn-signal). It can also monitor changes to external resource and invoke local action (using cfn-hup with hooks). CloudFormation Init requires several components to work together:

The cloudformation resource should have metadata. The metadata must have a key AWS::CloudFormation::Init in which configsets are declared.
The UserData must use helper script (cfn-init) to invoke configuration jobs
The UserData can use helper script (cfn-signal) to signal with a CreationPolicy or WaitCondition (of the same or different resource), so you can synchronize other resources in the stack when the prerequisite resource or application is ready.
The cfn-hup service on the instance can be configured, to check for updates to metadata and execute custom hooks when changes are detected.

Comparison

While there are overlaps between the functionalities of Cloud Init and CloudFormation Init, the major difference is the latter support extended features (signal, update, etc); whereas the former is vendor neutral. The table below summarized some the differences:

	Cloud Init	CloudFormation Init
Works on	Linux OS distribution	CloudFormation resource, in combination with cfn helper scripts
Usecase	Initialization only	Both initialization and resource update
Trigger	cloud-init systemd service	Initial: from UserData Update: by cfn hook
Adoption	Multiple cloud vendors and bare-metal system	AWS cloud instances
Action Playbook	/var/lib/cloud/ Instance Metadata -> User Data, encoded in base 64	CloudFormation Resource -> Metadata section -> AWS::CloudFormation::Init -> configSets and configs
Log file and stdout	/var/log/cloud-init.log /var/log/cloud-init-output.log	/var/log/cfn-init.log /var/log/cfn-init-cmd.log

Comparison between cloud-init and cfn-init

AWS Cloud Development Toolkit (CDK)

Traditionally, AWS CloudFormation uses template in YAML or JSON for resource declaration. As the size of system grows, the amount of resource involved grows quickly and the size of such declaration file may grow beyond manageable. Nested stacks and export of output are mechanisms designed to combat the template sprawling, but to a very limited extent. Two reasons it is hard to control template size are:

In declarative statements, each line carries very small piece of information. Without flow controls such as if-else, loops, object oriented structure, the level of code reusability is very low;
Some auxiliary resources (such as AWS::EC2::VPCGatewayAttachment) must be declared explicitly, even though they are insignificant to the stack functionality

To address these challenges, AWS introduced AWS CDK (cloud development tookkit), which supports multiple languages (JavaScript, TypeScript, Python, Java, and C#). The CDK was natively developed in TypeScript, which is supposed to be the preferred development language. A tutorial is provided here, with detailed API documentation here.

To install aws cdk and create a hello world project, follow this example.

An Example in Python

I have created an example for AWS CDK in Python. The purpose is to create some EC2 instance to complete a lab for Kubernetes (without using managed EKS service). The example provisions the followings:

VPC, a public and private subnets, Internet and NAT gateways;
Relevant security groups and permissions
Bastion host, public instances in public subnet
Private instances in private subnet, with public route through NAT gateway

The private instances forms a cluster for Kubernetes lab. We will use kubespray to initialize these instances. During the bootstraping, we download kubespray, install ansible, etc.

Here is the code repo for this example. With CloudFormation only, the single template could go well beyond 1000 lines. With CDK, the code are organized into several different python files, each representing a stack. The stacks can be stood up with command:

cdk deploy vpc-stack
cdk deploy security-stack
cdk deploy bastion-stack
cdk deploy private-stack

Although the documentation in Python is available, there are generally not a lot of examples built out on the Internet. The pypi site provides some Python specific examples for each module (e.g. core and aws-ec2). Given these libraries are available for only 2 years (since 2018), many advocates TypeScript as the language. However, I have implemented some CloudFormation init, used helper script, and UserData in this example, without running into any language specific issues.It should be noted that the EC2 instance by default will call cfn-init. So there is no need to explicitly run cfn-signal or cfn-init from user data in python code (example). This can be verified in file /var/lib/cloud/instances/<instance-id>/user-data.txt which automatically includes the following lines:

# fingerprint: e1b32ead13878deb
(
 set +e
 /opt/aws/bin/cfn-init -v --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 -c config_set_1,config_set_2
 /opt/aws/bin/cfn-signal -e $? --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456
 cat /var/log/cfn-init.log >&2
)

In addition to Python, AWS CDK also supports other languages. In the next post, we will discuss use of CDK in Typescript.

Docker storage

Tue, 03 Nov 2020 20:22:00 -0400

Microservices are all about stateless and ephemeral workloads, and containers are great microservices. This may suggest that that Docker is all about ephemeral storage. In fact, Docker supports both non-persistent and persistent storage, such as database, kafka, etc.

Non-persistent storage is automatically created, alongside the container and is tied to the lifecycle of the container. On Linux system, it is /var/lib/docker/ as part of container. This is referred to as local storage.

Docker has a concept of volume, which is essentially a file or a directory. Volumes are for persistent data. they are de-coupled from containers and are not tied to the lifecycle of any container. Volume allows process in docker container to bypass the default uionFS, and stores file or directory on host machine. It also allows different containers to share data. You may mount a volume to a container. even if container is deleted, volume persists.

By default, Docker creates new volumes with the built-in local driver. Local volumes are only available to containers on the node they’re created on. There are also third-party drivers as plugins that provides advanced options to integrate external storage system with Docker. (NAS, SAN, etc)

There are more than 25 volume plugins that you can specify with -d switch, to cover all three categories of storage

Block storage tends to be high performance and good for small-block random access workloads.
File storage is high performance, shared amongs multiple containers with NFS or SMB protocols.
Object storage is good for long term storage of large data blobs that do not change frequently. It is often content addressable and relatively low performance.

Note that if you share volume with multiple containers, the application needs to worry about data collision.

You may use docker volume create command to create volume. Note that there is no quota management within docker so the partition needs to be managed at operating system level.

Implementation of Volume

Remember that Docker image is built on multi-layer file system. When we run a container, Docker places a read-write layer on top of the image, such that the active files in running container are all placed in this read-write layer. When container is deleted, so are the files. The file system in Docker is a pseudo file system implemented in unionFS. Volumes bypasses the uionFS and directly accesses the host file system.

When we create a Docker volume, Docker places the volume data to /var/lib/docker/volumes and under each directory named after volume, creates a directory _data, which is attached to the corresponding container.

You can even mount an NFS volume to container. Reference here.

We mentioned UnionFS a couple times so far. UnionFS is a light-weight, layered file system. It can mount the contents of multiple directories to the same directory, to form a single file system. User can use unionFS like a directory. It is the foundation of Docker image and container and enables saving of spaces.

There are three common types of union FS: AUFS, DeviceMapper, and OverlayFS.

AUFS file system

AUFS is the earliest driver that Docker uses for file system, most common in Ubuntu and Debian. To check if the system support AUFS, check out the documentation here.

AUFS is recommended in Ubuntu or Debian. For CentOS and Redhat, it needs to be installed and make sure the command above returns aufs. To configure AUFS, create file /etc/docker/daemon.json and add:

{
 "storage-driver":"aufs"
}

Then restart docker service. Run “docker info” and examine the Storage Driver section, as documented here.

AUFS layers multiple directories on a single Linux host and presents them as a single directory. These directories are called branches in AUFS terminology, and layers in Docker terminology. The unification process is referred to as a union mount.

Layers of a Ubuntu container

This section describes how the layers work and this section describes how it reads and writes files (Copy-on-Write (CoW) strategy to maximize storage efficiency and minimize overhead). CoW characterized AUFS.

AUFS has not been adopted in the Linux kernel mainline for lack of maintainability. So for CentOS, the recommended file system driver is devicemapper.

Devicemapper file system

Devicemapper is a technical framework to map physical block device to virtual block device, introduced since kernel 2.6.9. So it’s essentially different from AUFS. The Logical Volume Manager (LVM) in Linux is also implemented based on devicemapper.

The three critical components in devicemapper are:

mapped device: a virtual device that devicemapper provides to client
target device: the underlying physical device or a section of it.
map table: keeps track of the offset, range, etc between mapped and target devices.

Devicemapper uses target driver to block, filter, and forward I/O requests (e.g. Raid, encryption, think provisioning, etc). In thin provisioning, storage driver only assigns spaces that are needed. Docker uses snapshot technology in thin provisioning. This part of the documentation provides further details as to how device mapper works.

Ubuntu and busybox image layers

Devicemapper has to modes:

loop-lvm: in dev and test environment
direct-lvm: recommended in production

Here is the performance best practice. To configure devicemapper, create /etc/docker/daemon.json file and add:

{
 "storage-driver":"devicemapper"
 "storage-opts":[
 "dm.directlvm_device=/dev/xdf",
 "dm.thinp_percent=95",
 "dm.thinp_metapercent=1",
 "dm.thinp_autoextend_threshold=80",
 "dm.thinp_autoextend_percent=20",
 "dm.directlvm_device_force=false"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure direct-lvm mode is on.

Since devicemapper uses block device to store files, it is faster than directly operate on file system. It is adopted as default driver as unionFS for a long time, ensuring stable performance under Red Hat and CentOS.

OverlayFS file system

Earlier versions of OverlayFS (known as overlay driver) is not stable. Later version is known as overlay2, which is very stable and recommended in overlay2. It requires:

Docker version higher than 17.06.02;
Kernel version higher than 3.10.0-514 for CentOS and RHEL; or higher than 4.0 for other distributions of Linux;
Using with xfs file system with d_type turned on

In production environment, it is recommended to moutn /var/lib/docker to separate disk or partition, to prevent the directory getting full from impacting the host OS. The option pquota is recommended for mounting options in /etc/fstab.

To configure storage driver, create file /etc/docker/daemon.json, with the following content:

{
 "storage-driver":"overlay2",
 "storage-opts":[
 "overlay2.size=20G",
 "overlay2.override_kernel_check=true"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure storage driver is overlay2 and d_type is true.

The way overlay2 works is similar to AUFS, involving union mount process, with lowerdir, upperdir and merged. More details are here, including how overlay2 works with file read and file write (e.g. CopyOnWrite).

Today, overlay2 driver is officially recommended by Docker for its stability and performance, it should be used if all the conditions are met.

Setup WSL2 (and Docker) on Windows 10

Tue, 02 Jun 2020 22:01:00 -0400

This is not for Linux snobs, but rather for those who are stuck with a Windows work laptop, have to deal with Linux on a daily basis, and are not a fan of PuTTY. This posting provides the steps to setup Windows 10 so you get a work environment closer to a Linux one. The environment to begin with should be Windows 10 version 2004 and above in order to use WSL2. Here’s the comparison between WSL and WSL2. This presentation is a great deep dive into how WSL2 works. The architecture diagram below is from that presentation. Note that WSL2 operates on a “true” Linux Kernel, therefore giving WSL2 the ability to run Docker.

Install WSL2

WSL2 was officially released in Windows 10 version 2004 (build 19041 or higher) and we will use Ubuntu 20.04 LTS. If you’re upgraded from older version of Windows 10 you will need to upgrade. The steps are:

Install Ubuntu 20.04 LTS from Microsoft Store, which requires Windows components “Windows Subsystem for Linux” and “Virtual Machine Platform”; or, if you already have an older version of Ubuntu such as 18.04, you will need to run a distribution upgrade with “sudo apt-get dist-upgrade”;
Upgrade Linux Virtual Machine from WSL to WSL2, following the official instruction here; Once completed, use this command to confirm the WSL version installed: wsl.exe –list -v

There’s also plenty of Youtube videos with step-by-step instruction on upgrading to WSL2 and its advantage over WSL. Once you’re there, you will realize one problem: the command terminal for Ubuntu is ugly:

The colour looks awful and it does not support multi-tabs. If you want better palette, there are a number of options. Check out ColorTool project and iTerms2 Colors project. The former gives you away to configure color scheme (from command prompt but takes effect in WSL as well) and the latter gives you rich choices of color schemes. My personal favourite is “Banana Blueberry”. The command to set color scheme is:

C:\ColorTool.exe -b "iTerm2-Color-Schemes-master\schemes\Banana Blueberry.itermcolors"

Windows Terminal

For a better terminal, consider Windows Terminal from Microsoft Store. Windows terminal is an open-source project and still a little glitchy as of June 2020. However it is heading to the right direction. Once installed, you may customize it by clicking settings:

A json file will open and you may edit it for customization, for example, you can adjust each profile (e.g. PowerShell, Windows command or WSL); you can specify default profile and default open location.

// This file was initially generated by Windows Terminal 1.0.1401.0
// It should still be usable in newer versions, but newer versions might have additional
// settings, help text, or changes that you will not see unless you clear this file
// and let us generate a new one for you.

// To view the default settings, hold "alt" while clicking on the "Settings" button.
// For documentation on these settings, see: https://aka.ms/terminal-documentation
{
 "$schema": "https://aka.ms/terminal-profiles-schema",

 "defaultProfile": "{2c4de342-38b7-51cf-b940-2309a097f518}",

 // You can add more global application settings here.
 // To learn more about global settings, visit https://aka.ms/terminal-global-settings

 // If enabled, selections are automatically copied to your clipboard.
 "copyOnSelect": false,

 // If enabled, formatted data is also copied to your clipboard
 "copyFormatting": false,

 // A profile specifies a command to execute paired with information about how it should look and feel.
 // Each one of them will appear in the 'New Tab' dropdown,
 // and can be invoked from the commandline with `wt.exe -p xxx`
 // To learn more about profiles, visit https://aka.ms/terminal-profile-settings
 "profiles":
 {
 "defaults":
 {
 // Put settings here that you want to apply to all profiles.
 },
 "list":
 [
 {
 "guid": "{2c4de342-38b7-51cf-b940-2309a097f518}",
 "hidden": false,
 "name": "Ubuntu",
 "source": "Windows.Terminal.Wsl",
 "startingDirectory": "//wsl$/Ubuntu/home/myuser/"
 },
 {
 // Make changes here to the powershell.exe profile.
 "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
 "name": "Windows PowerShell",
 "commandline": "powershell.exe",
 "hidden": false
 },
 {
 // Make changes here to the cmd.exe profile.
 "guid": "{0caa0dad-35be-5f56-a8ff-afceeeaa6101}",
 "name": "Command Prompt",
 "commandline": "cmd.exe",
 "hidden": true
 },
 {
 "guid": "{b453ae62-4e3d-5e58-b989-0a998ec441b8}",
 "hidden": true,
 "name": "Azure Cloud Shell",
 "source": "Windows.Terminal.Azure"
 }
 ]
 },

 // Add custom color schemes to this array.
 // To learn more about color schemes, visit https://aka.ms/terminal-color-schemes
 "schemes": [],

 // Add custom keybindings to this array.
 // To unbind a key combination from your defaults.json, set the command to "unbound".
 // To learn more about keybindings, visit https://aka.ms/terminal-keybindings
 "keybindings":
 [
 // Copy and paste are bound to Ctrl+Shift+C and Ctrl+Shift+V in your defaults.json.
 // These two lines additionally bind them to Ctrl+C and Ctrl+V.
 // To learn more about selection, visit https://aka.ms/terminal-selection
 { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" },
 { "command": "paste", "keys": "ctrl+v" },

 // Press Ctrl+Shift+F to open the search box
 { "command": "find", "keys": "ctrl+shift+f" },

 // Press Alt+Shift+D to open a new pane.
 // - "split": "auto" makes this pane open in the direction that provides the most surface area.
 // - "splitMode": "duplicate" makes the new pane use the focused pane's profile.
 // To learn more about panes, visit https://aka.ms/terminal-panes
 { "command": { "action": "splitPane", "split": "auto", "splitMode": "duplicate" }, "keys": "alt+shift+d" }
 ]
}

With these configuration, we have a quite comfortable Linux work environment on Windows.

Update to Zsh and beautify it

I’m no so big a fan of zsh (for lack of default wildcard support) but I do like one of its themes. We can install zsh on WSL2 (assuming Ubuntu distribution). This is a good instruction. I do like the theme name agnoster. However, the theme displays username at the beginning of the prompt, which takes a lot of screen real estate if your username is long. To remove it, the trick is to add a line in ~/.zshrc, as answered in this thread.

Docker runtime

There is a lot of fun to have in WSL2 on Windows 10. For example, you can configure Docker runtime on Windows according to this post. However, you cannot route traffic to the container in the absence of docker0 bridge for Docker on WSL2, as indicated in the known limitations.

Update: Docker also brings a Kubernetes cluster named docker-desktop. This allows you configure Kubernetes cluster, or install Helm and Rancher (instruction) for cluster management.

Dual-Boot?

With WSL2 on top of a real Linux kernel, I do not find a need for a dual-boot system on my workstation. To be fair, only in the following scenarios should one consider installing a dual-boot system.

You need to work on a specify version of Linux Kernel. The kernel provided by Microsoft is called “microsoft-standard-WSL2”, and its pretty up to date (5.x)
You need to run GUI applications on Linux. Update: Microsoft has started developing GUI application support on WSL2, even though it usually takes time to mature.
You need to work on a distribution not available in Windows Store. WSL2 provides a kernel and it’s up to developers to provide Linux distros in Windows store. Popular (and official) ones are: Ubuntu 20.04 LTS by Canonical Group, Debian by the Debian project, SUSE Linux Enterprise Server by SUSE, Kali Linux.

Certificate Issuers

I have to use WSL2 mostly because I’m forced to use a Windows laptop on my contracts. That also means that the Windows laptop has been configured to corporate firewall’s packet inspection. When that is the case, they corporate IT usually automatically imports a firewall certificate to the Windows trust store. However, the WSL is missing this certificate.

The symptom would be that you cannot run certain curl command. For example, when I do the followings:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This error message basically means that the website certificate cannot be trusted. To view the certificate, use Open SSL command:

openssl s_client -connect dl.k8s.io:443 -servername dl.k8s.io | openssl x509 -text -noout | less

In my case, I noticed that the certificate is a Zscaler issued certificate (firewall). While running the same command from my personal device returns a Goolge Trust issued certificate. This indicates that the certificate is changed by the proxy managed by the corporation. To fix this I need to export the certificate from Windows, and then import it in WSL. The answer in this post has good detailed steps.

CentOS – Remove Swap safely

Sat, 14 Apr 2018 16:15:43 -0400

If the default installation has swap on, you will see it from block device list:

[root@server /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 500G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 499G 0 part
 ├─centos-root 253:0 0 50G 0 lvm /
 ├─centos-swap 253:1 0 7.9G 0 lvm [SWAP]
 └─centos-home 253:2 0 441.1G 0 lvm /home
sr0 11:0 1 1024M 0 rom
[root@server /]# free -h
 total used free shared buff/cache available
Mem: 7.6G 217M 7.2G 11M 206M 7.2G
Swap: 7.9G 0B 7.9G

Many installations require swap to be off for performance reasons (although some advocate turning off swappiness of the application, instead of removing swap partition from operating system, which is a separate topic). This can be turned off by a simple command:

swapoff -a

Then if you run free command after rebooting, you will see the Swap is set to 0G. Great. But lsblk and df commands still shows the space being used by swap. This disk space is not being used at all.

[root@c7v-vitvcast02 ~]# free -m
 total used free shared buff/cache available
Mem: 7802 237 7380 11 185 7333
Swap: 0 0 0
[root@c7v-vitvcast02 ~]# lvdisplay
 --- Logical volume ---
 LV Path /dev/centos/swap
 LV Name swap
 VG Name centos
 LV UUID 7zHtvT-zGYn-sNBc-JRlT-nTfU-O9hN-r07y3J
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:21 -0500
 LV Status available
 # open 0
 LV Size <7.88 GiB
 Current LE 2016
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:1

 --- Logical volume ---
 LV Path /dev/centos/home
 LV Name home
 VG Name centos
 LV UUID 4XMTa7-uR44-qH0u-Oag4-PDyK-Z1BC-3oUSiz
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 0
 LV Size <441.12 GiB
 Current LE 112926
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:2

 --- Logical volume ---
 LV Path /dev/centos/root
 LV Name root
 VG Name centos
 LV UUID tkMCM4-cWxV-FaAC-hgJx-r1vS-DiY5-fyZvjt
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 1
 LV Size 50.00 GiB
 Current LE 12800
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:0

We can then remove the line for swap from /etc/fstab, this will keep the OS from mounting it upon reboot. However, the logical volume for swap is there, as a partition that’s not exposed.

If you need to reclaim space from swap partition, delete the logical volume:

lvremove /dev/centos/swap
Do you really want to remove active logical volume centos/swap? [y/n]: y
 Logical volume "swap" successfully removed

Now, using command lvdisplay or lvs, the swap space is not displayed anymore. So I rebooted the server in the hopes that the swap space is not being presented and everything is hunky-dory. But no… the server doesn’t boot up at all. It took a long time at the splash screen only to enter dracut, where /boot isn’t present.

On further reading, this is because the grub file is still referencing /dev/centos/swap somewhere. I need to somehow get to the grub file and fix that before it can reboot again. Specifically, I pressed “e” at the menu, which gave me an opportunity to edit grub file. From there I removed section for swap. Then I saved it with “Ctrl + X” (as suggested on the screen) so the OS booted into normal mode. Alternatively, I entered single user mode in order to get to the point I can edit the grub file.

Once booted, I have to make change to the actual grub file at /boot/grub2/grub.cfg so the change persists. In the file, locate the lines starting with linux16 and remove the section “rd.lvm.lv=centos/swap”. There are two appearances of this section on my machine. Then the server will boot just normally.

In summary, the clean way to turn off swap on CentOS involves the following steps:

turn off swap
Remove swap mount from /etc/fstab (and umount)
remove the logical volume for swap
remove reference to swap from grub

Out of those steps, 3 and 4 are needed to reclaim swap spaces. They must be done together. If you rebooted the server before step 4. The server will not be able to boot complaining about missing swap.