virtualization on Digi Hunch

Docker components

Wed, 28 Oct 2020 20:23:00 -0400

The previous post about virtualization and containerization brought up some underlying technologies which Docker build containers on, including:

namespaces – a Linux kernel mechanism to isolate resources. It allows a process to run within an isolated environment (mnt, pid, net, ipt, uts, user, cgroup)
cgroups – a Linux kernel mechanism to limit resource usage of a process or process group
unionFS (this will be further discussed under Docker storage)

In this post we further discuss the components in Docker, the dominant and popular player in container technology, as shown in the diagram below:

The component names can be seen under docker install directory. It consists of three groups:

Docker related: docker, dockerd, docker-init and docker-proxy
Containerd related: containerd, containerd-shim and ctr
Container runtime: runc

Now we discuss each group:

Docker-related components

docker is just an implementation of docker client, it supports commands to achieve all functions between client and server. Alternatively, user may use REST API, or Docker SDK to communicate with Docker server.

dockerd is the server process, to receive requests from docker (client), SDK library or REST API caller. It executes the request and returns status to client. There are three ways for docker (client) to communicate with dockerd.

By Unix Socket (unix://socket_path). The default socket path used by dockerd is /var/run/docker.sock, which is why only root can use docker after installation.
TCP request (tcp://host:port). It is recommended to configure TLS communication in production environment.
By file descriptor (fd://) used in systemd service.

Unix socket is the default communication method. To allow remote access to dockerd, use -H to specify HOST and PORT when starting dockerd.

docker-init is used by Docker as PID 1 process for containers, in case it needs to recycle zombie containers. To use this, specify –init when running container.

docker-proxy is used for port mapping. When you use -p switch with docker run, this docker-proxy is the service that maps the container port to host port. It does so by modifying the iptables nat rule.

Containerd related components

containerd component was separated from dockerd since Docker 1.11, in compliance with OCI standard. It is responsible for life cycle management of containers, it also manages images (e.g. pulling from repo), request from dockerd to call runc, storage and network resources.

dockerd uses UNIX socket to send request to containerd. The default socket path for containerd is /run/containerd/containerd.sock. containerd execute the task and return status to dockerd. You may also directly use containerd to manage containers.

ctr (containderd-ctr) is the client of containerd, mostly used only in development and testing. If the environment does not have dockerd, then you can use ctr as client, to send request directly to containerd.

containerd-shim is used to decouple containerd from the containers. containerd-shim is the parent process of containers. This is so that restarting containerd does not impact the running containers.

Container runtime

runc is a standard implementation of OCI container runtime. It is a command-line tool to create and run containers.

Host legacy application in Docker 2 of 2

Thu, 22 Oct 2020 17:54:00 -0400

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

Application service	TCP port to bind
Business service A	8030
Business service B	8040
Business service C	8050
……	…

TCP port requirement

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
The OS needs to host 6 processes of the same application all running at the same time;
The OS does not allow multiple processes to bind to a single TCP port (duh!);
It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

OS	Container ID	Application Service	container port	published port
Host CentOS	Container 1 (Instance #1)	Business Service A	8030	9301
		Business Service B	8040	9401
		Business Service C	8050	9501
	Container 2 (Instance #2)	Business Service A	8030	9302
		Business Service B	8040	9402
		Business Service C	8050	9502
	Container 3 (Instance #3)	Business Service A	8030	9601
		Business Service B	8040	9602
		Business Service C	8050	9603

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
 dapp1:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp1
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9301:8030 # BUSINESS SERVICE A
 - 9401:8040 # BUSINESS SERVICE B
 - 9501:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:cf
 environment:
 - INSTANCE_TAG=dapp1 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp2:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp2
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9302:8030 # BUSINESS SERVICE A
 - 9402:8040 # BUSINESS SERVICE B
 - 9502:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d0 
 environment:
 - INSTANCE_TAG=dapp2 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp3:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp3
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9601:8030 # BUSINESS SERVICE A
 - 9602:8040 # BUSINESS SERVICE B
 - 9603:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d1 
 environment:
 - INSTANCE_TAG=dapp3 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
networks:
 vcnet:
 driver: bridge
 driver_opts:
 com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
- Initialization work that cannot be done in Dockerfile, such as setting environment variable
- Launch the application, including pointing log file to stdout
- Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
build_image.sh: this script makes the image build process smoother
- check if image to build already exist, and ask permission to delete the existing image if so;
- build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

	Dockerfile instruction for container	Permission issue during mount by docker-compose
Bad config	RUN mkdir -p /var/lib/dapp && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.
Good config	RUN mkdir -p /var/lib/dapp/dcontainer/archive && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue.

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.

Virtualization 4 of 4 – Networking

Fri, 21 Aug 2020 21:53:32 -0400

Virtual LAN (VLAN)

Although VLAN emerged before virtualization and is technically not part of virtualization topic. I’d just like to start from here as a refresher. Suppose we have computers from finance department and computers from sales department all connected to a single layer-2 switch. There are at least three problems: 1) too many devices on the same broadcast domain causes traffic congestion; 2) security can be compromised and 3) each department might have several physical locations. We introduce multi-layer switch to address these with two main features: 1) the VLAN feature can map ports to logical networks, so that all hosts are physically connected to a single switch, but logically to their own network (VLAN) 2) the SVI (switch virtual interface) feature allows inter-VLAN routing at layer 3. Such multi-layer switch is sometimes referred to as layer-3 switch.

VLAN is local to a switch and a tag is required in ethernet frame in order to pass VLAN info across switches. This link between switches is called a trunk. IEEE 802.1q (aka dot1q) is the networking standard for VLAN, which standardizes the tagging traffic between switches to tell which traffic belongs to which VLAN. The dot1q trunk (aka dot1q link) provides VLAN IDs fro frames traversing between switches. A trunk can be configured between two switches, or between a switch and a router. Trunking is the process of traversing different VLAN traffic over the trunk. The ports on each switch need to be configured to enable trunking. While Cisco calls such ports trunk port, others call them tagged port. Their function is to add the VLAN tag to ethernet frame. In contrast, regular ports that send and receive frames without VLAN tag are called access port or untagged port. Trunk port carries traffic for multiple VLANS whereas access port carries traffic for a single VLAN. A network device connected to access port has no idea about its VLAN belonging. VLAN creation and management are the responsibility of the switch. Common trunking protocols include VTP (VLAN trunking protocol) and DTP (dynamic trunking protocol)

This video and this video have good explanations on VLAN.

Virtual Extensible LAN (VXLAN)

VXLAN is an overlay protocol. Remember that in the standard TCP/IP stack, you normally encapsulate layer-3 IP datagram into a layer-2 ethernet frame. With the VXLAN encapsulation technique however, layer-2 frames can be encapsulated within layer-4 UDP packet.

VXLAN allows you to stretch layer 2 connection over an intervening layer 3 network. VXLAN tunnel endpoints (VTEPs) are the endpoint device that terminate VXLAN tunnels and it can be either virtual or physical switch ports. It encapsulate VXLAN traffic and de-encapsulate the traffic when it leaves the VXLAN tunnel.

The VXLAN encapsulation includes the followings:

Outer Ethernet Header (source and dest MAC for underlay VTEPs)
Outer IP header (source and dest IP on underlay network)
Outer UDP header (including source and dest ports, 4789 default)
VXLAN Header (including VNI)
Inner Ethernet Frame (with source and dest MAC for overlay interfaces)

The VNI (VXLAN network identifier, aka VNID) included in the VXLAN header is 24-bit long. It is conceptually similiar to VLAN ID in VLAN but only with 12-bit length.

VXLAN Enapsulation

The VXLAN protocol is documented in RFC7348. Its specification was originally created by VMware, Cisco and Arista. As it became more common in network virtualization (with data centre virtualization, and application containerization) several other players joined the list of contributors and they manufacture switches that support VXLAN. This is a section on VXLAN from the document of Huawei Cloud Engine 5800 switch.

Open vSwitch is an example of a software-based virtual network switch that supports VXLAN overlay networks.

In summary, the main benefits of VXLAN over VLAN are:

VXLAN scales up to 16 million logical networks, thanks to the 24-bit length of VNI
VXLAN supports layer 2 adjacency across IP networks. A VM belonging to existing layer 2 domain can be created in different data centre (where more computing resources are available), without being constrained by layer 2 boundaries, or being forced to create geographically stretched layer 2 domains (stretched VLAN).

Virtual Machine Networking

For VM to connect to each other, within or across hosts, we need not only vNIC on VM, but also vSwitch to connect vNICs. A vSwitch (aka bridge) is a logically defined layer-2 device that passes frames between vNICs. On the same host, vNICs are directly connected to vSwitch, which is then connected to the physical NIC. Each vSwtich connects a broadcast domain. When we setup vNIC there are three modes:

Bridged networking: VM connect to outside network using host’s physical NIC, which acts as a bridge between vNIC and outside network. The VM is a full participant in the network as if it were a physical computer on the network. i.e. it obtain IP addressing information from a DHCP server on the outside (physical) network. The VM’s IP address is also visible and directly accessible by other computers on the network. bridge networking is common for servers as VMs.

NAT networking: The VM relies on the host to act as NAT device to make outgoing network connection. The IP address of VM is assigned by virtual DHCP server on host. The guest VMs form a private network and computers on the outside network are external. The host translates private IP address into the host’s IP address on the way out, and listens for returning traffic. Outside network sees traffic from VM guest as if it were from the host. This network mode is common when the VMs are mainly used as a client workstation.

Host-only networking: creates a network that is completely contained within the host computer. The vSwtich is the hub of the private network and the physical NIC on the host is not involved. The VM will not have access to the outside network. This mode is useful when the VMs needs to be isolated from outside network, and only need to communicate with peers on the same host.

The difference between NAT networking and host-only networking is the exposure of VM guest to external network. All of these networking modes are available on VMWare fusion, for example.

Advanced virtualization platform such as vSphere usually support multi-hosting. Multiple host can also be configured to form a distributed vSwitch, such as vSphere Distributed Switch, in addition to standard switches.

Docker Networking

I had a brief on Docker network covering three modes. Out of the three modes, single-host bridge network is the equivalent of host-only networking. MacVLAN driver is similar to bridged networking, in the sense that container may connect to external network, using host NIC as a bridge. However, the external network is still bound by physical location. This is when overlay network comes in handy.

CNM and CNI

In the Docker networking, container needs to map its own port to host, of which the port resource is implemented by IP tables, which limits the scale and performance of the solution. Also, those networking modes do not address the problem of multi-host networking. As multi-host networking became a real need for containers, the industry started looking into different solutions. Container project favour a model where networking is decoupled from the container runtime. This also greatly improves application mobility. In this model, networking is handled by a ‘plugin’ or ‘driver’ that manages the network interface, and how the containers are connected to the network. The plugin also assigns the IP address to the container’s network interfaces. In order for this model to succeed, there needs to be a well-defined interface or API between the container runtime and the network plugins.

Docker, the company behind the Docker container runtime, came up with the Container Network Model (CNM). Around the same time, CoreOS, the company responsible for creating the rkt container runtime, came up with the Container Network Interface (CNI). Kubernetes originally seeks to use CNM for its plugins, but they eventually decided to go with CNI. The primary reason was that CNM was still seen as something designed with Docker container runtime in mind and was hard to decouple from it. After this decision, several other open source project also turned to CNI for their container runtimes.

This article expands further into the difference between CNM and CNI.

Virtualization 3 of 4 – Containers

Tue, 18 Aug 2020 20:44:35 -0400

In broad terms, virtualization of computing resource is about isolation of resources at different levels. We have covered hypervisor-based virtualization in the other post. In this article, we continue to dive into OS level virtualization.

Remember again that the gist of virtualization is isolation of resource. To support OS level virtualization, the OS must have its own capability to isolate computing resource. There are many implementations of OS level virtualization.

Linux Kernel provides low-level mechanisms some two kernel features(namespaces, cgroups and chroot) for building various lightweight tools that can virtualize the system environment. Docker is such framework that builds on chroot namespaces and cgroups.

Chroot

Traditionally, root directory (/) is the top directory shared amongst all processes in the OS. There was a chroot() system call that allows each process to have its own idea of root directory. A chroot is an operation that changes the apparent root directory(/) for the current running process and their children. A program that is run in such a modified environment cannot access files and commands outside that environmental directory tree. This modified environment is called a chroot jail. By separating a process using chroot() we ensure security by restricting the process from accessing outside its environment (breaking the jail). This short video is a great lab.

Although chroot() has a basic idea of isolation, it simply modifies pathname lookups for a process and its children (by prepending the new root path to any name starting with /). Relative paths can still refer any locations outside of the new root. So chroot() does not intend to defend against intentional tampering by privileged users.

Namespace Isolation

Namespaces are fundamentally the mechanisms to abstract, isolate, and limit the visibility that a group of processes has over various system entities such as process trees, network interfaces, user IDs and file system mounts. So there are several categories of namespaces:

Mount namespaces – traditionally, there is one global mount namespace seen by all processes. The mount namespaces confine the set of filesystem mount points visible within a process namespace, enabling one process group in a mount namespace to have an exclusive view of the filesystem list, compared to another process.
UTS namespaces – allows isolation of hostname per namespace. Each namespace can have its own hostname on the network
User namespaces – allow a process to use unique user and group IDs
Cgroup namespaces – processes inside a cgroup namespace are only able to view paths relative to their namespace root.
IPC namespaces – isolates the System V inter-process communication between namespaces, as well as POSIX message queues within each namespace. POSIX message queue allow process to exchange data in the form of messsages.
PID namespaces – traditionally, *nix kernels spawn the init process with PID 1 during system boot, which in turn starts other user-mode process and is considered the root of the process tree (all the other processes start below this process in the tree). The PID namespace allows a process to spin off a new tree of processes under it with its own root process (PID=1). PID namespaces isolate process ID numbers, and allow duplication of PID numbers across different PID namespaces. The process IDs only needs to be unique within a PID namespace, and are assigned sequentially starting with PID 1. PID namespaces are used in containers.
Network namespaces – traditionally, all processes in the entire OS share a single set of network interfaces and routing table entries. The routing table entries can be modified at operating system level. With network namespace, this assumption is no longer valid. Network namespace provides abstraction and virtualization of network protocol and interfaces. Each network namespace will have its own network device instances that can be configured with individual network addresses. Other network services, such as routing table, port number, are isolated as well.

Namespaces are created with the “unshare” command or syscall, or as new flags in a clone() syscall. The flags are listed here in the man page for namespace. Note that the clone() syscall is a more generic implementation of fork() syscall.

Cgroup

cgroups is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc) of a collection of processes (not to be confused with process group, which has its own meaning). Cgroup has two versions. The control groups functionality (version 1) was merged into Linux kernel mainline in version 2.6.24, released in 2008, and version 2 in kernel 4.5 (March 2016), with significant changes to the interface and internal functionality.

Using cgroups, you can allocate resources such as CPU time, network and memory. Similiar to the process model in Linux, where each process is a child to a parent and relatively descends from the init process thus forming a single-tree like structure, cgroups are hierarchical, where child cgroups inherit the attributes of the parent, but what makes it different is that multiple cgroup hierarchies can exist within a single system, with each having distinct resource prerogatives.

Applying cgroups on namespaces results in isolation of processes into containers within a system, where resources are managed distinctly. Each container is a lightweight virtual machine, all of which run as individual entities and are oblivious of other entities within the same system.

Container Implementation

Above we covered some kernel features that enables container technology. There are many ways to use these technologies to implement the isolation. We call them container runtime.

LXC is a user space interface for those Linux kernel containment features. It allows for running isolated containers on a control host using a single kernel. Users can launch a system init for each containers, also referred to as virtual environment (as opposed to virtual machines). The author of this article regard LXC as a suprcharged chroot on Linux. LXC has rest API tool called LXD. LXC was targeting sysadmin’s use cases (not developer) to isolate users’ own private workloads from one another. In early days Docker was built on LXC.

Docker’s target market is developers, and it moved beyond LXC with its own execution environment called libcontainer. With the initial success of Docker, a large community (Docker, CoreOS, Google, etc) emerged around the idea of using containers as the standard unit of software delivery. They started the Open Container Initiative (OCI) to define industry standards around container runtime (runtime spec) and image format (image spec). Docker donated the libcontainer codebase to run independently under OCI, as runc. Docker implements isolation using the following technologies:

Namespace: to isolate process ID, networking, mount points, IPC, host and domain name;
Cgroups: to isolate the usage of CPU and memory between containers
UnionFS: isolate file system

Another container runtime technology is OpenVZ, which includes an extension of the Linux kernel. It uses container for entire operating systems (not just application and processes). All OpenVZ containers have to share the same Linux kernel version as host. The adoption of OpenVZ is not high.

Framework	Runtime implementation	Management tool
LXC	libvert LXC	LXD (rest API)
OCI	Docker’s runc CoreOS’s rtk	docker engine (daemon and cli) rtk cli

container runtimes

Docker is now widely adopted for application hosting in production environment.

Container and Cloud

Public cloud vendors also has managed services around Docker. Here are some examples:

	Managed Container	Image Registry	Managed Orchestration
AWS	Elastic Container Service	Elastic Container Registry	Elastic Kubernetes Services
Azure	Container Instances	Container Registry	Azure Kubernetes Service
GCP	CloudRun	Container Registry	Google Kubernetes Engine
Digital Ocean	N/A	Container Registry	Kubernetes

Container services from public cloud

Cloud service was originally developed with VM as a unit of computing resource to service. OS level virtualization allows container to be a unit of computing resource. All these new technologies breed the serverless architecture and cloud-native deployment model. This has significant impact on the creation and delivery of software services. The cloud native landscape page illustrates more tools around containers.

Virtualization 2 of 4 – Graphics Computing

Sat, 01 Aug 2020 18:24:00 -0400

We covered hypervisor in previous post. In this article we focus on the virtualization of graphics computing resource.

GPU vs CPU

GPU is a specialized type of microprocessor primarily designed for quick image rendering. GPU appeared as a response to graphically intense applications that put a burden on the CPU and degrated computer performance. They became a way to offload those tasks from CPUs, but modern graphics processors are powerful enough to perform rapid mathematical calculations for many other purposes apart from rendering.

CPU consists of a few cores (up to 23) optimized for sequential serial processing, which is designed to maximize the performance of a single task within a job. GPU uses thousands of smaller and more efficient cores for massively parallel architecture aimed at handling multiple functions at the same time. Typical uses cases for GPUs, in addition to graphics display, includes Games, 3D visualization, Image processing, big data and deep machine learning.

Moving to virtualization world, the most primitive mechanism for graphics acceleration is Soft 3D, which is commonly used in virtual desktops, or DaaS (desktop as a service). The Software 3D renderer (Soft 3D) uses the Soft 3D graphics driver to provide support for software-accelerated 3D graphics without any physical GPUs being installed in the ESXi host. With respect to GPU in virtualized environment, VMware developed a few technologies.

vSGA (Virtual Shared Graphics Acceleration)

The physical GPUs in the server are virtualized and shared across multiple guest VMs. This option involves installing an Nvidia driver into the hypervisor itself, and each guest VM uses a proprietary VMware SVGA 3D driver that communicates with the Nvidia driver in ESX. The biggest limitation here is that these drivers only work with DirectX up to 9.0c, and OpenGL up to 2.1. This technology was introduced in early 2013 and is used in light workload for knowledge worker, such as PowerPoint, Visio and web browsing.

vSGA

vDGA (Virtual Dedicated Graphics Acceleration)

vDGA, also known as “GPU passthrough”. It provides each VM with unrestricted, fully dedicated access to one of the host’s GPUs. The hypervisor is drilling a direct hole in itself between the GPU and the guest. This technology allows you to present an internal PCI GPU directly to a VM guest. The device acts as if it were directly driven by the VM guest, and the guest detects the PCI device as if it were physically connected, using the “real” driver. There is no special drivers in the hypervisor. vDGA offers the highest level of performance for users with the most intensive graphics computing needs.

GPU passthrough

The main advantage to vDGA is that since the GPU is passed through to the guest and the guest uses regular Nvidia drivers, it fully supports everything the Nvidia driver can do natively. This enables all versions of DirectX, OpenGL and even CUDA. The downside is that vDGA is expensive, since you need one GPU per user. There is also a lack of vMotion support.

VMware added support for vDGA in late 2013. The target market is high-end users with intensive graphical applications (oil&gas, scientific simulations, CAD/CAM, etc

vGPU (Virtual GPU)

vGPU is also known as Virtual Shared Pass-Through Graphics Acceleration. This technology sites somewhere in between the two previously introduced, as an option to strike a balance between cost-effectiveness and resource-sharing. It is essentially vDGA but with multiple users per GPU, instead of one-to-one mapping. Like vDGA, with vGPU you install the real Nvidia driver in guest VMs, and the hypervisor passes the graphics commands directly to the hypervisor without any translation.

vGPU gives you all that plus the ability to share a GPU across up to 8 VMs. The idea of vGPU is that you get better performance than vSGA option, with a portion of cost when compared to vDGA. The use case for vGPU will be the higher-end knowledge workers who need real “GPU” access but don’t need full-on multi-thousand dollar graphics workstations.

VMware partners with Nvidia on vGPU development. Below is the use-case chart from previous VMware white paper:

The diagram below illustrates the architecture of virtual GPU (NVIDIA Grid):

high-level architecture of GRID vGPU

The best white paper about the three technologies and their use cases is on VMware website.

Identify Graphics driver

On Linux VM, we can simply use lspci to identify graphics driver.

[root@ghrender ~]# lspci | grep VGA
03:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04)
3b:00.0 VGA compatible controller: NVIDIA Corporation GP104GL [Quadro P5000] (rev a1)In the result, the far left column is specified domain, e.g. 3b:00.0

To display details on graphics card by specified domain (3b:00.0 for example) with memory information:

[root@ghrender ~]# lspci -v -s 3b:00.0
3b:00.0 VGA compatible controller: NVIDIA Corporation GP104GL [Quadro P5000] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device 11b2
	Flags: bus master, fast devsel, latency 0, IRQ 190, NUMA node 0
	Memory at ab000000 (32-bit, non-prefetchable) [size=16M]
	Memory at 382fe0000000 (64-bit, prefetchable) [size=256M]
	Memory at 382ff0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at 6000 [size=128]
	[virtual] Expansion ROM at ac080000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [420] Advanced Error Reporting
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] #19
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

The lshw command can also identify onboard Intel/AMD or Nvidia dedicated GPU:

[root@ghrender ~]# lshw -C display
 *-display
 description: VGA compatible controller
 product: Integrated Matrox G200eW3 Graphics Controller
 vendor: Matrox Electronics Systems Ltd.
 physical id: 0
 bus info: pci@0000:03:00.0
 version: 04
 width: 32 bits
 clock: 66MHz
 capabilities: pm vga_controller bus_master cap_list rom
 configuration: driver=mgag200 latency=64 maxlatency=32 mingnt=16
 resources: irq:16 memory:91000000-91ffffff memory:92808000-9280bfff memory:92000000-927fffff
 *-display
 description: VGA compatible controller
 product: GP104GL [Quadro P5000]
 vendor: NVIDIA Corporation
 physical id: 0
 bus info: pci@0000:3b:00.0
 version: a1
 width: 64 bits
 clock: 33MHz
 capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
 configuration: driver=nvidia latency=0
 resources: iomemory:382f0-382ef iomemory:382f0-382ef irq:190 memory:ab000000-abffffff memory:382fe0000000-382fefffffff memory:382ff0000000-382ff1ffffff ioport:6000(size=128) memory:ac080000-ac0fffff

Virtualization 1 of 4 – Hypervisor

Mon, 27 Jul 2020 22:52:00 -0400

In broad terms, virtualization of computing resource is about isolation of resources, at different levels. There are five levels of virtualization:

Application level, such as JVM, .NET CLR
Library (user-level API) level
Operating system level, such as LXC, Docker, OpenVZ
Hardware abstraction layer (HAL) level, such as VMware, Xen, etc
Instruction set architecture (ISA) level

In my context I deal mostly with OS level and HAL (hardware abstraction layer) level of virtualization. In loose terms, the word containerization refers to OS level virtualization, while the word virtualization is exclusively reserved for HAL level virtualization, also referred to as hypervisor-based virtualization. This post will just focus on this family of technology and loosely refers to it as virtualization.

Virtualization technology evolved from on-premise data centre environment and now is the backbone of cloud computing. The challenges of IT operation in the era of virtualization involves managing VM sprawling, investigating performance issues, planning capacity and addressing storage I/O block. The idea of virtualization is sharing (thus isolating) resources for better utilization, leading to better return on investment. This posting is to cover only the very basics of virtualization.

Hypervisor

Hypervisor is the software layer which provides the capability to run multiple virtual machines on the same physical host. It is broken down into two categories:

Type I hypervisor (aka bare metal hypervisor): directly run on physical hardware. They control the hardware as well as manage the virtual machines. For example, Linux KVM, VMware ESXi, Xen and Microsoft Hyper-V
Type II hypervisor: runs as an application or service on top of the host operating system, which is installed on the bare metal. Guest operating system calls need to traverse via the host operating system stack to reach hardware resource. For example, Oracle Virtual Box, VMware Fusion and Linux Containers (LXC)

Hypervisor Types

Virtualization Techniques

The most primitive form of technology that can be arguably categorized under virtualization is hardware emulation, where a piece of (more accessible) hardware imitates another (less accessible). The architecture limits itself in functional testing only, and is not built for performance or production at all.

The original virtualization technology deals with CPU and memory virtualization. In this well-written whitepaper fromVMware, there are three CPU virtualization techniques introduced for x86 architecture.

The x86 architecture offers four levels of privilege known as Ring 0,1,2 and 3 to operating system and applications to manage access to the computer hardware. User-level applications typically run in Ring 3, the OS must execute its privileged instructions in Ring 0 since it needs to have direct access to memory and hardware. The two main challenges with virtualizing x86 architecture are:

A virtualization layer between hardware operating system who expects Ring 0 privilege;
Some instructions with different semantics when not executed in Ring 0 cannot be virtualized effectively. They need to be translated at runtime.

These challenges makes true virtualization of x86 architecture impossible and thus VMware developed three alternative technologies.

Full virtualization (using binary translation): virtual machine presents a complete simulation of the actual hardware environment so that an unmodified guest OS can run in isolation. The Guest OS is not aware that the underlying environment it is running on is virtualized, and issues hardware calls to communicate with (what it thinks as) hardware. The virtual processors have to understand guest CPU instruction, and reproduce the equivalent CPU instructions of the host machine. VMware’s technology to address this is called Binary Translation. This overhead makes true full virtualization difficult to achieve. In real life, a virtual environment that provides “enough representation of the underlying hardware” can be considered to provide full virtualization as long as it allows guest OS to run without modification. Full virtualization comes with a performance penalty.

Paravirtualization (aka OS assisted virtualization): refers to communication between the guest OS and the hypervisor to improve performance and efficiency. In this technology, guest OS is modified with an interface to host hardware to be able to communicate and operate seamlessly. Since the guest OS is modified, the VM does not need to be a complete simulation of the hardware. The modified guest OS knows it is running on a virtualized environment, and (vm driver) makes API calls (known as ‘hyper calls’) to the hypervisor. This allows para-virtualization technology to achieve performance closer to non-virtualized environment. However, since paravirtualization cannot support unmodified operating systems, its compatibility and portability is poor.

Hardware-Assisted Virtualization: hardware vendors such as Intel and AMD both have developed extensions (new features) to simplify virtualization techniques, for example, the introduction of privileged instructions with new CPU execution mode feature to allow hypervisor to run in a new root mode below ring 0. This removed the need for full virtualization and paravirtualization. With VMware originally as a promoter of full virtualization and Xen for paravirtualization, most virtualization technologies today utilizes hardware-assisted virtualization feature, for example, Linux KVM, VMware workstation, VMware fusion, Xen, VirtualBox, etc. Intel’s virtualization extension is VT-x. AMD’s counterpart is AMD-V technology.

To virtualize memory, another level of memory virtualization is required (similar to the virtual memory support in Linux). Hypervisor is responsible for mapping guest physical memory to the actual machine memory, and it uses shadow page tables to accelerate the mappings, usually at a performance cost.

Popular hypervisors

On the market there are a few popular hypervisor technologies. They are all type 1 hypervisors:

Xen is an open-source hypervisor project originally developed in Cambridge University, licensed under GPLv2. . Based on that, Citrix developed its commercial product XenServer, a bare-metal virtualization platform with enterprise-grade features for x86 and AMD environments. Oracle VM is another commercial implementation of Xen. The Xen project also supports many cloud platforms such as Openstack, Cloudstac, etc. Xen project supports paravirtualization (Xen-PV) as well as hardware-assisted virtualization (Xen-HVM) for virtualization of X86, IA64, ARM and other CPU architectures. The earlier versions does not support memory overcommit (aka “dynamic memory optimization”, “memory ballooning“, or as Citrix calls it “dynamic memory control, DMC”). This delivers better performance but also has higher budgetary requirement on hardware since there isn’t room for over-subscription.

Hyper-V is a Microsoft product. It executes in high CPU privilege (Microsoft calls it ring -1 which is equivalent to root mode as Intel calls it). On the guest VM, OS kernel and drivers run in ring 0, application rin in ring 3. This eliminates the need for binary translation. Hyper-V does not support memory overcommit either. Hyper-V is well integrated with Windows platform. It supports Linux as well although with some performance penalty.

Linux KVM (Kernel-based Virtual Machine) is a full open-source virtualization solution for GNU/Linux. What makes KVM a special hypervisor is that it uses a loadable kernel module kvm.ko that turns itself into a hypervisor and provides VMs with direct access to the hardware. So it is a type 1 hypervisor despite of the presence of Linux OS. KVM also contains a processor specific module, kvm-intel.ko or kvm-amd.ko. KVM leverages qemu to access devices. Because KVM runs as a process inside of Linux OS, KVM can use many existing feature in Linux kernel. Redhat has an enterprise solution based on KVM.

Xen vs KVM

VMware ESXi is VMware’s premium hypervisor product (not open-source) and is available for ~~free download~~, although the advanced features are not free. (Update no free download link anymore.) VMware vSphere is virtualization platform built on top of ESXi, including a whole family of virtualization products.

Market segments and players

Virtualization involves many market segments such as virtual desktop infrastructure (VDI, for desktop virtualization), server virtualization is the predominant domain in the virtualization of data centre environment. This effort led to Hyper-Converged Infrastructure (HCI) where almost all the traditional hardware resources are software-defined through the virtualization layer. The management of infrastructure is abstracted away from the physical hardware management. The three most fundamental areas in HCI are:

Server (compute) virtualization: the previous section covers the virtualization of memory and x86 CPU, which are the main focus on computing resource virtualization. Additionally, graphics computing resources can be virtualized today. Example products include: VMware vShpere (compute virtualization based on ESXi hypervisor).

Storage Virtualization: the technology to abstract physical data storage resource to make them appear as if they were a centralized resource. Storage virtualization takes place at three levels depending on the use case: block-level, file-level and object level. Example products include: VMWare vSAN (vSphere-native storage), HPE 3PAR (Tier-1 storage), EMC VxRail, PureStorage Flash Array (Tier 1), etc. Storage Virtualization enables Software-Defined Storage (SDS), the provisioning and management of data storage independent of the underlying hardware.

Network Virtualization: the technology to abstract network resources that were traditionally delivered in hardware to software. Network virtualization decouples network services from the underlying hardware management and allows virtual provisioning of an entire network. VLAN is a classic example of network virtualization. There are also various overlay technologies such as VXLAN, which provides an industry framework for overlaying virtualized layer 2 network over layer 3 network (used in Docker network) using an encapsulation mechanism and a control plane. Example products include: VMware NSX Data Center (L2-L7 network and security virtualization platform), Cisco ACI, Palo Alto Panorama. Network Virtualization enables Software-Defined Network (SDN), an approach to network management that enables dynamic, programmatically efficient network configuration in order to improve network performance and monitoring, making it more like cloud computing than traditional network management.

Delivery model

Virtualization allows managed service providers (MSPs) to deliver IT service in the following three models:

Iaas (Infrastructure as a Service): MSP delivers VM to customers.
PaaS (Platform as a Service): MSP delivers environments to customers (e.g. Database as a Service, managed RabbitMQ service, etc).
SaaS (Software as a Service): MSP delivers entire application for the customer.

IT service delivery models enabled by virtualization technology

Since virtualization is the backbone of cloud computing. This model is also referred to as cloud computing delivery model.

Virtualization and Containerization

These two concepts are similar and could be confusing to beginners. Both provide a mechanism to isolate computing resource for different applications, for the purpose of higher utilization of resource. The difference lies in how and where the isolation is made. Virtualization requires a guest operating system per VM (OS level isolation), whereas the container technology isolates application processes along with its runtime into a container (dependency level isolation), using some new Linux kernel features such as namespaces and cgroups. All containers make their system calls to the container engine on the host operating system. So they share a kernel on the same host. In this sense, container engine running on OS could be considered as type 2 hypervisor.

From VMs to containers

VMware is a major player in enterprise data centre virtualization, which is facing fierce competition from public and private cloud vendors. VMware also has its own private cloud services. Docker is the most popular container technology that conforms to the specifications of Open Container Initiative (OCI), a governance structure for industry standards around container formats and runtimes.

Virtualization and Cloud

Among public cloud vendors, AWS EC2 used Xen PV and Xen HVM in its earlier implementations. It has transitioned to AWS bare metal. The history is well summarized here. Microsoft Azure runs Azure Hypervisor as the native hypervisor in Azure Cloud Services platform. It is a customized version of Microsoft Hyper-V specifically for Azure platform. With GCP, Google Compute Engine (GCE) instance runs VMs on KVM as hypervisor. It can also enable nested virtualization.

The scope of cloud computing is evolving overtime. It originally only refers to a business model of offering IT services (in one of the three delivery models outlined above) based on virtualization technology. Therefore I cannot make comparison between a technology and a business model. Today, with public cloud vendor extending their offerings (with various managed services and platforms) and people’s misuse of the terms, the buzz-word “cloud” seems to suggest anything that is offered in public cloud service. The essence still remain the same where managed services and managed platforms are built on top of virtualized compute unit under the hood, which are driven by virtualization technologies.

Performance Analysis

Fri, 19 Jun 2020 16:47:01 -0400

Overview

In 2015, Brendan Gregg posted two great articles on Netflix blog: Linux Performance Analysis in 60 seconds, and Linux Perfomrance Tools. They have great value when I was in a urgency to spot performance issues. The articles cover the essential tools for performance troubleshooting, including:

Check out load averages: w or uptime
Print kernel ring buffer: dmesg -T
Virtual memory status: vmstat 1
Multiple processor staticstics: mpstat -P ALL 1
Task status: pidstat 1
CPU and I/O status: iostat -xz 1
Free memory check: free -m
Network Activity record: sar -n DEV 1
TCP activity record: sar -n TCP,ETCP 1
Display processes: top

We will dive into each of them in the next section.

Basic Troubleshooting

The command w is equivalent of uptime (which shows uptime since boot) and who (which shows logged-in users). It also displays load average for the last 1 minute, 5 minutes and 15 minutes. The number of load average reflects the overall system load (CPU + disks), and it is further discussed in this post with a simple take away:

If the averages are 0.0, then your system is idle.
If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
If they are higher than your CPU count, then you might have a performance problem (it depends).

When Linux load averages increase, you know you have higher demand for resources (CPUs, disks, and some locks), but you aren’t sure which. You will need to switch to other metrics. Brendan recommend don’t spend more than 5 seconds on these numbers.

[dhunch@c7v-ghintapp01 ~]$ w
 12:14:10 up 46 days, 16:41, 3 users, load average: 2.69, 2.44, 2.29
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dhunch pts/0 w6v-ghas01 24Jun20 3days 0.36s 0.30s ssh c7v-bastion
dhunch pts/1 202.95.88.111 12:02 2.00s 0.00s 0.00s w
dhunch pts/4 w6v-ghas01 17Jun20 15days 0.15s 0.07s view readme.txt

Before moving to more insightful metrics, it is also worth a quick look into the kernel ring buffer with dmesg command (dmesg -T | less +G). This will allow us to capture obvious issues such as oom-killer or TCP request dropping.

The vmstat tool reports the statistics of virtual memory. Servers have a fixed amount of physical memory, but they can run a set of applications that use a much larger amount of virtual memory. Application tend to reserve more memory than they need, and they usually operate on only a subset of their memory. In both cases, the operating system can keep the unused parts of memory on disk, and page it into physical memory only if it is needed. For the most part, this kind of memory management works well. But it doesn’t always with Java applications due to Java heap. Once a system start swapping – moving pages of data from main memory to disk, and vice versa, the performance tend to be bad. Systems must be configured so that swapping never occurs.

[dhunch@c7v-ghintapp01 ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 3 0 239360 385928 0 36734692 0 0 1 200 0 0 6 1 93 0 0
 3 0 239360 387732 0 36734704 0 0 0 43 8017 8524 14 0 85 0 0
 3 0 239360 387608 0 36734904 0 0 0 57 6768 7680 14 1 86 0 0
 2 0 239360 389008 0 36734904 0 0 0 44 6366 7300 14 0 86 0 0
 3 0 239360 421728 0 36700144 0 0 0 0 8141 7957 13 0 86 0 0
 2 0 239360 421984 0 36702048 0 0 0 467 8994 8362 14 0 85 0 0

The tool prints key server statistics each line, with the first line showing the average since boot. Here lists the explanation of some columns:

r: number of processes running on CPU and waiting for a turn. This provides a better signal than load averages for determining CPU saturation, as it does not include I/O. To interpret: an “r” value greater than the CPU count is saturation.
swpd: the amount of virtual memory used. This number should align with the used column for Swap row from free command.
buff, cache: buffer and cache. They should align with the buff/cache column form Mem row from free command.
free: free memory in kilobytes. This number should align with the free column for Mem row from free command.
si, so: swap-ins and swap-outs. As mentioned, if these are non-zero, you’re out of memory.
bi, bo: blocks received from and sent to a blcok device (# of block per second)
in, cs: number of interrupt, and context switches per second.
us, sy, id, wa, st: user, system, idle, wait I/O and stolen times. These are breakdowns of CPU time, on average across all CPUs. They should add up to 100% (or close). stolen time is amount of CPU time needed by a guest virtual machine that is not provided by the host. IO wait time is the CPU time waiting for I/O activity. Idle time could be several things: the process may be waiting for something (e.g. a response from database); the process may be blocked by a thread lock; or the process simply has nothing to do. user and system times are CPU times spent on user tasks and kernel tasks, respectively.

Out of these columns, watch for columns r, free, buff, cache, us, sy, id and wa at minimum. The combination of us and sy confirms if CPUs are busy. A constant degree of wa points to a disk bottleneck with too much time spent on pending disk I/O. The sy (kernel time) is necessary for I/O processing but sy stays high (e.g. constantly over 20%), it becomes interesting. Perhaps the kernel is processing I/O inefficiently.

For further per-CPU stats, use mpstat command (-P ALL), to prind CPU time breakdowns per CPU and check for imbalance. A single host CPU can be evidence of a single-threaded application. Here is an example output from a system of 16 CPU cores.

[dhunch@c7v-ghintapp01 ~]$ mpstat -P ALL 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

13:47:23 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
13:47:24 all 13.77 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 86.05
13:47:24 0 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 1 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 3 98.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
13:47:24 4 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 5 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 6 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 7 2.94 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 96.08
13:47:24 8 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 9 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 10 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
13:47:24 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 12 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 13 1.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 14 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

For a per process summary of CPU consumption, use pidstat command. It can be thought of a periodical snapshot of top command, allowing you to watch for patterns. The %CPU column is the total across all CPUs so 5 CPUs have a maximum value of 500.

If vmstate indicates some I/O issue, iostat tool can help us understand block devices, on both the workload applied and the resulting performance. Key columns are:

r/s, w/s, rkB/s, wkB/s: delivered reads, writes, read Kbytes, and write Kbytes per second to the device. Use these for workload characterization. A performance problem may simply be due to an excessive load applied.
await: the average wait time for I/O in milliseconds. This is the time that the application suffers, as it includes both time queued and time being serviced. Larger than expected average times can be an indicator of device saturation, or malfunction.
avgqu-sz: the average number of requests issued to device. Values greater than 1 can be evidence of saturation (although devices can typically operate on requests in parallel, especially virtual devices which front multiple back-end disks.)
%util: device utilization. This is really a busy percent, showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance (which should be seen in await), although it depends on the device. Values close to 100% usually indicate saturation.

I/O problem may either be inefficiencies in application that issues I/O request, or slowing disk unable to keep up with I/O requests. We review two examples here to illustrate each situation. The first output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 23.45 0.00 37.89 0.10 0.00 38.56

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 11.60 0.60 24.20 0.02 0.14 13.35 0.15 6.06 5.33 6.08 0.42 1.04

In the first example, the disk stat loosk up at first glance. The w_await (time to service I/O write) is fairly low at 6.08ms. However, the system is spending 37.89% of its time in the kernel. If all that system time is from the application, it suggest something inefficient is happening. The fact that the system is doing 24.2 writes per second is another clue: that is alot when writing only 0.14 MB per second (MBps). I/O has become a bottleneck, and the next step would be to look into how the application is performing its writes.

The second example output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 35.05 0.00 7.85 47.89 0.00 9.20

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 0.20 1.00 163.40 0.00 81.09 1010.19 142.74 866.47 97.60 871.17 6.08 100.00

In this example, it tells us that processes are spending 47.89% of their time in iowait, and the data to complete the I/O (w_await) is 871ms, the queue size is large, and the disk is writing at 81MB per second. This all points to disk I/O as a problem and that the amount of I/O in the application (or elsewhere in the system) must be reduced.

Bear in mind that poor performing disk I/O isn’t necessarily an application issue. Many techniques are typically used to perform I/O asynchronously, so that the application doesn’t block and suffer the latency directly (e.g. read-ahead for reads, and buffering for writes, also refer to “tail latency“).

Note that the acceptable utilization metric depends on the configuration of block device. If the storage is a logical disk device fronting many back-end disks (e.g. RAID 0), then 100% utilization may just mean that some I/O is being processed 100% of the time, however, the back-end disks may be far from being saturated, and may even be able to handle more work.

The free command gives the breakdown of memory usage. The right two columns are:

buffers: for the buffer cache, used for block device I/O.
cached: for the page cache, used by file systems.

We just want to check that these aren’t near-zero in size, which can lead to higher disk I/O (confirm using iostat), and worse performance. Linux uses free memory for the caches, but can reclaim it quickly if applications need it. So in a way the cached memory should be included in the free memory column. In this case, it’s included in the available column. This website has further details.

To check interface stat, nicstat is a great tool but it isn’t available by default in Linux. Instead we can run sar (-n DEV) to retrieve stats.

[dhunch@c7v-ghintapp01 ~]$ sar -n DEV 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:51:25 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:26 eth0 3089.00 934.00 3815.33 834.61 0.00 0.00 0.00
16:51:26 lo 464.00 464.00 2289.07 2289.07 0.00 0.00 0.00

16:51:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:27 eth0 956.00 586.00 826.66 211.34 0.00 0.00 0.00
16:51:27 lo 213.00 213.00 196.00 196.00 0.00 0.00 0.00

16:51:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:28 eth0 349.00 181.00 52.32 147.19 0.00 0.00 0.00
16:51:28 lo 244.00 244.00 81.13 81.13 0.00 0.00 0.00

Here rxkB/s/s and txkB/s represents receive and transmission throughput, as a measure of workload. If they reach the limit then the interface is the bottleneck.

On top of interface is the TCP layer, which can be monitored with sar again (-n ECP, ETCP). The key metrics are:

active/s: number of locally-initiated (e.g. via connect()) TCP connections per second
passive/s: number of remotely-initiated (e.g. via accept()) TCP connections per second
retrans/s: number of TCP retransmits per second

The active and passive counts are often useful as a rough measure of server load. It might help to think of active as outbound, and passive as inbound, but this isn’t strictly true (e.g. consider a localhost to localhost connection). Retransmits are a sign of network or server issue; it may be an unreliable network (e.g. public Internet), or it may be due to a server being overloaded and dropping packets.

[dhunch@c7v-ghintapp01 ~]$ sar -n TCP,ETCP 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:52:29 active/s passive/s iseg/s oseg/s
16:52:30 0.00 1.00 28.00 35.00

16:52:29 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:30 0.00 0.00 0.00 0.00 0.00

16:52:30 active/s passive/s iseg/s oseg/s
16:52:31 8.00 8.00 200.00 317.00

16:52:30 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:31 0.00 1.00 1.00 0.00 3.00

Last but not least is our favourite command top, which includes many of the metrics covered in previous tools. The downside to top is it is harder to see patterns over time, which may be more clear in tools like vmstat and pidstat, both of which produce rolling output.

Several tools introduced here involves sar, which is a great monitoring tool on its own that we need to be familiar with.

System Activity Report (SAR)

Further to the basic tools, the sar command is very helpful as it stores historical stat every 10 minutes. Sar keeps 18 types of reports, all stored in /var/log/sa/. When viewing the report file, you may pipe the result to less command so it only prints header once. For example, if you would like to print CPU report for the 2nd of the month:

# sar -u -f /var/log/sa/sar02 | less

If you check NFS client statistics for the 31st

 # sar -n NFS -f /var/log/sa/sar31

If you need to check network server statistics for today

# sar -n NFS -f /var/log/sa/sar31

Below are all types of reports:

-u CPU utilization
-w task creation and system switching activity
-W swapping statistics
-B report paging
-b report I/O and transfer rate statistics
-R report memory statistics
-r memory utilization
-S swap space utilization
-H huge pages utilization statistics
-v inode
-q queue length and load average
-y TTY device activity
-d activity for each block device
-n network statistics, DEV (per interface)
-n network statistics, EDEV (error per interface)
-n network statistics, NFS (NFS client)
-n network statistics, NFSD (NFS server)
-n network statistics, SOCK (socket usage)

Berkeley Packet Filter (BPF) Compiler Collection (bcc) tools

For advanced, low-level performance troubleshooting, the BCC tools provide a suite of tools. Here we only cover the installation of it.

In CentOS, install bcc-tools package with yum. When you try to run a command, such as cachestat, if you come across this error:

-bash: cachestat: command not found

Then you will need to add its path to default:

export PATH=$PATH:/usr/share/bcc/tools

Now if you run into this error:

chdir(/lib/modules/3.10.0-1062.12.1.el7.x86_64/build): No such file or directory
Traceback (most recent call last):

The file listed is a symbolic link, and if it is missing, you just need to install kernel-headers that matches the kernel version:

yum install kernel-headers

Then you may use tools in /usr/share/bcc/tools. For example, cachestat help you display page cache hit ratio; gethostlatency shows DNS resolution latency; tcpconnect prints out active tcp connections (made via connect system call):

[root@dhunch ~]# /usr/share/bcc/tools/tcpconnect -t -P 8080 | gawk '{ print strftime("%F %T  "), $0 }'
2020-06-13 00:16:57   TIME(s)  PID    COMM         IP SADDR            DADDR            DPORT
2020-06-13 02:16:57   0.000    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.064    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.438    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080

The command above outputs a time and pid stamped log line every time a TCP connection is made to port 8080; tcpaccept traces passive tcp connections (via accept system call). These tools are not as intrusive as tcpdump.

It is beyond the purpose of this article to get into details of each tool in the BFP suite. The tools are covered in detail in books “BPF Performance Tools” and “Linux Observability with BPF”.

Setup WSL2 (and Docker) on Windows 10

Tue, 02 Jun 2020 22:01:00 -0400

This is not for Linux snobs, but rather for those who are stuck with a Windows work laptop, have to deal with Linux on a daily basis, and are not a fan of PuTTY. This posting provides the steps to setup Windows 10 so you get a work environment closer to a Linux one. The environment to begin with should be Windows 10 version 2004 and above in order to use WSL2. Here’s the comparison between WSL and WSL2. This presentation is a great deep dive into how WSL2 works. The architecture diagram below is from that presentation. Note that WSL2 operates on a “true” Linux Kernel, therefore giving WSL2 the ability to run Docker.

Install WSL2

WSL2 was officially released in Windows 10 version 2004 (build 19041 or higher) and we will use Ubuntu 20.04 LTS. If you’re upgraded from older version of Windows 10 you will need to upgrade. The steps are:

Install Ubuntu 20.04 LTS from Microsoft Store, which requires Windows components “Windows Subsystem for Linux” and “Virtual Machine Platform”; or, if you already have an older version of Ubuntu such as 18.04, you will need to run a distribution upgrade with “sudo apt-get dist-upgrade”;
Upgrade Linux Virtual Machine from WSL to WSL2, following the official instruction here; Once completed, use this command to confirm the WSL version installed: wsl.exe –list -v

There’s also plenty of Youtube videos with step-by-step instruction on upgrading to WSL2 and its advantage over WSL. Once you’re there, you will realize one problem: the command terminal for Ubuntu is ugly:

The colour looks awful and it does not support multi-tabs. If you want better palette, there are a number of options. Check out ColorTool project and iTerms2 Colors project. The former gives you away to configure color scheme (from command prompt but takes effect in WSL as well) and the latter gives you rich choices of color schemes. My personal favourite is “Banana Blueberry”. The command to set color scheme is:

C:\ColorTool.exe -b "iTerm2-Color-Schemes-master\schemes\Banana Blueberry.itermcolors"

Windows Terminal

For a better terminal, consider Windows Terminal from Microsoft Store. Windows terminal is an open-source project and still a little glitchy as of June 2020. However it is heading to the right direction. Once installed, you may customize it by clicking settings:

A json file will open and you may edit it for customization, for example, you can adjust each profile (e.g. PowerShell, Windows command or WSL); you can specify default profile and default open location.

// This file was initially generated by Windows Terminal 1.0.1401.0
// It should still be usable in newer versions, but newer versions might have additional
// settings, help text, or changes that you will not see unless you clear this file
// and let us generate a new one for you.

// To view the default settings, hold "alt" while clicking on the "Settings" button.
// For documentation on these settings, see: https://aka.ms/terminal-documentation
{
 "$schema": "https://aka.ms/terminal-profiles-schema",

 "defaultProfile": "{2c4de342-38b7-51cf-b940-2309a097f518}",

 // You can add more global application settings here.
 // To learn more about global settings, visit https://aka.ms/terminal-global-settings

 // If enabled, selections are automatically copied to your clipboard.
 "copyOnSelect": false,

 // If enabled, formatted data is also copied to your clipboard
 "copyFormatting": false,

 // A profile specifies a command to execute paired with information about how it should look and feel.
 // Each one of them will appear in the 'New Tab' dropdown,
 // and can be invoked from the commandline with `wt.exe -p xxx`
 // To learn more about profiles, visit https://aka.ms/terminal-profile-settings
 "profiles":
 {
 "defaults":
 {
 // Put settings here that you want to apply to all profiles.
 },
 "list":
 [
 {
 "guid": "{2c4de342-38b7-51cf-b940-2309a097f518}",
 "hidden": false,
 "name": "Ubuntu",
 "source": "Windows.Terminal.Wsl",
 "startingDirectory": "//wsl$/Ubuntu/home/myuser/"
 },
 {
 // Make changes here to the powershell.exe profile.
 "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
 "name": "Windows PowerShell",
 "commandline": "powershell.exe",
 "hidden": false
 },
 {
 // Make changes here to the cmd.exe profile.
 "guid": "{0caa0dad-35be-5f56-a8ff-afceeeaa6101}",
 "name": "Command Prompt",
 "commandline": "cmd.exe",
 "hidden": true
 },
 {
 "guid": "{b453ae62-4e3d-5e58-b989-0a998ec441b8}",
 "hidden": true,
 "name": "Azure Cloud Shell",
 "source": "Windows.Terminal.Azure"
 }
 ]
 },

 // Add custom color schemes to this array.
 // To learn more about color schemes, visit https://aka.ms/terminal-color-schemes
 "schemes": [],

 // Add custom keybindings to this array.
 // To unbind a key combination from your defaults.json, set the command to "unbound".
 // To learn more about keybindings, visit https://aka.ms/terminal-keybindings
 "keybindings":
 [
 // Copy and paste are bound to Ctrl+Shift+C and Ctrl+Shift+V in your defaults.json.
 // These two lines additionally bind them to Ctrl+C and Ctrl+V.
 // To learn more about selection, visit https://aka.ms/terminal-selection
 { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" },
 { "command": "paste", "keys": "ctrl+v" },

 // Press Ctrl+Shift+F to open the search box
 { "command": "find", "keys": "ctrl+shift+f" },

 // Press Alt+Shift+D to open a new pane.
 // - "split": "auto" makes this pane open in the direction that provides the most surface area.
 // - "splitMode": "duplicate" makes the new pane use the focused pane's profile.
 // To learn more about panes, visit https://aka.ms/terminal-panes
 { "command": { "action": "splitPane", "split": "auto", "splitMode": "duplicate" }, "keys": "alt+shift+d" }
 ]
}

With these configuration, we have a quite comfortable Linux work environment on Windows.

Update to Zsh and beautify it

I’m no so big a fan of zsh (for lack of default wildcard support) but I do like one of its themes. We can install zsh on WSL2 (assuming Ubuntu distribution). This is a good instruction. I do like the theme name agnoster. However, the theme displays username at the beginning of the prompt, which takes a lot of screen real estate if your username is long. To remove it, the trick is to add a line in ~/.zshrc, as answered in this thread.

Docker runtime

There is a lot of fun to have in WSL2 on Windows 10. For example, you can configure Docker runtime on Windows according to this post. However, you cannot route traffic to the container in the absence of docker0 bridge for Docker on WSL2, as indicated in the known limitations.

Update: Docker also brings a Kubernetes cluster named docker-desktop. This allows you configure Kubernetes cluster, or install Helm and Rancher (instruction) for cluster management.

Dual-Boot?

With WSL2 on top of a real Linux kernel, I do not find a need for a dual-boot system on my workstation. To be fair, only in the following scenarios should one consider installing a dual-boot system.

You need to work on a specify version of Linux Kernel. The kernel provided by Microsoft is called “microsoft-standard-WSL2”, and its pretty up to date (5.x)
You need to run GUI applications on Linux. Update: Microsoft has started developing GUI application support on WSL2, even though it usually takes time to mature.
You need to work on a distribution not available in Windows Store. WSL2 provides a kernel and it’s up to developers to provide Linux distros in Windows store. Popular (and official) ones are: Ubuntu 20.04 LTS by Canonical Group, Debian by the Debian project, SUSE Linux Enterprise Server by SUSE, Kali Linux.

Certificate Issuers

I have to use WSL2 mostly because I’m forced to use a Windows laptop on my contracts. That also means that the Windows laptop has been configured to corporate firewall’s packet inspection. When that is the case, they corporate IT usually automatically imports a firewall certificate to the Windows trust store. However, the WSL is missing this certificate.

The symptom would be that you cannot run certain curl command. For example, when I do the followings:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This error message basically means that the website certificate cannot be trusted. To view the certificate, use Open SSL command:

openssl s_client -connect dl.k8s.io:443 -servername dl.k8s.io | openssl x509 -text -noout | less

In my case, I noticed that the certificate is a Zscaler issued certificate (firewall). While running the same command from my personal device returns a Goolge Trust issued certificate. This indicates that the certificate is changed by the proxy managed by the corporation. To fix this I need to export the certificate from Windows, and then import it in WSL. The answer in this post has good detailed steps.

Balloon steals memory from virtual machines

Thu, 07 May 2020 19:49:00 -0400

This article is my experience with memory balloon on virtual machine.

I came across an ElasticSearch server (ESXi guest with 32GB physical memory) where the main process keeps dying of OOM. Even worse, after the OOM event, the free memory left is about 10G and Elastic Search cannot start because its JVM is set with -Xms16g in /etc/elasticsearch/jvm.options. So I need to address the OutOfMemory error.

To understand what triggered OOM, we can use dmesg or just check /var/log/message, where the memory snapshot by OOM killer is displayed during the kernel panic:

Apr 26 04:18:15 elastichost kernel: kworker/7:1 invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: kworker/7:1 cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 7 PID: 13968 Comm: kworker/7:1 Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Workqueue: events_freezable vmballoon_work [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc05f1f84>] vmballoon_work+0x454/0x6ff [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22b9d4f>] process_one_work+0x17f/0x440
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bade6>] worker_thread+0x126/0x3c0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1c31>] kthread+0xd1/0xe0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2974c37>] ret_from_fork_nospec_begin+0x21/0x21
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3649243 inactive_anon:439380 isolated_anon:0#012 active_file:480 inactive_file:693 isolated_file:0#012 unevictable:0 dirty:2 writeback:0 unstable:0#012 slab_reclaimable:77845 slab_unreclaimable:10219#012 mapped:10086 shmem:9596 pagetables:16727 bounce:0#012 free:50116 free_pcp:238 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122768kB min:5972kB low:7464kB high:8956kB active_anon:471052kB inactive_anon:471040kB active_file:232kB inactive_file:460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:584kB shmem:300kB slab_reclaimable:23908kB slab_unreclaimable:3872kB kernel_stack:656kB pagetables:6440kB unstable:0kB bounce:0kB free_pcp:288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:214 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61804kB min:61576kB low:76968kB high:92364kB active_anon:14125920kB inactive_anon:1286480kB active_file:1688kB inactive_file:2312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:8kB writeback:0kB mapped:39760kB shmem:38084kB slab_reclaimable:287472kB slab_unreclaimable:36988kB kernel_stack:5680kB pagetables:60468kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:698 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 407*4kB (UEM) 506*8kB (UEM) 431*16kB (UM) 364*32kB (M) 272*64kB (UM) 179*128kB (UM) 99*256kB (UM) 35*512kB (M) 11*1024kB (M) 2*2048kB (M) 0*4096kB = 123164kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15576*4kB (UM) 39*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62616kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 44652 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 34162 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884045, delete 10845872, find 3294122/3678936
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 12996 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 83 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 50 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 47 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 113 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 39 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 132 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 4770 100 482 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6332 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 30 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3994129 15027 457539 0 java
Apr 26 04:18:15 elastichost kernel: [16264] 495 16264 18032 0 31 162 0 controller
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16070 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16264 (controller) total-vm:72128kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:15 elastichost kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: java cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 2 PID: 16265 Comm: java Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b6347>] __page_cache_alloc+0x97/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b8fa8>] filemap_fault+0x298/0x490
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484d0e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c2dc0>] ? wake_bit_function+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484f0c>] xfs_filemap_fault+0x2c/0x30 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e444a>] __do_fault.isra.59+0x8a/0x100
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e49fc>] do_read_fault.isra.61+0x4c/0x1b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e93a4>] handle_pte_fault+0x2f4/0xd10
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ebedd>] handle_mm_fault+0x39d/0x9b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f5e3>] __do_page_fault+0x203/0x500
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f915>] do_page_fault+0x35/0x90
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296b758>] page_fault+0x28/0x30
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3607073 inactive_anon:480522 isolated_anon:0#012 active_file:8 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:75170 slab_unreclaimable:10131#012 mapped:2070 shmem:9592 pagetables:16696 bounce:0#012 free:50006 free_pcp:72 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122560kB min:5972kB low:7464kB high:8956kB active_anon:471000kB inactive_anon:471076kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:292kB shmem:288kB slab_reclaimable:22832kB slab_unreclaimable:3752kB kernel_stack:672kB pagetables:6408kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:215 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61572kB min:61576kB low:76968kB high:92364kB active_anon:13957292kB inactive_anon:1451012kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:0kB writeback:4kB mapped:7988kB shmem:38080kB slab_reclaimable:277848kB slab_unreclaimable:36756kB kernel_stack:5664kB pagetables:60376kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:500 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 393*4kB (UEM) 490*8kB (UEM) 453*16kB (EM) 369*32kB (UEM) 269*64kB (M) 180*128kB (M) 98*256kB (M) 35*512kB (M) 12*1024kB (UM) 1*2048kB (M) 0*4096kB = 122148kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15440*4kB (UM) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61768kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 42975 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 33308 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884233, delete 10846914, find 3294127/3678944
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 4016 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 78 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 46 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 42 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 91 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 36 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 130 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 1215 100 485 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6230 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 27 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3993992 15027 457699 0 java
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16291 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16070 (java) total-vm:188490524kB, anon-rss:15975968kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:16 elastichost systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL
Apr 26 04:18:16 elastichost systemd: Unit elasticsearch.service entered failed state.
Apr 26 04:18:16 elastichost systemd: elasticsearch.service failed.

The line “Free swap = 0kB” suggest the trigger of OOM is out of swap. So does ElasticSearch contribute to the swap shortage?

ElasticSearch was the main process on the VM and I noticed that the OS did not disable swap, which does not align with the best practice from ElasticSearch community. In order to find out whether Elastic Search is pushed to use swap, we can get the process ID:

pidof java

ElasticSearch happens to be the only Java based process and the PID is 2283, the following command shows the swap usage by this process:

cat /proc/2283/status | grep VmSwap

If it shows a non-zero value, then ElasticSearch is using swap and you should expect some performance issues. We definitely should disable swap on ElasticSearch but does that solve the problem? Probably not because we didn’t address what caused the memory pressure in the first place. Disabling swap most likely makes ElastciSearch last longer before something else such as shortage of available memory triggers OOM.

In search for the source of memory shortage, I checked the result of free command and the top command. The free command (-mh) shows the following:

 total used free shared buff/cache available
Mem: 31G 20G 10G 117M 243M 10G
Swap: 2.0G 249M 1.8G

I tried to follow my example from a previous post to make sense of the memory reads. When I monitor process with top and watch for RSS column, I cannot identify a single process that even takes more than a few hundred megabytes. I’ve downloaded a tool smem from epel repo, and the result of “smem -kt” suggest that the total RSS is about 372.2M:

So there is about 19.6GB of memory usage unaccounted for. There is something that takes this much memory in the VM and haunting round even after OOM killer!

Inspired by this post, I was able to identify the culprit, that is the memory balloon. We don’t have access to the host but from the Guest OS, we can tell by vmware-toolbox-cmd (need to install yum package open-vm-tools):

vmware-toolbox-cmd stat balloon

The result displays a whopping 20807 MB as memory balloon! This needs to be sent to PaaS vendor for investigation but it is likely a result of memory over-allocation/over-commitment at host level, as well as the setup where the memory of guest is not reserved. Looking at the original log snippet, the line with “events_freezable vmballoon_work” is also an indicator of balloon causing the OOM.

Hypervisor needs memory ballooning to reclaim memory from guest. Since the guest OS does not expect the amount of physical memory to change, hypervisor has to maintain the illusion that the guest has its fixed amount of physical memory. The hypervisor first computes the amount of memory that needs to reclaim, then it leverage some low-level mechanism such as a balloon driver (a pseudo-device driver) installed on guest. The driver communicates with hypervisor and is told to allocate or de-allocate memory. If the driver is told to allocate memory to host, it tells the guest OS to pin the allocated pages into physical memory so they are locked and the physical memory available to guest OS is decreased. All these low level mechanisms explains why it is hard to account for memory consumption based on process running in Guest OS.

Memory balloon is a dynamic process and the hypervisor adjusts the size of balloon. However, the hypervisor, the guest OS and the application process (ElasticSearch in this case) may enter a nuance interaction:

The hypervisor reclaims memory from guest OS (ballooning);
Guest OS panicked with OOM exception;
OOM picks a process to kill, based on oom_score. Unfortunately, the true culprit (balloon) is not visible to guest OS, hence exempted from being considered to kill.
By killing the application process (usually the main application for VM), a fair chunk of guest memory is freed up;
the balloon then became more aggressive on the guest as seeing more memory becomes “available”;
the application on guest then does not have minimum memory on guest OS to start;

How to solve this problem? There are several things to consider. On the hypervisor, keep the entire or part of the guest memory reserved for start of application. Tune the OOM scoring so the non-critical application get killed. Implement application daemon, etc.

Storage Nitty-Gritty 3 of 5 – NAS and Object Storage

Sat, 13 Jul 2019 23:31:00 -0400

NAS (network attached storage)

NAS server is dedicated to file-serving. NAS device runs its own specialized operating system that is optimized for file I/O, integrated hardware and software component that meets specific file-service needs, and performs file I/O better than a general-purpose server. NAS device can serve more clients than general-purpose servers and provide the benefit of server consolidation (centralized storage).

NAS uses network and file-sharing protocols to provide access to the file data. These protocols include TCP/IP for data transfer, and Common Internet File System (CIFS) and Network File System (NFS) for network file service.

Network File Sharing – user who creates a file determines the type of access to be given to other user. When multiple users try to access a shared file at the same time, a locking scheme is required to maintain data integrity and, at the same time, make this sharing possible. Examples of file sharing method (FTP, DFS, NFS, CIFS, P2P)

Components of NAS – NAS head (CPU, memory, NIC, optimized OS, ports, applications that supports CIFS/NFS) and Storage Array

Typical NAS components

NAS I/O operation:

Client packages an I/O request into TCP/IP and forwards it through network stack. NAS head receives this request from network;
NAS head converts the I/O request into an appropriate physical storage request, which is a block-level I/O, and then performs the operation on the physical storage;
When NAS head receives data from the storage array, it processes and repackages the data into an appropriate NFS/CIFS response;
NAS head packages this response into TCP/IP again and forwards it to the client through the network

NAS I/O operation

NAS implementation

Unified NAS – consolidate NAS-based and SAN-based data access within a unified storage platform and provides a unified management interface for managing both the environments.

Unified NAS connectivity

Gateway implementation – similar to unified NAS, the storage is shared with other applications that use block-level I/O. The gateway NAS is more scalable compared to unified NAS because NAS heads and storage arrays can be independently scaled up when required. For example, NAS heads can be added to scale up the NAS device performance.

When the storage limit is reached, it can scale up, adding capacity on the SAN, independent of NAS heads. Similar to a unified NAS, a gateway NAS also enables high utilization of storage capacity by sharing it with the SAN environment.

Gateway NAS connectivity

Scale-out NAS – enables grouping multiple nodes together to construct a clustered NAS system. A scaled-out NAS provides the capability to scale its resources by simply adding nodes to a clustered NAS architecture. The cluster works as a single NAS device and is managed centrally. Scaled-out NAS creates a single file system that runs on all nodes in the cluster. All information is shared among nodes, so the entire file system is accessible by clients connecting to any node in the cluster. Scale-out NAS stripes data across all nodes in a cluster along with mirror or parity protection. As data is sent from clients to the cluster, the data is divided and allocated to different nodes in parallel. When a client sends a request to read a file, the scale-out NAS retrieves the appropriate blocks from multiple nodes, recombines the blocks into a file, and presents the file to the client. As nodes are added, the file system grows dynamically and data is evenly distributed to every node. Each node added to the cluster increases the aggregate storage, memory, CPU, and network capacity. Hence, cluster performance also increases.

Scale-out NAS use separate internal and external networks for back-end and front-end connectivity, respectively. The internal network offers high throughput and low-latency and uses high-speed networking technology, such as InfiniBand or Gigabit Ethernet.

Scale-out NAS with dual internal and single external networks

NFS protocol – originally based on UDP, uses RPC as a method of inter-process communication between two computers. NFS provides a set of RPCS to access remote file system for the following operations:

Searching files and directories
Opening, reading, writing to and closing a file
Changing file attributes
Modifying file links and directories

NFSv3 and earlier is stateless protocol. Each call provides a full set of arguments to access files on the server. NFSv3 is most commonly used version, based on UDP or TCP.

NFSv4 uses TCP and is based on stateful protocol design.

CIFS – a public, or open variation of SMB protocol. Filenames in CIFS are encoded using unicode characters. It is stateful protocol because the server maintain connection information regarding every connected client. If a network failure or CIFS server failure occurs, the client receives a disconnection notification. If application has embedded intelligence to restore the connection, then the storage solution is fault tolerant. If the embedded intelligence is missing, the user must take steps to reestablish the CIFS connection.

NAS Performance – network congestion is one of the most significant sources of latency in NAS environment. Other factors

number of hops
authentication with AD
Retransmission – speed and duplex settings on the network devices and NAS heads must match
Over-utilized routers and switches
File system lookup and metadata request – deep directory structure could cause delay.
Over-utilized NAS devices – client accessing multiple files can cause high utilization levels on a NAS device
Over-utilized clients – if a client is busy itself, it requires a longer time to process the request and responses.

NAS latency

NFS server manages privilege and does not require username and password from the client at the time of mounting. CIFS share does require username and password.

Common network optimization practices for network contestion:

A VLAN is a logical segment of a switched network or logical grouping of end devices connected to different physical networks. The segmentation or grouping can be done based on business functions, project teams, or applications. VLAN is a Layer 2 (data link layer) construct and works similar to a physical LAN. A network switch can be logically divided among multiple VLANs, enabling better utilization of the switch and reducing overall cost of deploying a network infrastructure.

The broadcast traffic on one VLAN is not transmitted outside that VLAN, which substantially reduces the broadcast overhead, makes bandwidth available for applications, and reduces the network’s vulnerability to broadcast storms.

MTU setting determines the size of the largest packet that can be transmitted without data fragmentation. Path maximum transmission unit discovery is the process of discovering the maximum size of a packet that can be sent across a network without fragmentation. The default MTU setting for an Ethernet interface card is 1,500 bytes. A feature called jumbo frames sends, receives or transports Ethernet frames with an MTU of more than 1,500 bytes. The most common deployments of jumbo frames have an MTU of 9,000 bytes. However, not all vendors use the same MTU size for jumbo frames. Servers send and receive larger frames more efficiently than smaller ones in heavy network traffic conditions. Jumbo frames ensure increased efficiency because it takes fewer, larger frames to transfer the same amount of data. Larger packets also reduce the amount of raw network bandwidth being consumed for the same amount of payload. Larger frames also help to smooth sudden I/O burst.

The TCP window size is the maximum amount of data that can be sent at any time for a connection. For example, if a pair of hosts is talking over a TCP connection that has a TCP windows size of 64KB, the sender can send only 64KB of data and must then wait for an acknowledgement from the receiver. If the receiver acknowledges that all the data has been received, then the sender is free to send another 64 KB of data. If the sender receives an acknowledgment from the receiver that only the first 32 KB of data has been received, which can happen only if another 32 KB of data is in transit or was lost, the sender can send only another 32 KB of data because the transmission cannot have more than 64 KB of unacknowledged data outstanding.

In theory, the TCP window size should be set to the product of the available bandwidth of the network and the round-trip time of data sent over the network. For example, if a network has a bandwidth of 100 Mbps and the round-trip time is 5 milliseconds, the TCP window should be as follows:

100 Mb/s x .005 seconds = 524,288 bits or 65,536 bytes

The size of the TCP window fi eld that controls the fl ow of data is between 2 bytes and 65,535 bytes

Link aggregation is the process of combining two or more network interfaces into a logical network interface, enabling higher throughput, load sharing or load balancing, transparent path failover, and scalability. Due to link aggregation, multiple active Ethernet connections to the same switch appear as one link. If a connection or a port in the aggregation is lost, then all the network traffic on that link is redistributed across the remaining active connections.

File-level virtualization

File-level virtualization eliminates the dependencies between the data accessed at the file level and the location where the files are physically stored. Implementation of file-level virtualization is common in NAS or file-server environments. It provides non-disruptive file mobility to optimize storage utilization.

It provides user or application independence from the location where the files are stored. File-level virtualization creates a logical pool of storage, enabling users to use a logical path, rather than a physical path, to access files. While the files are being moved, clients can access their files non-disruptively. Clients can also read their files from the old location and write them back to the new location without realizing that the physical location has changed. A global namespace is used to map the logical path of a file to the physical path names.

File-serving environment before and after file-level virtualization

Object-based storage

In NAS, metadata are stored as part of the file distributed throughout the environment, which adds to the complexity and latency in searching and retrieving files. Object-based storage, on the other hand, stores file data in the form of objects based on its content and other attributes, rather than the name and location.

Hierarchical File System and Flat Address Space

OSD – object-based storage devices, stores data in the form of objects using flat address space. There is no hierarchy of directories and file. Object is identified by objectID, which is usually generated using hash function.

In block storage, when file system receives the IO from an application, the file system maps the incoming I/O to the disk blocks. The block interface is used for sending the I/O over the channel or network to the storage device. The I/O is then written to the block allocated on the disk drive. When an application accesses data stored in OSD, the request is sent to the file system user component. The file system user component communicates to the OSD interface, which in turn sends the request to the storage device. The storage device has the OSD storage component responsible for managing the access to the object on a storage device.

Benefit of object storage

security and reliability: OSD can use special algorithm for strong encryption capacity. Request authentication is performed at the storage device rather than with an external authentication mechanism
platform independence: standard web access via REST or SOAP
scalability: Both storage and OSD nodes can be scaled independently in terms of performance and capacity

Block-level access vs object-level access

OSD components:

nodes: a server with OSD operating environment to provide services to store, retrieve and manage data. Two key services are metadata service (generating objectID and maintaining the mapping between objectID and file) and storage service (manage a set of disks where data are stored).
private network: provides node-to-node connectivity and node-to-storage connectivity.
storage device

OSD system components

Storage mechanism

The application server presents the file to be stored to the OSD node.
The OSD node divides the file into two parts: user data and metadata.
The OSD node generates the object ID using a specialized algorithm. The algorithm is executed against the contents of the user data to derive an ID unique to this data.
For future access, the OSD node stores the metadata and object ID using the metadata service.
The OSD node stores the user data (objects) in the storage device using the storage service.
An acknowledgment is sent to the application server stating that the object is stored.

OSD: object storage

Retrieval mechanism

The application server sends a read request to the OSD system.
The metadata service retrieves the object ID for the requested file.
The metadata service sends the object ID to the application server.
The application server sends the object ID to the OSD storage service for object retrieval.
The OSD storage service retrieves the object from the storage device.
The OSD storage service sends the file to the application server.

OSD object retrieval

OSD usage: data archival, especially long-term; and cloud storage, storage as service

CAS – content addressed storage, a special type of OSD designed for secure online storage and retrieval of fixed content. Data access in CAS differs from other OSD devices. In CAS, the application server access the CAS device only via the CAS API running on the application server. However, the way CAS stores data is similar to the other OSD systems.

CAS Use case

Healthcare: storing patient studies – size of radiology study ranges from 15MB to more than 1GB. Newly acquired studies are retained for 60 days and moved to long term storage.
Finance: storing financial records – bank stores images of cheques (~25KB each) for about 90 millions a month. Images are processed in transaction system for 5 days. For the next 60 days images are requested for verifications. After 60 days access requirements drop drastically. Retention policy manages life-cycle of the images.

Unified storage

Components

storage controller: The storage controller provides block-level access to application servers through iSCSI, FC, or FCoE protocols.
NAS head: a dedicated file server that provides file access to NAS clients
OSD node: accesses the storage through the storage controller using a FC or FCoE connection.
Storage

Unified storage platform

Storage Nitty-Gritty 2 of 5 – SAN

Mon, 06 May 2019 22:06:02 -0400

In direct attached storage (DAS), storage is server centric and the host owns the storage. The storage is fully dedicated to the server that owns it.

With DAS, storage is server-centric

Storage has evolved into information centric model. In this model, when a new server is deployed in the environment, storage is assigned from the same shared storage pool to the new server.

Network based centralized storage solution

A network-based storage solution is the centralized storage pool. No single host owns the entire storage pool. The storage solution consists of two categories based on the interface with the host: SAN (storage area network) and NAS (network attached storage). To a client OS on the host, SAN typically appears as a local disk, allowing block-level access from the client OS, and therefore is more suited for structured workload such as database storage. It operates on its own storage network independent of the host network. NAS on the other hand, typically appears as a file share to the client OS, identified by an IP address and path. This is because NAS operates on the same TCP/IP network where the hosts are operated on. The client has file level access to storage, therefore NAS is better for unstructured data such as video and medical images. It is very important to understand the difference between SAN and NAS.

SAN deployment consists of two categories based on the connection technology. FC SAN is based on Fibre Channel network; and IP SAN is based on Internet protocol (iSCSI, FCIP, FCoE).

Fibre Channel SAN

Cable types: MMF (multimode fibre, usually for short distance within data centre because of signal attenuation due to modal dispersion) and SMF (single mode fibre, carries a single ray of light, used for long-distance cable runs;
Connector: standard connector (SC), lucent connector (LC) and straight tip connector (ST);
Interconnect device
- FC hub – for FC-AL implementation, but no longer in use
- FC switch – directly route data from one physical port to another (more intelligent than hub)
- Director – high end switches with redundant components to provide high availability
FC connectivity
- point-to-point: two devices connected directly to each other;
- arbitrated loop (FC-AL): devices are attached to a shared loop;

FC- AL (rarely used today)

switched fabric (FC-SW): uses switches that can switch data traffic between nodes directly through switch ports. Frames are routed between source and destination by the fabric

Fibre Channel Switched Fabric

Protocol: Fibre Channel Protocol (FCP): defines protocol stack (five layers, FC-0 through FC-4), addressing, identification (world wide name), frame, data structure, flow control, fabric services

Fibre Channel Protocol stack

FC frame

Topology: mesh topology and core-edge fabric topology
Block-level virtualization: aggregates block storage devices (LUNs) and enables provisioning of virtual storage volumes, independent of underlying physical storage. The virtualization layer maps the virtual volumes to the LUNs on the individual arrays. Block-level storage virtualization not only enables extending the storage volumes online; it consolidates heterogeneous storage arrays and enables transparent volume access. It also provides the advantage of non-disruptive data migration, where the virtualization layer handles the back-end migration of data, which enables the LUNs to remain online during migration.

Block-level virtualization (classic)

Federation of block storage across data centers (new generation)

Virtual SAN (VSAN, aka virtual fabric) – a logical fabric on an FC SAN, which enables communication among a group of nodes regardless of physical location in the fabric.

IP SAN

iSCI (one of the IP SAN protocols) – an IP based protocol that establishes and manages connections between host and storage over IP. iSCSI encapsulates SCSI commands and data into an IP packet and transport them using TCP/IP. It is relatively inexpensive and easy to implement so widespread in environments without FC SAN.

Topology
- Native connectivity (without FC components)
- Bridged connectivity (including FC components in the configuration)
- Combined connectivity (most common because a storage array usually comes with both FC and iSCSI ports)

iSCSI topologies

Protocol
- stack: SCSI is the command protocol that works at the application layer of OSI model; iSCSI is session-layer protocol that initiates a reliable session between devices that recognize SCSI commands and TCP/IP. The iSCSI session-layer interface is responsible for handling login, authentication, target discovery, and session management.

iSCSI protocol stack

iSCSI session and PDU encapsulation

PDU encapsulation

iSCSI discovery – an initiator must discover the location of its targets on the network and the names of the targets available to it before session establishment. Two types are SendTargets discovery and internet Storage Name Service
iSCSI names:
- IQN, iSCSI Qualified Name such as iqn.2008-02.com.example:optional_string;
- EUI, extended unique identifier such as eui.0300732A32598D26
iSCSI command sequencing – A command sequence may generate multiple PDUs. A command sequence number (CmdSN) within an iSCSI session is used for numbering all initiator-to-target command PDUs belonging to the session. This number ensures that every command is delivered in the same order in which it is transmitted, regardless of the TCP connection that carries the command in the session.

FCIP (one of the IP SAN protocols) – transporting FC block data over the IP infrastructure.

protocol stack and packet encapsulation

FCIP protocol stack

FCIP encapsulation

Topology (FCIP gateway involved):

FCIP topology

FCoE (one of the IP SAN protocols) – consolidation of LAN and SAN traffic over a single physical interface infrastructure. FCoE helps organizations address the challenges of having multiple discrete network infrastructures.

CNA (converged network adapters) replaces both HBAs and NICs in the server and consolidates both the IP and FC traffic
Special requirement on cables and switches
protocol stack and encapsulation

FCoE field mapping

CEE (converged enhanced Ethernet, or lossless Ethernet) provides new specification to existing Ethernet standard that eliminates the lossy nature of Ethernet. This makes 10Gb Ethernet a viable storage networking option, similar to FC. It features the following functionalities as part of IEEE 802.1:
- PFC (priority-based flow control)
- ETS (enhanced transmission selection)
- CN (congestion notification)

A review of vSphere virtualization technologies

Sat, 21 Jul 2018 15:54:00 -0400

This post is a summary of VMware’s white paper Introduction to VMware vSphere.

ESXi is the hypervisor (virtualization layer) on bare metal servers that abstracts processor, memory, storage and networking resources into multiple virtual machines. It was previously known as ESX and VMware discontinued ESX at version 4.1 so only ESXi is provided at and above version 5.0. vSphere is the platform to view, configure and manage the key aspects of virtualization, including:

computing and memory resources (hosts, clusters and resource pools)
storage resources (data stores)
networking resources (networks)
virtual machines

Under vSphere product family, vCenter Server is the central point for configuring, provisioning and managing the virtual environment. vShphere client is a client application to connect remotely to vCenter Server, or ESXi from any Windows PC. There is also vSphere Web Access for users from non-Windows environment.

For each aspects of virtualization, there is some vSphere features.

Computing

A host is a virtual representation of of the computing and memory resources of a physical machine running ESXi. When two or more physical machines are grouped to work and be managed as a whole, the aggregate computing and memory resources form a cluster. Physical machines can be dynamically added to or removed from a cluster. A cluster acts and can be managed as a single entity. It represents the aggregate computing and memory resources of a group of physical x86 servers sharing the same network and storage arrays. Computing and memory resources from hosts and clusters can be finely partitioned into a hierarchy of resource pools.

You can dynamically change resource allocation policies without shutting down the associated VMs. When reserved resources are not being used by a resource pool or a VM, the resources can be shared. This helps to maximize resource use while also ensuring that reservations are met and resource policies enforced.

ESXi provides a memory compression cache to improve VM performance when you use memory overcommitment. Memory compression is enabled by default. When a hosts memory becomes overcommitted, ESXi compresses virtual pages and stores them in memory. This is because accessing compressed memory is faster than accessing memory that has been swapped out to disk. Memory compression in ESXi allows you to overcommit memory without hindering performance. When a virtual page needs to be swapped, ESXi first attempts to compress the page. Pages that can be compressed to 2KB or smaller are stored in the VM’s compression cache, increasing the capacity of the host.

High Availability

VMware vMotion enables the migration of running VMs from one physical server to another without service interruption. The effect is a more efficient assignment of resources across physical servers.

Storage vMotion enables the migration of VMs from one datastore to another datastore without service interruption. This allows administrators to off-load VMs from one storage array to another.

VMware DRS (distributed resource scheduler) helps you manage a cluster of physical hosts as a single compute resource. You can configure DRS to execute VM placement, VM migration, and host power actions. When you create a VM on a cluster, DRS places the VM in such a way as to ensure that load across the cluster is balanced, and cluster-wide resource allocation policies (e.g. reservations, priorities, and limits) are enforced. When you add a new physical server to a cluster, DRS enables VMs to immediately take advantage of the new resources. When a VM is powered on, DRS performs an initial placement of the VM on a host. As cluster conditions (e.g. load and available resources) change over time, DRS migrates (using vMotion) VMs to other hosts as necessary.

VMware DRS

When DPM (distributed power management) is enabled, the system compares cluster-level and host-level capacity to the demands of VMs running in the cluster. If the resource demands of the running VMs can be met by a subset of hosts in the cluster, DPM migrates the VMs to this subset and powers down the hosts that are not needed. When resource demands increase, DPM powers these hosts back on and migrates the VMs to them. This dynamic cluster right-sizing that DPM performs reduces the power consumption of the cluster, without sacrificing VM performance or availability.

Storage I/O control congestion management allows cluster-wide storage I/O prioritization and enables administrator to set congestion thresholds for I/O shares.

VMware HA enables quick automated restart of virtual machines on a different physical server within a cluster if a host fails. HA monitors all physical hosts in a cluster and detects host failures. An agent placed on each physical host maintains a heartbeat with the other hosts in the resource pool. Loss of a heartbeat initiates the process of restarting all affected VMs on that host. HA also provides a VM monitoring feature that monitors the status of VM in an HA cluster. If a VM does not generate heartbeats within a specified time, VM monitoring identifies it as having failed and restarts it. HA is configured centrally through vCenter Server and once configured, it operates continuously and in a distributed manner on every ESXi host without needing vCenter Server to stay up.

VMware HA

VMware vLockstep technology and VMware Fault Tolerance provides continuous availability by protecting a VM with a shadow copy that runs in virtual lockstep on a separate host. Inputs and events performed on the primary VM are recorded and replayed on the secondary VM to ensure identical state. The secondary VM in virtual lockstep can take over execution at any point without interruption or loss of data.

Storage

Datastores are virtual representations of combinations of underlying physical storage resources in the data center. These physical storage resources include:

Local SCSI, SAS, or SATA disks attached to the physical machines
Fibre Channel or iSCSI SAN disk arrays
Network Attached Storage (NAS) arrays

Storage subsystem appears as a virtual SCSI controller connected to one or more virtual SCSI disks. These virtual controllers (BusLogic Parallel, LSI Logic Parallel, LSI Logic SAS and VMware Paravirtual) are the only types of SCSI controllers that a VM can see and access. The virtual SCSI disks are provisioned from datastore. This datastore abstraction is a model that assigns storage space to VMs while insulating the guest from the complexity of the underlying physical storage technology. The guest VM however, is not exposed to Fibre Channel SAN, iSCSI SAN, direct attached storage or NAS.

Storage Architecture

Each datastore is a VMFS volume on a storage device. Datastore can span multiple physical storage subsystems. A single VMFS volume can contain one or more LUNs from a local SCSI disk array on a physical host, a Fibre Channel disk farm, or iSCSI SAN disk farm. New LUNs added to any of the physical storage subsystems are detected and made available to all existing new datastores. Storage capacity on a previously created datastore can be extended without powering down physical hosts or storage subsystems. If any of the LUNs within a VMFS volume fails, only VMs that use that LUN are affected.

Each VM is stored as a set of files in a directory in the datastore. The disk storage associated with each VM is a set of files within the guest’s directory. You can operate on the guest disk storage as an ordinary file, which can be copied, moved, or backed up. New virtual disks can be added to a virtual machine without powering it down. In that case, a virtual disk file (.vmdk) is created in VMFS to provide new storage for the added virtual disk

VMFS is a clustered file system that leverages shared storage to allow multiple physical hosts to read and write the same storage simultaneously. VMFS provides on-disk locking to ensure that the same virtual machine is not powered on by multiple servers at the same time. If a physical host fails, the on-disk lock for each VM is released so that VMs can be restarted on other physical hosts. VMFS also features failure consistency and recovery mechanisms, such as distributed journaling, a failure-consisten VM I/O path, and VM state snapshots. These mechanisms can aid quick identification of the cause and recovery from VM, physical host and storage subsystem failures. VMFS also supports raw device mapping (RDM), which is a mechanism for a VM to have direct access to a LUN on the physical storage subsystem (Fibre Channel or iSCSI only). An RDM is a symbolic link from a VMFS volume to a raw LUN. The mapping makes LUNs appear as files in a VMFS volume. The mapping file, not the raw LUN, is referenced in the VM configuration. When a LUN is opened for access, the mapping file is read to obtain the reference to the raw LUN. Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.

Raw Device Mapping

Networking

Each VM has one or more vNICs (virtual network interface cards). The guest OS and application program communicate with a vNIC through either a commonly available device driver or a VMware device driver optimized for the virtual environment. In either case, communication in the guest OS occurs just as it would with a physical device. On the network, the vNIC responds to standard Ethernet protocol as would a physical NIC. An outside agent does not detect that it is communicating with a virtual machine.

A virtual switch (vSwitch) works like a layer 2 physical switch. Each server has its own virtual switches. One one side of the virtual switch are port groups that connect to virtual machines. On the other side are uplink connections to physical Ethernet adapters on the physical server where the virtual switch resides. VMs connect to the outside world through the physical Ethernet adapters that are connected to the virtual switch uplinks. A virtual switch can connect its uplinks to more than one physical Ethernet adapter to enable NIC teaming.

Port group is a unique concept in the virtual environment. A port group is a mechanism for setting policies that govern the network connected to it. A vSwitch can have multiple port groups. A VM connects its vNIC to a port group instead of to a particular port on the vSwitch, for better network segmentation.

A vNetwork Distributed Swtich (vDs) function as a single virtual switch across all associated hosts. This functionality allows VMs to maintain consistent network configuration as they migrate across multiple hosts. Like vSwitch, each VDS is a network hub that VMs can use and it can route traffic internally between VMs or link to an external network by connecting to physical Ethernet adapters. Each vDS can also hae one or more dvPort groups assigned to it. dvPort groups aggregate multiple ports under a common configuration and provide a stable anchor point for VMs connecting to labeled networks.

When network resource management is enabled, vDS traffic is divided into six network resource pools: FT traffic, iSCSI traffic, vMotion traffic, management traffic, NFS traffic, and VM traffic. You can control the priority of each of these network resource pools.

vCenter

vCenter Server provides centralized managed for data centers. It communicates with the ESXi host agent through the VMware vSphere API. When you first add a host to vCenter Server sends a vCenter Server agent to run on the host. The vCenter Server agent acts as a small vCenter Server to perform many fundamental management functions.