cache on Digi Hunch

GitHub Action Gotchas

Fri, 27 Jan 2023 01:15:02 -0400

I started with GitHub Actions a couple years ago. Recently I came across a few interesting use cases while I was trying to setup Terraform workflow with GitHub actions. These use cases prompted me to make use some new features in GitHub Action. So I put them in a post here.

Runners can assume IAM Role in AWS

In many scenarios we want to execute AWS CLI command from GitHub action. Also, executables such as terraform inherits credential from AWS CLI. The credential should be a temporary role-based credential instead of an IAM user based on access keys.

There is a GitHub Action called configure-aws-credentials-for-github-actions that can help configure GitHub runner using OIDC identity provider (since Nov 2021 v1.6.0). With the action, the GitHub runner can assume an IAM role as an IAM user (with access key), or using a web identity.

For a GitHub runner to have a web identity thereby assume an IAM role, we should configure OIDC provider in AWS. We can do that from AWS console (i.e. under IAM), or using CloudFormation code. Below is a snippet as an example:

Resources:
 GitHubOIDC:
 Type: AWS::IAM::OIDCProvider
 Properties:
 Url: https://token.actions.githubusercontent.com
 ClientIdList: 
 - sts.amazonaws.com
 ThumbprintList:
 - 6938fd4d98bab03faadb97b34396831e3780aea1

Then from the configured OIDC provider, we can obtain a thumbprint. GitHub action gives the thumbprint here. In AWS, we configure an IAM role whose AssumeRolePolicyDocument will reference the thumbprint. Here is an example. In the condition section of AssumeRolePolicyDocument, we can also specify a specific GitHub repository so that only Actions from that repository can assume the IAM role with their web identities.

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1-node16
 with:
 role-to-assume: ${{ vars.IAM_ROLE_ARN }}
 aws-region: ${{ vars.AWS_REGION }}

This way, we map a GitHub runner’s web identity to an IAM role with a step using the Action above. We also filter what GitHub org and what repo can trigger actions that assumes the role, by the condition clause in the role statement. If the step fails, we can look at CloudTrail on the AWS side for causes. Look for entries with AssumeRoleWithWebIdentity as Event Name.

Reusable workflows

For better reusability of Action steps, GitHub introduced reusable workflows (generally available since Nov 2021). It is particularly helpful when we need to run a workflow for different environments. The reusable workflows files (YML) can be placed in separate repositories, and be reference as such. This allows enterprises to centralize the management of reusable workflows.

I have been using the act project to emulate GitHub action locally on MacBook. As of Jan 2023, act does not support reusable workflow.

With the split between caller and reusable workflows, we have a new challenge of passing secrets and variables between them. It is not straightforward and GitHub documentation needs improvement to get the documentation clear. Also because the word “environment” is used in different contexts, it is ambiguous and therefore difficult to Google relevant information.

Passing variables

First, there are several types of variables in GitHub action:

Environment variable: declared under env keyword in a workflow. To use environment variable, use the env context. For example: ${{ env.MY_VARIABLE }}
Configuration variable: introduced in Jan 2023, configuration variables are defined at repository, environment and organization levels. To use configuration variable, use vars context, and ensure the workflow job specifies a value for environment attribute.
Secrets: GitHub also calls it Environment secret when defined at environment level. It works the same way as a configuration variable because it is also specific to an environment. The content is not viewable once set.

The reason GitHub action makes this so confusing, is that on one page, its documentation distinguishes between environment variable and configuration variable:

On another page, the document refers to configuration variables at environment level as environment variable:

It seems that “configuration variable” is too new for GitHub to refine its documentation as of January. This semantical confusion gave me a hard time investigating how to pass “Environment variable” to reusable workflows. I will stick to the meaning on the first page to distinguish environment variable and configuration variable at environment level.

Passing environment variable isn’t straightforward. In this discussion thread, people discussed how inconvenient it is. I used the workaround in this comment, where I had to create a job for the sake of storing variable values to output.

Pass secret is easier. This is an insightful blog post (Dec 2021) about passing secret to reusable workflow. The attempt 3 in the post works for me. First, we pass the value of environment to the reusable workflow as an input, then at job level specify the environment with the value. Then in the jobs we can reference secrets as ${{ secrets.NAME }}. The job will pick up the secret based on the correct environment.

It appears that since May 2022, GitHub introduced secrets: inherit keyword to address this. However, the method above still works for configuration variable.

Authentication of GitHub Actions

By default, a GitHub action can access the code repository that triggers the action and no other repositories (with GITHUB_TOKEN). However, in many cases we need to access external repositories. For example, terraform init command from a GitHub action implicitly calls git clone to pull module code from external repositories. A GitHub workflow may also reference a workflow file from external repositories.

The question is how to authenticate GitHub workflow to access external repo. This post has a thorough discussion. We may create a Personal Access Token and pass it to set-git-credentials action. We are essentially sharing a personal credential (and repo access) with a GitHub action, which is not a good practice.

The proper way to solve this problem, is to create a separate GitHub App and grant the access only the repo that the workflow needs to access. The GitHub App will generate a private key. Then we supply the private key to workflow-application-token-action so the workflow can act as the GitHub App, thereby access the external repos. The post has more details in the GitHub App section. Suppose we have terraform get command to clone external repo, the actions may look like this:

 - name: HashiCorp - Setup Terraform
 uses: hashicorp/setup-terraform@v2

 - name: Get RepoReader App Token
 id: get_repo_reader_token
 uses: peter-murray/workflow-application-token-action@v2
 with:
 application_id: ${{ vars.REPO_READER_APPLICATION_ID }}
 application_private_key: ${{ secrets.REPO_READER_PRIVATE_KEY }}

 - name: Cache Git Creds
 uses: de-vri-es/setup-git-credentials@v2
 with:
 credentials: https://x-access-token:${{ steps.get_repo_reader_token.outputs.token }}@github.com/

 # Terraform Get implicitly calls git clone which uses the credential cached as above
 - name: Terraform Get 
 run: terraform get

Another benefit of using GitHub App is that the token is a short-lived credential that expires as the job is finished, whereas a PAT will expire on a preset date. In this use case we can think of GitHub App as a service account with minimized privilege to read a short list of repos.

Final words

I came across this reddit post against pipeline use and there are some legit points. For example, the user is frustrated with the limitation with variable passing and unclear documentation. The post wasn’t specific to GitHub action but I do share some frustration while working with GitHub Actions.

On a positive note, since I first used GitHub actions, it has evolved quite a bit with new features, although the documentation is somewhat lagging. It is still very helpful as all of these are free to personal use. I look forward to more interesting features.

EMC Isilon storage product

Wed, 08 Jul 2020 20:04:00 -0400

EMC has several product lines for different use cases in enterprise data storage. Like may other IT solutions, the website is clouded with marketing terms and slogans, and is purposefully not technical. This makes it difficult for technical staff to grasp the advantage of its product in a glimpse. I personally have to know their product (mostly with Isilon and ECS) well in order to make integration decisions. So I’m putting together this note (updated as of July 2020), with lots of details from their technical white paper.

Overview of EMC storage

At the highest level, the EMC enterprise data storage product lines are categorized into two groups: primary storage (along the lines of block-level storage) and unstructured storage (mostly file and object storage). The primary storage includes the following product:

PowerMax for OLTP database (Oracle, MicrosoftSQL and SAP)
PowerFlex: for Software defined storage, Oracle RAC, Elastic Stack, Kubernetes, Splunk
XtremIO for VMware, VDI, SAP
PowerStore for Database, VMware
PowerVault for Entry-level SAN and DAS environment

This post only expands on the unstructured storage product line, which mainly consists of PowerScale and ECS. ECS (elastic cloud storage) is EMC’s object storage. PowerScale (aka Isilon) is scale-out NAS platform for high-volume storage (up to 50 PB in a single file system), backup and archiving of unstructured data. For the rest of this post, I will still refer to PowerScale as Isilon.

Dell’s official support website is the most resourceful place to get information. For example, when I want to read about Isilon. I start with Dell support, then click on knowledgebase at the top, then go to “servers, storage and networking“, then “storage technical documents and videos“. There I can select a productline such as Isilon.

Overview of Isilon Family

Isilon is a clustered storage system consisting of three or more nodes. A node is a server with OneFS as its operating system. Based on FreeBSD, OneFS is EMC’s proprietary operating system to unify a cluster of nodes into a single shared resource. So OneFS is for Isilon only. It is the basis of Isilon. Isilon has three series of products:

F series: F200, F600, F800 and F810.
H series: typical models are H400, H500 and H600, which seeks to balance performance and capacity
A series: typical models are A200 and A2000 for active and deep archive storage

In June 2020, Dell decoupled OneFS software (with 9.0 released) from server hardware (referred to as PowerScale). Going forward EMC will refer to Isilon as PowerScale for OneFS version newer than 9.0 in spec sheets and white papers.

F200 is the cost-effective choice with SSD for remote office, small hospital, retail outlets, IOT or factory floor. F600 uses NVMe drives instead, and has more ECC memory and faster ethernet backend network. and is higher than F200 in its use case. Both F200 and F600 provide inline data compression and deduplication capabilities. F800 and F810 both use SSD and they are similar. F800 comes with InfiniBand backend network and F810 provides inline data compression and deduplication capabilities. H series tries to strike a balance between performance and capacity so they are pretty much everything in betwee. On the other end, A200 and A2000 are almost the same except for capacity difference.

Isilon’s advantage

Isilon has lots of intelligence built into its solution compared to a traditional NAS. Here are some aspects from its product white paper:

Aspects of Design	Isilon OneFS Scale-Out NAS	Traditional NAS
Network	Separation of front-end and back-end network to isolate node-to-node communication to a private low-latency network. Front-end traffic load balanced with SmartConnect	Single network for both external and internal traffic
File system structure and NameSpace	The storage is completely virtualized to users as a truly single file system with one namespace. There is no partitioning or volumes. The single file tree can grow organically without requiring planning or oversight about how the tree grows. SmartPool handles tiering of files to appropriate disk, without disrupting the single file tree.	An appearance of single namespace is typically achieved through namespace aggregation, where files are still managed in separate volumes, and a simple “veneer” layer glues individual directories to a “top-level” tree via symbolic links. LUNs and volumes, as well as volume limits are still present. Files have to be manually moved from volume-to-volume to load-balance.
Data Layout	OneFS controls the placement of file directly, down to the sector-level on any drive anywhere in the cluster. The addressing scheme for data and metadata is indexed at physical level by a tuple of {node, drive, offset}	Data are sent through RAID and volume management layers, introducing inefficiencies in data layout and providing non-optimized block access.
Redundancy Control	OneFS can flexibly control the type of striping as well as the redundancy level of the storage system at the system, directory and even file-levels.	The entire RAID volume is dedicated to a particular performance type and protection setting.

Isilon terms

The Isilon technology re-implemented the read and write path during file storage and introduced several terms along with its technology.

SmartPools – Job that runs and moves data between the tiers of nodes within the same cluster. Also executes the CloudPools functionality if licensed and configured. FilePolicy is changelist-based SmartPools file pool policy job. SmartPoolsTree enforces SmartPools file policies on a subtree.

Storage Pools – Storage pools provide the ability to define subsets of hardware within a single cluster, allowing file layout to be aligned with specific sets of nodes through the configuration of storage pool policies. The notion of Storage pools is an abstraction that encompasses disk pools, node pools, and tiers.

Disk Pools – Disk pools are the smallest unit within the storage pools hierarchy. OneFS provisioning works on the premise of dividing similar nodes’ drives into sets, or disk pools, with each pool representing a separate failure domain. Disk pools are laid out across all five sleds in each node.

Node Pools – groups of disk pools, spread across similar storage nodes (or equivalent classes). Multiple groups of different node types can work together in a single, heterogeneous cluster. For example, one node pool of all-flash F-Series anodes, one node pool of H-series, and one node pool of A-series. Each node pool only contains disk pools from the same type of storage nodes.

Tiers – groups of nodepools combined into a logical superset to optimize data storage, according to OneFS platform type. this allows customers who consistently purchase highest capacity nodes available to consolidate a variety of node styles within a single tier, and manage them as one logical group. SmartPools users typically deploy 2 to 4 tiers. different node pools under a tier needs to be compatible.

Global Namespace Acceleration (GNA)’s principal goal is to help accelerate metadata read operations by keeping a copy of a cluster’s metadata on high performance, low latency SSD media.

SmartConnect is a load balancer that works at the front-end Ethernet layer to evenly distribute client connections across the cluster. SmartConnect supports dynamic NFS failover and failback to ensure that when a node failure occurs, or preventative maintenance is performed, all in-flight reads and writes are handed off to another node in the cluster to finish its operation without any user or application interruption.

Auto Balance reallocates and rebalances data and make storage space more usable and efficient.

SmartQuotas is directory-level quota management. Note: there is no partitioning, and no need for volume creation in OneFS.

SmartRead creates a data “pipeline” from L2 cache, prefetching into a local “L1” cache, on the captain node, in order to greatly improve sequential-read performance. For high-sequential cases, SmartRead can very aggressively prefetch ahead. SmartRead can control how aggresive the pre-fetching is, and how long data stays in the cache, and optimizes where data is cached.

In-line Data Reduction – the write path involves zero block removal, in-line deduplication, and in-line compression. This is supported in some models only.

Smart Dedupe – post-process, asynchronous deduplication. Smart Dedupe scans the on-disk data for identical blcoks and then eliminate the duplicates. After duplicate blocks are discovered, SmartDedupe movees a single copy of those blocks to a special set of files known as shadow stored. With post-process deduplication, new data is first stored on the storage device and then a subsequent process analyzes the data looking for commonality. This means that initial file write or modify performance is not impacted, since no additional computation is required in the write path, as opposed to in-line deduplication. This is supported on some models only.

OneFS SSD strategy – How OneFS leverage the SSD for performance. It has these options:

L3 cache (implemented at nodepool level)
metadata read
metadata read/write
Global Namespace Acceleration (GNA)
Data on SSD

L3 cache consumes all the SSD in node pool. L3 cannot coexist with other SSD strategies, with the exception of GNA just because L3 cache node pool SSD cannot participate in GNA.

Isilon’s High Availability

The OneFS is distributed across all nodes in the cluster and is accessible by clients connecting to any node in the cluster. Metadata and locking tasks are managed by all nodes collectively and equally in a peer-to-peer architecture. This symmetry is key to the simplicity and resiliency of the architecture. There is no single metadata server, lock manager or gateway node.

The entire cluster forms a single file system with a single namespace that runs across every node equally. No one node controls or “masters” the cluster; all nodes are true peers.

During failover, clients are evenly redistributed across all remaining nodes in the cluster, ensuring minimal performance impact. If a node is brought down for any reason, including a failure, the virtual IP addresses on that node is seamlessly migrated to another node in the cluster. When the offline node is brought back online, SmartConnect automatically rebalances the NFS and SMB3 clients across the entire cluster to ensure maximum storage and performance utilization. This functionality allows for per-node rolling upgrades affording full-availability throughout the duration of the maintenance window.

There are two logical roles in processing an I/O request from client:

The initiator: the node that the client connects to with front-end protocol. The initiator acts as the ‘captain’ for the entire I/O operation.
The participant: Every node in the cluster is a participant for a particular I/O operation.

File Write in Isilon

OneFS employs a patented transaction system during write to eliminate single point of failure. In a write operation, the initiator “captains” or orchestrates the layout of data and metadata, the creation of erasure codes, and the normal operations of lock management and permission control.

When a client connects to a node to write a file, it is connecting to the Initiator. OneFS breaks the file down into atomic units. An atomic unit is a smaller logical chunk of data, also called stripe, or protection groups in the context of data protection. The size of each file chunk is referred to as the stripe unit size. After this division, OneFS then write the stripe individually to the Participant (with disks). This design ensures that data is protected at the specified level as soon as it is being written. Redundancy is built into protection groups, such that if every protection group of a file is safe, then the entire file is safe. In terms of protection mechanism, OneFS can use either Reed-Solomon erasure coding system, or simply mirroring for data protection. Erasure coding is the predominant mechanism with very high performance without sacrificing on-disk efficiency.

The initiator node uses a modified two-phase commit transaction to safely distribute writes to multiple NVRAMs across the cluster. As client initiates write to OneFS cluster, instead of immediately writing to disk, OneFS temporarily writes the data to an NVRAM-based journal cache on the initiator node and acknowledge the write the client. As outlined above, these writes are also mirrored to participant nodes’ NVRANM journals to satisfy the file’s protection requirement. Later, at a more convenient time, OneFS then flush these cached writes to disks asynchronously.

Since NVRADM journals all the transactions that are occurring across every node in the storage cluster. If a node fails mid-transaction, and then re-joins the cluster, the uncommitted cached writes are fully protected, and the only required actions for the node, are to replay its journal from NVRAM, and occasionally for AutoBalance to rebalance files that were involved in the transaction. Writes are never blocked due to a failure. There is no ‘fsck’ or ‘disk-check’ process.

OneFS file system block size is 8KB. A file smaller than 8KB will use a full 8KB block. For larger files, OneFS can maximize sequential performance by taking advantage of a stripe unit consisting of 16 contiguous blocks, for a total of 128KB per stripe unit.

Cache in Isilon

OneFS aggregates the cache present on each node in a cluster into one globally accessible pool of memory by using a messaging system similar to NUMA (non-uniform memory access). This allows all the nodes’ memory cache to be available to each and every node in the cluster. Remote memory is access over internal network with much lower latency than accessing hard disk drives. The internal network as distributed system bus, is a redundant, under-subscribed flat Ethernet up to 40Gb. The oneFS caching subsystem is coherent across the cluster, due to the use of MESI protocol to maintain cache coherency. If the same content exists in the private caches of multiple nodes, this cached data is consistent across all instances.
OneFS uses up to three levels of read cache, plus an NVRAM-backed write cache, or coalescer.

OneFS Caching Hierarchy

L1 cache – prefetches data from remote nodes. Data is prefetched per file, and this is optimized in order to reduce the latency associated with the nodes’ back-end network. The L1 cache refers to memory on the same node as the initiator. It is only accessible to the local node, and typically the cache is not the master copy of the data.

L1 is also known as remote cache because it contains data retrieved from other nodes in the cluster. It is coherent across the cluster but is used only by the node on which it resides and is not accessible by other nodes. Data in L1 cache on storage nodes is aggressively discarded after it is used. L1 cache uses file-based addressing, in which data is accessed via an offset into a file object.

OneFS also uses a dedicated inode cache in which recently requested inodes are kept. The inode cache frequently has a large impact on performance, because clients often cache data, and many network I/O activities are primarily requests for file attributes and metadata, which can be quickly returned from the cached inode.

L2 cache (backend cache) refers to local memory on the node on which a particular block of data is stored. L2 cache is globally accessible from any node in the cluster and is used to reduce the latency of a read operation by not requiring a seek directly from the disk drives.

L2 cache is also known as local cache because it contains data retrieved from disk drives located on that node and then made available for requests from remote nodes. Data in L2 cache is evicted according to a Least Recently Used (LRU) algorithm.

L3 cache, or Smart Flash, is configurable on nodes that contain solid state drives. Smart Flash (L3) is an eviction cache that is populated by L2 cache blocks as they are aged out from memory.

During I/O request, clients talk to L1 cache and write coalescer; L1 cache talks to L2 cache on all cluster nodes. L2 cache buffers to and from disks. L3 cache is optionally enabled per node pool, as an extension from L2. L3 and L2 communicate in backend network.

Name	Medium	Description
L1 Cache (aka front-end cache or remote cache)	RAM (volatile)	holds clean, cluster coherent copies of file system data and metadata blocks requested by clients via front-end network
L2 Cache (aka back-end cache or local cache)	RAM (volatile)	contains clean copies of file system data and metadata on a local node
SmartCache (Write Coalescer)	Battery-backed NVRAM (Persistent)	a persistent journal cache that buffers any pending writes to front-end files that have not been committed to disk
SmartFlash or L3 Cache	SSD (persistent)	contains file data and metadata blocks evicted from L2 cache, effectively increasing L2 cache capacity

File Read in Isilon

The high-level steps for fulfilling a read request with cache interaction involves:

Step 1 – on local node, determine whether part of the requested data is in the local L1 cache:

if so, return to client
if not, request data from remote nodes

Step 2 – on remote nodes, determine whether requested data is in the local L2 or L3 cache:

if so, return to the requesting node
if not, read from disk and return to requesting node

During a read operation, the “captain” node gathers all of the data from the various nodes in the cluster and presents it in a cohesive way to the requestor.

The cluster provides a high ratio of cache to disk (multiple GB per node) that is dynamically allocated for read and write operations as needed. This RAM-based cache is unified and coherent across all nodes in the cluster, allowing a client read request on one node to benefit from I/O already transacted on another node. As the cluster grows larger, the cache benefit increases. For this reason, the amount of I/O to disk on a cluster is generally substantially lower than it is on traditional platforms.

For files marked with an access pattern of concurrent or streaming, OneFS can take advantage of pre-fetching of data based on heuristics used by the SmartRead component

Conclusion

This post provided a high level introduction to EMC storage product line and expanded into some technical details in the read write operation in OneFS/Isilon. Some of the features can be seen in OneFS simulator which is a free tool from EMC.

Performance Analysis

Fri, 19 Jun 2020 16:47:01 -0400

Overview

In 2015, Brendan Gregg posted two great articles on Netflix blog: Linux Performance Analysis in 60 seconds, and Linux Perfomrance Tools. They have great value when I was in a urgency to spot performance issues. The articles cover the essential tools for performance troubleshooting, including:

Check out load averages: w or uptime
Print kernel ring buffer: dmesg -T
Virtual memory status: vmstat 1
Multiple processor staticstics: mpstat -P ALL 1
Task status: pidstat 1
CPU and I/O status: iostat -xz 1
Free memory check: free -m
Network Activity record: sar -n DEV 1
TCP activity record: sar -n TCP,ETCP 1
Display processes: top

We will dive into each of them in the next section.

Basic Troubleshooting

The command w is equivalent of uptime (which shows uptime since boot) and who (which shows logged-in users). It also displays load average for the last 1 minute, 5 minutes and 15 minutes. The number of load average reflects the overall system load (CPU + disks), and it is further discussed in this post with a simple take away:

If the averages are 0.0, then your system is idle.
If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
If they are higher than your CPU count, then you might have a performance problem (it depends).

When Linux load averages increase, you know you have higher demand for resources (CPUs, disks, and some locks), but you aren’t sure which. You will need to switch to other metrics. Brendan recommend don’t spend more than 5 seconds on these numbers.

[dhunch@c7v-ghintapp01 ~]$ w
 12:14:10 up 46 days, 16:41, 3 users, load average: 2.69, 2.44, 2.29
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dhunch pts/0 w6v-ghas01 24Jun20 3days 0.36s 0.30s ssh c7v-bastion
dhunch pts/1 202.95.88.111 12:02 2.00s 0.00s 0.00s w
dhunch pts/4 w6v-ghas01 17Jun20 15days 0.15s 0.07s view readme.txt

Before moving to more insightful metrics, it is also worth a quick look into the kernel ring buffer with dmesg command (dmesg -T | less +G). This will allow us to capture obvious issues such as oom-killer or TCP request dropping.

The vmstat tool reports the statistics of virtual memory. Servers have a fixed amount of physical memory, but they can run a set of applications that use a much larger amount of virtual memory. Application tend to reserve more memory than they need, and they usually operate on only a subset of their memory. In both cases, the operating system can keep the unused parts of memory on disk, and page it into physical memory only if it is needed. For the most part, this kind of memory management works well. But it doesn’t always with Java applications due to Java heap. Once a system start swapping – moving pages of data from main memory to disk, and vice versa, the performance tend to be bad. Systems must be configured so that swapping never occurs.

[dhunch@c7v-ghintapp01 ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 3 0 239360 385928 0 36734692 0 0 1 200 0 0 6 1 93 0 0
 3 0 239360 387732 0 36734704 0 0 0 43 8017 8524 14 0 85 0 0
 3 0 239360 387608 0 36734904 0 0 0 57 6768 7680 14 1 86 0 0
 2 0 239360 389008 0 36734904 0 0 0 44 6366 7300 14 0 86 0 0
 3 0 239360 421728 0 36700144 0 0 0 0 8141 7957 13 0 86 0 0
 2 0 239360 421984 0 36702048 0 0 0 467 8994 8362 14 0 85 0 0

The tool prints key server statistics each line, with the first line showing the average since boot. Here lists the explanation of some columns:

r: number of processes running on CPU and waiting for a turn. This provides a better signal than load averages for determining CPU saturation, as it does not include I/O. To interpret: an “r” value greater than the CPU count is saturation.
swpd: the amount of virtual memory used. This number should align with the used column for Swap row from free command.
buff, cache: buffer and cache. They should align with the buff/cache column form Mem row from free command.
free: free memory in kilobytes. This number should align with the free column for Mem row from free command.
si, so: swap-ins and swap-outs. As mentioned, if these are non-zero, you’re out of memory.
bi, bo: blocks received from and sent to a blcok device (# of block per second)
in, cs: number of interrupt, and context switches per second.
us, sy, id, wa, st: user, system, idle, wait I/O and stolen times. These are breakdowns of CPU time, on average across all CPUs. They should add up to 100% (or close). stolen time is amount of CPU time needed by a guest virtual machine that is not provided by the host. IO wait time is the CPU time waiting for I/O activity. Idle time could be several things: the process may be waiting for something (e.g. a response from database); the process may be blocked by a thread lock; or the process simply has nothing to do. user and system times are CPU times spent on user tasks and kernel tasks, respectively.

Out of these columns, watch for columns r, free, buff, cache, us, sy, id and wa at minimum. The combination of us and sy confirms if CPUs are busy. A constant degree of wa points to a disk bottleneck with too much time spent on pending disk I/O. The sy (kernel time) is necessary for I/O processing but sy stays high (e.g. constantly over 20%), it becomes interesting. Perhaps the kernel is processing I/O inefficiently.

For further per-CPU stats, use mpstat command (-P ALL), to prind CPU time breakdowns per CPU and check for imbalance. A single host CPU can be evidence of a single-threaded application. Here is an example output from a system of 16 CPU cores.

[dhunch@c7v-ghintapp01 ~]$ mpstat -P ALL 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

13:47:23 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
13:47:24 all 13.77 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 86.05
13:47:24 0 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 1 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 3 98.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
13:47:24 4 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 5 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 6 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 7 2.94 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 96.08
13:47:24 8 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 9 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 10 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
13:47:24 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 12 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 13 1.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 14 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

For a per process summary of CPU consumption, use pidstat command. It can be thought of a periodical snapshot of top command, allowing you to watch for patterns. The %CPU column is the total across all CPUs so 5 CPUs have a maximum value of 500.

If vmstate indicates some I/O issue, iostat tool can help us understand block devices, on both the workload applied and the resulting performance. Key columns are:

r/s, w/s, rkB/s, wkB/s: delivered reads, writes, read Kbytes, and write Kbytes per second to the device. Use these for workload characterization. A performance problem may simply be due to an excessive load applied.
await: the average wait time for I/O in milliseconds. This is the time that the application suffers, as it includes both time queued and time being serviced. Larger than expected average times can be an indicator of device saturation, or malfunction.
avgqu-sz: the average number of requests issued to device. Values greater than 1 can be evidence of saturation (although devices can typically operate on requests in parallel, especially virtual devices which front multiple back-end disks.)
%util: device utilization. This is really a busy percent, showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance (which should be seen in await), although it depends on the device. Values close to 100% usually indicate saturation.

I/O problem may either be inefficiencies in application that issues I/O request, or slowing disk unable to keep up with I/O requests. We review two examples here to illustrate each situation. The first output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 23.45 0.00 37.89 0.10 0.00 38.56

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 11.60 0.60 24.20 0.02 0.14 13.35 0.15 6.06 5.33 6.08 0.42 1.04

In the first example, the disk stat loosk up at first glance. The w_await (time to service I/O write) is fairly low at 6.08ms. However, the system is spending 37.89% of its time in the kernel. If all that system time is from the application, it suggest something inefficient is happening. The fact that the system is doing 24.2 writes per second is another clue: that is alot when writing only 0.14 MB per second (MBps). I/O has become a bottleneck, and the next step would be to look into how the application is performing its writes.

The second example output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 35.05 0.00 7.85 47.89 0.00 9.20

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 0.20 1.00 163.40 0.00 81.09 1010.19 142.74 866.47 97.60 871.17 6.08 100.00

In this example, it tells us that processes are spending 47.89% of their time in iowait, and the data to complete the I/O (w_await) is 871ms, the queue size is large, and the disk is writing at 81MB per second. This all points to disk I/O as a problem and that the amount of I/O in the application (or elsewhere in the system) must be reduced.

Bear in mind that poor performing disk I/O isn’t necessarily an application issue. Many techniques are typically used to perform I/O asynchronously, so that the application doesn’t block and suffer the latency directly (e.g. read-ahead for reads, and buffering for writes, also refer to “tail latency“).

Note that the acceptable utilization metric depends on the configuration of block device. If the storage is a logical disk device fronting many back-end disks (e.g. RAID 0), then 100% utilization may just mean that some I/O is being processed 100% of the time, however, the back-end disks may be far from being saturated, and may even be able to handle more work.

The free command gives the breakdown of memory usage. The right two columns are:

buffers: for the buffer cache, used for block device I/O.
cached: for the page cache, used by file systems.

We just want to check that these aren’t near-zero in size, which can lead to higher disk I/O (confirm using iostat), and worse performance. Linux uses free memory for the caches, but can reclaim it quickly if applications need it. So in a way the cached memory should be included in the free memory column. In this case, it’s included in the available column. This website has further details.

To check interface stat, nicstat is a great tool but it isn’t available by default in Linux. Instead we can run sar (-n DEV) to retrieve stats.

[dhunch@c7v-ghintapp01 ~]$ sar -n DEV 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:51:25 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:26 eth0 3089.00 934.00 3815.33 834.61 0.00 0.00 0.00
16:51:26 lo 464.00 464.00 2289.07 2289.07 0.00 0.00 0.00

16:51:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:27 eth0 956.00 586.00 826.66 211.34 0.00 0.00 0.00
16:51:27 lo 213.00 213.00 196.00 196.00 0.00 0.00 0.00

16:51:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:28 eth0 349.00 181.00 52.32 147.19 0.00 0.00 0.00
16:51:28 lo 244.00 244.00 81.13 81.13 0.00 0.00 0.00

Here rxkB/s/s and txkB/s represents receive and transmission throughput, as a measure of workload. If they reach the limit then the interface is the bottleneck.

On top of interface is the TCP layer, which can be monitored with sar again (-n ECP, ETCP). The key metrics are:

active/s: number of locally-initiated (e.g. via connect()) TCP connections per second
passive/s: number of remotely-initiated (e.g. via accept()) TCP connections per second
retrans/s: number of TCP retransmits per second

The active and passive counts are often useful as a rough measure of server load. It might help to think of active as outbound, and passive as inbound, but this isn’t strictly true (e.g. consider a localhost to localhost connection). Retransmits are a sign of network or server issue; it may be an unreliable network (e.g. public Internet), or it may be due to a server being overloaded and dropping packets.

[dhunch@c7v-ghintapp01 ~]$ sar -n TCP,ETCP 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:52:29 active/s passive/s iseg/s oseg/s
16:52:30 0.00 1.00 28.00 35.00

16:52:29 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:30 0.00 0.00 0.00 0.00 0.00

16:52:30 active/s passive/s iseg/s oseg/s
16:52:31 8.00 8.00 200.00 317.00

16:52:30 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:31 0.00 1.00 1.00 0.00 3.00

Last but not least is our favourite command top, which includes many of the metrics covered in previous tools. The downside to top is it is harder to see patterns over time, which may be more clear in tools like vmstat and pidstat, both of which produce rolling output.

Several tools introduced here involves sar, which is a great monitoring tool on its own that we need to be familiar with.

System Activity Report (SAR)

Further to the basic tools, the sar command is very helpful as it stores historical stat every 10 minutes. Sar keeps 18 types of reports, all stored in /var/log/sa/. When viewing the report file, you may pipe the result to less command so it only prints header once. For example, if you would like to print CPU report for the 2nd of the month:

# sar -u -f /var/log/sa/sar02 | less

If you check NFS client statistics for the 31st

 # sar -n NFS -f /var/log/sa/sar31

If you need to check network server statistics for today

# sar -n NFS -f /var/log/sa/sar31

Below are all types of reports:

-u CPU utilization
-w task creation and system switching activity
-W swapping statistics
-B report paging
-b report I/O and transfer rate statistics
-R report memory statistics
-r memory utilization
-S swap space utilization
-H huge pages utilization statistics
-v inode
-q queue length and load average
-y TTY device activity
-d activity for each block device
-n network statistics, DEV (per interface)
-n network statistics, EDEV (error per interface)
-n network statistics, NFS (NFS client)
-n network statistics, NFSD (NFS server)
-n network statistics, SOCK (socket usage)

Berkeley Packet Filter (BPF) Compiler Collection (bcc) tools

For advanced, low-level performance troubleshooting, the BCC tools provide a suite of tools. Here we only cover the installation of it.

In CentOS, install bcc-tools package with yum. When you try to run a command, such as cachestat, if you come across this error:

-bash: cachestat: command not found

Then you will need to add its path to default:

export PATH=$PATH:/usr/share/bcc/tools

Now if you run into this error:

chdir(/lib/modules/3.10.0-1062.12.1.el7.x86_64/build): No such file or directory
Traceback (most recent call last):

The file listed is a symbolic link, and if it is missing, you just need to install kernel-headers that matches the kernel version:

yum install kernel-headers

Then you may use tools in /usr/share/bcc/tools. For example, cachestat help you display page cache hit ratio; gethostlatency shows DNS resolution latency; tcpconnect prints out active tcp connections (made via connect system call):

[root@dhunch ~]# /usr/share/bcc/tools/tcpconnect -t -P 8080 | gawk '{ print strftime("%F %T  "), $0 }'
2020-06-13 00:16:57   TIME(s)  PID    COMM         IP SADDR            DADDR            DPORT
2020-06-13 02:16:57   0.000    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.064    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.438    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080

The command above outputs a time and pid stamped log line every time a TCP connection is made to port 8080; tcpaccept traces passive tcp connections (via accept system call). These tools are not as intrusive as tcpdump.

It is beyond the purpose of this article to get into details of each tool in the BFP suite. The tools are covered in detail in books “BPF Performance Tools” and “Linux Observability with BPF”.

Balloon steals memory from virtual machines

Thu, 07 May 2020 19:49:00 -0400

This article is my experience with memory balloon on virtual machine.

I came across an ElasticSearch server (ESXi guest with 32GB physical memory) where the main process keeps dying of OOM. Even worse, after the OOM event, the free memory left is about 10G and Elastic Search cannot start because its JVM is set with -Xms16g in /etc/elasticsearch/jvm.options. So I need to address the OutOfMemory error.

To understand what triggered OOM, we can use dmesg or just check /var/log/message, where the memory snapshot by OOM killer is displayed during the kernel panic:

Apr 26 04:18:15 elastichost kernel: kworker/7:1 invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: kworker/7:1 cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 7 PID: 13968 Comm: kworker/7:1 Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Workqueue: events_freezable vmballoon_work [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc05f1f84>] vmballoon_work+0x454/0x6ff [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22b9d4f>] process_one_work+0x17f/0x440
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bade6>] worker_thread+0x126/0x3c0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1c31>] kthread+0xd1/0xe0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2974c37>] ret_from_fork_nospec_begin+0x21/0x21
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3649243 inactive_anon:439380 isolated_anon:0#012 active_file:480 inactive_file:693 isolated_file:0#012 unevictable:0 dirty:2 writeback:0 unstable:0#012 slab_reclaimable:77845 slab_unreclaimable:10219#012 mapped:10086 shmem:9596 pagetables:16727 bounce:0#012 free:50116 free_pcp:238 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122768kB min:5972kB low:7464kB high:8956kB active_anon:471052kB inactive_anon:471040kB active_file:232kB inactive_file:460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:584kB shmem:300kB slab_reclaimable:23908kB slab_unreclaimable:3872kB kernel_stack:656kB pagetables:6440kB unstable:0kB bounce:0kB free_pcp:288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:214 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61804kB min:61576kB low:76968kB high:92364kB active_anon:14125920kB inactive_anon:1286480kB active_file:1688kB inactive_file:2312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:8kB writeback:0kB mapped:39760kB shmem:38084kB slab_reclaimable:287472kB slab_unreclaimable:36988kB kernel_stack:5680kB pagetables:60468kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:698 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 407*4kB (UEM) 506*8kB (UEM) 431*16kB (UM) 364*32kB (M) 272*64kB (UM) 179*128kB (UM) 99*256kB (UM) 35*512kB (M) 11*1024kB (M) 2*2048kB (M) 0*4096kB = 123164kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15576*4kB (UM) 39*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62616kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 44652 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 34162 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884045, delete 10845872, find 3294122/3678936
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 12996 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 83 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 50 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 47 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 113 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 39 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 132 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 4770 100 482 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6332 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 30 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3994129 15027 457539 0 java
Apr 26 04:18:15 elastichost kernel: [16264] 495 16264 18032 0 31 162 0 controller
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16070 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16264 (controller) total-vm:72128kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:15 elastichost kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: java cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 2 PID: 16265 Comm: java Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b6347>] __page_cache_alloc+0x97/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b8fa8>] filemap_fault+0x298/0x490
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484d0e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c2dc0>] ? wake_bit_function+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484f0c>] xfs_filemap_fault+0x2c/0x30 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e444a>] __do_fault.isra.59+0x8a/0x100
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e49fc>] do_read_fault.isra.61+0x4c/0x1b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e93a4>] handle_pte_fault+0x2f4/0xd10
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ebedd>] handle_mm_fault+0x39d/0x9b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f5e3>] __do_page_fault+0x203/0x500
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f915>] do_page_fault+0x35/0x90
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296b758>] page_fault+0x28/0x30
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3607073 inactive_anon:480522 isolated_anon:0#012 active_file:8 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:75170 slab_unreclaimable:10131#012 mapped:2070 shmem:9592 pagetables:16696 bounce:0#012 free:50006 free_pcp:72 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122560kB min:5972kB low:7464kB high:8956kB active_anon:471000kB inactive_anon:471076kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:292kB shmem:288kB slab_reclaimable:22832kB slab_unreclaimable:3752kB kernel_stack:672kB pagetables:6408kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:215 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61572kB min:61576kB low:76968kB high:92364kB active_anon:13957292kB inactive_anon:1451012kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:0kB writeback:4kB mapped:7988kB shmem:38080kB slab_reclaimable:277848kB slab_unreclaimable:36756kB kernel_stack:5664kB pagetables:60376kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:500 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 393*4kB (UEM) 490*8kB (UEM) 453*16kB (EM) 369*32kB (UEM) 269*64kB (M) 180*128kB (M) 98*256kB (M) 35*512kB (M) 12*1024kB (UM) 1*2048kB (M) 0*4096kB = 122148kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15440*4kB (UM) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61768kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 42975 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 33308 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884233, delete 10846914, find 3294127/3678944
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 4016 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 78 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 46 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 42 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 91 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 36 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 130 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 1215 100 485 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6230 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 27 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3993992 15027 457699 0 java
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16291 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16070 (java) total-vm:188490524kB, anon-rss:15975968kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:16 elastichost systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL
Apr 26 04:18:16 elastichost systemd: Unit elasticsearch.service entered failed state.
Apr 26 04:18:16 elastichost systemd: elasticsearch.service failed.

The line “Free swap = 0kB” suggest the trigger of OOM is out of swap. So does ElasticSearch contribute to the swap shortage?

ElasticSearch was the main process on the VM and I noticed that the OS did not disable swap, which does not align with the best practice from ElasticSearch community. In order to find out whether Elastic Search is pushed to use swap, we can get the process ID:

pidof java

ElasticSearch happens to be the only Java based process and the PID is 2283, the following command shows the swap usage by this process:

cat /proc/2283/status | grep VmSwap

If it shows a non-zero value, then ElasticSearch is using swap and you should expect some performance issues. We definitely should disable swap on ElasticSearch but does that solve the problem? Probably not because we didn’t address what caused the memory pressure in the first place. Disabling swap most likely makes ElastciSearch last longer before something else such as shortage of available memory triggers OOM.

In search for the source of memory shortage, I checked the result of free command and the top command. The free command (-mh) shows the following:

 total used free shared buff/cache available
Mem: 31G 20G 10G 117M 243M 10G
Swap: 2.0G 249M 1.8G

I tried to follow my example from a previous post to make sense of the memory reads. When I monitor process with top and watch for RSS column, I cannot identify a single process that even takes more than a few hundred megabytes. I’ve downloaded a tool smem from epel repo, and the result of “smem -kt” suggest that the total RSS is about 372.2M:

So there is about 19.6GB of memory usage unaccounted for. There is something that takes this much memory in the VM and haunting round even after OOM killer!

Inspired by this post, I was able to identify the culprit, that is the memory balloon. We don’t have access to the host but from the Guest OS, we can tell by vmware-toolbox-cmd (need to install yum package open-vm-tools):

vmware-toolbox-cmd stat balloon

The result displays a whopping 20807 MB as memory balloon! This needs to be sent to PaaS vendor for investigation but it is likely a result of memory over-allocation/over-commitment at host level, as well as the setup where the memory of guest is not reserved. Looking at the original log snippet, the line with “events_freezable vmballoon_work” is also an indicator of balloon causing the OOM.

Hypervisor needs memory ballooning to reclaim memory from guest. Since the guest OS does not expect the amount of physical memory to change, hypervisor has to maintain the illusion that the guest has its fixed amount of physical memory. The hypervisor first computes the amount of memory that needs to reclaim, then it leverage some low-level mechanism such as a balloon driver (a pseudo-device driver) installed on guest. The driver communicates with hypervisor and is told to allocate or de-allocate memory. If the driver is told to allocate memory to host, it tells the guest OS to pin the allocated pages into physical memory so they are locked and the physical memory available to guest OS is decreased. All these low level mechanisms explains why it is hard to account for memory consumption based on process running in Guest OS.

Memory balloon is a dynamic process and the hypervisor adjusts the size of balloon. However, the hypervisor, the guest OS and the application process (ElasticSearch in this case) may enter a nuance interaction:

The hypervisor reclaims memory from guest OS (ballooning);
Guest OS panicked with OOM exception;
OOM picks a process to kill, based on oom_score. Unfortunately, the true culprit (balloon) is not visible to guest OS, hence exempted from being considered to kill.
By killing the application process (usually the main application for VM), a fair chunk of guest memory is freed up;
the balloon then became more aggressive on the guest as seeing more memory becomes “available”;
the application on guest then does not have minimum memory on guest OS to start;

How to solve this problem? There are several things to consider. On the hypervisor, keep the entire or part of the guest memory reserved for start of application. Tune the OOM scoring so the non-critical application get killed. Implement application daemon, etc.

How memory usage adds up in Linux

Sun, 19 Apr 2020 21:12:56 -0400

There are too many metrics that describes some aspects about memory in Linux. This posting will make sense of those common metrics in Linux, CentOS as an example.

The most fundamental command is free and my favourite switch is -h for human readable reads. You can use -m, -k, -b for different units. The result looks like this:

 total used free shared buff/cache available
Mem: 32780168 16832160 3200408 101356 12747600 15399528
Swap: 2097148 2055148 42000

Swap is essentially disk space and many application such as Cassandra, ElasticSearch recommend disabling swap as best practice and they do not want disk speed to drag the performance of memory. Many suggest that swap is not needed in today’s era at all given the amount of memory for cheap. This is debatable.

With the row for Mem, the four columns should add up to the total, as suggested in the chart below.

total = free + used + shared + buff/cache

The four columns from free command output are supposed to always add up to the physical memory size. This command simplifies things quite a bit and each of these values are actually taken from certain lines in /proc/meminfo:

Metric from free command	Metric in /proc/meminfo
total	MemTotal
used	??
free	MemFree
shared	Shmem
buff/cache	Cached + Slab
available	MemAvailable

The buffer and cache (and even swap) can be freed by command. The value of used doesn’t seem to come from anwhere in /proc/meminfo, but it should be calculable from the memory used per process, which can be seen from top command.

In the result of top command, the column RSS (resident set size) is from the VmRSS value in /proc/<pid>/status, it is the actual physical memory consumed by the process. This value is originally from the second read in /proc/<pid>/statm, which represents the number of pages. For example:

[ghunch@centos ~]$ cat /proc/6495/status | grep VmRSS ; cat /proc/6495/statm
VmRSS:	20852916 kB
49829626 5213229 1212275 1 0 5773980 0
[ghunch@centos ~]$ getconf PAGE_SIZE
4096

Linux default page size is 4096 or 4K, so in the result from above, 5213229 x 4kB = 20852916 kB, which is the size of memory taken by process ID 6459. Therefore if we go through all processes and add up the VmRSS, we should get (close to) the used memory?

But wait a second, we have not account for slab info (memory used by kernel) yet, which is displayed in /proc/slabinfo. To calculate the total size taken by slab, we use <num_objs> and <objsize> columns from /proc/slabinfo.

Apart from that there is page table, the table that stores the mapping between virtual address and physical address, is stored in the physical memory as well and the size is specified in the PageTables entry in /proc/meminfo.

Now our equation becomes:

Used Memory = (RSS for all processes) + (all objects in slab) + (page table)

We can use the following script to calculate the used memory and compare it with free command output.

#/bin/bash
for PROC in `ls /proc/|grep "^[0-9]"`
do
 if [ -f /proc/$PROC/statm ]; then
 TEP=`cat /proc/$PROC/statm | awk '{print ($2)}'`
 RSS=`expr $RSS + $TEP`
 fi
done
RSS=`expr $RSS \* 4`
PageTable=`grep PageTables /proc/meminfo | awk '{print $2}'`
SlabInfo=`cat /proc/slabinfo |awk 'BEGIN{sum=0;}{sum=sum+$3*$4;}END{print sum/1024/1024}'`
 
echo $RSS"KB", $PageTable"KB", $SlabInfo"MB"
printf "rss+pagetable+slabinfo=%sMB\n" `echo $RSS/1024 + $PageTable/1024 + $SlabInfo|bc`
free -m

Running it require root access and the bc package installed. The result is most likely greater than the used memory value. Below is the result from my server:

89925884KB, 201788KB, 3303.92MB
rss+pagetable+slabinfo=91318.92MB
 total used free shared buff/cache available
Mem: 128772 87032 726 498 41013 40346
Swap: 2047 3 2044

So the result is over by (91318 – 87032) = 4286M. This is due to shared memory. The RSS value from above include memory from shared libraries as long as the pages from those libraries are in the memory. If multiple processes use the same library, the memory from shared library is counted multiple times. Check out the difference between RSS and PSS (proportional set size)

[Disclaimer] The chart and script are stolen from this authors post.

Storage Nitty-Gritty 1 of 5 – Disk and RAID

Sat, 09 Mar 2019 22:25:52 -0500

On my trip I have been through several pre-sales discussions on storage. Therefore I’m taking this opportunities to write up a series of postings deep diving into storage technologies. In this first section, we lay out the foundation of storage technology, from physical device to RAID, focusing on the concepts. Some contents are excerpts from Information Storage and Management.

Volume Manager – In early days, a file system occupies the entire disk drive, and presents continuous disk blocks directly to operating systems. Logical Volume Manager (LVM) was then introduced to bring a layer of abstraction (logical volume) on top of disks. The layers are shown as follows:

With all these layers, a byte in user file maps to disk sectors through several layers:

Mapping from user file to physical disk

File System – a hierarchical structure of files. It organizes data in a structural hierarchical manner. It includes files, directories as well as metadata. metadata must be consistent for the file system to be considered healthy. In Linux, metadata consists of:

Superblock: important information about file system, e.g. type, creation and modification dates, size, mount status flag
Inodes: a data structure that contains information associated with every file or directory
list of data blocks free and in use

Host connects to storage through various Interface Protocols. Common interface protocols include:

IDE/ATA and Serial ATA
SCSI (Small Computer System Interface)
FC (Fibre Channel)
IP (Internet Protocol per se is a network protocol traditionally used for host-to-host traffic in the early days. In the virtualization era, it has become a viable option for host-to-storage communication. Examples are iSCSI and FCIP)

The most prevalent disk drive types are SSD (solid state drive) and HDD (hard disk driveA). SSD (solid state drive) is newer, flash-based technology. Without seek and rotational latencies they deliver a high number of IOPS with low response times. They are especially suited for applications with small block size and random-read workloads requiring constant latency <1ms.

HDD is traditional and more cost effective. Its physical components are illustrated in the following two graphs:

Hard disk component

For HDD, disk service time (time taken by a disk to complete an I/O request) is determined by the following factors:

Seek time (aka access time) describes the time taken to position the R/W heads across the platter with a radial movement (moving along the radius of the platter). In other words, it is the time taken to position and settle the arm and the head over the correct track.
Rotation latency is the time taken by the platter to rotate and position the data under the R/W head. It depends on the rotation speed of the spindle and is measured in milliseconds.
(Data) transfer rate is the average amount of data per unit time that the drive can deliver from disk controller to the HBA (on the host).

Zone Bit Recording – a mechanism to use disk efficiently by grouping tracks into zones based on their distance from the disk.

Logical Block Addressing (LBA) – disk controller translates LBA to a physical address (CHS, cylinder, head and sector). The host only needs to know the size of disk drive in terms of number of blocks. The logical blocks are mapped to physical sectors on a 1:1 basis

IO request processing – I/O controller is introduce to improve response time for I/O request, in this model, The I/O requests arrive at the controller at the rate generated by the application. This rate is also called the arrival rate. These requests are held in the I/O queue, and the I/O controller processes them one by one, as shown here:

The relationship between controller utilization and average response time is:

Average response time = Service time / (1 – Utilization)

where service time is the time taken by disk controller to service the I/O request. This results in a classic relation between response time and utilization, as plotted below:

The graph indicates that the response time changes are nonlinear as the utilization increases. When the average queue sizes are low, the response time remains low. The response time increases slowly with added load on the queue and increases exponentially when the utilization exceeds 70 percent. Therefore, for performance-sensitive applications, it is common to utilize disks below their 70 percent of I/O serving capability.

Command queuing is a technique implemented on modern disk drives that determines the execution order of received I/Os and reduces unnecessary drive-head movements to improve disk performance. When an I/O is received for execution at the disk controller, the command queuing algorithms assign a tag that defines a sequence in which the commands should be executed. With command queuing, commands are executed based on the organization of data on the disk, regardless of the order in which the commands are received. Below is an example:

Disk Command Queuing

RAID is a technology that leverages multiple drives as part of a set that provides data protection against drive failures. It may also improve performance by serving I/Os from multiple disks simultaneously. It is primarily used in HDD but SSD may still benefit from it. RAID may be implemented by software but hardware RAID with a controller is widespread. RAID is built on three basic techniques:

Striping: spread data across multiple drives (more than one) to use the drives in parallel.
Mirroring: same data is stored on two different disk drives, yielding two copies of the data.
Parity: a method to protect striped data from disk drive failure without the cost of mirroring. An additional disk drive is added to hold parity, a mathematical construct that allows re-creation of the missing data.

Basic RAID levels are summarized here:

RAID summary

RAID 0, 1, 5 and 6 are pretty common in data centre operations. In addition to these levels above. If you hear RAID 1+0, 5+0 and RAID 6+0, they are called nested RAID. They are simply a RAID 0 on top of RAID1, RAID 5 and RAID 6, respectively.

RAID 0 – data striping technique utilizes full capacity of drives. Although it is a good option for applications that need high I/O throughput. It lacks data protection so it cannot drive application requiring high availability.

RAID 0

RAID 1 – mirroring technique ensures data duplication. In the event of disk failure, it introduces minimal impact to the disk array. It is suitable for applications that require high availability and cost is no constraint.

RAID 1

RAID 1+0 or RAID 10 – performs well for workloads with small, random, write-intensive I/Os. Some applications that benefit from RAID 1+0 include the following:

High transaction rate Online Transaction Processing (OLTP)
Large messaging installations
Database applications with write intensive random access workloads

RAID 1+0

RAID 3 – RAID 3 stripes data for performance and uses parity for fault tolerance. the total disk space required is 1.25 times the size of the data disks. RAID 3 always reads and writes complete stripes of data across all disks because the drives operate in parallel. RAID 3 provides good performance for applications that involve large sequential data access, such as data backup or video streaming.

RAID 5 – RAID5 is similar to RAID 4 because it uses striping. The drives (strips) are also independently accessible. The difference between RAID 4 and RAID 5 is the parity location. In RAID 4, parity is written to a dedicated drive, creating a write bottleneck for the parity disk. In RAID 5, parity is distributed across all disks to overcome the write bottleneck of a dedicated parity disk.

RAID 5

RAID 5 is good for random, read-intensive I/O applications and preferred for messaging, data mining, medium-performance media serving, and relational database management system (RDBMS) implementations, in which database administrators (DBAs) optimize data access.

RAID 6 – works the same way as RAID 5, except that RAID 6 includes a second parity element to enable survival if two disk failures occur in a RAID set. The write penalty in RAID 6 is more than that in RAID 5; therefore, RAID 5 writes perform better than RAID 6. The rebuild operation in RAID 6 may take longer than that in RAID 5 due to the presence of two parity sets.

RAID 6

Intelligent Storage System involves cache as the core component. An intelligent storage system involves front end, cache, back end and physical disks, as shown here:

Intelligent Storage System components

A variety of physical disk types and speed (e.g. mix of FC, SATA, SAS and flash) can be supported in a modern intelligent storage system. The front end provides the interface between the storage system and the host. It consists of ports and controllers, with redundancy. The back end provides an interface between cache and the physical disks. It consists of ports and controllers. For high data protection and high availability, storage systems are configured with dual controllers with multiple ports.

Cache improves storage system performance by isolating hosts from mechanical delays associated with hard disks. In intelligent storage system, read and write are first attempted on cache.

Page is the basic unit of cache, and the size of page is configured based on application I/O size. Cache consists of:

data store: holds the actual data temporarily
tag RAM: mainly serves three purposes:
- it tracks locations of data in cache and on disk;
- it maintains dirty bit flag to indicate whether data in cache has been committed to disk;
- it keeps time based information such as last access time, for cache management

Read Operation with Cache: When host issues a read request, the storage controller reads the tag RAM first to determine whether required data is available in cache:

Read cache hit: data is sent to host without any disk operation;
Read cache miss: back end access the disk to retrieve the requested data. Data is then placed in cache and sent to host through front end.

Cache miss increases I/O response time, to increase read hit ratio, read-ahead algorithm can be used when read requests are sequential. In a sequential read request, a contiguous set of associated blocks is retrieved. Several other blocks that have not yet been requested by the host can be read from the disk and placed into cache in advance. When the host subsequently requests these blocks, the read operations will be read hits.

Fixed prefetch – the intelligent storage system prefetches a fixed amount of data. It is most suitable when host I/O sizes are uniform.
Variable prefetch, the storage system prefetches an amount of data in multiples of the size of the host request.
Maximum prefetch limits the number of data blocks that can be prefetched to prevent the disks from being rendered busy with prefetch at the expense of other I/Os.

Write Operation with Cache: When an I/O is written to cache and acknowledged, it is completed in far less time (from the host’s perspective) than it would take to write directly to disk. Sequential writes also offer opportunities for optimization because many smaller writes can be coalesced for larger transfers to disk drives with the use of cache. Write operation with cache can be implemented in two ways:

Write-back cache: Data is placed in cache and an acknowledgment is sent to the host immediately. Later, data from several writes are committed (de-staged) to the disk. Write response times are much faster because the write operations are isolated from the mechanical delays of the disk. However, uncommitted data is at risk of loss if cache failures occur.
Write-through cache: Data is placed in the cache and immediately written to the disk, and an acknowledgment is sent to the host. Because data is committed to disk as it arrives, the risks of data loss are low, but the write-response time is longer because of the disk operations.

If the size of an I/O request exceeds the write aside size, writes are sent to the disk directly to reduce the impact of large writes consuming a large cache space. This is helpful where cache resources are constrained and cache is required for small random I/Os.

Cache space can be assigned in two ways:

dedicated cache: separate sets of locations are reserved for read and write;
global cache: user may specify percentage of cache for read and write based on application workload pattern; or the system set is dynamically.

Cache Management algorithm is used to determine when, and what pages of the cache need to be free up during maintenance. Most commonly used algorithms are:

LRU (least recently used): assuming data not accessed for a while will not be requested by host any more;
MRU (most recently used): assuming data recently accessed will not be requested by host again

As cache fills, the storage system must take action to flush dirty pages by committing data from cache to disk. There are several triggers for cache management action:

Idle flushing – occurs continuously at modest rate when cache utilization level is between high and low watermark;
High watermark flushing – activated when utilization hits high watermark; and stops at low watermark; this has impact to I/O processing;
Forced flushing – occurs in the event of large I/O burst when cache reaches 100% capacity; this significantly impacts I/O response time

Types of Flushing

Cache data protection is the mechanism to prevent losing uncommitted data held in cache. Common mechanisms are:

Cache mirroring – Each write to cache is held in two different memory locations on two independent memory cards. If a cache failure occurs, the write data will still be safe in the mirrored location and can be committed to the disk. The array operating environment needs to maintain cache coherency between the redundant memory locations. Read cache does not need mirroring.

Cache vaulting – In the event of server power failure, use battery power to write the cache content to the disk (vault drive). When power is restored, data from these disks is written back to write cache and then written to the intended disks.