memory on Digi Hunch

Optimize CPU and Memory for Kubernetes Pod

Fri, 13 Jan 2023 11:47:00 -0400

When optimizing workload performance, it is important to understand how on earth operating system allocates CPU and memory to processes. This helps understand how to set resource limit Kubernetes Pod in an optimal way.

CPU resource assignment

The OS distributes CPU resource to processes by the unit of time share of CPU time. Most of the time, many processes with CPU instructions (machine code) are waiting in the Job queue, for their share of CPU time in order to execute their instructions. As soon as CPU becomes idle, the CPU scheduler selects a process from the ready queue to run next:

Ideally, OS should schedule CPU in a way that it should not waste any CPU cycle. It should also minimizes waiting time and response time of processes. At a high level, there are two types of CPU scheduling:

Preemptive: OS allocate CPU resources to a process for only a limited period of time and then takes those resources back. It could interrupt a running process to execute a higher priority process.
Non-preemptive: New processes are executed only after the current executing process has completed its execution.

Here is more information about preemptive and non-preemptive scheduling.

CPU is compressible resource in Linux

In the Linux world, all scheduling is preemptive. We also call it kernel preemption. As the wikipedia entry states: the scheduler is permitted to forcibly perform a context switch (on behalf of a runnable and higher-priority process) on a driver or other part of the kernel during its execution, rather than co-operatively waiting for the driver or kernel function (such as a system call) to complete its execution and return control of the processor to the scheduler when done.

The Linux scheduler implements a number of scheduling policies, which determine when and for how long a thread runs on a particular CPU core. The scheduling policies in RHEL include real time policies such as SCHED_FIFO and SCHED_RR where processes have a sched_priority value in the range of 1 (low) to 99 (high); and normal policies such as SCHED_OTHER, SCHED_BATCH and SCHED_IDLE, where sched_priority (specified as 0) is not used in scheduling decisions.

It is important to understand preemptive CPU scheduling on Linux. When OS allocate CPU resource to a process for one time slot, it is not committed to the same process for the next time slot. The OS reserves the ability to revoke the next CPU use and re-assign it for processes of higher priority.

Because of this, we regard CPU as a compressible resource. The compressible characteristic impacts how we optimize CPU utilization for a process, including setting CPU request and limit for Kubernetes workload.

Memory is non-compressible resource

A few years ago, I discussed how to calculate memory usage. A process requests memory from OS using memory allocation functions (the malloc family), and return memory to OS using free functions. The design of Linux OS knows that processes have a tendency to request more memory than they use, which causes under-utilization. In combat against under-utilization, the Linux OS supports memory overcommitment (on by default), allowing processes to request more memory than what is available. The processes have access to virtual memory space and the OS may swap some pages out to disks. The overcommitment mechanism also prevents processes from crashing due to insufficient memory assignment. The kernel can also OOM kill a process when the entire system is in a crisis.

Memory is non-compressible resource. When OS assigns memory pages to a process, the process has to right to keep those pages, until the OS takes them away. Unlike assigning CPU cycles, the assignment of memory pages to processes does not have an expiry time. This is the non-compressible characteristic of memory assignment.

CPU limit and requests for Kubernetes workload

It was considered best practice to set request and limit for memory and CPU. However, knowing CPU is compressible resource and memory isn’t, we should re-consider this practice. In short, for CPU, we should set request only, without setting limit. For memory, we should set limit to exactly the same as request.

A process has different level of demands for CPU at different times. Depending on the activity in the process, the level of demand can even be spiky. If there is a lot of iowait, it may not need a lot of CPU. But when there are lots of computing-bound activities, the program is CPU-thirsty as it is programmed to to more. The last thing we want is to throttle the CPU use for a process in such legit situations. When throttling happens, the process does not get sufficient time share of CPU time. At the platform level, we can’t control when the Pod (process) gets busy. The best thing it can do, is trying to fit more CPU time shares to this process when it becomes CPU thirsty. When we apply a limit of CPU in workload setting, we are potentially throttling the CPU use for a process at the times it needs more CPU time shares, which is counter-productive.

We should still configure CPU request, so that kube-scheduler factors it in when scheduling multiple Pods to a Node. The CPU request alone ensures the number of Pods are not excessive. This is the only thing we can do about controlling CPU assignment for Pods. We should also monitor CPU throttling.

Memory limit and request for Kubernetes workload

Memory is not compressible, therefore we should set both limit and request to the same value. We set memory request so that kube-scheduler has an idea assigning Pods. We set the limit so that no single Pod takes more memory than its fair share. Unlike CPU, once a Pod takes more memory than its fair share, the platform will have to be aggressive to reclaim it back, which may impacts the running of the Pod (process). In contrast, CPU scheduler never guarantees the assignment of CPU time share to a Pod beyond the end of the current CPU cycle.

When we’re setting memory limit and request with different values, we’re sending a confusing signal. We’re inviting Pods to use more memory than they requested. This increases the chance of memory shortage at the node level, and hence the need to OOM kill a Pod.

Horizontal autoscaling and Cluster Autoscaling

The native HPA is metrics-based. As I previously discussed, neither CPU nor memory metrics are good indicators of time to scale. A process or a Pod may have a temporary high demand of CPU purely due to how programmers write the code. Even if we followed the best practices as above, I would still not regard CPU and memory metrics as a reliable indicator to drive auto scaling. If a service is a potential point of congestion, we should use a queue in front and the queue size is almost always a much better indicator of the timing to scale.

As to cluster autoscaler, on it FAQ, it says flat out that you should NOT use a CPU usage based scaling mechanism. I guess this is for a similar reason (compressibility). As discussed, when a Pod is pending for schedule for too long, it emits and event that drives the cluster autoscaler.

Summary

When I first worked on Kubernetes workload I did not give this much thought and proposed the use of CPU limit. As of January 2023 I still find static code analysis tools that requires CPU limit for Pods in the check (e.g. CKV_K8S_11 on Checkov), which leads me to investigate the issue further, and noticed more voices advocating the correct use of resource limit (such as this post) in 2022. For existing deployments, it is worth a review the resource limit configuration.

Java Garbage Collection

Fri, 07 Aug 2020 23:19:17 -0400

Tuning the garbage collector is the most important thing that can be done to improve the performance of a Java application. GC is typically caused when the JVM decides GC is necessary, specifically when:

a minor GC will be triggered when the new generation is full;
a full GC will be triggered when the old generation is full;
a concurrent GC (if applicable) will be triggered when the heap starts to fill up

OpenJDK has three collectors suitable for production, with different performance characteristics. In order to study the GC behaviours in application, it is important to turn on GC logging. The detailed step is different in JDK 8 and JDK 11 (read about java version here).

Java developers don’t need to manage life cycle of objects explicitly as the JVM automatically fress the object. In order to track objects that are still in use, it is insufficient to count references to objects. Instead, the JVM must periodically search the heap for unused objects. Once it finds unused objects, the JVM frees the memory occupied by those objects. It also needs to compact the memory to prevent memory fragmentation. The performance of GC is dominated by these basic operations (finding unused objects; freeing up their memory; compacting the heap), no matter which collector is used. Some algorithms delay compaction until absolutely necessary, some compact entire sections of the heap at a time, and some compact the heap by relocating small amounts of memory at a time. These different approaches are why different algorithms have different performance characteristics.

Java programs are typically heavily multithreaded, and the garbage collector itself often runs multiple threads too. We refer to the application logic threads as mutator threads, since they are mutating objects as part of the application logic. When GC threads track object references or move objects around in memory, they must make sure application threads are not using those objects on the move. This introduces a pause when all application threads are stopped (known as stop-the-world pauses), which generally has the greatest impact on the performance of an application. Minimizing those pauses is one important consideration when tuning GC.

Garbage collectors are generational

Most garbage collectors work by splitting the heap into generations. These are called the old (or tenured) generation, and the young generation, which is further divided into sections known as eden and survivor spaces, with eden taking up the vast majority of the young generation. The rationale for having separate generations is that many objects are used for a very short period of time in the real life of application programming. Garbage collector is designed to take advantage of this. Objects are first allocated in the young generation, which is a subset of the entire heap. When the young generation fills up, the garbage collector will stop all the application threads and empty out the young generation. Objects that are no longer in use are discarded, and objects that are still in use are moved elsewhere. This operation is called a minor GC or a young GC. Common GC algorithms have stop-the-world pauses during collection of the young generation.

Heap Generation

This design has two performance advantages. First, Cleaning up young generation as a only a portion of the entire heap causes shorter pause than cleaning up the entire heap. Second, by moving used objects to survivor spaces or old generation, and discarding unused objects, compatction is achieved.

With used objects moved to the old generation, eventually it woo will fill up, and the JVM will needt o find any objects within the old generation that are no longer in use to discard. This is where GC algorithms have their biggest differences. The simpler alghorithms stop all application threads, find the unused objects, free their memory, and then compact the heap. This process is called a full GC, and it generally causes a relatively long pause for the application threads.

On the other hand, sophisticated alghrithms are able to find unused objects while application threads are running. These algorithms are called concurrent collectors, or low-pause collectors. A concurrent collector typically allows an application to experience fewer and shorter pauses. The biggest trade-off here is the overall CPU required by the sophisticated algorithms.

The three main algorithms

Serial GC is the simplest and the default for single core host (e.g. client-class machine, single-processor VM or Docker container). The serial collector uses a single thread to process the heap. It will stop all application threads as the heap is processed (for either a minor or full GC). During a full GC, it will fully compact the old generation. The serial collector is enabled by using the -XX:+UseSerialGC flag.

The throughput collector (aka parallel collector) is the default collector for any 64-bit machine with two or more CPUs. The throughput collector uses multiple threads to collect the young generation, which makes minor GCs much faster than when the serial collector is used. This uses multiple threads to process the old generation as well. The throughput collector stops all application threads during both minor and full GCs, and it fully compacts the old generation during a full GC. Since it is the default in most situations where it would be used, it needen’t be expliticly enabled. To enable it where necessary, use the flag -XX:+UseParallelGC

The G1 GC (or garbage first garbage collector) uses a concurrent collection strategy to collect the heap with minimal pauses. It is the default collector in JDK 11 and later for 64-bit JVMs on machines with two or more CPUs. G1 GC divides the heap into regions, but it still considers the heap to have two generations. Some of those regions make up the young generation, and the young generation is still collected by stopping all application threads and moving all objects that are alive into the old generation or the survisor spaces, using multiple threads. In G1 GC, the old generation is processed by background threads that don’t need to stop the application threads to perform most of their work.

In G1 GC, the old generation is processed by background threads that don’t need to stop the application threads to perform most of their work. Because the old generation is divided into regions, G1 GC can clean up objects from the old generation by copying from one region into another, which means that it compacts the heap during normal processing. This helps keep G1 GC heaps from becoming fragmented.

The trade-off for avoiding the full GC cycles is CPU time; the multiple background threads G1 GC uses to process the old generation requires CPU cycles available at the same time the application threads are running. G1 GC is enabled by specifying the flag -XX:+UseG1GC. It is the default in JDK 11, and functional in JDK 8 as well, with some performance feature missing.

In all cases, GC is caused when the JVM decices GC is necessary; a minor GC will be triggered when the new generation is full; a full GC will be triggered when the old generation is full, or a concurrent GC (if applicable) will be triggered when the heap starts to fill up. Java also provides a mechanism for applications to force a GC to occur: the System.gc() method, although it is always a bad idea to call that method explicitly because it triggers a full GC which hangs the application threads. This method can be disabled by including -XX:+DisableExplicitGC in the JVM arguments.

As to choosing GC algorithm, the rule of thumb is that G1 GC is the better choice. However, in JDK 8, the ability of G1 GC to avoid a full GC is also a key consideration. In this case we may need to choose betwen serial collectors and throughput collectors, based on the number of CPUs on the machine.

The serial collector makes sense when running CPU-bound applications on a machine with a single CPU, even if that single CPU is hyper-threaded. The throughput collector makes sens on multi-CPU machines running jobs that are CPU bound. Even for jobs that are not CPU bound, the throughput collector can be the better choice if it does relatively few full GCs or if the old generation is generally full.

Basic GC tuning

Sizing the heap

If the heap is too small, the program will spend too much time performing GC and not enough time performing application logic. On the contrary, a very large heap will increase the time spent in GC pauses, even thought the pauses occur less frequently. It is also potentially dangerous due to interaction with memory swap. If a Java program with a 12 GB heap is running on a system where swap is enabled, the OS may handle it by keeping 8GB of the heap in RAM and 4GB on disk. The JVM does not know about this because swapping is handled by the OS. The JVM will happily fill up all 12GB of heap it has been told to use. This can cause a sever performance penalty when OS swaps data from disk to RAM. Worse, the one time this swapping is guaranteed to occur is during a full GC, when the JVM must access the entire heap. Swapping during full GC makes the pause an order of magnitude longer.
So heap size (total for all JVMs) should never exceed the amount of physical memory on the machine. Size of heap is controled by two values (Xms as initial value and Xmx as maximum value). Having an initial and maximum size for the heap allows the JVM to tune its behaviour depending on the workload. If the JVM sees that it is doing too much GC, with the initial heap size, it will continually increase the heap until the JVM is doing the “correct” amount of GC, or until the heap hits its maximum size.

A good rule of thumb is to size the heap so that it is 30% occupied after a full GC. To calculate this, start your application and push it to high load. Then connect to the application with jconsole, force a full GC, and observe how much memory is used when the full GC completes.

Sizing the generations

The JVM must also decide how much of the heap to allocate respectively to they young generation and old generation. THe JVM usually does this automatically and usually does a good job in determining the optimal ratio. In some cases you might hand-tune these values.
In general, if there is a relatively larger young generation, young GC pause times will increase, but the young generation will be collected less often, and fewer objects will be promoted into the old genration. But on the other hand, older generation will be relatively smaller and fill up more frequently and do more full GCs. The command-line flag to tune the generation sizes are:
-XX:NewRatio=N
-XX:NewSize=N
-XX:MaxNewSize=N
-Xmn N
The size of initial young generation is determined by initial heap size and new ratio:
Initial Young Gen Size = Initial Heap Size / (1 + NewRatio)
The young generation will grow in tandem with the overall heap size, but it can also fluctuate as a percentage of the total heap (based on the initial and maximum size of the young generation). Adaptive sizing controls how the JVM alters the ratio of young genration to old gneeration within the heap. It should be kept enabled in general. For finely tuned heaps, adaptive sizing can be disabled for a small performance boost.

Sizing the metaspace

When the JVM loads classes, it must keep track of certain metadata about those classes. This occupies a separate heap space called the metaspace. In older JVMs this was handled by a different implementation called permgen. To end users, the metaspace is opaque. It does not hold the actual instance of the class. The objects are held in the regular heap. Information in the metaspace is used only by the compiler and JVM runtime, and the data it holds is referred to as class metadata.
Tuning the metaspace is fairly rare these days because the default values for the size of metaspace are very generous. It is sized dynamically based on an initial size (-XX:MetaspaceSize=N) and will increase as needed to a maximum size (-XX:MaxMetaspaceSize=N).
Resizing the metaspace requires a full GC, so it is an expensive operation. If there are a lot of full GCs during the startup of a program (as it it loading classes), it is often because permgen or metaspace is being resized, so increasing the initial size is a good idea to improve startup in that case.

Controlling Parallelism

All GC alghorithms except the serial collector use multiple threads. The number of these threads is controlled by the -XX:ParallelGCThreads=N flag. Bacuase these GC operations stop all application threads from executing, the JVM attempts to use as many CPU resources as it can in order to minimize the pause time. By default, that means the JVM will run one thread for each CPU on a machine, up to eight. Once that threashold has reached, the JVM adds new thread for only every 1.6 CPus. Sometimes this number is too large relative to the heap size and hand tuning is needed.

Reference: Java Performance by Scott Oaks

This post also contains some helpful information, where the original Oracle white paper about GC was cited. Further than GC, this website from Oracle describes more about JVM.

Balloon steals memory from virtual machines

Thu, 07 May 2020 19:49:00 -0400

This article is my experience with memory balloon on virtual machine.

I came across an ElasticSearch server (ESXi guest with 32GB physical memory) where the main process keeps dying of OOM. Even worse, after the OOM event, the free memory left is about 10G and Elastic Search cannot start because its JVM is set with -Xms16g in /etc/elasticsearch/jvm.options. So I need to address the OutOfMemory error.

To understand what triggered OOM, we can use dmesg or just check /var/log/message, where the memory snapshot by OOM killer is displayed during the kernel panic:

Apr 26 04:18:15 elastichost kernel: kworker/7:1 invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: kworker/7:1 cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 7 PID: 13968 Comm: kworker/7:1 Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Workqueue: events_freezable vmballoon_work [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc05f1f84>] vmballoon_work+0x454/0x6ff [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22b9d4f>] process_one_work+0x17f/0x440
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bade6>] worker_thread+0x126/0x3c0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1c31>] kthread+0xd1/0xe0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2974c37>] ret_from_fork_nospec_begin+0x21/0x21
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3649243 inactive_anon:439380 isolated_anon:0#012 active_file:480 inactive_file:693 isolated_file:0#012 unevictable:0 dirty:2 writeback:0 unstable:0#012 slab_reclaimable:77845 slab_unreclaimable:10219#012 mapped:10086 shmem:9596 pagetables:16727 bounce:0#012 free:50116 free_pcp:238 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122768kB min:5972kB low:7464kB high:8956kB active_anon:471052kB inactive_anon:471040kB active_file:232kB inactive_file:460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:584kB shmem:300kB slab_reclaimable:23908kB slab_unreclaimable:3872kB kernel_stack:656kB pagetables:6440kB unstable:0kB bounce:0kB free_pcp:288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:214 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61804kB min:61576kB low:76968kB high:92364kB active_anon:14125920kB inactive_anon:1286480kB active_file:1688kB inactive_file:2312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:8kB writeback:0kB mapped:39760kB shmem:38084kB slab_reclaimable:287472kB slab_unreclaimable:36988kB kernel_stack:5680kB pagetables:60468kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:698 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 407*4kB (UEM) 506*8kB (UEM) 431*16kB (UM) 364*32kB (M) 272*64kB (UM) 179*128kB (UM) 99*256kB (UM) 35*512kB (M) 11*1024kB (M) 2*2048kB (M) 0*4096kB = 123164kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15576*4kB (UM) 39*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62616kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 44652 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 34162 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884045, delete 10845872, find 3294122/3678936
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 12996 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 83 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 50 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 47 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 113 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 39 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 132 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 4770 100 482 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6332 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 30 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3994129 15027 457539 0 java
Apr 26 04:18:15 elastichost kernel: [16264] 495 16264 18032 0 31 162 0 controller
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16070 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16264 (controller) total-vm:72128kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:15 elastichost kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: java cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 2 PID: 16265 Comm: java Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b6347>] __page_cache_alloc+0x97/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b8fa8>] filemap_fault+0x298/0x490
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484d0e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c2dc0>] ? wake_bit_function+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484f0c>] xfs_filemap_fault+0x2c/0x30 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e444a>] __do_fault.isra.59+0x8a/0x100
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e49fc>] do_read_fault.isra.61+0x4c/0x1b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e93a4>] handle_pte_fault+0x2f4/0xd10
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ebedd>] handle_mm_fault+0x39d/0x9b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f5e3>] __do_page_fault+0x203/0x500
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f915>] do_page_fault+0x35/0x90
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296b758>] page_fault+0x28/0x30
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3607073 inactive_anon:480522 isolated_anon:0#012 active_file:8 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:75170 slab_unreclaimable:10131#012 mapped:2070 shmem:9592 pagetables:16696 bounce:0#012 free:50006 free_pcp:72 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122560kB min:5972kB low:7464kB high:8956kB active_anon:471000kB inactive_anon:471076kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:292kB shmem:288kB slab_reclaimable:22832kB slab_unreclaimable:3752kB kernel_stack:672kB pagetables:6408kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:215 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61572kB min:61576kB low:76968kB high:92364kB active_anon:13957292kB inactive_anon:1451012kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:0kB writeback:4kB mapped:7988kB shmem:38080kB slab_reclaimable:277848kB slab_unreclaimable:36756kB kernel_stack:5664kB pagetables:60376kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:500 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 393*4kB (UEM) 490*8kB (UEM) 453*16kB (EM) 369*32kB (UEM) 269*64kB (M) 180*128kB (M) 98*256kB (M) 35*512kB (M) 12*1024kB (UM) 1*2048kB (M) 0*4096kB = 122148kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15440*4kB (UM) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61768kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 42975 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 33308 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884233, delete 10846914, find 3294127/3678944
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 4016 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 78 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 46 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 42 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 91 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 36 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 130 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 1215 100 485 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6230 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 27 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3993992 15027 457699 0 java
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16291 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16070 (java) total-vm:188490524kB, anon-rss:15975968kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:16 elastichost systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL
Apr 26 04:18:16 elastichost systemd: Unit elasticsearch.service entered failed state.
Apr 26 04:18:16 elastichost systemd: elasticsearch.service failed.

The line “Free swap = 0kB” suggest the trigger of OOM is out of swap. So does ElasticSearch contribute to the swap shortage?

ElasticSearch was the main process on the VM and I noticed that the OS did not disable swap, which does not align with the best practice from ElasticSearch community. In order to find out whether Elastic Search is pushed to use swap, we can get the process ID:

pidof java

ElasticSearch happens to be the only Java based process and the PID is 2283, the following command shows the swap usage by this process:

cat /proc/2283/status | grep VmSwap

If it shows a non-zero value, then ElasticSearch is using swap and you should expect some performance issues. We definitely should disable swap on ElasticSearch but does that solve the problem? Probably not because we didn’t address what caused the memory pressure in the first place. Disabling swap most likely makes ElastciSearch last longer before something else such as shortage of available memory triggers OOM.

In search for the source of memory shortage, I checked the result of free command and the top command. The free command (-mh) shows the following:

 total used free shared buff/cache available
Mem: 31G 20G 10G 117M 243M 10G
Swap: 2.0G 249M 1.8G

I tried to follow my example from a previous post to make sense of the memory reads. When I monitor process with top and watch for RSS column, I cannot identify a single process that even takes more than a few hundred megabytes. I’ve downloaded a tool smem from epel repo, and the result of “smem -kt” suggest that the total RSS is about 372.2M:

So there is about 19.6GB of memory usage unaccounted for. There is something that takes this much memory in the VM and haunting round even after OOM killer!

Inspired by this post, I was able to identify the culprit, that is the memory balloon. We don’t have access to the host but from the Guest OS, we can tell by vmware-toolbox-cmd (need to install yum package open-vm-tools):

vmware-toolbox-cmd stat balloon

The result displays a whopping 20807 MB as memory balloon! This needs to be sent to PaaS vendor for investigation but it is likely a result of memory over-allocation/over-commitment at host level, as well as the setup where the memory of guest is not reserved. Looking at the original log snippet, the line with “events_freezable vmballoon_work” is also an indicator of balloon causing the OOM.

Hypervisor needs memory ballooning to reclaim memory from guest. Since the guest OS does not expect the amount of physical memory to change, hypervisor has to maintain the illusion that the guest has its fixed amount of physical memory. The hypervisor first computes the amount of memory that needs to reclaim, then it leverage some low-level mechanism such as a balloon driver (a pseudo-device driver) installed on guest. The driver communicates with hypervisor and is told to allocate or de-allocate memory. If the driver is told to allocate memory to host, it tells the guest OS to pin the allocated pages into physical memory so they are locked and the physical memory available to guest OS is decreased. All these low level mechanisms explains why it is hard to account for memory consumption based on process running in Guest OS.

Memory balloon is a dynamic process and the hypervisor adjusts the size of balloon. However, the hypervisor, the guest OS and the application process (ElasticSearch in this case) may enter a nuance interaction:

The hypervisor reclaims memory from guest OS (ballooning);
Guest OS panicked with OOM exception;
OOM picks a process to kill, based on oom_score. Unfortunately, the true culprit (balloon) is not visible to guest OS, hence exempted from being considered to kill.
By killing the application process (usually the main application for VM), a fair chunk of guest memory is freed up;
the balloon then became more aggressive on the guest as seeing more memory becomes “available”;
the application on guest then does not have minimum memory on guest OS to start;

How to solve this problem? There are several things to consider. On the hypervisor, keep the entire or part of the guest memory reserved for start of application. Tune the OOM scoring so the non-critical application get killed. Implement application daemon, etc.