swap on Digi Hunch

How memory usage adds up in Linux

Sun, 19 Apr 2020 21:12:56 -0400

There are too many metrics that describes some aspects about memory in Linux. This posting will make sense of those common metrics in Linux, CentOS as an example.

The most fundamental command is free and my favourite switch is -h for human readable reads. You can use -m, -k, -b for different units. The result looks like this:

 total used free shared buff/cache available
Mem: 32780168 16832160 3200408 101356 12747600 15399528
Swap: 2097148 2055148 42000

Swap is essentially disk space and many application such as Cassandra, ElasticSearch recommend disabling swap as best practice and they do not want disk speed to drag the performance of memory. Many suggest that swap is not needed in today’s era at all given the amount of memory for cheap. This is debatable.

With the row for Mem, the four columns should add up to the total, as suggested in the chart below.

total = free + used + shared + buff/cache

The four columns from free command output are supposed to always add up to the physical memory size. This command simplifies things quite a bit and each of these values are actually taken from certain lines in /proc/meminfo:

Metric from free command	Metric in /proc/meminfo
total	MemTotal
used	??
free	MemFree
shared	Shmem
buff/cache	Cached + Slab
available	MemAvailable

The buffer and cache (and even swap) can be freed by command. The value of used doesn’t seem to come from anwhere in /proc/meminfo, but it should be calculable from the memory used per process, which can be seen from top command.

In the result of top command, the column RSS (resident set size) is from the VmRSS value in /proc/<pid>/status, it is the actual physical memory consumed by the process. This value is originally from the second read in /proc/<pid>/statm, which represents the number of pages. For example:

[ghunch@centos ~]$ cat /proc/6495/status | grep VmRSS ; cat /proc/6495/statm
VmRSS:	20852916 kB
49829626 5213229 1212275 1 0 5773980 0
[ghunch@centos ~]$ getconf PAGE_SIZE
4096

Linux default page size is 4096 or 4K, so in the result from above, 5213229 x 4kB = 20852916 kB, which is the size of memory taken by process ID 6459. Therefore if we go through all processes and add up the VmRSS, we should get (close to) the used memory?

But wait a second, we have not account for slab info (memory used by kernel) yet, which is displayed in /proc/slabinfo. To calculate the total size taken by slab, we use <num_objs> and <objsize> columns from /proc/slabinfo.

Apart from that there is page table, the table that stores the mapping between virtual address and physical address, is stored in the physical memory as well and the size is specified in the PageTables entry in /proc/meminfo.

Now our equation becomes:

Used Memory = (RSS for all processes) + (all objects in slab) + (page table)

We can use the following script to calculate the used memory and compare it with free command output.

#/bin/bash
for PROC in `ls /proc/|grep "^[0-9]"`
do
 if [ -f /proc/$PROC/statm ]; then
 TEP=`cat /proc/$PROC/statm | awk '{print ($2)}'`
 RSS=`expr $RSS + $TEP`
 fi
done
RSS=`expr $RSS \* 4`
PageTable=`grep PageTables /proc/meminfo | awk '{print $2}'`
SlabInfo=`cat /proc/slabinfo |awk 'BEGIN{sum=0;}{sum=sum+$3*$4;}END{print sum/1024/1024}'`
 
echo $RSS"KB", $PageTable"KB", $SlabInfo"MB"
printf "rss+pagetable+slabinfo=%sMB\n" `echo $RSS/1024 + $PageTable/1024 + $SlabInfo|bc`
free -m

Running it require root access and the bc package installed. The result is most likely greater than the used memory value. Below is the result from my server:

89925884KB, 201788KB, 3303.92MB
rss+pagetable+slabinfo=91318.92MB
 total used free shared buff/cache available
Mem: 128772 87032 726 498 41013 40346
Swap: 2047 3 2044

So the result is over by (91318 – 87032) = 4286M. This is due to shared memory. The RSS value from above include memory from shared libraries as long as the pages from those libraries are in the memory. If multiple processes use the same library, the memory from shared library is counted multiple times. Check out the difference between RSS and PSS (proportional set size)

[Disclaimer] The chart and script are stolen from this authors post.

CentOS – Remove Swap safely

Sat, 14 Apr 2018 16:15:43 -0400

If the default installation has swap on, you will see it from block device list:

[root@server /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 500G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 499G 0 part
 ├─centos-root 253:0 0 50G 0 lvm /
 ├─centos-swap 253:1 0 7.9G 0 lvm [SWAP]
 └─centos-home 253:2 0 441.1G 0 lvm /home
sr0 11:0 1 1024M 0 rom
[root@server /]# free -h
 total used free shared buff/cache available
Mem: 7.6G 217M 7.2G 11M 206M 7.2G
Swap: 7.9G 0B 7.9G

Many installations require swap to be off for performance reasons (although some advocate turning off swappiness of the application, instead of removing swap partition from operating system, which is a separate topic). This can be turned off by a simple command:

swapoff -a

Then if you run free command after rebooting, you will see the Swap is set to 0G. Great. But lsblk and df commands still shows the space being used by swap. This disk space is not being used at all.

[root@c7v-vitvcast02 ~]# free -m
 total used free shared buff/cache available
Mem: 7802 237 7380 11 185 7333
Swap: 0 0 0
[root@c7v-vitvcast02 ~]# lvdisplay
 --- Logical volume ---
 LV Path /dev/centos/swap
 LV Name swap
 VG Name centos
 LV UUID 7zHtvT-zGYn-sNBc-JRlT-nTfU-O9hN-r07y3J
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:21 -0500
 LV Status available
 # open 0
 LV Size <7.88 GiB
 Current LE 2016
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:1

 --- Logical volume ---
 LV Path /dev/centos/home
 LV Name home
 VG Name centos
 LV UUID 4XMTa7-uR44-qH0u-Oag4-PDyK-Z1BC-3oUSiz
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 0
 LV Size <441.12 GiB
 Current LE 112926
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:2

 --- Logical volume ---
 LV Path /dev/centos/root
 LV Name root
 VG Name centos
 LV UUID tkMCM4-cWxV-FaAC-hgJx-r1vS-DiY5-fyZvjt
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 1
 LV Size 50.00 GiB
 Current LE 12800
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:0

We can then remove the line for swap from /etc/fstab, this will keep the OS from mounting it upon reboot. However, the logical volume for swap is there, as a partition that’s not exposed.

If you need to reclaim space from swap partition, delete the logical volume:

lvremove /dev/centos/swap
Do you really want to remove active logical volume centos/swap? [y/n]: y
 Logical volume "swap" successfully removed

Now, using command lvdisplay or lvs, the swap space is not displayed anymore. So I rebooted the server in the hopes that the swap space is not being presented and everything is hunky-dory. But no… the server doesn’t boot up at all. It took a long time at the splash screen only to enter dracut, where /boot isn’t present.

On further reading, this is because the grub file is still referencing /dev/centos/swap somewhere. I need to somehow get to the grub file and fix that before it can reboot again. Specifically, I pressed “e” at the menu, which gave me an opportunity to edit grub file. From there I removed section for swap. Then I saved it with “Ctrl + X” (as suggested on the screen) so the OS booted into normal mode. Alternatively, I entered single user mode in order to get to the point I can edit the grub file.

Once booted, I have to make change to the actual grub file at /boot/grub2/grub.cfg so the change persists. In the file, locate the lines starting with linux16 and remove the section “rd.lvm.lv=centos/swap”. There are two appearances of this section on my machine. Then the server will boot just normally.

In summary, the clean way to turn off swap on CentOS involves the following steps:

turn off swap
Remove swap mount from /etc/fstab (and umount)
remove the logical volume for swap
remove reference to swap from grub

Out of those steps, 3 and 4 are needed to reclaim swap spaces. They must be done together. If you rebooted the server before step 4. The server will not be able to boot complaining about missing swap.