centos on Digi Hunch

AWS CDK example in Typescript – provision an AWX server

Sat, 19 Dec 2020 17:14:00 -0400

This post provides an example of using AWS CDK in Typescript.

Ansible Tower and AWX

We have used open-source Ansible extensively in the past. While the automation is convenient, the lack of UI makes it not as suitable as a team collaboration tool. One way to allow team collaboration with open-source Ansible, is to use Jenkins to glue the components together, as discussed in the Automated Deployment Pipeline series. In this setup, the open-source Ansible remains command-line driven, with Jenkins building up the command, rather than a human user. There are many upsides in this configuration, but it is not built specifically for Ansible. Ansible is agent-less, and can be run from any host. This sounds appealing and can work well in smaller server fleet. However, since it requires some configuration on the controlling host for Ansible to function properly, it become unnecessary to configure Ansible environment on every single host (e.g. production). A typically environment only has Ansible environment configured on the bastion host. This brings the need for a dedicated controller server to drive all Ansible tasks.

Ansible Tower is Red Hat’s commercial enhancement to the open source Ansible, providing web-based console, REST API and other services such as Role-based Access Control (RBAC). Managing Ansible via REST API is still somewhat involving but this also enables other open-source contributions to simplify the use of API. For example Tower CLI allows you to use Ansible Tower with simplified command. Ansible Tower has an open-source upstream project called AWX, maintained by Red Hat. AWX is essentially a preview release of Ansible Tower without commercial support. AWX can serve as an engine for all Ansible related task. AWX server is essentially an Ansible control server. AWX, or Ansible Tower, also brings several concepts on top of Ansible:

Job template: defines how an Ansible playbook should be executed, including details such as machine credential, project, inventory, and playbook file.
Job: the actual execution of job template
Project: connects Ansible Tower to source control such as BitBucket. It is tied to a Git repository and a branch within that repository

To deploy AWX on EC2 instances, there is a reference deployment by AWS. However, it is provided as CloudFormation template and appears to be outdated (from 2018). In our project (late 2020, named ansible tower lab, or dubbed as “atlab”), we provide the infrastructure in AWS CDK (written in typescript), to provision the AWX environment.

The goal is that once the configuration is completed, you can run ansible ping against a target EC2 instance. The steps are as automated as possible. However, a number of key steps are purposefully left manual for learning purpose, such as the installation of AWX on EC2 instance.

Infrastructure as Code

In previous posting, I created infrastructure as code in AWS CDK with Python, so I decided to change to typescript in this project, with the assumption it is just a matter of syntax mapping. However, I underestimated the transition to a new language I never learned before. A fuzzy understanding of little details such as when to use let a=4 vs this.a=4, may produce elusive errors that takes hours to troubleshoot. I would therefore strongly recommend reading the basic syntax guide for typescript, before getting started.

In typescript, this example project provides an implementation of configuring autoscaling groups, including cloud init, user data, etc on AWS. Other than the language, everything else is very similar to the project in this post, which was developed in Python. Also, note that the project directory structure varies slightly based on the language being used.

If you are absolutely new to AWS cdk, start with this app. It is beyond the scope of this post, to cover extensively the installation and environment configuration of AWS CDK.

In the provision process for Bastion host, the cloudformation init script pulls a specific version from AWX repository, then makes slight modification. User will need to install it manually. Note that AWX can be installed on three types of platforms:

OpenShift
Kubernetes
Docker Compose

All are documented in their README file. For simplicity in this project, the installation is on standalone docker compose. This is the default mode so there is no need to modify the inventory file.

The code repo

The repository is version controlled here. To run the project, you need to have aws cli environment, then install the required packages including node js, and npm packages such as aws cdk. Once configured, validate with command:

cdk ls

This should display the stacks available. Use cdk deploy to deploy each stack.

When BastionStack is deployed, dependent packages should be installed with user data and cloud init. You will just need to SSH on to the server to manually install AWX, as explained in the instruction, to manually install AWX:

ansible-playbook -i inventory install.yml

Then you can browse to the server (at port 80 by default). Before the log-in page for the first time, the AWX will upgrade itself, with the following screen presented:

Now log on with default credential (in README.md), you will have the UI for AWX:

AWX Web Console

From here, you can edit inventory by adding the host. Or use the helper script (~/awxcompose-helper.sh) from bastion host to create a new inventory (named Private Instance Inventory), and populate it with the hosts in the stack. The helper script does so by querying aws resource, and isssue rest API calls to AWX. After executing the script, you can see a new inventory, and the Private instance inventory should contain all hosts in the stack:

Automatically populated inventory

You can then run Ansible ping against the host to validate connectivity. Note that during inventory creation, the ansible_user is already set to ec2-user (by the helper script):

Ping result

Some technical details

The initialization process on Bastion host creates an RSA key pair, stores the public key to AWS, for the upcoming private instances to uses. It keeps the private key locally in order to make outgoing SSH connection to the private instances. To ensure connectivity between AWX and private instances, there are a couple of (bash) helper scripts involved. Both reflects some technical details that I had to work through.

awxcompose-helper.sh: the initialization process pulls AWX installation file from git repo. The installation process will build a docker-compose file in ~/.awx/awxcompose, based on a template (~/awx-*/installer/roles/local_docker/templates/docker-compose.yml.j2). When user tells AWX to connect to private instance, the connection was made out of a docker container (instead of from the OS of bastion host), we need this script to map SSH key file from host to container, by modifying the template file. Without this helper, outgoing SSH connection will fail with error (Permission denied (publickey,gssapi-keyex,gssapi-with-mic)). This script is invoked in the cloud init process without requiring manual execution.
awxinvt-helper.sh: once the private stack is up and the installation has completed, we need to add the hosts to AWX inventory. This script gets the instance ID and IP addresses of the private instances, and uses Rest API calls to create inventory and populate it with hosts. Ansible has multiple ways of authentication. This script uses the non-stateful basic authentication with each curl command requiring credential. Ansible Rest API guide is provided here and be wary of the convention where URI must end with a slash.

This project is just a start of AWX on AWS CDK project using Typescript. In real life scenarios, there are some work to do to make this even more automated. For example, use cfn-hup service to monitor changes of private stack, and therefore update inventories accordingly.

Docker storage

Tue, 03 Nov 2020 20:22:00 -0400

Microservices are all about stateless and ephemeral workloads, and containers are great microservices. This may suggest that that Docker is all about ephemeral storage. In fact, Docker supports both non-persistent and persistent storage, such as database, kafka, etc.

Non-persistent storage is automatically created, alongside the container and is tied to the lifecycle of the container. On Linux system, it is /var/lib/docker/ as part of container. This is referred to as local storage.

Docker has a concept of volume, which is essentially a file or a directory. Volumes are for persistent data. they are de-coupled from containers and are not tied to the lifecycle of any container. Volume allows process in docker container to bypass the default uionFS, and stores file or directory on host machine. It also allows different containers to share data. You may mount a volume to a container. even if container is deleted, volume persists.

By default, Docker creates new volumes with the built-in local driver. Local volumes are only available to containers on the node they’re created on. There are also third-party drivers as plugins that provides advanced options to integrate external storage system with Docker. (NAS, SAN, etc)

There are more than 25 volume plugins that you can specify with -d switch, to cover all three categories of storage

Block storage tends to be high performance and good for small-block random access workloads.
File storage is high performance, shared amongs multiple containers with NFS or SMB protocols.
Object storage is good for long term storage of large data blobs that do not change frequently. It is often content addressable and relatively low performance.

Note that if you share volume with multiple containers, the application needs to worry about data collision.

You may use docker volume create command to create volume. Note that there is no quota management within docker so the partition needs to be managed at operating system level.

Implementation of Volume

Remember that Docker image is built on multi-layer file system. When we run a container, Docker places a read-write layer on top of the image, such that the active files in running container are all placed in this read-write layer. When container is deleted, so are the files. The file system in Docker is a pseudo file system implemented in unionFS. Volumes bypasses the uionFS and directly accesses the host file system.

When we create a Docker volume, Docker places the volume data to /var/lib/docker/volumes and under each directory named after volume, creates a directory _data, which is attached to the corresponding container.

You can even mount an NFS volume to container. Reference here.

We mentioned UnionFS a couple times so far. UnionFS is a light-weight, layered file system. It can mount the contents of multiple directories to the same directory, to form a single file system. User can use unionFS like a directory. It is the foundation of Docker image and container and enables saving of spaces.

There are three common types of union FS: AUFS, DeviceMapper, and OverlayFS.

AUFS file system

AUFS is the earliest driver that Docker uses for file system, most common in Ubuntu and Debian. To check if the system support AUFS, check out the documentation here.

AUFS is recommended in Ubuntu or Debian. For CentOS and Redhat, it needs to be installed and make sure the command above returns aufs. To configure AUFS, create file /etc/docker/daemon.json and add:

{
 "storage-driver":"aufs"
}

Then restart docker service. Run “docker info” and examine the Storage Driver section, as documented here.

AUFS layers multiple directories on a single Linux host and presents them as a single directory. These directories are called branches in AUFS terminology, and layers in Docker terminology. The unification process is referred to as a union mount.

Layers of a Ubuntu container

This section describes how the layers work and this section describes how it reads and writes files (Copy-on-Write (CoW) strategy to maximize storage efficiency and minimize overhead). CoW characterized AUFS.

AUFS has not been adopted in the Linux kernel mainline for lack of maintainability. So for CentOS, the recommended file system driver is devicemapper.

Devicemapper file system

Devicemapper is a technical framework to map physical block device to virtual block device, introduced since kernel 2.6.9. So it’s essentially different from AUFS. The Logical Volume Manager (LVM) in Linux is also implemented based on devicemapper.

The three critical components in devicemapper are:

mapped device: a virtual device that devicemapper provides to client
target device: the underlying physical device or a section of it.
map table: keeps track of the offset, range, etc between mapped and target devices.

Devicemapper uses target driver to block, filter, and forward I/O requests (e.g. Raid, encryption, think provisioning, etc). In thin provisioning, storage driver only assigns spaces that are needed. Docker uses snapshot technology in thin provisioning. This part of the documentation provides further details as to how device mapper works.

Ubuntu and busybox image layers

Devicemapper has to modes:

loop-lvm: in dev and test environment
direct-lvm: recommended in production

Here is the performance best practice. To configure devicemapper, create /etc/docker/daemon.json file and add:

{
 "storage-driver":"devicemapper"
 "storage-opts":[
 "dm.directlvm_device=/dev/xdf",
 "dm.thinp_percent=95",
 "dm.thinp_metapercent=1",
 "dm.thinp_autoextend_threshold=80",
 "dm.thinp_autoextend_percent=20",
 "dm.directlvm_device_force=false"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure direct-lvm mode is on.

Since devicemapper uses block device to store files, it is faster than directly operate on file system. It is adopted as default driver as unionFS for a long time, ensuring stable performance under Red Hat and CentOS.

OverlayFS file system

Earlier versions of OverlayFS (known as overlay driver) is not stable. Later version is known as overlay2, which is very stable and recommended in overlay2. It requires:

Docker version higher than 17.06.02;
Kernel version higher than 3.10.0-514 for CentOS and RHEL; or higher than 4.0 for other distributions of Linux;
Using with xfs file system with d_type turned on

In production environment, it is recommended to moutn /var/lib/docker to separate disk or partition, to prevent the directory getting full from impacting the host OS. The option pquota is recommended for mounting options in /etc/fstab.

To configure storage driver, create file /etc/docker/daemon.json, with the following content:

{
 "storage-driver":"overlay2",
 "storage-opts":[
 "overlay2.size=20G",
 "overlay2.override_kernel_check=true"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure storage driver is overlay2 and d_type is true.

The way overlay2 works is similar to AUFS, involving union mount process, with lowerdir, upperdir and merged. More details are here, including how overlay2 works with file read and file write (e.g. CopyOnWrite).

Today, overlay2 driver is officially recommended by Docker for its stability and performance, it should be used if all the conditions are met.

Host legacy application in Docker 2 of 2

Thu, 22 Oct 2020 17:54:00 -0400

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

Application service	TCP port to bind
Business service A	8030
Business service B	8040
Business service C	8050
……	…

TCP port requirement

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
The OS needs to host 6 processes of the same application all running at the same time;
The OS does not allow multiple processes to bind to a single TCP port (duh!);
It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

OS	Container ID	Application Service	container port	published port
Host CentOS	Container 1 (Instance #1)	Business Service A	8030	9301
		Business Service B	8040	9401
		Business Service C	8050	9501
	Container 2 (Instance #2)	Business Service A	8030	9302
		Business Service B	8040	9402
		Business Service C	8050	9502
	Container 3 (Instance #3)	Business Service A	8030	9601
		Business Service B	8040	9602
		Business Service C	8050	9603

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
 dapp1:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp1
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9301:8030 # BUSINESS SERVICE A
 - 9401:8040 # BUSINESS SERVICE B
 - 9501:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:cf
 environment:
 - INSTANCE_TAG=dapp1 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp2:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp2
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9302:8030 # BUSINESS SERVICE A
 - 9402:8040 # BUSINESS SERVICE B
 - 9502:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d0 
 environment:
 - INSTANCE_TAG=dapp2 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp3:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp3
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9601:8030 # BUSINESS SERVICE A
 - 9602:8040 # BUSINESS SERVICE B
 - 9603:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d1 
 environment:
 - INSTANCE_TAG=dapp3 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
networks:
 vcnet:
 driver: bridge
 driver_opts:
 com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
- Initialization work that cannot be done in Dockerfile, such as setting environment variable
- Launch the application, including pointing log file to stdout
- Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
build_image.sh: this script makes the image build process smoother
- check if image to build already exist, and ask permission to delete the existing image if so;
- build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

	Dockerfile instruction for container	Permission issue during mount by docker-compose
Bad config	RUN mkdir -p /var/lib/dapp && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.
Good config	RUN mkdir -p /var/lib/dapp/dcontainer/archive && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue.

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.

Automated Deployment Pipeline 3 of 3

Wed, 14 Oct 2020 17:27:00 -0400

Background

We have previously covered a pipeline example with Jenkins calling Ansible to leverage OpenSSH configuration and Ansible inventory. We also discussed a use case with declarative pipeline.

In this posting, I will provide another advanced example, built on declarative pipeline. The pipeline file will be pulled from Git repository. Also, the script is executed on a remote agent, instead of the Jenkins master server. The reason this example is important, is that:

Jenkinsfile is now version controlled (pipeline as code);
Service script (e.g. python) is also version controlled from a central repository;
Computing resource is provided by a remote agent. Since the script is pulled before running, the agent is still fungible;
The result from service script execution is archived (similar to the way build artifact is stored in Jenkins) in master.

The architecture of this pipeline now becomes the followings:

Configure Agent

Jenkins has a plugin called SSH Build Agent, that allows you to configure a Linux agent, communicating with Jenkins master in SSH. For Window agent, it uses JNLP to communicate with master, which is outside of our scope of discussion. As I touched on in previous post, Jenkins uses its own implementation of SSH protocol to achieve this. This means that it cannot re-use the configurations in ~/.ssh/config and thus the ability to do SSH chaining is eliminated. This is incompatible with our architecture so I have to register Jenkins agent using a different launch method “Launch agent via execution of command on the master”. The execution of command on master can still leverage OpenSSH config file. In order to do so, we must copy the agent.jar file to the remote agent first (URL is ${JENKINS_URL}/jnlpJars/agent.jar). Then use SSH command to call the jar file from agent (aka slave) machine. You may add some java argument for troubleshooting. Below is an example configuration for the node.

I shall also note that if the remote agent is a different operating system where the path of bash might be different, then you need to include the directory of bash executable in PATH environment variable. That can be done as in the screenshot above (PATH=${PATH:/usr/bin}). If this is incorrect, you might run into issues when running sh step in Jenkins pipeline. Here is an article about this. To translate that page, the symptom of this issue includes:

Error from pipeline execution that says the following, which is very generic:

process apparently never started in /home/dhunch/jenkins/workspace/site-remote-job@tmp/durable-b997d26c 
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

Job status shows:

hudson.AbortException: script returned exit code -2
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:659)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:605)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:549)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Threaddump (only available during execution) shows:

Thread #6
 at DSL.sh(awaiting process completion in /home/dhunch/jenkins/workspace/site-remote-job/durable-b997d26c; recurrence period: 9543ms; check task scheduled; cancelled? false done? false)
 at WorkflowScript.run(WorkflowScript:9)
 at DSL.script(Native Method)

This issue is tricky because none of the symptoms above make you think about the environment variable defined for the agent, and that the issue does not occur until you actually execute a Jenkins pipeline, usually well after the node agent is registered, and only impacts shell step (sh). It is recommended to test this with sh steps in pipeline.

Configure Repository

Our example job executes service script pulled from SCM, on a remote agent. Then the master pulls the result file to itself. The git repository can be set in Jenkins job, where you specify that the script named Jenkinsfile (from the repo) is the pipeline file that needs to be executed.

This is fairly simple, but what makes the situation more complex is the following few requirements:

it’s the agent node that needs git (installed) and clone to repo;
the master does not need (and should not attempt to) clone from repo;
the master needs to do its job without having to pull SCM;

We need to be able to specify whether each step needs to pull from SCM. The following pipeline syntax shows how this is done:

def agent_dir = 'initial_value'
pipeline {
 agent none
 options { skipDefaultCheckout(true) }
 stages {
 stage('Execute Job') {
 agent {
 label 'remote-agent-customer1'
 }
 options { skipDefaultCheckout(false) }
 steps {
 echo 'Executing job on node'
 sh 'whoami && pwd'
 script {
 agent_dir = sh(returnStdout: true, script: 'echo -n ${WORKSPACE}')
 // the variable should not include carriage return
 }
 sh "echo ${agent_dir}"
 }
 }
 stage('Pull Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 echo 'Pulling job below'
 sh "echo ${agent_dir}"
 sh "scp dhunch@site1:\"${agent_dir}\"/result.csv ./"
 }
 }
 }
}

In this example pipeline file, we declare agent none for the pipeline, then an agent for each specific stage. we also specify the option skipDefaultCheckout as true at the step where pulling from SCM is not needed. This allows us to finish job with multiple agent, and only pull from SCM as needed. This snippet also exemplifies how to declare a variable, assign it from stdout from one agent, and persist the value across ensuing stages.

Store Result

The reason we run pipeline jobs on this remote agent is because it is sitting in customer network and has local direct access to data. So this is perfect for situation such as data analytical jobs, which access to database on local network and store result. We need to pull the result file back to agent and make it available on Jenkins. This is so similar to “archive artifact” task (commonly seen in CI process) that we can simply use its plugin to achieve what we need. Before archive artifact, we need to pull it to local (master), as shown in the example code above. After that, we need another stage to archive the result.

 stage('Archive Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 archiveArtifacts artifacts: '*.csv',onlyIfSuccessful: true,fingerprint: true
 }
 }

I believe there is plugins to compress artifacts as well.

Conclusion

At the end I’d like to reiterate my perception about Jenkins. It is a very generic and adaptive automation platform that originally evolved from use cases in build automation. Due to this original root, many components in Jenkins are named around Continuous Integration use cases, such as the “build” button, and the “archiveArtifacts” step. These misnomers underplays what Jenkins can potentially do in continuous deployment or other automation scenarios. It is important for automation engineers to understand Jenkins components and plugins, through their functionalities and not by the name, and therefore make creative use of Jenkins as automation engine in all scenarios.

An alternative to this proposed pipeline would be Ansible Tower, a commercial project based on open-source Ansible, but with nice UI support. Ansible Tower is Ansible oriented, and it does not have everything that Jenkins can do. It should still be a decent alternative given the proposes pipeline uses Ansible a lot.

Automated Deployment Pipeline 2 of 3

Tue, 06 Oct 2020 22:05:00 -0400

In this posting, we continue to discuss Jenkins’ ability to automate deployment routines. Jenkins supports freestyle project out of the box, as well as Pipeline with several plugins. Freestyle project allows user to specify multiple steps on UI. This does not scale well when your entire process involves many steps. As explained on Jenkins’ website:

While standard Jenkins “freestyle” jobs support simple continuous integration by allowing you to define sequential tasks in an application lifecycle, they do not create a persistent record of execution, enable one script to address all the steps in a complex workflow, or confer the other advantages of pipelines.
In contrast to freestyle jobs, pipelines enable you to define the whole application lifecycle. Pipeline functionality helps Jenkins to support continuous delivery (CD). The Pipeline plugin was built with requirements for a flexible, extensible, and script-based CD workflow capability in mind.

So although freestyle projects are easy to set up, and can do technically everything that Jenkins pipeline can do, the major advantage of Jenkins Pipeline is the ability to manage multiple-step as code, and version control the pipeline-as-code. Here is some more information.

We will start with a freestyle project to understand Jenkins’ ability and then advance to building pipelines.

Freestyle projects

As mentioned, Jenkins calls a task a “build”, and the build can be triggered in a variety of ways:

Time schedule (with or without parameter)
Remotely via API
On completion of other projects
Poll SCM for changes
Commit to SCM

There are also several ways to execute a job:

Execute Shell command, batch command or groovy script
Invoke Ansible adhoc command, playbook or vault
Conditional on specified boolean value, file existence, etc

After the job one can specify post-build jobs, for example:

start other build projects
notification of various means
publishing result file

In deployment, it is common task to execute a task over SSH, using SSH command or invoke Ansible command with Ansible plugin. The former fits simple command line tasks. For example:

The Ansible plugin is good for more steps and more complicated inventory hierarchies. This post includes an example of an inventory involving multiple layers. The other limitation with SSH command is lack of a straightforward configuration to escalate privilege and run remote command. On the other hand, Ansible addressed this with become method. Below is a screenshot

In order to use Ansible command, you will also need to specify where the Ansible binary in Global Tool Configuration. You will also need to store vault credential in Jenkins credential store so it’s not being prompted.

Jenkins Pipeline

Jenkins pipeline allows one to describe actions in a pipeline in groovy Domain Specific Language (DSL). There are two styles of pipelines:

Declarative pipeline is identified by a block named ‘pipeline’, it is relatively new and supports the pipeline-as-code concept. It can be stored as Jenkinsfile in code repository or edited in Jenkins’ UI.
Scripted pipeline is identified by a block named ‘node’, it is the conventional format and can only be edited in Jenkins’ UI.

To fully understand the two styles of pipelines you also need to know declarative programming and imperative programming. Read this instruction for a better explanation. Here is another good one that focus on their differences.

Jenkins’ plugin provides UI components for user to input information (e.g. Invoke Ansible Command), now with Jenkins pipelines, plugins can provide step functions in order to help Jenkins user. This page from Jenkins lists the most common ones, many of which requires plugin installation.

One example of step function is SSH Pipeline Steps. It allows one to issue SSH connection from Jenkinsfile. However, in our deployment scenarios, it has some limitations:

Unable to match a group of host by specific pattern
Cannot use host name aliases
SSH tunneling is not supported
No means of privilege escalation

Out of these 1 through 3 are due to the fact that SSH step function does not use the OpenSSH configuration on the machine. The jenkins-ssh-slaves plugin uses trilead SSH2 implementation written in Java. Only OpenSSH implementation uses ~/.ssh/config file. This creates problem whenever SSH tunneling is needed. For example, to register a remote slave node via SSH tunneling, we need to select “Launch agent via execution of command on the master” instead of “Launch agent via SSH” as launch method. Here is an instruction and below is what it looks like:

Similarly, if we use ssh-agent plugin as tool for deployment, we cannot use any configuration made by OpenSSH. Therefore Ansible in Jenkins Pipeline is a better tool for deployment because it can use OpenSSH.

Pipelines with Ansible

In this section I demonstrate the use of Ansible playbook and adhoc command in pipeline through two examples.

In example 1, the job pulls authorized key file from SCM, and then use a playbook from SCM, to push the key file to all servers in the specified inventory.

Here is the playbook file for example 1:

---
# Example:
# ansible-playbook -l all -i ~/ansible/inventories/bh.yml push-key.yml --ask-vault-pass
- name: push key to target
 hosts: all
 tasks:
 - name: sync key to host
 copy:
 src: "{{authorized_keys_src}}"
 dest: "/home/dhunch/.ssh/authorized_keys"
 force: yes
 mode: 0600

Here is the pipeline script (note ampersand is mistakenly displayed as & in the box below):

import java.net.URLEncoder;
 
pipeline {
 agent any
 
 options {
 skipDefaultCheckout(true)
 }
 environment {
 BITBUCKET_CREDS = credentials('bitbucket') 
 //BITBUCKET_CREDS_USR and BITBUCKET_CREDS_PSW are set
 BITBUCKET_CREDS_USR = "${BITBUCKET_CREDS_USR}"
 BITBUCKET_CREDS_PSW = URLEncoder.encode("${BITBUCKET_CREDS_PSW}", "UTF-8") 
 // if password contains special character we need to url encode it. e.g. @->%40
 }
 stages {
 stage('Start') {
 steps {
 echo "Starting pipeline ..."
 deleteDir()
 }
 }
 stage('Prep') {
 steps{
 wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: "${BITBUCKET_CREDS_PSW}", var: 'RANDOM']]]) {
 // MaskPasswordsBuildWrapper requires Mask Passwords Plugin and is to mask specific string in console output. 
 // Otherwise BITBUCKET_CREDS_PSW will display in the clear
 sh "git init && git config core.sparsecheckout true"
 sh "git remote add origin https://$BITBUCKET_CREDS_USR:${BITBUCKET_CREDS_PSW}@bitbucket.org/vendorcompoany/configmanagerepo.git"
 sh "echo 'public_keys/*' >> .git/info/sparse-checkout"
 echo "Downloading key file"
 sh "git pull --depth=1 origin master" 
 }
 }
 }
 
 stage('Deploy to Site 1') {
 steps {
 echo '> Deploying to Site 1 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site1.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 1 ...'
 }
 }
 
 stage('Deploy to Site 2') {
 steps {
 echo '> Deploying to Site 2 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site2.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 2 ...'
 }
 }
 
 stage('Deploy to Site 3') {
 steps {
 echo '> Deploying to Site 3 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site3.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 3 ...'
 }
 }
 }
 post {
 always {
 cleanWs()
 }
 }
}

In this example, we pull a sub-directory from git repo. We use two tricks to minimize amount of traffic. First, we use git pull with depth=1 so only the required recent commits are pulled, not the entire history. Second we use sparse checkout to get result from a sub-directory, not the entire repo.

When connecting to repo, password is required. We pull the credential from Jenkins’ credential store and they are masked by default. However, the password must be converted to URL string when used in git remote add origin. Otherwise if the password contains special character the URL will not work.

This modification to password brings about another challenge, password masking during the job execution. We will have to build our own wrapper function using class MaskPasswordsBuildWrapper in order to mask any variable.

In example 2, we wrap an Ansible step function to call adhoc command to check version on all hosts. Here is the pipeline script:

def HunchVersionQuery(pattern, siteinventory){
 step([
 $class: 'AnsibleAdHocCommandBuilder',
 ansibleName: 'Ansible on Mac',
 inventory: [$class: 'InventoryPath',path: siteinventory],
 hostPattern: pattern,
 module: 'shell',
 command: 'cat /etc/*release',
 forks: 1,
 vaultCredentialsId: 'ansible-vault-pass'
 ]);
}

pipeline {
 agent any

 stages {
 stage('Start') {
 steps {
 echo 'Staring Pipeline'
 }
 }
 stage('Query Site1') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site1.yml')
 }
 }
 stage('Query Site2') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site2.yml')
 }
 }
 stage('Query Site3') {
 steps {
HunchVersionQuery('*app','~/ansible/inventories/site3.yml')
 }
 }
 }
}

Because Ansible plugin does not provide a warpper function for adhoc command step, we will have to build our own wrapper function in the above code.

Automated Deployment Pipeline 1 of 3

Wed, 30 Sep 2020 22:04:00 -0400

The business case

You launched a software application. You installed it on two customer sites. You support the application mostly by SSH to customer server and run Bash commands, or slightly better, Bash scripts.

The product is a hit to the market. You hired 20 support specialists in a customer service department. The dream client came through: an enterprise giving you a fleet of 100 servers to deploy your application on.

More staff, more business, more installations, more incidents, but the same old command driven steps. Problems:

Non-standard support procedures. Every one takes notes and everyone’s notes are slightly different.
Information sharing among team members are ad hoc, and at high level.
Post-mortem discussion is driven by memory and command fragments, instead of evidence end-to-end

If that looks like your organization, chances are you also suffer from some secondary damages over the long term, such as:

Downtime resolutions rely on the knowledgeable few
Documentation helps. But it never catches up to the latest version of application
Lack of auditing of commands during support

I propose an automation scheme to existing support and deployment practice. This automation scheme combines a suite of common technologies, such as Bash, Python, Ansible, OpenSSH and Jenkins. The automation allows the department to, either fully or partially, operationalize the steps in support and deployment, and eventually shift towards agile practice.

Bash, Python and Ansible

Bash script is based on shell command, perfect for running critical system tasks such as volume management. When it turns into a script, it can be cumbersome, especially with complex data structure. Python, as a tool for system administration, is a good complement to that.

Python 2 comes with most Linux distributions, and is also a dependency of other built-in tools such as YUM. Python3 can be installed easily from default YUM repositories. Both Python2 and Python3 can exist on the same operating system, although new module development are now shifted to Python3. Python’s syntax is very simple and offers object-oriented programming ability. Moreover, there is an entire open-source community behind Python, which offers modules in every aspect of IT (for example, Datastax has a driver module for connecting to Cassandra). Those modules are installed with PIP tool, or PIP3 for python3.

Both bash and Python executes on local machine. To run them on remote servers over SSH. You want to have a list of target hosts, and specify which one to execute the script against. This is where Ansible comes in handy. Ansible is superior in the following aspects:

Free and open-source, with commercial alternative (Towers);
Inventory management (inventory);
Desire state engine (roles)

Ansible is built on Python and is agent-less. Connectivity to remote host is done via secure shell so it can take advantage of existing SSH configurations. Job execution on the target machine is done through Python. With Python you can also develop custom module in Ansible. For some use cases in customer support with Ansible, refer to my two previous postings about Ansible at scale.

Jenkins

The tools above forms a package for automation. The issue is that all of them are command-line based. Any task that requires Ansible requires the IT professional craft up long command, such as running playbook, executing a role, or ad-hoc command. This is inconvenient when a task needs to be done during an incident. Such tasks also require trained professional with the relevant skills.

These tasks can be stored in, or initiated by Jenkins. Although Jenkins is well known for build automation in continuous integration, it is automation engine for any command-line based IT tasks. The button to start such tasks in Jenkins UI is called “Build”, which is also a misnomer that underplays Jenkins’ versatility: building application from source code is just one of the many IT tasks that involves multiple long running commands. In this and next article we introduce Jenkins as an engine for deployment automation.

The infrastructure architecture is diagramed as above, and with the connection across Internet, the target hosts must be hardened properly in the following aspects:

Connectivity to remote host is via SSH chaining, through an SSH proxy;
Root login must be disabled for remote session or by password;
Service user may be shared, but must be authenticated by individual RSA key pair;
Service user connected remotely needs to escalate privilege by su if needed;

I want to make a theoretical distinction between our topic here and continuous deployment. We simply focus on the technical side of deployment automation. Essentially automating a few bash scripts. On the other hand, a continuous deployment process is an extension to an existing continuous integration pipeline, with the vision to streamline the process end-to-end from code commit to production rollout. Implementing CI/CD pipelines should be approached as an organizational program rather than an individual technical initiative. Here is a good technical overview on CI/CD pipeline with Jenkins and Ansible.

Security

The security mechanism of this system is based on OpenSSH because the connectivity between servers are through SSH chaining. RSA key authentication must be used in order to encrypt traffic with password-less login. Connection to an SSH host can be done through a proxy server. Below is an example of SSH configuration:

Include customer1.config
Include customer2.config

Host *
 IdentityFile ~/.ssh/id_rsa
 ServerAliveInterval 60
 ServerAliveCountMax 3
 Compression yes
 ControlPersist 3h
 ControlPath ~/.ssh/sockets/%r@%h-%p

Host gateway
 Hostname support.digihunch.com
 User jdoe
 Port 2223

Host customer-server-0
 Hostname 192.168.201.12
 User support
 ProxyCommand ssh -W %h:%p gateway

Open SSH configuration file (~/.ssh/config) needs to be configured properly with useful host names and aliases. To prevent the config files from growing too long, include statement can be used to reference other configuration file (available with OpenSSH v7.3 sp1 and up).

The host names (as well as aliases) listed in SSH configuration can be directly referenced in Ansible inventory, allowing Ansible (and Jenkins) to reference site by alias and connect to target host through proxy.

Plugins

Jenkins has a community that develops a variety of plugins, which makes Jenkins the most powerful automation platform. Here are some examples of useful plugins:

Audit Trail: output job execution history to file or Elasticsearch;
Credentials: stores credentials in Jenkins;
Pipeline: build declarative (new) or scripted (old) pipeline for Jenkins jobs;
Simple Theme: just a theme but allows console output to be dark (using CSS);
Job Configuration History: job configuration audit;
Mask password: mask variables (including password) from console output
Ansible: invokes ad-hoc commands and playbooks
SSH agent, SSH pipeline steps, SSH credentials: features related in SSH in Jenkins pipelines.
Purge Job History: purge all of build history, or purge by time and number of old builds.
Parameterized Scheduler: schedule to run a job and provide parameter
Workspace cleanup: clean up workspace when invoked.

In the next article, we will go over some common job configurations.

Log file navigator (lnav)

Wed, 23 Sep 2020 21:03:00 -0400

I’ve used a number of log viewers in command terminal, on MacOS and Linux server. I read system logs, log4j formats, as well as json formats. Unfortunately, I have not found an ideal (free) log viewer, either on UI or in command terminal.

Out of those I tried lnav is one of the better ones. It’s been around for more than 10 years and is configurable for a variety of formats. It is available in EPEL-repository for Linux and home brew for Mac.

For example, if our log (produced by log4j) looks like this:

2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): start ARTIM 5000ms
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): Client closed connecti
on without sending data
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550) enter state: Sta1
2020-10-06 20:59:25,471,INFO,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): close Socket[addr=/10.100.101.10,port=24976,localport=44104]
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.AssociationReaper - [platform-dicomServer-44104-574917] Stop check for idle Association(552550)

We can introduce custom formatting, for example:

{
 "dapp" : {
 "title" : "dapp log4j format",
 "description" : "dapp log4j format",
 "regex" : {
 "dapp" : {
 "pattern": "^(?<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}),(?<level>\\w+),(?<component>[\\w-.]+) - \\[(?<thread>[^ ]+)\\] (?<body>.*)$"
 }
 },
 "level-field" : "level",
 "level" : {
 "error" : "ERROR",
 "warning" : "WARN",
 "info" : "INFO",
 "debug" : "DEBUG"
 },
 "value" : {
 "level" : { "kind" : "string", "identifier" : true },
 "component" : { "kind" : "string", "identifier" : true },
 "thread" : { "kind" : "string", "identifier" : true },
 "body" : { "kind" : "string" }
 },
 "highlights" : {
 "DIMSE" : {
 "pattern" : "A-(ASSOCIATE-(RQ|AC)|RELEASE-(RQ|RP)|ABORT)|C-(STORE|MOVE|FIND|ECHO)-(RQ|RSP)",
 "color" : "Red"
 }
 },
 "sample" : [
 {
 "line" : "2020-10-06 12:00:28,500,INFO,dicom.dicom-main - [main] Start listening on port 44104"
 }
 ]
 }
}

Save the content above as ~/.lnav/formats/installed/dapp.json, then load the log file with lnav, lnav will display the log by presenting columns in different colours.

One of the default behaviours is highlighting the IPv4 address, as shown above. This behaviour is however, not optional and currently cannot be turned off, which is reported here.

Kafka high-level Overview

Tue, 21 Jul 2020 23:19:00 -0400

Zookeeper

General definition of distributed system: a software system that is composed of independent computing entities linked together by a computer network whose components communicate and coordinate with each other to achieve a common computational goal. Implementing coordination among components of a distributed system is hard. For example, designated master node becomes single point of failure; cluster needs to detect availability of new nodes as it joins cluster.

Zookeeper is designed to simplify cluster coordination. Zookeeper implements key aspects in cluster coordination, such as distributed consensus, group management, presence protocols and leader election. In order to coordinate a cluster, zookeeper itself also runs in its own cluster, called ensemble. Zookeeper exposes a simple but powerful interface of primitives. Applications can be designed on these primitives implemented through ZooKeeper APIs to solve the problems of distributed synchronization, cluster configuration management, group membership, etc.

Zookeeper Ensemble

Clients can connect to a Zookeeper service by connecting to any member of the ensemble. The members of the ensemble are aware of each other’s state. As long as a majority of the nodes are available, the service will be available. Zookeeper cli (zkCli.sh) can be used to connect to Zookeeper server. they can be downloaded from here.

Zookeeper is integrated with many other services apart from Kafka, such as Nifi and Hadoop.

Kafka

Kafka is a messaging system that is horizontally scalable, fault tolerant. It can also serve as queue storage system and stream processing system. It is distributed and use Zookeeper for cluster coordination. Each node is called a broker.

Topics in Kafka (think of table in database) is a category or feed name to which messages (records) are published. Topic is broken up into ordered commit logs called partitions. Each partition has an ID. Each message in a partition is assigned an offset. Topics that are created in Kafka are distributed across brokers based on the partition, replication, and other factors. Each partition is replicated across several brokers depending on replication factor. For each partition, Kafka elect one replica as the leader of partition.

Writes to a partition is generally sequential. Reading messages can either be from the beginning, or rewind or skip to any port in partition given an offset value. Data in a topic is retained for a configurable period of time.

A message is a unit of data in Kafka, in the format of key-value pair. A key is used to control the message that is to be written to partitions. Messages with the same keys are always written to the same partition (hash map)

A producer publishes new message to a topic. Producers do not care which partition the message is written to and will balance messages over every partition of a topic evenly. Directing messages to a partition is done using the message key and a partitioner, this will generate a hash of the key and map it to a partition.

A consumer is subscribed to one or more topics and read messages sequentially. The consumer keeps track of messages it has consumed by keeping track on the offset of the message. The offset is a bit of metadata (an integer value that continually increases) that kafka adds to each message. Each partition has a unique offset which is stored with the offset of the last consumed message. A consumer can stop and start without losing its current state.

A Kafka broker is designed to operate as part of a cluster. One broker in the cluster also function as the cluster’s controller, which is responsible for administrative operations such as: assigning partitions to brokers; monitoring for broker failures in cluster. A particular partition is owned by a broker and that broker is called the leader of the partition.

All consumers and producers operating on that partition must connect to the leader.

Kafka cluster may replicate across cluster using MirrorMaker.

Reference: Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale

How RPC and NFS work

Wed, 15 Jul 2020 10:45:00 -0400

I touched on NFS in several previous postings, and here is a deeper dive of this particular protocol. NFS is built on top of Remote Procedure Call (RPC) and therefore it is important to understand RPC first. In fact NFS is one of the most prominent user of RPC and the best example for learning RPC.

RPC overview

According to Wikipedia, an RPC is when a computer program causes a procedure to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote. This is a form of client–server interaction (caller is client, executor is server), typically implemented via a request–response message-passing system. In the object-oriented programming paradigm, RPCs are represented by remote method invocation (RMI), such as Java RMI API.

RPCs are a form of inter-process communication (IPC), in that other processes have a different address spaces: if on the same host machine, they have distinct virtual address spaces, even though the physical address space is the same; while if they are on different hosts, the physical address space is different.

RPC is a request–response protocol, and therefore synchronous. An RPC is initiated by the client, which sends a request message to a known remote server to execute a specified procedure with supplied parameters. The remote server sends a response to the client, and the application continues its process. While the server is processing the call, the client is blocked (it waits until the server has finished processing before resuming execution), unless the client sends an asynchronous request to the server. There are many variations and subtleties in various implementations, resulting in a variety of different (incompatible) RPC protocols.

NFS overview

NFS is defined as a set of RPCs, including their arguments, results and effects. RPC makes NFS protocol transparent. RPC is also stateless so the server does not keep the state of RPCs once the request has been served. Each RPC contains the necessary information to complete the call. In the event of server failure, client will need to resubmit requests. NFS has several versions, with v3 and v4 most popular. We will focus on v3 in this posting and brief on v4.

Portmap

RPC makes a remote call appears to client application as a local call, with the help of portmap. The utility for portmap is rpcbind. In RedHat/CentOS 5 or prior, it was even simply called portmap but they are essentially the same service for RPC port mapper.

The rpcbind service is required on both NFS client and NFS server. On the client, it talks to client application, as well as its counterpart on the server. Its main function is query its counterpart on the server, providing a RPC program number, and expects a port number in return. On the server, the rpcbind utility listens at port 111, waiting for request with RPC program number(service), and return the TCP or UDP port number on which the requested service is hosted. RPC program number is reserved numeric indicator of services as outlined in RFC5531. For example, 100005 for mountd, 100021 for nlockmgr, and 100003 for nfs.

The port that rpcbind service itself listens on is always at port 111, which is known to both client and server. This is also referred to as portmapper daemon. Other than this fixed port, each NFS-related service (with their respect reserved program number) may be hosted on different ports on the server. The client rpcbind service must first look up for the port for the requested program number, then it directs the client to initiate connection to the specified port for the specific service.

With the following command rpcinfo command you may look up the program to port number mapping on destination server isilon.company.com

# rpcinfo -p isilon.company.com

The following command is commonly used for displaying mountpoint and troubleshooting mount. Under the hood it is using the information from rpcinfo.

showmount -e isilon.dcb.digitalhunch.com

On the NFS server side, rpcbind service must start before nfs service start. Otherwise nfs service cannot register ports to rpcbind. If you restart rpcbind, every service that had registered to rpcbind must restart to register themselves again. By default, NFS server can dynamically assign a port for mountd, nlockmgr within a port range. This makes firewall setting a pain. These dynamically assigned port can be fixed via configuration files.

File handle

NFS uses file handles (or fhandle) to represent files. It is a better mechanism to reference a file object than pathname for three reasons: 1. file handle has fixed length (32bytes); 2. If the file is renamed, the file handle reference remain the same; 3. If a file is deleted, and then a new file is created with the same path, a new file handle will be created. A file handle has three parts:

volume ID: to identify the mounted file system
inode #: to identify the file within the mounted file system
generation #: to detect when file handle refers to an older version of inode. Traditional Unix filesystems may reuse inode and thus NFS client could mistakenly use an old file handle and access a new file.

The file handle information is only meaningful to the server. New file handles are returned to client by certain procedures, such as LOOKUP, CREATE, and MKDIR. The file handle for the root of the file system, is obtained by the client when it mounts the file system, as permission allows.

Permission and Locking

When accessing a file on server, the client passes uid/gid info in RPCs, and the server performs permission checks as if the user was performing the operation locally. So users and groups are represented as integers. There are two security problems:

The mapping from uid/gid to user must be the same on all clients. This is not practical in large deployment, although can be solved via Network Information Service (NIS);
Whether the root user on the client has root access to files on the server, is a server policy configuration. This can be addressed by enabling “root squashing” on server, so that client’s uid 0 (root) is mapped to 65534 (nobody).

Unix has two locking mechanisms (fcntl and flock). NFS protocol supports fcntl but not flock. The flock function is managed by a separate service (nfslock) to allow NFS to lock files. The nfslock daemon provides the ability to lock regions of NFS files. NFS service itself is still completely stateless with locking managed separately. This is changed in NFSv4.

Procedures used in NFS service

NFS service defines a list of procedures. Here is a list with brief summary of activities. The bottom five RPCs are introduced in v3.

Procedure	Activity
GETATTR(fh)	Returns the attributes of a file, similar to stat syscall.
SETATTR(fh, attr)	Sets the attributes of a file (mode, uid, gid, size, atime, mtime); setting the size to 0 truncates the file
STATFS(fh)	Returns the status of a filesystem, such as block size, number of free blocks. e.g. df command.
LOOKUP (dirfh, name)	Returns fhandle and attributes for the named file in the directory specified by dirfh
READ (fh, offset, count)	Reads from a file, with offset and count specified. In v2, the length is up to 8192 bytes; v3 support more.
WRITE (fh, offset, count, data)	Writes to a file, with offset and count specified, as well as a separate field called data. Returns the new attributes of the file after the write.
CREATE (dirfh, name, attr)	Creates a file with the name, in directory, returns new fhandle and attributes
REMOVE (dirfh, name)	Deletes the named file in from directory dirfh and returns status.
RENAME (dirfh, name, tofh, toname)	Renames name in directory dirfh, to toname in directory tofh.
LINK (dirfh, name, tofh, toname)	Creates a hard link toname, in directory tofh, that points to name, indirectory dirfh.
SYMLINK (dirfh, name, string)	Creates a symbolic link name, in the directory dirfh, with value string.
READLINK (fh)	Reads a symbolic link and get file name of the target.
MKDIR (dirfh, name, attr)	Creates a directory name in the directory dirfh, and returns the new fh and attributes.
RMDIR(dirfh, name)	Removes a directory with the name, from parent directory dirfh.
READDIR (dirfh, cookie, count)	Reads a directory and returns up to count bytes of directory entries from the directory dirfh. The cookie is used in subsequent readdir calls to start reading at a specific entry in the directory. Cookie of zero get the server to start with the first entry in the directory.
NULL	No activity. Used for testing only.
ACCESS	Helps with client caching.
MKNOD	Makes a device special file.
FSINFO	Returns information about the server’s capabilities.
READDIRPLUS	Returns both file handle and attributes to eliminate LOOKUP calls when scanning a directory
COMMIT	In NFSv3, the server can reply to WRITE RPCs immediately without syncing to disk. When client wants to ensure that the data is on stable storage, it sends a COMMIT RPC. This is used in asynchronous writes for better performance, which is an option negotiated at mount time.

The addition of COMMIT procedure in v3 offers the option to improve write performance in place of synchronous write. However, asynchronous write requires more coordination to ensure data integrity during transmission, in the event of server crash. NFSv3 uses write verifier for this purpose. A write verifier is an 8-bye value that the server must change if it crashes.

After an asynchronous write, the reply from WRITE RPC includes a write verifier, the client must keep it for later use;
The client then sends a COMMIT RPC and the reply contains another write verifier;
The client compares the verifiers from the two returns for crash detection. If the verifiers don’t match, the client must rewrite all uncommitted data.
The client must keep all uncommitted data in case of a server crash.

Additional daemon processes

In addition to the three essential services, (nfs, rpcbind and nfslock), there are several auxiliary processes that facilitates NFS services. Their functions are listed here:

Process	Description
rpc.mountd	Used by NFS server to process MOUNT requests from NFSv3 client. It checks that the requested NFS share is currently exported by the NFS server, and that the client is allowed to access it. If the mount request is allowed, the rpc.mountd server replies with a Success status and provides the File-Handle for this NFS share back to the NFS client.
rpc.nfsd	Allows explicit NFS versions and protocols the server advertises to be defined. It works with the Linux kernel to meet the dynamic demands of NFS clients, such as providing server threads each time an NFS client connects. This process corresponds to the nfs service.
rpc.lockd	A kernel thread which runs on both clients and servers. It implements the Network Lock Manager (NLM) protocol, which allows NFSv3 clients to lock files on the server, using procedures such as NLM_NULL, NLM_TEST, NLM_LOCK, NLM_GRANTED, NLM_UNLOCK, NLM_FREE. The service is started automatically whenever the NFS server is run and whenever an NFS file system is mounted.
rpc.statd	This process implements the Network Status Monitor (NSM) RPC protocol, which notifies NFS clients when an NFS server is restarted without being gracefully brought down. rpc.statd is started automatically by the nfslock service, and does not require user configuration. This is not used with NFSv4.
rpc.rquotad	This process provides user quota information for remote users. rpc.rquotad is started automatically by the nfs service and does not require user configuration.
rpc.idmapd	provides NFSv4 client and server upcalls, which map between on-the-wire NFSv4 names (strings in the form of user@domain) and local UIDs and GIDs. For idmapd to function with NFSv4, the /etc/idmapd.conf file must be configured. At a minimum, the “Domain” parameter should be specified, which defines the NFSv4 mapping domain. If the NFSv4 mapping domain is the same as the DNS domain name, this parameter can be skipped. The client and server must agree on the NFSv4 mapping domain for ID mapping to function properly.

NFSv4

Even NFSv4 was introduced in 20 years ago, it improves access and performance of NFS on the Internet. It should be the default option for any new deployment.

NFSv4 is TCP only protocol and it is stateful.
NFSv4 combines mount and lock protocols into NFS so only one port is being used.
Users and groups are identified with strings (user@domain, or group@domain where domain represents a registered DNS domain or sub-domain), instead of integers. The access control policies are compatible with both Unix and Windows.
NFSv4 mandates strong RPC security built on cryptography, with negotiation at the time of mount
NFSv4 adopted a framework for authentication, integrity and privacy at RPC level
Introduced new RPC COMPOUND, which allows for several operations in one go. At the server, operations are evaluated in order, and each has a return value.

NFSv4.1 was release in 2010, and 4.2 in 2016. Both AWS EFS and Azure File storage supports 4.1.

Performance Analysis

Fri, 19 Jun 2020 16:47:01 -0400

Overview

In 2015, Brendan Gregg posted two great articles on Netflix blog: Linux Performance Analysis in 60 seconds, and Linux Perfomrance Tools. They have great value when I was in a urgency to spot performance issues. The articles cover the essential tools for performance troubleshooting, including:

Check out load averages: w or uptime
Print kernel ring buffer: dmesg -T
Virtual memory status: vmstat 1
Multiple processor staticstics: mpstat -P ALL 1
Task status: pidstat 1
CPU and I/O status: iostat -xz 1
Free memory check: free -m
Network Activity record: sar -n DEV 1
TCP activity record: sar -n TCP,ETCP 1
Display processes: top

We will dive into each of them in the next section.

Basic Troubleshooting

The command w is equivalent of uptime (which shows uptime since boot) and who (which shows logged-in users). It also displays load average for the last 1 minute, 5 minutes and 15 minutes. The number of load average reflects the overall system load (CPU + disks), and it is further discussed in this post with a simple take away:

If the averages are 0.0, then your system is idle.
If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
If they are higher than your CPU count, then you might have a performance problem (it depends).

When Linux load averages increase, you know you have higher demand for resources (CPUs, disks, and some locks), but you aren’t sure which. You will need to switch to other metrics. Brendan recommend don’t spend more than 5 seconds on these numbers.

[dhunch@c7v-ghintapp01 ~]$ w
 12:14:10 up 46 days, 16:41, 3 users, load average: 2.69, 2.44, 2.29
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dhunch pts/0 w6v-ghas01 24Jun20 3days 0.36s 0.30s ssh c7v-bastion
dhunch pts/1 202.95.88.111 12:02 2.00s 0.00s 0.00s w
dhunch pts/4 w6v-ghas01 17Jun20 15days 0.15s 0.07s view readme.txt

Before moving to more insightful metrics, it is also worth a quick look into the kernel ring buffer with dmesg command (dmesg -T | less +G). This will allow us to capture obvious issues such as oom-killer or TCP request dropping.

The vmstat tool reports the statistics of virtual memory. Servers have a fixed amount of physical memory, but they can run a set of applications that use a much larger amount of virtual memory. Application tend to reserve more memory than they need, and they usually operate on only a subset of their memory. In both cases, the operating system can keep the unused parts of memory on disk, and page it into physical memory only if it is needed. For the most part, this kind of memory management works well. But it doesn’t always with Java applications due to Java heap. Once a system start swapping – moving pages of data from main memory to disk, and vice versa, the performance tend to be bad. Systems must be configured so that swapping never occurs.

[dhunch@c7v-ghintapp01 ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 3 0 239360 385928 0 36734692 0 0 1 200 0 0 6 1 93 0 0
 3 0 239360 387732 0 36734704 0 0 0 43 8017 8524 14 0 85 0 0
 3 0 239360 387608 0 36734904 0 0 0 57 6768 7680 14 1 86 0 0
 2 0 239360 389008 0 36734904 0 0 0 44 6366 7300 14 0 86 0 0
 3 0 239360 421728 0 36700144 0 0 0 0 8141 7957 13 0 86 0 0
 2 0 239360 421984 0 36702048 0 0 0 467 8994 8362 14 0 85 0 0

The tool prints key server statistics each line, with the first line showing the average since boot. Here lists the explanation of some columns:

r: number of processes running on CPU and waiting for a turn. This provides a better signal than load averages for determining CPU saturation, as it does not include I/O. To interpret: an “r” value greater than the CPU count is saturation.
swpd: the amount of virtual memory used. This number should align with the used column for Swap row from free command.
buff, cache: buffer and cache. They should align with the buff/cache column form Mem row from free command.
free: free memory in kilobytes. This number should align with the free column for Mem row from free command.
si, so: swap-ins and swap-outs. As mentioned, if these are non-zero, you’re out of memory.
bi, bo: blocks received from and sent to a blcok device (# of block per second)
in, cs: number of interrupt, and context switches per second.
us, sy, id, wa, st: user, system, idle, wait I/O and stolen times. These are breakdowns of CPU time, on average across all CPUs. They should add up to 100% (or close). stolen time is amount of CPU time needed by a guest virtual machine that is not provided by the host. IO wait time is the CPU time waiting for I/O activity. Idle time could be several things: the process may be waiting for something (e.g. a response from database); the process may be blocked by a thread lock; or the process simply has nothing to do. user and system times are CPU times spent on user tasks and kernel tasks, respectively.

Out of these columns, watch for columns r, free, buff, cache, us, sy, id and wa at minimum. The combination of us and sy confirms if CPUs are busy. A constant degree of wa points to a disk bottleneck with too much time spent on pending disk I/O. The sy (kernel time) is necessary for I/O processing but sy stays high (e.g. constantly over 20%), it becomes interesting. Perhaps the kernel is processing I/O inefficiently.

For further per-CPU stats, use mpstat command (-P ALL), to prind CPU time breakdowns per CPU and check for imbalance. A single host CPU can be evidence of a single-threaded application. Here is an example output from a system of 16 CPU cores.

[dhunch@c7v-ghintapp01 ~]$ mpstat -P ALL 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

13:47:23 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
13:47:24 all 13.77 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 86.05
13:47:24 0 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 1 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 3 98.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
13:47:24 4 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 5 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 6 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 7 2.94 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 96.08
13:47:24 8 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 9 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 10 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
13:47:24 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 12 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 13 1.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 14 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

For a per process summary of CPU consumption, use pidstat command. It can be thought of a periodical snapshot of top command, allowing you to watch for patterns. The %CPU column is the total across all CPUs so 5 CPUs have a maximum value of 500.

If vmstate indicates some I/O issue, iostat tool can help us understand block devices, on both the workload applied and the resulting performance. Key columns are:

r/s, w/s, rkB/s, wkB/s: delivered reads, writes, read Kbytes, and write Kbytes per second to the device. Use these for workload characterization. A performance problem may simply be due to an excessive load applied.
await: the average wait time for I/O in milliseconds. This is the time that the application suffers, as it includes both time queued and time being serviced. Larger than expected average times can be an indicator of device saturation, or malfunction.
avgqu-sz: the average number of requests issued to device. Values greater than 1 can be evidence of saturation (although devices can typically operate on requests in parallel, especially virtual devices which front multiple back-end disks.)
%util: device utilization. This is really a busy percent, showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance (which should be seen in await), although it depends on the device. Values close to 100% usually indicate saturation.

I/O problem may either be inefficiencies in application that issues I/O request, or slowing disk unable to keep up with I/O requests. We review two examples here to illustrate each situation. The first output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 23.45 0.00 37.89 0.10 0.00 38.56

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 11.60 0.60 24.20 0.02 0.14 13.35 0.15 6.06 5.33 6.08 0.42 1.04

In the first example, the disk stat loosk up at first glance. The w_await (time to service I/O write) is fairly low at 6.08ms. However, the system is spending 37.89% of its time in the kernel. If all that system time is from the application, it suggest something inefficient is happening. The fact that the system is doing 24.2 writes per second is another clue: that is alot when writing only 0.14 MB per second (MBps). I/O has become a bottleneck, and the next step would be to look into how the application is performing its writes.

The second example output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 35.05 0.00 7.85 47.89 0.00 9.20

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 0.20 1.00 163.40 0.00 81.09 1010.19 142.74 866.47 97.60 871.17 6.08 100.00

In this example, it tells us that processes are spending 47.89% of their time in iowait, and the data to complete the I/O (w_await) is 871ms, the queue size is large, and the disk is writing at 81MB per second. This all points to disk I/O as a problem and that the amount of I/O in the application (or elsewhere in the system) must be reduced.

Bear in mind that poor performing disk I/O isn’t necessarily an application issue. Many techniques are typically used to perform I/O asynchronously, so that the application doesn’t block and suffer the latency directly (e.g. read-ahead for reads, and buffering for writes, also refer to “tail latency“).

Note that the acceptable utilization metric depends on the configuration of block device. If the storage is a logical disk device fronting many back-end disks (e.g. RAID 0), then 100% utilization may just mean that some I/O is being processed 100% of the time, however, the back-end disks may be far from being saturated, and may even be able to handle more work.

The free command gives the breakdown of memory usage. The right two columns are:

buffers: for the buffer cache, used for block device I/O.
cached: for the page cache, used by file systems.

We just want to check that these aren’t near-zero in size, which can lead to higher disk I/O (confirm using iostat), and worse performance. Linux uses free memory for the caches, but can reclaim it quickly if applications need it. So in a way the cached memory should be included in the free memory column. In this case, it’s included in the available column. This website has further details.

To check interface stat, nicstat is a great tool but it isn’t available by default in Linux. Instead we can run sar (-n DEV) to retrieve stats.

[dhunch@c7v-ghintapp01 ~]$ sar -n DEV 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:51:25 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:26 eth0 3089.00 934.00 3815.33 834.61 0.00 0.00 0.00
16:51:26 lo 464.00 464.00 2289.07 2289.07 0.00 0.00 0.00

16:51:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:27 eth0 956.00 586.00 826.66 211.34 0.00 0.00 0.00
16:51:27 lo 213.00 213.00 196.00 196.00 0.00 0.00 0.00

16:51:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:28 eth0 349.00 181.00 52.32 147.19 0.00 0.00 0.00
16:51:28 lo 244.00 244.00 81.13 81.13 0.00 0.00 0.00

Here rxkB/s/s and txkB/s represents receive and transmission throughput, as a measure of workload. If they reach the limit then the interface is the bottleneck.

On top of interface is the TCP layer, which can be monitored with sar again (-n ECP, ETCP). The key metrics are:

active/s: number of locally-initiated (e.g. via connect()) TCP connections per second
passive/s: number of remotely-initiated (e.g. via accept()) TCP connections per second
retrans/s: number of TCP retransmits per second

The active and passive counts are often useful as a rough measure of server load. It might help to think of active as outbound, and passive as inbound, but this isn’t strictly true (e.g. consider a localhost to localhost connection). Retransmits are a sign of network or server issue; it may be an unreliable network (e.g. public Internet), or it may be due to a server being overloaded and dropping packets.

[dhunch@c7v-ghintapp01 ~]$ sar -n TCP,ETCP 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:52:29 active/s passive/s iseg/s oseg/s
16:52:30 0.00 1.00 28.00 35.00

16:52:29 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:30 0.00 0.00 0.00 0.00 0.00

16:52:30 active/s passive/s iseg/s oseg/s
16:52:31 8.00 8.00 200.00 317.00

16:52:30 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:31 0.00 1.00 1.00 0.00 3.00

Last but not least is our favourite command top, which includes many of the metrics covered in previous tools. The downside to top is it is harder to see patterns over time, which may be more clear in tools like vmstat and pidstat, both of which produce rolling output.

Several tools introduced here involves sar, which is a great monitoring tool on its own that we need to be familiar with.

System Activity Report (SAR)

Further to the basic tools, the sar command is very helpful as it stores historical stat every 10 minutes. Sar keeps 18 types of reports, all stored in /var/log/sa/. When viewing the report file, you may pipe the result to less command so it only prints header once. For example, if you would like to print CPU report for the 2nd of the month:

# sar -u -f /var/log/sa/sar02 | less

If you check NFS client statistics for the 31st

 # sar -n NFS -f /var/log/sa/sar31

If you need to check network server statistics for today

# sar -n NFS -f /var/log/sa/sar31

Below are all types of reports:

-u CPU utilization
-w task creation and system switching activity
-W swapping statistics
-B report paging
-b report I/O and transfer rate statistics
-R report memory statistics
-r memory utilization
-S swap space utilization
-H huge pages utilization statistics
-v inode
-q queue length and load average
-y TTY device activity
-d activity for each block device
-n network statistics, DEV (per interface)
-n network statistics, EDEV (error per interface)
-n network statistics, NFS (NFS client)
-n network statistics, NFSD (NFS server)
-n network statistics, SOCK (socket usage)

Berkeley Packet Filter (BPF) Compiler Collection (bcc) tools

For advanced, low-level performance troubleshooting, the BCC tools provide a suite of tools. Here we only cover the installation of it.

In CentOS, install bcc-tools package with yum. When you try to run a command, such as cachestat, if you come across this error:

-bash: cachestat: command not found

Then you will need to add its path to default:

export PATH=$PATH:/usr/share/bcc/tools

Now if you run into this error:

chdir(/lib/modules/3.10.0-1062.12.1.el7.x86_64/build): No such file or directory
Traceback (most recent call last):

The file listed is a symbolic link, and if it is missing, you just need to install kernel-headers that matches the kernel version:

yum install kernel-headers

Then you may use tools in /usr/share/bcc/tools. For example, cachestat help you display page cache hit ratio; gethostlatency shows DNS resolution latency; tcpconnect prints out active tcp connections (made via connect system call):

[root@dhunch ~]# /usr/share/bcc/tools/tcpconnect -t -P 8080 | gawk '{ print strftime("%F %T  "), $0 }'
2020-06-13 00:16:57   TIME(s)  PID    COMM         IP SADDR            DADDR            DPORT
2020-06-13 02:16:57   0.000    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.064    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.438    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080

The command above outputs a time and pid stamped log line every time a TCP connection is made to port 8080; tcpaccept traces passive tcp connections (via accept system call). These tools are not as intrusive as tcpdump.

It is beyond the purpose of this article to get into details of each tool in the BFP suite. The tools are covered in detail in books “BPF Performance Tools” and “Linux Observability with BPF”.

Setup WSL2 (and Docker) on Windows 10

Tue, 02 Jun 2020 22:01:00 -0400

This is not for Linux snobs, but rather for those who are stuck with a Windows work laptop, have to deal with Linux on a daily basis, and are not a fan of PuTTY. This posting provides the steps to setup Windows 10 so you get a work environment closer to a Linux one. The environment to begin with should be Windows 10 version 2004 and above in order to use WSL2. Here’s the comparison between WSL and WSL2. This presentation is a great deep dive into how WSL2 works. The architecture diagram below is from that presentation. Note that WSL2 operates on a “true” Linux Kernel, therefore giving WSL2 the ability to run Docker.

Install WSL2

WSL2 was officially released in Windows 10 version 2004 (build 19041 or higher) and we will use Ubuntu 20.04 LTS. If you’re upgraded from older version of Windows 10 you will need to upgrade. The steps are:

Install Ubuntu 20.04 LTS from Microsoft Store, which requires Windows components “Windows Subsystem for Linux” and “Virtual Machine Platform”; or, if you already have an older version of Ubuntu such as 18.04, you will need to run a distribution upgrade with “sudo apt-get dist-upgrade”;
Upgrade Linux Virtual Machine from WSL to WSL2, following the official instruction here; Once completed, use this command to confirm the WSL version installed: wsl.exe –list -v

There’s also plenty of Youtube videos with step-by-step instruction on upgrading to WSL2 and its advantage over WSL. Once you’re there, you will realize one problem: the command terminal for Ubuntu is ugly:

The colour looks awful and it does not support multi-tabs. If you want better palette, there are a number of options. Check out ColorTool project and iTerms2 Colors project. The former gives you away to configure color scheme (from command prompt but takes effect in WSL as well) and the latter gives you rich choices of color schemes. My personal favourite is “Banana Blueberry”. The command to set color scheme is:

C:\ColorTool.exe -b "iTerm2-Color-Schemes-master\schemes\Banana Blueberry.itermcolors"

Windows Terminal

For a better terminal, consider Windows Terminal from Microsoft Store. Windows terminal is an open-source project and still a little glitchy as of June 2020. However it is heading to the right direction. Once installed, you may customize it by clicking settings:

A json file will open and you may edit it for customization, for example, you can adjust each profile (e.g. PowerShell, Windows command or WSL); you can specify default profile and default open location.

// This file was initially generated by Windows Terminal 1.0.1401.0
// It should still be usable in newer versions, but newer versions might have additional
// settings, help text, or changes that you will not see unless you clear this file
// and let us generate a new one for you.

// To view the default settings, hold "alt" while clicking on the "Settings" button.
// For documentation on these settings, see: https://aka.ms/terminal-documentation
{
 "$schema": "https://aka.ms/terminal-profiles-schema",

 "defaultProfile": "{2c4de342-38b7-51cf-b940-2309a097f518}",

 // You can add more global application settings here.
 // To learn more about global settings, visit https://aka.ms/terminal-global-settings

 // If enabled, selections are automatically copied to your clipboard.
 "copyOnSelect": false,

 // If enabled, formatted data is also copied to your clipboard
 "copyFormatting": false,

 // A profile specifies a command to execute paired with information about how it should look and feel.
 // Each one of them will appear in the 'New Tab' dropdown,
 // and can be invoked from the commandline with `wt.exe -p xxx`
 // To learn more about profiles, visit https://aka.ms/terminal-profile-settings
 "profiles":
 {
 "defaults":
 {
 // Put settings here that you want to apply to all profiles.
 },
 "list":
 [
 {
 "guid": "{2c4de342-38b7-51cf-b940-2309a097f518}",
 "hidden": false,
 "name": "Ubuntu",
 "source": "Windows.Terminal.Wsl",
 "startingDirectory": "//wsl$/Ubuntu/home/myuser/"
 },
 {
 // Make changes here to the powershell.exe profile.
 "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
 "name": "Windows PowerShell",
 "commandline": "powershell.exe",
 "hidden": false
 },
 {
 // Make changes here to the cmd.exe profile.
 "guid": "{0caa0dad-35be-5f56-a8ff-afceeeaa6101}",
 "name": "Command Prompt",
 "commandline": "cmd.exe",
 "hidden": true
 },
 {
 "guid": "{b453ae62-4e3d-5e58-b989-0a998ec441b8}",
 "hidden": true,
 "name": "Azure Cloud Shell",
 "source": "Windows.Terminal.Azure"
 }
 ]
 },

 // Add custom color schemes to this array.
 // To learn more about color schemes, visit https://aka.ms/terminal-color-schemes
 "schemes": [],

 // Add custom keybindings to this array.
 // To unbind a key combination from your defaults.json, set the command to "unbound".
 // To learn more about keybindings, visit https://aka.ms/terminal-keybindings
 "keybindings":
 [
 // Copy and paste are bound to Ctrl+Shift+C and Ctrl+Shift+V in your defaults.json.
 // These two lines additionally bind them to Ctrl+C and Ctrl+V.
 // To learn more about selection, visit https://aka.ms/terminal-selection
 { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" },
 { "command": "paste", "keys": "ctrl+v" },

 // Press Ctrl+Shift+F to open the search box
 { "command": "find", "keys": "ctrl+shift+f" },

 // Press Alt+Shift+D to open a new pane.
 // - "split": "auto" makes this pane open in the direction that provides the most surface area.
 // - "splitMode": "duplicate" makes the new pane use the focused pane's profile.
 // To learn more about panes, visit https://aka.ms/terminal-panes
 { "command": { "action": "splitPane", "split": "auto", "splitMode": "duplicate" }, "keys": "alt+shift+d" }
 ]
}

With these configuration, we have a quite comfortable Linux work environment on Windows.

Update to Zsh and beautify it

I’m no so big a fan of zsh (for lack of default wildcard support) but I do like one of its themes. We can install zsh on WSL2 (assuming Ubuntu distribution). This is a good instruction. I do like the theme name agnoster. However, the theme displays username at the beginning of the prompt, which takes a lot of screen real estate if your username is long. To remove it, the trick is to add a line in ~/.zshrc, as answered in this thread.

Docker runtime

There is a lot of fun to have in WSL2 on Windows 10. For example, you can configure Docker runtime on Windows according to this post. However, you cannot route traffic to the container in the absence of docker0 bridge for Docker on WSL2, as indicated in the known limitations.

Update: Docker also brings a Kubernetes cluster named docker-desktop. This allows you configure Kubernetes cluster, or install Helm and Rancher (instruction) for cluster management.

Dual-Boot?

With WSL2 on top of a real Linux kernel, I do not find a need for a dual-boot system on my workstation. To be fair, only in the following scenarios should one consider installing a dual-boot system.

You need to work on a specify version of Linux Kernel. The kernel provided by Microsoft is called “microsoft-standard-WSL2”, and its pretty up to date (5.x)
You need to run GUI applications on Linux. Update: Microsoft has started developing GUI application support on WSL2, even though it usually takes time to mature.
You need to work on a distribution not available in Windows Store. WSL2 provides a kernel and it’s up to developers to provide Linux distros in Windows store. Popular (and official) ones are: Ubuntu 20.04 LTS by Canonical Group, Debian by the Debian project, SUSE Linux Enterprise Server by SUSE, Kali Linux.

Certificate Issuers

I have to use WSL2 mostly because I’m forced to use a Windows laptop on my contracts. That also means that the Windows laptop has been configured to corporate firewall’s packet inspection. When that is the case, they corporate IT usually automatically imports a firewall certificate to the Windows trust store. However, the WSL is missing this certificate.

The symptom would be that you cannot run certain curl command. For example, when I do the followings:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This error message basically means that the website certificate cannot be trusted. To view the certificate, use Open SSL command:

openssl s_client -connect dl.k8s.io:443 -servername dl.k8s.io | openssl x509 -text -noout | less

In my case, I noticed that the certificate is a Zscaler issued certificate (firewall). While running the same command from my personal device returns a Goolge Trust issued certificate. This indicates that the certificate is changed by the proxy managed by the corporation. To fix this I need to export the certificate from Windows, and then import it in WSL. The answer in this post has good detailed steps.

Balloon steals memory from virtual machines

Thu, 07 May 2020 19:49:00 -0400

This article is my experience with memory balloon on virtual machine.

I came across an ElasticSearch server (ESXi guest with 32GB physical memory) where the main process keeps dying of OOM. Even worse, after the OOM event, the free memory left is about 10G and Elastic Search cannot start because its JVM is set with -Xms16g in /etc/elasticsearch/jvm.options. So I need to address the OutOfMemory error.

To understand what triggered OOM, we can use dmesg or just check /var/log/message, where the memory snapshot by OOM killer is displayed during the kernel panic:

Apr 26 04:18:15 elastichost kernel: kworker/7:1 invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: kworker/7:1 cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 7 PID: 13968 Comm: kworker/7:1 Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Workqueue: events_freezable vmballoon_work [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc05f1f84>] vmballoon_work+0x454/0x6ff [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22b9d4f>] process_one_work+0x17f/0x440
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bade6>] worker_thread+0x126/0x3c0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1c31>] kthread+0xd1/0xe0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2974c37>] ret_from_fork_nospec_begin+0x21/0x21
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3649243 inactive_anon:439380 isolated_anon:0#012 active_file:480 inactive_file:693 isolated_file:0#012 unevictable:0 dirty:2 writeback:0 unstable:0#012 slab_reclaimable:77845 slab_unreclaimable:10219#012 mapped:10086 shmem:9596 pagetables:16727 bounce:0#012 free:50116 free_pcp:238 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122768kB min:5972kB low:7464kB high:8956kB active_anon:471052kB inactive_anon:471040kB active_file:232kB inactive_file:460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:584kB shmem:300kB slab_reclaimable:23908kB slab_unreclaimable:3872kB kernel_stack:656kB pagetables:6440kB unstable:0kB bounce:0kB free_pcp:288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:214 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61804kB min:61576kB low:76968kB high:92364kB active_anon:14125920kB inactive_anon:1286480kB active_file:1688kB inactive_file:2312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:8kB writeback:0kB mapped:39760kB shmem:38084kB slab_reclaimable:287472kB slab_unreclaimable:36988kB kernel_stack:5680kB pagetables:60468kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:698 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 407*4kB (UEM) 506*8kB (UEM) 431*16kB (UM) 364*32kB (M) 272*64kB (UM) 179*128kB (UM) 99*256kB (UM) 35*512kB (M) 11*1024kB (M) 2*2048kB (M) 0*4096kB = 123164kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15576*4kB (UM) 39*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62616kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 44652 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 34162 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884045, delete 10845872, find 3294122/3678936
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 12996 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 83 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 50 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 47 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 113 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 39 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 132 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 4770 100 482 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6332 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 30 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3994129 15027 457539 0 java
Apr 26 04:18:15 elastichost kernel: [16264] 495 16264 18032 0 31 162 0 controller
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16070 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16264 (controller) total-vm:72128kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:15 elastichost kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: java cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 2 PID: 16265 Comm: java Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b6347>] __page_cache_alloc+0x97/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b8fa8>] filemap_fault+0x298/0x490
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484d0e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c2dc0>] ? wake_bit_function+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484f0c>] xfs_filemap_fault+0x2c/0x30 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e444a>] __do_fault.isra.59+0x8a/0x100
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e49fc>] do_read_fault.isra.61+0x4c/0x1b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e93a4>] handle_pte_fault+0x2f4/0xd10
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ebedd>] handle_mm_fault+0x39d/0x9b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f5e3>] __do_page_fault+0x203/0x500
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f915>] do_page_fault+0x35/0x90
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296b758>] page_fault+0x28/0x30
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3607073 inactive_anon:480522 isolated_anon:0#012 active_file:8 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:75170 slab_unreclaimable:10131#012 mapped:2070 shmem:9592 pagetables:16696 bounce:0#012 free:50006 free_pcp:72 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122560kB min:5972kB low:7464kB high:8956kB active_anon:471000kB inactive_anon:471076kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:292kB shmem:288kB slab_reclaimable:22832kB slab_unreclaimable:3752kB kernel_stack:672kB pagetables:6408kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:215 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61572kB min:61576kB low:76968kB high:92364kB active_anon:13957292kB inactive_anon:1451012kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:0kB writeback:4kB mapped:7988kB shmem:38080kB slab_reclaimable:277848kB slab_unreclaimable:36756kB kernel_stack:5664kB pagetables:60376kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:500 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 393*4kB (UEM) 490*8kB (UEM) 453*16kB (EM) 369*32kB (UEM) 269*64kB (M) 180*128kB (M) 98*256kB (M) 35*512kB (M) 12*1024kB (UM) 1*2048kB (M) 0*4096kB = 122148kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15440*4kB (UM) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61768kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 42975 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 33308 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884233, delete 10846914, find 3294127/3678944
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 4016 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 78 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 46 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 42 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 91 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 36 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 130 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 1215 100 485 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6230 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 27 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3993992 15027 457699 0 java
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16291 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16070 (java) total-vm:188490524kB, anon-rss:15975968kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:16 elastichost systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL
Apr 26 04:18:16 elastichost systemd: Unit elasticsearch.service entered failed state.
Apr 26 04:18:16 elastichost systemd: elasticsearch.service failed.

The line “Free swap = 0kB” suggest the trigger of OOM is out of swap. So does ElasticSearch contribute to the swap shortage?

ElasticSearch was the main process on the VM and I noticed that the OS did not disable swap, which does not align with the best practice from ElasticSearch community. In order to find out whether Elastic Search is pushed to use swap, we can get the process ID:

pidof java

ElasticSearch happens to be the only Java based process and the PID is 2283, the following command shows the swap usage by this process:

cat /proc/2283/status | grep VmSwap

If it shows a non-zero value, then ElasticSearch is using swap and you should expect some performance issues. We definitely should disable swap on ElasticSearch but does that solve the problem? Probably not because we didn’t address what caused the memory pressure in the first place. Disabling swap most likely makes ElastciSearch last longer before something else such as shortage of available memory triggers OOM.

In search for the source of memory shortage, I checked the result of free command and the top command. The free command (-mh) shows the following:

 total used free shared buff/cache available
Mem: 31G 20G 10G 117M 243M 10G
Swap: 2.0G 249M 1.8G

I tried to follow my example from a previous post to make sense of the memory reads. When I monitor process with top and watch for RSS column, I cannot identify a single process that even takes more than a few hundred megabytes. I’ve downloaded a tool smem from epel repo, and the result of “smem -kt” suggest that the total RSS is about 372.2M:

So there is about 19.6GB of memory usage unaccounted for. There is something that takes this much memory in the VM and haunting round even after OOM killer!

Inspired by this post, I was able to identify the culprit, that is the memory balloon. We don’t have access to the host but from the Guest OS, we can tell by vmware-toolbox-cmd (need to install yum package open-vm-tools):

vmware-toolbox-cmd stat balloon

The result displays a whopping 20807 MB as memory balloon! This needs to be sent to PaaS vendor for investigation but it is likely a result of memory over-allocation/over-commitment at host level, as well as the setup where the memory of guest is not reserved. Looking at the original log snippet, the line with “events_freezable vmballoon_work” is also an indicator of balloon causing the OOM.

Hypervisor needs memory ballooning to reclaim memory from guest. Since the guest OS does not expect the amount of physical memory to change, hypervisor has to maintain the illusion that the guest has its fixed amount of physical memory. The hypervisor first computes the amount of memory that needs to reclaim, then it leverage some low-level mechanism such as a balloon driver (a pseudo-device driver) installed on guest. The driver communicates with hypervisor and is told to allocate or de-allocate memory. If the driver is told to allocate memory to host, it tells the guest OS to pin the allocated pages into physical memory so they are locked and the physical memory available to guest OS is decreased. All these low level mechanisms explains why it is hard to account for memory consumption based on process running in Guest OS.

Memory balloon is a dynamic process and the hypervisor adjusts the size of balloon. However, the hypervisor, the guest OS and the application process (ElasticSearch in this case) may enter a nuance interaction:

The hypervisor reclaims memory from guest OS (ballooning);
Guest OS panicked with OOM exception;
OOM picks a process to kill, based on oom_score. Unfortunately, the true culprit (balloon) is not visible to guest OS, hence exempted from being considered to kill.
By killing the application process (usually the main application for VM), a fair chunk of guest memory is freed up;
the balloon then became more aggressive on the guest as seeing more memory becomes “available”;
the application on guest then does not have minimum memory on guest OS to start;

How to solve this problem? There are several things to consider. On the hypervisor, keep the entire or part of the guest memory reserved for start of application. Tune the OOM scoring so the non-critical application get killed. Implement application daemon, etc.

How memory usage adds up in Linux

Sun, 19 Apr 2020 21:12:56 -0400

There are too many metrics that describes some aspects about memory in Linux. This posting will make sense of those common metrics in Linux, CentOS as an example.

The most fundamental command is free and my favourite switch is -h for human readable reads. You can use -m, -k, -b for different units. The result looks like this:

 total used free shared buff/cache available
Mem: 32780168 16832160 3200408 101356 12747600 15399528
Swap: 2097148 2055148 42000

Swap is essentially disk space and many application such as Cassandra, ElasticSearch recommend disabling swap as best practice and they do not want disk speed to drag the performance of memory. Many suggest that swap is not needed in today’s era at all given the amount of memory for cheap. This is debatable.

With the row for Mem, the four columns should add up to the total, as suggested in the chart below.

total = free + used + shared + buff/cache

The four columns from free command output are supposed to always add up to the physical memory size. This command simplifies things quite a bit and each of these values are actually taken from certain lines in /proc/meminfo:

Metric from free command	Metric in /proc/meminfo
total	MemTotal
used	??
free	MemFree
shared	Shmem
buff/cache	Cached + Slab
available	MemAvailable

The buffer and cache (and even swap) can be freed by command. The value of used doesn’t seem to come from anwhere in /proc/meminfo, but it should be calculable from the memory used per process, which can be seen from top command.

In the result of top command, the column RSS (resident set size) is from the VmRSS value in /proc/<pid>/status, it is the actual physical memory consumed by the process. This value is originally from the second read in /proc/<pid>/statm, which represents the number of pages. For example:

[ghunch@centos ~]$ cat /proc/6495/status | grep VmRSS ; cat /proc/6495/statm
VmRSS:	20852916 kB
49829626 5213229 1212275 1 0 5773980 0
[ghunch@centos ~]$ getconf PAGE_SIZE
4096

Linux default page size is 4096 or 4K, so in the result from above, 5213229 x 4kB = 20852916 kB, which is the size of memory taken by process ID 6459. Therefore if we go through all processes and add up the VmRSS, we should get (close to) the used memory?

But wait a second, we have not account for slab info (memory used by kernel) yet, which is displayed in /proc/slabinfo. To calculate the total size taken by slab, we use <num_objs> and <objsize> columns from /proc/slabinfo.

Apart from that there is page table, the table that stores the mapping between virtual address and physical address, is stored in the physical memory as well and the size is specified in the PageTables entry in /proc/meminfo.

Now our equation becomes:

Used Memory = (RSS for all processes) + (all objects in slab) + (page table)

We can use the following script to calculate the used memory and compare it with free command output.

#/bin/bash
for PROC in `ls /proc/|grep "^[0-9]"`
do
 if [ -f /proc/$PROC/statm ]; then
 TEP=`cat /proc/$PROC/statm | awk '{print ($2)}'`
 RSS=`expr $RSS + $TEP`
 fi
done
RSS=`expr $RSS \* 4`
PageTable=`grep PageTables /proc/meminfo | awk '{print $2}'`
SlabInfo=`cat /proc/slabinfo |awk 'BEGIN{sum=0;}{sum=sum+$3*$4;}END{print sum/1024/1024}'`
 
echo $RSS"KB", $PageTable"KB", $SlabInfo"MB"
printf "rss+pagetable+slabinfo=%sMB\n" `echo $RSS/1024 + $PageTable/1024 + $SlabInfo|bc`
free -m

Running it require root access and the bc package installed. The result is most likely greater than the used memory value. Below is the result from my server:

89925884KB, 201788KB, 3303.92MB
rss+pagetable+slabinfo=91318.92MB
 total used free shared buff/cache available
Mem: 128772 87032 726 498 41013 40346
Swap: 2047 3 2044

So the result is over by (91318 – 87032) = 4286M. This is due to shared memory. The RSS value from above include memory from shared libraries as long as the pages from those libraries are in the memory. If multiple processes use the same library, the memory from shared library is counted multiple times. Check out the difference between RSS and PSS (proportional set size)

[Disclaimer] The chart and script are stolen from this authors post.

Introduction to Authentication Frameworks (PAM and SSPI)

Tue, 24 Mar 2020 20:19:00 -0400

This article gives a very brief high-level introduction to PAM (Pluggable Authentication Module) and SSPI (Security Support Provider Interface) as authentication frameworks in Linux and Windows respectively.

PAM

The Pluggable Authentication Module (PAM) architecture provides a powerful abstraction for user IAM using pluggable authentication model Unix platforms. It defines a generic API for authentication and hides the underlying mechanisms. Thanks to PAM, administrators can plug different authentication modules and protocols into Linux. This makes different authentication methods and protocols available to applications running on Linux. Here is some of authentication methods and protocols that PAM supports:

Unix file-based authentication (using /etc/passwd or /etc/shadow)
LDAP-based authentication
Kerbero-based authentication
NTLM-based authentication

PAM obviates the need for a separate authentication schemes. It exports methods of the various libraries under its auspices to calling applications. Here is a diagram for PAM on Redhat.

PAM can also enbable single-sign-on(SSO) on the UNIX platform. If the password used for different services are identical, PAM can be used to share the password transparently between the application’s possibly different authentication mechanisms. PAM is configured in /etc/pam.d/ directory.

The Windows equivalent of PAM is the Security Support Provider Interface (SSPI) and its Security Support Provider (SSP) Modules. For example /etc/pam./login manages login module, /tec/pam.d/imap manages imap module.

SSPI

Windows Authentication Architecture involves Local Security Authority (LSA, to authenticate users to local computer only and is managed in local security policy) and Security Support Provider Interface (SSPI).

SSPI is the API that obtains integrated security service for authentication, message integrity, message privacy, and security quality-of-service for any distributed application protocol. SSPI is the implementation of the Generic Security Service API (GSS-API) in Windows operating system. Applications and infrastructure services authenticate users by using the SSPI to abstract calls for authentication. This way, developers do not need to understand the complexities of specific authentication protocols or build authentication protocols into their applications. Here is the architecture diagram:

The SSPI in Windows provides a mechansim that carries authentication token over the existing communication channel between the client computer and the server. When two computers or devices need to be authenticated so that they can communicate securely, the requests for authentication are routed to the SSPI, which completes the authentication process, regardless of the network protocol currently in use.

Here is some exampls of SSPs that are supported by SSPI:

Kerberos SSP (default for Active Directory)
NTLM SSP
Digest SSP
Negotiate SSP (based on SPNEGO, RFC4178)
Credential SSP
Negotiate Extension SSP

Some SSPs such as Kerberos SSP and NTLM SSP use a single protocol. Some (e.g. Negotiate SSP and Credential SSP) combine several protocols to allow application to select what security mechanism they wish to use and negotiate with authentication services.

High Availability and Load Balancer

Wed, 22 Jan 2020 20:49:00 -0400

Overview

Fault tolerance and high availability are two architectural characteristics that people often confuse with each other. High availability focuses on minimizing downtime. It guarantees uptime, but not performance in the event of component failures. Fault tolerance, on the other hand, focuses on stable capacity even in the event of component failures. Fault tolerance has higher bar, and therefore is more expensive.

Suppose an application requires four servers to meet performance goal. Placing two servers in each of the two AZs will meet HA criteria but not FT requirement. In the event of an AZ failure, application can operate at degraded performance yet still be highly available. However, FT requires stable capacity and to meet FT requirement, we’d have to place four servers in each AZ.

High availability can be achieved either by clustering, or load balancing. A cluster involves several nodes, all able to perform the same function, but may take different roles at different times (e.g. primary, standby) in order for the cluster to perform its function as a single system. In Linux, clustering is implemented by pacemaker or corosync. With a high load system, it is common to set up load balancing system to achieve high availability (and fault tolerance).

Load balancing

The idea of load balancing is simple: load goes high and we want to scale horizontally instead of simply upgrading server hardware. At a high level, there has been three approaches to load balancing:

DNS rotating: (aka. DNS round robin) DNS record resolves to multiple IPs, very simple and cheap to implement. Since DNS is cached, the load distribution will come imbalanced and it’s hard to re-balance, making this a very limited approach;
Hardware Load Balancer: using dedicated hardware device to configure load balancing. This option is expensive and only enterprises can afford it (here‘s some pricing information). A classic load balancer operates at layer 3 and 4, which is also known as POLB (plain old load balancer). It is the core functionality of hardware load balancer. The hardware load balancer on the market today usually come with a variety of add-on features, such as advanced load balancing (L4, L7 path-based, script driven), compression, caching, SSL offloading, and even DDoS mitigation, etc. The whole suite of features makes it an Application Delivery Controller (ADC). Therefore many refer to hardware load balancer as hardware-based ADC to highlight the features in addition to POLB. Hardware-based ADCs ship with manufactures hardware, with specialized processors, advanced network hardware, and often ASIC (application specific integrated circuit). At a higher expense, they have better reliability and capacity. Some major market players are:
- F5 – Big IP, F5 also has a good article about history of load balancer.
- Cisco – Citrix Ahttps://www.citrix.com/products/citrix-adc/DC (formerly NetScaler ADC)
- A10 Networks – Thunder (general) and Lightning (cloud)
Software Load Balancer: using software to achieve load balancing. These solutions are affordable, and usually open-source. They can be loaded on commodity hardware (including NIC). Some (e.g. Nginx) refers to themselves as software-based ADC. Major players are:
- HA Proxy
- Nginx
- Linux Virtual Server (LVS, L4 only)

The hardware ADCs are usually supported commercially and there are plenty of resources from their white papers. There is an ongoing debate about whether one is better than the other. However, there is no doubt that a software-based load balancer is more approachable as open-source tools. The line between software and hardware load balancers becomes blurred today as hardware vendors try to adapt their software appliance to commodity hardware. Check out this article. The rest of this post, will focus on software-based load balancer.

Software-based load balancer

We explained that ADC (application delivery controller) is an expanded set of features from load balancer, and will only cover the load balancer part of the feature set in this article.

HAProxy supports both layer 4 and layer 7 load balancing. It supports load balancing based on cookie and session, as well as health check. Since it is layer 4 load balancing, it supports any TCP protocol such as read traffic for MySQL.

Nginx is a high-performance, event-driven, cross-platform layer 7 load balancing application. It works as a reverse proxy where it receives request for the Internet and forwards it to (upstream) internal servers. It consumes less memory than many of its alternatives for layer 7 load balancing. There are many strategies for load balancing such as round robin, by weight, by hash of requesting IP, by upstream response time, or by URL hash. It supports 20-30 k concurrent connections, and support compression and health check. It is known to be very stable and common for small and medium volume. Nginx has a commercial counterpart Nginx Plus with advanced features.

Nginx and HA proxy are commonly used in front end load balancing. For backend traffic such as database (e.g. separating read write traffic), LVS can be used.

Linux Virtual Server

LVS (Linux Virtual Server) is part of standard Linux kernel. It performs layer 4 load balancing based on TCP or UDP and therefore consumes less memory and CPU. Compared to layer 7 load balancing, the performance is generally higher, and the configuration is less complex (with simpler routing rules). LVS is usually configured in a common cluster architecture involving these components:

Load balancer: the front-end machine of the whole cluster systems, and balances requests from clients among a set of servers, so that the clients consider that all the services is from a single IP address.
Server cluster: set of servers running actual business workload
Shared storage: a shared storage space for the servers, such as NFS

Load balancer is the single entry-point of server cluster systems, it can run IPVS that implements IP load balancing techniques inside the Linux kernel, or KTCPVS that implements application-level load balancing inside the Linux kernel. When IPVS is used, all the servers are required to provide the same services and contents, the load balancer forward a new client request to a server according to the specified scheduling algorithms and the load of each server. No matter which server is selected, the client should get the same result. When KTCPVS is used, servers can have different contents, the load balancer can forward a request to a different server according to the content of request. Since KTCPVS is implemented inside the Linux kernel, the overhead of relaying data is minimal, so that it can still have high throughput.

IPVS is also called layer-4 switching, it directs TCP/UDP requests to the real servers behind load balancer. It works in three modes:

Network Address Translation (NAT)
Direct Routing (DR)
Tunnel mode (TUN)

These are three packet-forwarding methods in IPVS. The IPVS is implemented as a module over the netfilter framework, similar to iptables, which is also built on top of netfilter, based on chain and rules.

Summary

We had an overview of high availability, and then expanded on load balancing, an important mechanism to implement high availability. We touched on both hardware-based and software-based load balancing technologies, and dived a little more into Linux Virtual Server. It is worth-noting that LVS is also the foundation of kube-proxy, the load balancing mechanism used in Kubernetes.

Nginx as a reverse proxy for Nifi web UI and Kibana

Thu, 16 Jan 2020 22:22:51 -0400

Nginx can act as a application neutral proxy. One example is to front Nifi. The nifi default configuration provides an HTTP access point, specified in the following entries in nifi.properties:

nifi.web.http.host=192.168.133.5
nifi.web.http.port=8080

Nifi can provide secure port by commenting out the lines above and provide the followings:

nifi.web.https.host=192.168.133.5
nifi.web.https.port=8083

However, it does require configuring JKS keystore for Java, as well as authentication. Customers with existing AD servers are likely to require authentication via LDAP. While Nifi does support LDAP integration according to its administration guide. The configuration is quite involving. You need to configure the identity provider, as well as authorizes. I have personally spent a couple days on this without much progress. The information in the logging isn’t to the point. Restarting nifi also is a long process, making it painful to troubleshoot.

I then moved to Nginx (open-source) as an alternative and it is quite enlightening. I already knew that the SSL termination in nginx is super easy to configure. This time I learned that the opensource community even has a support for LDAP integration. Here is a diagram of how it works:

This approach is outlined in a blog post on Nginx website. The ldap-auth daemon is implemented in Python can can be wrapped up as a systemd service. Once a client sends a request in https, security layer is terminated in nginx, and an authentication request in http is sent to the ldap-auth daemon, which proxies converts the request into LDAP searches and proxies it over to customer’s Active Directory server, for authentication. Once authenticated, the http request can make to one of the backend container or server which hosts Nifi. Below is an example of how this can be configure on RedHat.

Install python3 and python-ldap

RedHat may have both python2 and python3 pre-installed, python2 being the default. We do not want to change the default because other applications such as yum still depends on python2 as of early 2020.

yum -y install python3
yum -y install gcc python3-devel openldap-devel
pip3 install python-ldap

Once python3 is installed, pip3 will be available and we use that to install python-ldap. This is a module in Python3 that will be used by the script that act as ldap daemon.

Configure ldap-auth daemon as systemd service

In the github project for ldap-auth, download nginx-ldap-auth-daemon.py to local location such as /usr/bin, then we create nginx-ldap-auth.service in /etc/systemd/system/ with the following content.

[Unit]
Description=LDAP authentication helper for Nginx
After=network.target network-online.target
 
[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/var/run
ExecStart=/usr/bin/python3 /usr/bin/nginx-ldap-auth-daemon.py
KillMode=process
KillSignal=SIGINT
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

Then, run the following command to load, start and check nginx-ldap-auth service.

systemctl reload-daemon
systemctl start nginx-ldap-auth
systemctl status nginx-ldap-auth

This service will be up and listening to port 8888 for http traffic.

Configure Nginx

Then configure nginx with the following entries in its default.conf file, typically located in /etc/nginx/conf.d.

upstream nifibackend {
 # default: round robin
 server container1.nifi.digihunch.com:8080;
 server container2.nifi.digihunch.com:8080;
}
proxy_cache_path cache/ keys_zone=auth_cache:10m;
# nifi proxy
server {
 listen 8083ssl;
 include /etc/nginx/ssl/default.conf;
 
 location / {
 auth_request /auth-proxy;
 
 proxy_pass http://nifibackend;
 proxy_set_header Host $host:$server_port;
 proxy_set_header X-ProxyScheme https;
 proxy_set_header X-ProxyHost $1;
 proxy_set_header X-ProxyPort 8083;
 proxy_set_header X-ProxyContextPath /;
 }
 
 location /auth-proxy {
 internal;
 proxy_pass http://127.0.0.1:8888;
 
 proxy_pass_request_body off;
 proxy_set_header Content-Length "";
 proxy_cache auth_cache;
 proxy_cache_valid 200 10m;
 
 proxy_cache_key "$http_authorization$cookie_nginxauth";
 proxy_set_header X-Ldap-URL "ldaps://ldap.digihunch.com:636";
 proxy_set_header X-Ldap-BaseDN "OU=Corporate User Accounts,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindDN "CN=Digi Hunch Service Account,OU=Digi,OU=ServiceAccounts,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindPass "myownpasswordtricks";
 proxy_set_header X-CookieName "nginxauth";
 proxy_set_header Cookie nginxauth=$cookie_nginxauth;
 proxy_set_header X-Ldap_Starttls "true";
 proxy_set_header X-Ldap-Template "(&(sAMAccountName=%(username)s)(objectClass=organizationalPerson)(memberOf=CN=GH_SYSADMIN,OU=GHCO,OU=Groups,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org))";
 }
}

We need the full distinguished name of bind user to get this to work. Once configured properly, and user attempts to connect through a browser, Nginx will pop up a prompt for username and password. The username will be plugged into the X-Ldap-Template for further queries. The same HTTP header also allows you to filter by membership that the user is associated with.

Personal Vim cheatsheet

Mon, 07 Oct 2019 17:52:00 -0400

This is my personal cheatsheet as intermediate Vim user so I skipped the ones that I consider basic. All the commands listed are used in command mode for fast editing.

Command execution

In command mode, use colon to start ex command. Here are some examples of ex commands:

e: edit file
g: global command
q: quit
w: write
s: substitute

The commands (g)lobal and (s)ubstitute are heavily used in string manipulation. The rest are summarized here:

:e!	reload current file discarding all unsaved changes
:e newfile.txt	open file newfile.txt for editing
:e .	load current directory
:w!	force write (if permission allows)
shift + zz	equivalent to :wq!
shift + zq	equivalent to :q!

Operator

.	repeat last operation
fx	find next character x on the same line

Text Editing

y for yank(copy), i for inside, a for around, d for delete, w for word, p for paragraph or paste. Examples:

yyp	copy current line and insert after
diw	delete the entire word where the cursor sits in (dw deletes from cursor to end of word; db deletes from cursor to beginning of word)
shift + V	select entire line in visual mode (v selects character in visual mode)
2>	visual mode: indent twice on all selected lines edit mode: indent once for 2 lines
3<	visual mode: outdent three times on all selected lines edit mode: outdent once for 3 lines
di”	delete everything between the double quotes surrounding the cursor (exclusive); use c instead of d to finish the same effect with insert mode
di>	delete everything between < and > surrounding the cursor (exclusive); use (c)hange instead of (d)elete to finish the same effect with insert mode
dit	delete everything between tags. e.g. <xml>contenttodelete</xml>
dip	delete the entire paragraph
da’	delete everything between the single quote surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert mode
da}	delete everything between { and } surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert mode
dt.	delete all characters until the next .
i	insert at cursor location
shift + I	move cursor to first non-blank character of line and start in insert mode
a	insert at the location next to cursor
shift + A	move cursor to last non-blank character of line and start in insert mode
0	move to beginning of line. ^ moves to first non-blank character in the line. $ moves to the end of line

Note: wherever d is used in this table, c can be used instead for the same effect but switch to editing mode at the end.

String Manipulation

The general patterns are:

[range]g/pattern/cmd
[range]s/match/replacement/option

If range is not specified, it applies to current line only! To specify the whole file, use % range. You may also specify line range such as “10,20”. Here are some examples:

:%s/bacon/lettuce	For every line of the file, replace the first occurrence of bacon in each line to lettuce
:%s/bacon/lettuce/g	For every line of the file, replace all occurrences of bacon to lettuce
:s/bacon/lettuce	For current line, replace the first occurrence of bacon to lettuce
:s/bacon/lettuce/gi	For current line, replace all occurrences of bacon to lettuce, case insensitive
:g/bacon/d	delete all lines that contain pattern ‘bacon’
:g!/lettuce/d	delete all lines that do not contain pattern ‘lettice’; or use :v/lettuce/d instead
:g/^\s*$/d	delete all blank lines. \s* represents zero or more white spaces

Bookmarking

ma	mark cursor line as bookmark a
`a	jump to cursor position at line a
‘a	jump to beginning of line a
`.	jump to last line where change occurred
“	jump back

Insert Mode

Ctrl + N

Auto complete

Edit and run

While tmux and screen can help split screen in Bash, we sometimes need to split a bash screen to run a quick command when we’re already in vim. This can be done with some simple commands.

:term	Open up a terminal above vim. You can also spell :ter or :terminal
Ctrl+W; Ctrl+W	Press Ctrl+W twice can help you toggle between the terminal and vim buffer
Ctrl+D	Close the terminal

This can be very helpful when you are debugging code and need it run repeatedly. You don’t need to exit vim just to run a command and come back. Note that while you’re in terminal, you can’t use Ctrl+W as a shortcut key to backspace a word. Use Alt + Delete instead.

AWS Certified DevOps Engineer Exam Tips

Sat, 17 Aug 2019 23:07:59 -0400

The last 30 days have been exhausting for me. I studied hard on the new AWS Certified DevOps Engineer exam and thank goodness I passed (750 out of 1000 is required). This was the hardest professional certification I ever worked on. The exam was re-launched recently in March 2019 so there is still a shortage of information around the community. I was hoping to share my experience to help demystify this new exam.

I had a somewhat solid background to begin with, having taken the AWS Certified Solution Architect Professional exam (before the 2019 update), and worked quite a bit on CloudFormation, automation and Git. However, I still did not anticipate the exam to be this difficult until I was halfway through and already had the exam and materials paid for.

Without clear guideline on study material other than the white papers, I first skimmed through the ACloudGuru course, which helped me form a high level sense of exam coverage. Nonetheless it does not cover any topic in-depth and therefore by no means makes an essential part of my study. I checked out LinuxAcademy course and they are much more in-depth for the major topics indeed. I like the course material in Lucid chart. However, the LinuxAcademy course along does not cover everything you need to know.

What I found extremely helpful is the free training videos from the official training website. I strongly recommend the 7-hour course Exam Readiness: AWS Certified DevOps Engineer – Professional. The instructor did a great job outlining the services and knowledge areas in the assessment . The course also comes with quality sample questions with answers and explanations on what the thinkings are behind the correct answers or why some choices are obviously wrong. I went through these questions twice and feel much better at not only understanding the question, but also understanding the intent of the question.

Apart from the Exam Readiness course, other free introductory courses from the official training website are helpful as well especially for those services that you only need to know the basics. Most of those courses are 5 ~ 10 minutes long, with brief but sufficient introduction and a demo session. The other extremely helpful resource is the official practice questions. The practice exam is harder than the actual exam but they closely resemble the actual question style in the exam. Unfortunately, no answer is provided but they made me spend time finding answers across the documentations and blogs. It is worth-noting that the AWS blogs provides plenty of use cases that are covered in the questions.

When it comes to the real exam, it covers a lot more topics than its predecessor. Many questions are long and confusing. And I wish I could run a diff command to highlight the differences between choices. During exam preparation you really need to train yourself on reading efficiently. I found myself sometimes eyeball through all four choices at the same time, which get my mind scattered.

During my study, I divide all services into three categories based on the level of familiarity, and here is my list:

Category 1. Know these services very well, in and out:

AWS ElasticBeanstalk, OpsWorks Stacks, OpsWorks Chef Automate, CloudFormation, CloudWatch, CodeBuild, CodeCommit, CodeDeploy, CodePipeline, CodeStar, Lambda, API Gateway, Config, Trusted Advisor, CloudTrail, Systems Manager, Autoscaling Group in EC2, DynamoDB

Category 2. Know these services well, but not necessarily down to every single detail:

Amazon Kinesis Firehose, Kinesis Analytics, Kinesis Streams, Step Functions, Elastic Load Balancer, Secrets Manager, Serverless Application Model (SAM), Route53, RDS, Certificate Manager, ElasticSearch, ECS, ECR

Category 3. Know about these services at a high level, but do not skip any:

AWS Organization, X-Ray, GuardDuty, Macie, Inspector, Service Catalog, KMS, Batch, Athena, Single-Sign-On, Data LifeCycle Manager, CloudSearch, Health Dashboard, Glue, QuickSight, LightSail

Although I did not mention much about the white papers, I want to highlight their importance again. I would not attempt the exam without reading and understanding the required white papers.

If you aspire to take the AWS certified DevOps Engineer exam I hope this helps you a little bit. Good luck.

Set up automation with Ansible

Wed, 22 May 2019 22:11:00 -0400

Ansible is very flexible automation tools with many benefits. The free version is command-line based and here is an example to set it up.

Environment setup

Ansible 2.8 is required or some command may not work.
Ansible files (including playbooks, tasks and inventory files) are all located in /home/glowing/ansible
Default inventory file needs to be referenced in Ansible configuration /etc/ansible/ansible.cfg. This ensures ansible or ansible-playbook command can pick up hosts or host patterns without requiring inventory file through -i on every execution. Here is what the inventory config looks like in ansible.cfg:

[defaults]
 
# some basic default values...
inventory = /etc/ansible/hosts,/home/glowing/ansible/inventories/glowing_inventory.yml
 
host_key_checking = False

As best practice, servers involved should be able to ssh to each other on RSA key authentication. This can be achieved by adding a separate authorized keys file and reference it from /etc/ssh/sshd_config, at the line starting with AuthorizedKeysFile, and separated with the file name of existing authorized keys with a space. In this way, you may keep public keys of human user in one authorized key file and the public keys of machines in the other.
Build inventory file in ~/ansible/inventories/glowing_inventory.yml. The inventory file can declare some variables to use across hosts. If password is involved (e.g. synchronize module still requires password), it can be stored here with base64 encoded. Here is an example of glowing_inventory.yml

---
all:
 vars:
 ansible_user: glowing
 gh_sudo_pass: qGS0bWVuu3Jr
 gh_dir: /opt/glowing/etc
 tmp_dir: /tmp
 
 children:
 dc1_front_end:
 hosts:
 e9a-ghfe01:
 e9a-ghfe03:
 e9a-ghfe05:
 e9a-ghfe07:
 e9a-ghfe09:
 
 dc2_front_end:
 hosts:
 e9a-ghfe02:
 e9a-ghfe04:
 e9a-ghfe06:
 e9a-ghfe08:
 e9a-ghfe10:
 
 dc1_back_end:
 hosts:
 e9a-ghbe01:
 e9a-ghbe03:
 e9a-ghbe05:
 
 dc2_back_end:
 hosts:
 e9a-ghbe02:
 e9a-ghbe04:
 e9a-ghbe06:
 
 dc1_database:
 hosts:
 e9a-ghdb01:
 e9a-ghdb03:
 e9a-ghdb05:
 
 dc2_database:
 hosts:
 e9a-ghdb02:
 e9a-ghdb04:
 e9a-ghdb06:

That is the basic steps to set up Ansible. Now we can run adhoc commands. The command below allows me to copy a file from executing server to all destination servers that match a pattern:

# ansible 'dc*_back_end:!'`hostname -s` -e "file_name={{gh_dir}}/test.zip" -m copy -a "src={{file_name}} dest={{file_name}}"

In this command:

variable {{gh_dir}} is declared in the inventory file. It must be referenced by placing variable name between double curly bracket;
An extra variable {{file_name}} is declared at run time because this is the dynamic part of the command;
This adhoc ansible command uses copy module. With copy module, the src anddest files are in the same absolute path here so we use this variable to save some typing;
reference to the host support wildcard such as dc*_back_end;
hostname -s returns the host name of the server where the adhoc command is run
:! excluds the running host from being matched as destination server. this is in case that running machine is already in the dc*_back_end group, where copy source and destination are identical

Here is another example for deleting a file from destination servers:

# ansible dc2_database -e "fn=/tmp/file_to_delete" -m file -a "path={{fn}} state=absent"

If we can run adhoc command, then we can start writing some playbook and roles. Below is a simple playbook test-conn.yml to ping each server:

---
 - name: measure mint retrieval time from {{ ansible_limit }}
 hosts: '{{ ansible_limit }}'
 serial: 1
 order: sorted
 gather_facts: no
 ignore_errors: yes
 tasks:
 - name: measure time
 command: curl -s -XGET http://{{ inventory_hostname }}:8080/index.html -o /dev/null
 delegate_to: localhost
 register: curlout
 no_log: true
 - name: display result
 debug:
 msg: "Time to load webpage from {{ inventory_hostname }} is {{ curlout.delta }}."

Then you can run that playbook with the following command:

# ansible-playbook -l dc2_front_end test-conn.yml

Ansible automation is essentially editing yaml files for playbooks. Writing playbook involves a lot of module interaction and one needs to follow best practices. Understanding Ansible roles can help reuse some code.

In addition here are some common playbook keywords you should be familiar with:

serial
order
gather_facts
ignore_errors
when
run_once
local_action
register

Here are common modules to know

set_fact
state
touch
fail
synchoronize
lineinfile

Package Repository Management for Linux

Sat, 09 Feb 2019 22:14:56 -0400

RPM is the package manager tool in Linux. YUM is a repository management tool to fetch appropriate package for the particular version of Linux. YUM performs automatic dependency resolution when updating, installing or removing packages, and thus is able to automatically determine, fetch and install all available dependent packages. This posting is about common commands used for package management. Then we walk through the step to set up a local yum repository. Lastly, we summarize the best practice for patch management.

Tools

In addition to the basic yum install, update and remove commands, here are some more tools for repository management:

## check for installed packages with updates available
yum check-update

## search for packages
yum search package.name

## check for dependency
yum deplist package.name

## list installed or available packages:
yum list installed package.name
yum list available package.name

## list all available packages of a repo:
yum --disablerepo=* --enablerepo=repo.name list available

## display package info
yum info package.name

## list all repositories
yum repolist
yum repoinfo

## list all transactions
yum history list all

## display history of a package or transaction id
yum history package-list package.name
yum history info package.name
yum history info transaction.id

## undo or redo a transaction
yum history undo transaction.id
yum history redo transaction.id

RHEL provides a more complete cheatsheet for yum commands.

In addition to single package, YUM also manages package groups, and works with plugins. For example, you may enable security related packages only by using the yum-plugin-security. For security, YUM repositories can enable GPG check (based on public key cryptography).

Local Repo configuration

Now we configure a YUM repository server and reference that from a client to update packages. To begin with, we collect all RPM packages in /home/dhunch/upgrade/downloads/, then we install createrepo package and use it to create metadata:

yum install createrepo yum-utils
createrepo /home/dhunch/upgrade/downloads/

Examine the directory and you will find a new repodata directory created. Now we can configure nginx with the following configuration:

server {
 listen 8088;
 server_name rpmsource.digihunch.com;
 root /home/dhunch/upgrade/downloads/;
 location / {
 autoindex on; #enable listing of directory index
 }
}

Restart nginx and browse to server name at port 8088, you should see the directory in html. If you’re getting 403 error, most likely nginx has issues accessing the directory. Nginx process should be able to traverse each level of directory to serve the files. The parent directories should have executable permission for others. For example:

chmod o+x /home/dhunch/

Here is a reference to nginx permission requirement. Now we can continue to configure the client.

On the client, we need to add a file in /etc/repos.d/yum

[digirepo]
name=Local YUM Repository 
baseurl=http://rpmsource.digihunch.com:8088/
enabled=1
gpgcheck=0

Last we can check the available packages from the client:

yum --disablerepo=* --enablerepo=digirepo list available

Note that in order to keep the local repo up-to-date, there is additional maintenance work on the server. You may need to sync from official source, such as:

reposync -g -l -d -m --repoid=base-source --newest-only --download-metadata --download_path=/home/dhunch/upgrade/downloads/

You may also need to set up a daily job with yum-cron service so this is automated.

Sometimes the server is locked down and you need HTTP proxy to allow YUM to access repo. You may configure proxy in /etc/yum.conf in the proxy, proxy_username, and proxy_password entries. Also, you may export environment variable http_proxy. Here are more information.

Sometimes yum cache may introduce issues. To clean cache before installing, run:

yum clean all &amp;&amp; yum -y install python3

Python packages

Python3 is not installed by default on CentOS 7. So it needs to be installed with YUM. Do not replace the existing python2 with python3 by changing where symbolic link /bin/python points to, because YUM is dependent on python2.

For python3 use pip3 as package management. If the server does not have a public route, then you need to install pip3 package offline. For example, on a server with Internet, run

pip3 download -d ~/Downloads javaobj-py3

Then SCP the file to the offline server (e.g. to /home/dhunch/javaobj/), from which you can run:

pip3 install --no-index --find-links=/home/dhunch/javaobj/ javaobj-py3 --user

This will install the pip3 package offline. Note it is recommended to not run pip3 installer as root user. The switch –user allows you to run as non-root user.

Best Practices

To configure a patch management environment, we assume that the servers do not have access to the Internet, not even through proxy. This should be part of the security guideline anyways.

This requires a local repository server to be setup. Administrators may use createrepo tool as outline above to create such repo, use reposync to synchronize local source from official source, or yum-cron to automate this for daily task.

On the clients, administrators should manage the repo file in /etc/yum.repo.d, to ensure they are pointing to the correct source, using gpg check, etc. A local script to enable and disable different repos are also helpful. Otherwise, the enabelrepo and disablerepo switch can be used per command.

It is important to catalogue the state of operating system, before and after each patching activity, for verification and auditing purposes. This usually requires scripting work using yum history or package-cleanup tools.

The actual patch work can be done either manually or automatically (e.g. with Ansible’s yum module). RPM is also a good tool for troubleshooting. If the scope of patching is only for security, use the yum-plugin-security to limit the packages to only security related ones.

Roll-back should be prepared in case of inadvertent outcome. Roll-back scripting relies heavily on yum history commands. If the OS needs to boot with a previous kernel, use the grubby tool.

Happy Patching!

Redhat Firewall configuration: from iptables to firewalld

Fri, 12 Oct 2018 17:56:00 -0400

Tools to manage firewall

Packet filter rules in Linux Kernel is managed by an user-space application named iptables in CentOS and RedHat. Since CentOS 7, firewalld is introduced as an alternative to iptables. Firewalld can be installed and executed as a systemd service, and it is supposed to replace iptables. This article describes how to configure both.

There are several advantages in firewalld. One is is the support of zones. Here are some useful information. Also, iptables involves three different services for IPv4(iptables), IPv6(ip6tables), and software bridging (ebtables), whereas firewalld only involves a single service to manage all three. Firewalld allows user to add or remove rules/ports from running firewall, without restarting firewall. Unless you have specific reason to use iptables, always use firewalld service to manage firewall. Here is an instruction to firewalld service. In this posting however, we will be focusing on iptables to understand firewall managment. We also go through an example of opening a TCP port.

How does iptables work

When working with iptables, it is important to understand that its related concepts (tables->chains->rules->criteria and targets) and how the order of rules plays a factor. There are five independent tables, each contains a number of chains, either built-in or user-defined. Administrators mostly deals with built-in chains in filter and nat tables. The five tables are:

filter: If -t isn’t specified, this is the default table. It contains built-in chains:
- INPUT: for packet destined to local sockets
- FORWARD: for packets being routed through the box
- OUTPUT: for locally-generated packets
nat: this table is consulted when a packet that creates a new connection is encountered. It has three built-in chains:
- PREROUTING: for altering packets as soon as they come in
- OUTPUT: for altering locally generated packets before routing
- POSTROUTING: for altering packets as they are about to go out
mangle: this table is used for specialized packet alternation, with five built-in chains (since kernel 2.4.18): PREROUTING and OUTPUT, INPUT, FORWARD, and POSTROUTING
raw: this table is mainly for configuring exceptions from connection tracking with two built-in chains: PREROUTING and OUTPUT
security: for Mandatory Access Control (MAC) networking rules, with three built-in chains: INPUT, OUTPUT, and FORWARD.

Under the table (e.g. filter, nat), each chain (e.g. INPUT, OUTPUT, etc) consists of list of firewall rules. Each rule is made up of two parts defined for the packets:

Criteria: if the packet does not match the criteria, the next rule in the chain is examined; if it does match, then the next rule is specified by the value of the target.
Target: what to do if criteria is met. The target can be:
- user-defined chain,
- one of the target described in iptables-extensions, or
- in most cases, one of the special values ACCEPT, DROP or RETURN
  - ACCEPT – let the packet through
  - DROP – drop the packet on the floor
  - RETURN – stop traversing this chain, and resume at next rule in the previous (calling) chain

The rules, defined in each chain under their tables, can be found in file /etc/sysconfig/iptables. You can find tables (prefix with asterisk *), chains (prefix with colon :), rules under their chains and a statement COMMIT after each table. The iptables process flow illustrates how a packet interact with all these rules under different chains and tables defined in this file:

iptables Process Flow

Although this big picture looks formidable, an administrator commonly only deals with the green and purple blocks (filter and nat), with the big picture in mind. Here is an example of /etc/sysconfig/iptables file from a newly installed system:

# Generated by iptables-save v1.4.21 on Fri Sep 11 23:15:32 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [132:17200]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Fri Sep 11 23:15:32 2017

The rule simply allows SSH traffic. This file will be loaded up on every reboot (specifically, restart of iptables service). So if you have made some changes to rules and you want the change picked up on reboot. The rules should be saved to this file:

$ sudo iptables-save > /etc/sysconfig/iptables

Other than saving rule for reboot, if you simply want to edit the rules (e.g. order of rules is incorrect), you can save the rules to file, modify the file and restore the rule from file:

$ sudo iptables-save > ~/iptables.txt
$ sudo vi ~/iptables.txt
$ sudo iptables-restore &lt; ~/iptables.txt

Here is some further reading about iptables architecture.

Anatomy of a rule

The man page for iptables species the following synopsis:

iptables [-t table] {-A|-C|-D} chain rule-specification
rule-specification = [matches...] [target]
match = -m matchname [per-match-options]
target = -j targetname [per-target-options]

So when you append (-A), delete (-D), insert (-I) or replace (-R) a rule, you need to specify rule specification. The man page further explains that the following parameters make up a rule specification:

protocol (-p): the protocol of the rule of the packet to check. value can be tcp, udp, icmp, all or any name defined in /etc/protocols.
match (-m): specifies the name of a match to use and is followed by match options. The match refers to an extension module that tests for a specific property. Those extension modules are documented in the man page of iptables-extensions. You may specify -m multiple times for different match names, which together make up the condition under which a target is invoked. Matches are evaluated first to last as specified. We often use extensions tcp and state. According to iptables-extensions man page, we can specify –dport followed by port number for the tcp extension, and –state followed by value such as NEW or ESTABLISHED for the state extension.
jump (-j): specifies the target of the rule, such as ACCEPT, REJECT or DROP.
source and destination (-s and -d): source and destination IP address or masks. Hostname will work but not recommended since resolution is needed.
inbound and outbound interface (-i and -o): name of interface via which the packet was received and is going to be sent.
goto (-g): processing should continue in a user specified chain
Other parameters: -4/–ipv4, -6/–ipv6, -c/–set-counters, -f/–fragment

When we run iptables command to view rules, we need to specify the table (e.g. filter, nat, etc) followed by -S or –list-rules:

$ iptables -t nat -S

If you do not specify -t switch, the default (-t filter) is applied. Be aware that in this case, you’re only seeing rules under filter table, and not all rules under tall tables!

In the result, for example one line from command “iptables -S” may say:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 9200 -j ACCEPT

The interpretation: appending a rule to INPUT chain of filter table (implicitly specified). The protocol is tcp. The first match extension is state, and the state value shall be NEW. The second match extension is tcp, and the dport value shall be 9200. If the packet is a match, then the target (action) is ACCEPT.

Managing rules

As mentioned earlier, rules can be dumped to any file or /etc/sysconfig/iptables, in which the rules are assessed in order. Below is a real life iptables file with a nat table as well.

# Generated by iptables-save v1.4.21 on Wed Jan 15 13:58:39 2017
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [4:208]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7000:7001 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7199 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9042 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9160 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 161 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 162 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

*nat
:PREROUTING ACCEPT [1:328]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A PREROUTING -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
COMMIT
# Completed on Wed Jan 15 13:58:39 2017

In this example, the nat table defines traffic forwarding: traffic arriving at TCP port 2392 and 2393 are forwarded to port 2398; outgoing traffic to port 2392 and 2393 are also redirected to port 2398. These rules do not overlap each other so the rules probably don’t matter.

On the other hand, the tcp filter table lists the rules to open certain TCP and UDP ports. Its block starts with a couple accepting rules and ends with a couple reject rules (regardless of protocols or ports). This is a good way to close a chain of rules with security. However, if you need to add additional rules to open more TCP ports, the new rule should not be appended after the reject rules at the bottom since the order matter here!

Correct way to open a TCP port

It’s a common task for developers to open a TCP port simply for the purpose of bring up a web service and make it accessible to client. If we simply add a new rule to existing list, for example:

# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9870 -j ACCEPT
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

# systemctl reload iptables

You will notice that the rule is appended to the end of INPUT block, below the INPUT REJECT rule, which will never take effect.

To address this, you can use iptables-save and iptables-restore to export, edit to correct order and reload the rule, as illustrated above, instead of using iptables command to modify the rule directly.

# iptables-save > /tmp/rule.list
# vi /tmp/rule.list
# iptables-restore < /tmp/rule.list
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

Alternatively, you could use some advanced iptables command switches to add the new rule to certain line number with –line-number switch. Here is more information.

Linux Admin Basics 3 of 3 – text processing, regex, sed & awk

Wed, 06 Jun 2018 19:56:00 -0400

Most of the text processing can be processed by awk and sed. Sed is non-interactive stream editor that allows you to specify all editing instructions in one place and execute them on a single pass through the file. Awk is a pattern-matching programming language.

Using sed and awk requires some understanding of regular expressions. Here’s the basics of regular expression signs:

sign	operation
^	matches beginning of line
$	matches end of line
.	matches any single character (wildcard)
*	repeat previous token zero, one or more times.
+	repeat previous token one or more times
?	repeat previous token zero or one time
[…]	matches any one of the class of characters enclosed between the classes. ^ as first character reverses the match – is used to ndicate a range of characters
()	groups regex
\|	either preceding or following regex can be matched
{n,m}	matches a range of occurrences of the single character that immediately precedes it. {n} will match exactly n occurrences {n,} will match at least n occurrences {n,m} will match any number of occurrences between n and m

Common expressions

exp	interpretation
[^0-9]	excluding number
[15]00*	matches “10”, “50”, “100”, “500”, “1000”, “5000”. Here the first 0 is literal, the second is modified by *, see the table above
.*	any number (including 0) of any character
<.*>	any html tags
book	matches book with preceding and following spaces
books*	matches books, or book, but not “book.” “book?” etc
book.*	matches book, followed by any number of characters, or none followed by a space

Note that regular expression comes in several different flavours, which can be confusing and frustrating. This is a good summary. There are DFA (Deterministic Finite Automata) based engines and NFA (Non-Deterministic Finite Automata) based engines:

NFA based engines can “go back” in the regex, used in Perl, Python, vim, sed and GNU grep.
DFA based engines cannot “go back” in the regex, used in awk and BSD grep.

Standard	IEEE POSIX BRE	IEEE POSIX ERE	PCRE
Detail	Basic Regular Expressions	Extended Regular Expressions that add repetition, alternation on top of BRE	Perl Compatible regular expression.
Engine	DFA	DFA	NFA
GNU grep	grep by default, or grep -G	egrep grep -E	grep -P
BSD grep	grep	egrep
GNU sed	sed	sed -r	NA. Just use perl
BSD sed	sed	sed -E	NA. Just use perl
awk		awk

The best way to check isn on BSD manual and GNU.

Here are several examples of sed and awk I came across at work:

Find and remove duplicate lines:

awk '!x[$0]++' input_file.txt > output_file.txt

Remove white spaces at the beginning and end of each line:

awk '{$1=$1}1' input_file.txt > output_file.txt

Print with multiple dilimiters (;, , , and |)

awk -F '[;,|]' '{print $1, $3, $5}'

Print with calculation between columns

awk '{res=$1-$2;print res,$0}'

Print rows conditionally

awk '$1>20{print;}'

Prefix each line of a file

awk '$0="PREFIX|"$0' input.txt > prefix.input.txt

Replace string original to new in file

sed -i 's/original/new/g' file.txt

Remove multiple patterns

sed 's/pattern1\|pattern2\|pattern3//g'

Delete the first matching pattern only

sed 's/pattern//'

Remove blank lines:

sed -i '/^$/d' input_file.txt > output_file.txt

Copy from line 100 to line 500 of input file to output file

sed -n 100,500p input.log>output.log

Merge every three lines:

sed 'N;N;N; s/\n/ /g'

cron and anacron in RedHat Linux (How logrotate works)

Tue, 15 May 2018 14:11:00 -0400

Cron and anacron

We all know cron is a job scheduler. Many admin uses crontab to manage scheduled task. It is also important to know that crontab works at different levels as well, as well as the distinction between cron and anacron. They are similar, but different, managed by different sets of files.

Below is a brief description of how cron works from this article.

After Cron starts, it searches its spool area to find and load crontab files into the memory. It additionally checks the /etc/crontab and or /etc/cron.d directories for system crontabs.
After loading the crontabs into memory, Cron checks the loaded crontabs on a minute-by-minute basis, running the events which are due.
In addition to this, Cron regularly (every minute) checks if the spool directory’s modtime (modification time) has changed. If so, it checks the modetime of all the loaded crontabs and reloads those which have changed. That’s why we don’t have to restart the daemon when installing a new cron job.

Basically, in cron, you specify a particular time at which a job will run. These jobs are managed by files in /var/spool/cron/ directory. In this directory, each file is named by the username that owns the crontab file. These files shall not be edited directly by respective users. Instead, they are edited by crontab by each user. To understand the syntax, one can refer to RedHat document for automating system tasks. Note that you can specify periodical jobs here with special syntax. For example, */5 at the minute slot indicates every five minutes.

Running cron jobs can be allowed or disallowed for different users. For this purpose, use the /etc/cron.allow and /etc/cron.deny files. If the cron.allow file exists, a user must be listed in it to be allowed to use cron If the cron.allow file does not exist but the cron.deny file does exist, then a user must not be listed in the cron.deny file in order to use cron. If neither of these files exists, only the super user is allowed to use cron.

In addition, there is a system-wide crontab file in /etc/crontab, in which you need to not only specify tasks, but also the user to run those tasks. By default, the schedule in this file is empty.

The limitation of cron is it assumes the servers is up all the time, if a script misses the schedule while the server is down, it will not be executed when the server comes back up. This is where anacron comes in handy. Although anacron can only be used by superuser, it doesn’t expect system to be running 24×7. If a job is scheduled at a time system is down, it starts the job when system comes back up.

Both cron and anacron are run by systemd service named crond.service. Although they require different packages installed (cronie vs cronie-anacron). They are also managed by different sets of files as explained below:

package	file or directory	purpose	example
cronie	/var/spool/cron/	this directory accommodates files that represents cron jobs for each individual users.	if a file named digihunch contains a valid line, it means that Linux user digihunch has a scheduled task for the time specified.
cronie	/etc/crontab	This file keeps system-wide cronjob entries. Each line needs to sepcify users.	if a line specifies schedule, user and command, it means that at the scheduled time, that user will execute the command.
cronie-anacron	/var/spool/anacron/	This directory accommodates files such as cron.daily, in which a timestamp is kept to indicate last execution time.	if cron.daily in this directory reads 20180418, it indicates last daily execution time stamp is 20180418
	/etc/anacrontab	This file tells anacron where in the file system to go for directories for periodical jobs.	Example: `#period in days delay in minutes job-identifier command 1 5 cron.daily nice run-parts /etc/cron.daily 7 25 cron.weekly nice run-parts /etc/cron.weekly @monthly 45 cron.monthly nice run-parts /etc/cron.monthly` The file usually also indicates RANDOM_DELAY and START_HOURS_RANGE
	/etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/	These directories stores script files that anacron needs to execute at different intervals. this is configured in /etc/anacrontab	if script logrotate is present in /etc/cron.daily/, it means the script is to be executed daily

An anacron example: logrotate

Rotating logs is a common task in Linux that can be done by logrotate. To understand how this works, first, make sure cronie-anacron package is installed and crond.service is up. Then examine the /etc/anacrontab file:

# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly

This indicates that daily, weekly and monthly jobs are active. Go into /etc/cron.daily/, and examine script logrotate:

#!/bin/sh

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
 /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

This indicates that logrotate loads configuration from /etc/logrotate.conf, the man page of logrotate explains how this configuration works, along with an example. If you have any custom application where the log file needs rotated, it can be configured in this file.

Basics Terms in Linux OS

Tue, 24 Apr 2018 19:16:00 -0400

These are the things quite confusing or abstract while I was at school but now makes lots of sense after many years working with different flavours of OS.

GPL and BSD as software license types

The main difference is that BSD (Berkeley Software Distribution) is a permissive (non-protective) license, while GPL(GNU General Public License) is a copyleft (protective) license.

Permissive licenses do not protect the code from being used in non-open source apps and apply no restrictions on the derivatives, while copyleft licenses force the creator of derivatives or re-distributor of the software to open the modified code. Under GPL you can’t sub-license, meaning, you can’t change any of the original license terms or introduce any of your own. You’re also required to state all the changes you make to the original code. That is why components licensed under GPL and other copyleft licenses should be avoided in commercial products that would later be distributed under proprietary licenses.

The BSD license family (including the Modified BSD License), on the other hand, doesn’t compel you to do any of the above. They have fairly relaxed redistribution terms.

GNU/Linux and BSD as operating systems

Unix – the name of the original system designed at AT&T in the 1970s. At the time, it featured a great deal of novelties such as multi-tasking, multi-user support, time sharing, etc. It was made portable by using C language.

Linux Kernel – a free, open-source, monolithic, Unix-like OS kernel. It was conceived and created in 1991 by Linus Torvalds for his personal computer. The word Linux, technically, is just the kernel. By itself, Linux (Kernel) has no place for user to land (e.g. no apps, no commands)

GNU/Linux – GNU project has developed a comprehensive set of free software tools for use with Unix and Linux. GNU/Linux involves the software tools along with the Linux Kernel.

GNU/Linux distribution – ready-to-use full OS, including Linux Kernel, GNU library and tools, whose developers have made a commitment to follow GNU GFSD, X Window and desktop environment (e.g. KDE, GNOME) and includes many pieces of software. GNU/Linux distro is what many people refer to as “Linux”, and it includes Debian, Ubuntu, RedHat and CentOS.

BSD (Berkeley Software Distribution) – an OS based on Research Unix, originally developed at Bell Labs, eventually grown into a complete operating system. Today, “BSD” often refers to its decendants, such as FreeBSD, OpenBSD, NetBSD, or DragonFly BSD. Each of these are both a kernel and an operating system. Another famous BSD descendant is Darwin, which is what Mac OS X based on.

Comparing “BSD” with “Linux” – Linux is more popular and tends to support new hardware sooner. Typical users usually don’t feel the difference between them. FreeBSD as desktop OS uses the same GNOME, KDE, or Xfce desktop environments that many flavours of Linux use as well. Although you need to install the desktop environment yourself. Another important difference is the licensing model as mentioned above between GPL and BSD. This article is a good reference about all the differences bewteen “BSD” and “Linux”
https://www.educba.com/linux-vs-bsd/

Swap, Cache and Buffer

Swap – swap file or swap partition. The primary function is to substitute disk space for RAM memory when real RAM fills up and more space is needed. The kernel uses a memory management program that detects blocks, aka pages, of memory in which the contents have not been used recently. The memory management program swaps enough of these relatively infrequently used pages of memory out to a special partition on the hard drive specifically designated for “paging”, or swapping. This frees up RAM and makes room for more data to be entered into your spreadsheet. Those pages of memory swapped out to the hard drive are tracked by the kernel’s memory management code and can be paged back into RAM if they are needed.

Swapping moves entire process between main memory and secondary storage; this is the original Unix method and can cause severe performance loss;
Paging moves small unites of memory (i.e. pages with 4Kbytes). It is more efficient and was added to BSD

In both cases, least recently used memory is moved to secondary storage and back to main memory only when needed again. In Linux, the term swapping is used to refer to paging. Older Unix-style swapping of entire thread and process is no longer supported.

CPU Cache – a hardware cache used by CPU to reduce the average time to access data from the main memory. A cache is a smaller, faster memory, located closer to a processor core, which stores copies of the data from frequently used main memory locations. Most CPUs have different independent caches, inclusing insructions and data caches, where the data cache is usually organized as a hierarchy of more cache levels (L1, L2, L3, L4, etc)

Page Cache (or disk cache) – kept by the OS in computer’s main memory and controlled by the computer. The OS keeps a page cached in otherwise unused portions of the main memory, resulting in quicker access to the contents of cached pages and overall performance.

Disk Buffer – (or ambiguously called disk cache or cache buffer) the embedded memory in a hard disk drive acting as a buffer between the rest of the computer and the physical hard disk platter that is used for storage. Modern hard disk drives come with 8 to 256 MiB of such memory. Disk buffer is physically distinct from and is used differently from page cache. It is controlled by the microcontroller in the hard disk drive.

GRUB (GNU GRand Unified Bootloader) – a boot loader package from GNU project. It is predominantly used for Unix-like systems. Current version is GRUB2

System call – In computing, a system call is the programmatic way in which a computer program requests a service from the kernel of the operating system it is executed on. This may include hardware-related services (for example, accessing a hard disk drive), creation and execution of new processes, and communication with integral kernel services such as process scheduling. System calls provide an essential interface between a process and the operating system. Below is a list of key system calls:

System Call	Description
read()	read bytes
write()	write bytes
open()	open a file
close()	close a file
fork()	create a new process
exec()	execute a new program
connect()	connect to a network host
accept()	accept a network connection
stat()	fetch file statistics
ioctl()	set I/O properties, or other miscellaneous functions
mmap()	map a file to the memory address space
brk()	extend the heap pointer

strace is the tool to trace system calls and signals in Linux.

sysVinit, runit and systemd

init – in Unix-based OS, init is the first process started during booting of OS. Init is a daemon process that continues running until the system is shutdown. It is the direct or indirect ancestor of all other processes and automatically adopts all orphaned processes. Init is started by the kernel during the booting process; a kernel panic will occur if the kernel is unable to start it. Init is typically assigned process identifier 1 and its job is to start other programs that are essential to the operation of your system. All other processes are descended from init.

init systems
Linux has several options as init systems. For example: sysvinit, runit, systemd and upstart. Here is a comparison of commands involved in managing each.
http://unix.cafe/wp/en/2017/07/howto-manage-a-service-in-systemd-sysvinit-upstart-runit-and-openrc/

SysV init – System initialization process is handled by the init daemon. One of the original daemon is SysVinit, which is a collection of System V-style init programs. init process starts serially. It is a run-once process during the start of the OS. One task starts only after the last task startup was successful and it was loaded in the memory. This often resulted in delayed and long booting time.

Runit – an init scheme for Unix-like operating systems that initializes, supervises, and ends processes throughout the operating system. It is a replacement of sysvinit and features brevity and simplicity.

Systemd – A init replacement daemon designed to start process in parallel, implemented in a number of standard distribution – Fedora, OpenSuSE, Arch, RHEL, CentOS, etc. Its flexibility comes with more complexity. It is an event driven init system, that not only starts stuff at boot (hence managing dependencies), but also after that. It also keeps track of many things after boot, such as mounts, availability of services, integration with resource management, etc. Because of that, systemd is also good at logging and monitoring. Systemd allows services to start when:

the system boots
a hardware components attaches to the system
other service started
a timer fires

To determine which system initialization method your current Linux distribution is using (SysVinit or systemd), simply check process id 1:

$ ps -p 1

Soft link and hard link

inode – stores the attributes and disk block locations of a file or directory

Soft/Symbolic link – essentially a shortcut to another file. The link itself is a separate file, pointing to the destination file or directory. The inode of the file is different from that of the symbolic link. Deleting the destination file will leave the symbolic link file a “dangling link”. Symbolic file may also have different permissions from the destination file or directory.

Hard link – essentially an alias of a file. The link itself is not a separate file, and the destination can only be a file (no directory). The inode of the link the the same as the file itself. So there is actually no distinction between destination file and link. Both files are equal. If you delete the file, the link continue to work until the number of hard links to the file becomes zero.

Soft link points to a file by name whereas hard link points by inode number.

CentOS – Remove Swap safely

Sat, 14 Apr 2018 16:15:43 -0400

If the default installation has swap on, you will see it from block device list:

[root@server /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 500G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 499G 0 part
 ├─centos-root 253:0 0 50G 0 lvm /
 ├─centos-swap 253:1 0 7.9G 0 lvm [SWAP]
 └─centos-home 253:2 0 441.1G 0 lvm /home
sr0 11:0 1 1024M 0 rom
[root@server /]# free -h
 total used free shared buff/cache available
Mem: 7.6G 217M 7.2G 11M 206M 7.2G
Swap: 7.9G 0B 7.9G

Many installations require swap to be off for performance reasons (although some advocate turning off swappiness of the application, instead of removing swap partition from operating system, which is a separate topic). This can be turned off by a simple command:

swapoff -a

Then if you run free command after rebooting, you will see the Swap is set to 0G. Great. But lsblk and df commands still shows the space being used by swap. This disk space is not being used at all.

[root@c7v-vitvcast02 ~]# free -m
 total used free shared buff/cache available
Mem: 7802 237 7380 11 185 7333
Swap: 0 0 0
[root@c7v-vitvcast02 ~]# lvdisplay
 --- Logical volume ---
 LV Path /dev/centos/swap
 LV Name swap
 VG Name centos
 LV UUID 7zHtvT-zGYn-sNBc-JRlT-nTfU-O9hN-r07y3J
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:21 -0500
 LV Status available
 # open 0
 LV Size <7.88 GiB
 Current LE 2016
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:1

 --- Logical volume ---
 LV Path /dev/centos/home
 LV Name home
 VG Name centos
 LV UUID 4XMTa7-uR44-qH0u-Oag4-PDyK-Z1BC-3oUSiz
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 0
 LV Size <441.12 GiB
 Current LE 112926
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:2

 --- Logical volume ---
 LV Path /dev/centos/root
 LV Name root
 VG Name centos
 LV UUID tkMCM4-cWxV-FaAC-hgJx-r1vS-DiY5-fyZvjt
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 1
 LV Size 50.00 GiB
 Current LE 12800
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:0

We can then remove the line for swap from /etc/fstab, this will keep the OS from mounting it upon reboot. However, the logical volume for swap is there, as a partition that’s not exposed.

If you need to reclaim space from swap partition, delete the logical volume:

lvremove /dev/centos/swap
Do you really want to remove active logical volume centos/swap? [y/n]: y
 Logical volume "swap" successfully removed

Now, using command lvdisplay or lvs, the swap space is not displayed anymore. So I rebooted the server in the hopes that the swap space is not being presented and everything is hunky-dory. But no… the server doesn’t boot up at all. It took a long time at the splash screen only to enter dracut, where /boot isn’t present.

On further reading, this is because the grub file is still referencing /dev/centos/swap somewhere. I need to somehow get to the grub file and fix that before it can reboot again. Specifically, I pressed “e” at the menu, which gave me an opportunity to edit grub file. From there I removed section for swap. Then I saved it with “Ctrl + X” (as suggested on the screen) so the OS booted into normal mode. Alternatively, I entered single user mode in order to get to the point I can edit the grub file.

Once booted, I have to make change to the actual grub file at /boot/grub2/grub.cfg so the change persists. In the file, locate the lines starting with linux16 and remove the section “rd.lvm.lv=centos/swap”. There are two appearances of this section on my machine. Then the server will boot just normally.

In summary, the clean way to turn off swap on CentOS involves the following steps:

turn off swap
Remove swap mount from /etc/fstab (and umount)
remove the logical volume for swap
remove reference to swap from grub

Out of those steps, 3 and 4 are needed to reclaim swap spaces. They must be done together. If you rebooted the server before step 4. The server will not be able to boot complaining about missing swap.