Host on Digi Hunch

Docker components

Wed, 28 Oct 2020 20:23:00 -0400

The previous post about virtualization and containerization brought up some underlying technologies which Docker build containers on, including:

namespaces – a Linux kernel mechanism to isolate resources. It allows a process to run within an isolated environment (mnt, pid, net, ipt, uts, user, cgroup)
cgroups – a Linux kernel mechanism to limit resource usage of a process or process group
unionFS (this will be further discussed under Docker storage)

In this post we further discuss the components in Docker, the dominant and popular player in container technology, as shown in the diagram below:

The component names can be seen under docker install directory. It consists of three groups:

Docker related: docker, dockerd, docker-init and docker-proxy
Containerd related: containerd, containerd-shim and ctr
Container runtime: runc

Now we discuss each group:

Docker-related components

docker is just an implementation of docker client, it supports commands to achieve all functions between client and server. Alternatively, user may use REST API, or Docker SDK to communicate with Docker server.

dockerd is the server process, to receive requests from docker (client), SDK library or REST API caller. It executes the request and returns status to client. There are three ways for docker (client) to communicate with dockerd.

By Unix Socket (unix://socket_path). The default socket path used by dockerd is /var/run/docker.sock, which is why only root can use docker after installation.
TCP request (tcp://host:port). It is recommended to configure TLS communication in production environment.
By file descriptor (fd://) used in systemd service.

Unix socket is the default communication method. To allow remote access to dockerd, use -H to specify HOST and PORT when starting dockerd.

docker-init is used by Docker as PID 1 process for containers, in case it needs to recycle zombie containers. To use this, specify –init when running container.

docker-proxy is used for port mapping. When you use -p switch with docker run, this docker-proxy is the service that maps the container port to host port. It does so by modifying the iptables nat rule.

Containerd related components

containerd component was separated from dockerd since Docker 1.11, in compliance with OCI standard. It is responsible for life cycle management of containers, it also manages images (e.g. pulling from repo), request from dockerd to call runc, storage and network resources.

dockerd uses UNIX socket to send request to containerd. The default socket path for containerd is /run/containerd/containerd.sock. containerd execute the task and return status to dockerd. You may also directly use containerd to manage containers.

ctr (containderd-ctr) is the client of containerd, mostly used only in development and testing. If the environment does not have dockerd, then you can use ctr as client, to send request directly to containerd.

containerd-shim is used to decouple containerd from the containers. containerd-shim is the parent process of containers. This is so that restarting containerd does not impact the running containers.

Container runtime

runc is a standard implementation of OCI container runtime. It is a command-line tool to create and run containers.

Host legacy application in Docker 2 of 2

Thu, 22 Oct 2020 17:54:00 -0400

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

Application service	TCP port to bind
Business service A	8030
Business service B	8040
Business service C	8050
……	…

TCP port requirement

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
The OS needs to host 6 processes of the same application all running at the same time;
The OS does not allow multiple processes to bind to a single TCP port (duh!);
It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

OS	Container ID	Application Service	container port	published port
Host CentOS	Container 1 (Instance #1)	Business Service A	8030	9301
		Business Service B	8040	9401
		Business Service C	8050	9501
	Container 2 (Instance #2)	Business Service A	8030	9302
		Business Service B	8040	9402
		Business Service C	8050	9502
	Container 3 (Instance #3)	Business Service A	8030	9601
		Business Service B	8040	9602
		Business Service C	8050	9603

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
 dapp1:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp1
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9301:8030 # BUSINESS SERVICE A
 - 9401:8040 # BUSINESS SERVICE B
 - 9501:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:cf
 environment:
 - INSTANCE_TAG=dapp1 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp2:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp2
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9302:8030 # BUSINESS SERVICE A
 - 9402:8040 # BUSINESS SERVICE B
 - 9502:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d0 
 environment:
 - INSTANCE_TAG=dapp2 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp3:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp3
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9601:8030 # BUSINESS SERVICE A
 - 9602:8040 # BUSINESS SERVICE B
 - 9603:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d1 
 environment:
 - INSTANCE_TAG=dapp3 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
networks:
 vcnet:
 driver: bridge
 driver_opts:
 com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
- Initialization work that cannot be done in Dockerfile, such as setting environment variable
- Launch the application, including pointing log file to stdout
- Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
build_image.sh: this script makes the image build process smoother
- check if image to build already exist, and ask permission to delete the existing image if so;
- build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

	Dockerfile instruction for container	Permission issue during mount by docker-compose
Bad config	RUN mkdir -p /var/lib/dapp && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.
Good config	RUN mkdir -p /var/lib/dapp/dcontainer/archive && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue.

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.

Host legacy application in Docker 1 of 2

Fri, 04 Sep 2020 16:24:00 -0400

This is my notes from containerizing a legacy application with Docker compose. We have to run multiple instances of our application because we’re unable to secure additional VMs for this single-VM education environment. The application is target of containerization, because it requires mass reconfiguration (around TCP port) to run multiple instances of the application. We want to use the same application configuration file for multiple containers, and map the TCP port to different groups of ports on the host, leveraging port mapping in Docker. On the other hand, the auxiliary services are not being containerized, such as Cassandra database and ElasticSearch because they can be shared for multiple application instances. In other words, we use Docker to isolate processes of the same application.

Prepare environment

The CentOS server needs to have docker-ce (through YUM) as well as docker-compose (direct download). They can be installed this way:

$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum install docker-ce docker-ce-cli containerd.io

$ curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo systemctl start docker

Our Docker registry is not publicly available. So we need to port the Docker image we need to remote server and load it into the local registry. We first examine the registry locally:

$ curl -XGET https://admin:password@docker.digihunch.com/v2/dhunch/tags/list | python -m json.tool

Once we identify the image, we export it to a tar file:

$ docker save docker.digihunch.com/dhunch > dhunch_image.tar

SCP the file to remote server and load it locally:

$ docker load -i /home/dhunch/dhunch_image.tar
$ docker image ls

We need to distinguish these commands:

docker save: saves an (non-running) image with all layers to file
docker export: saves a running or paused container to file
docker import: import the contents from a tarball to create a filesystem image, most used with docker export
docker load: load an image from a tar archive or STDIN, most used with docker save

Build docker-compose file

I need to cater to the customer environment with a newly create docker-compose file. The customer environment includes specific storage and networking configurations. Docker compose’s official documentation is here. We repeat the following commands for our troubleshooting:

$ docker-compose up -d
$ docker-compose exec -it dhunch1 bash
$ docker container ls

Once we start the container, the status might go unhealthy after it starts. The documentation explains two reasons you’re seeing an unhealthy container:

a single run of the command takes longer than the specified timeout
health check fails; the health check command will retry a number of times before it declares the container as unhealthy.

In our case, It is most likely because it does not pass a built-in health check mechanism. We need to understand where the health check was defined. There are four ways to enable health check:

Dockerfile instruction when building the image
Docker run command
Docker-compose or docker stack yaml file
Docker service

With #1, unfortunately, you can’t reverse engineer an image and view the Dockerfile that were used to built it and review the health check statement. What you can do is check docker events, or inspect the container, and go to the log files as specified under logPath section in the inspection result and look for HealthCheck section. We determined it is the case, then we can disable, or override the built-in healthcheck command from image, with a statement in docker compose.

For network interface, docker compose also allows us to specify MAC address for each container with mac_address keyword (for license key). MAC address generator are available on the internet.

EntryPoint vs CMD

The difference between EntryPoint and CMD is very important when launching container. Some literature also mentions RUN, which is only used when building a new layer of images so it is not relevant here (in the context of launching a container from image). EntryPoint and CMD has similar functionalities both allowing you to specify a command to run. The difference is whether they can be overwritten by command line arguments that user provide to docker-compose or docker run in an ad-hoc manner. As their names suggests, EntryPOINT means what is specified under it must be executed as it launches into the container, regardless of any adhoc commands. On the other hand, CMD is just an entry to save users from typing in a command every time they run docker compose or docker run. Should user prefer a different command, it can be provided as an explicit argument and it will be respected overwriting the pre-defined CMDentry in Dockerfile or command entry in docker-compose.yml.

Both CMD and EntryPoint supports shell and exec forms. More details here.

Choice of Networking

With single-host deployment, the containerized application needs to communicate with other existing, non-containerized service on host, such as database or elastic search. If docker uses host network, the container shares interface with the host and it does not have its own IP address. Host network removes isolation between container and host. This allows container to run the application that was licensed to the host based on MAC address. There is also no port mapping from container to host network. Container simply uses port on host, and is subject to the availability of TCP/UDP port on host.

We will have to use bridge network here. We can force MAC address the app container, and pre-generate license. For container to communicate with a service on host, through bridge network, there are two problems to address:

Container knows the IP of the host (layer-3 connectivity, ping);
Making host service available to container (layer-4 connectivity, telnet);

Docker creates its own interface for bridge network. If it’s an unnamed network, i.e. not explicitly declared under networks section in docker compose, then interface docker0 is used. If it’s a named network, then an interface name starting with br- is used.

The first problem is easier to address, we simply needs to IP address of the host on the interface. We can validate by pinging from container to host. Docker can also use host.docker.internal to reference the host. Unfortunately, this stopped working for linux since 18.09.3.

It is reportedly to be fixed in 20.04 and until it is available, we may add it to manual dns. The following command outputs the entry to add to /etc/hosts in container.

# ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}'

To do this automatically in docker compose, we need some tricks:

Store the Host IP in host environment variable ( use an export command)
Use compose to pass host environment variable to container environment variable
Have the container write its environment variable to /etc/hosts

The compose file will contain a line like this:

services:
 myenv1:
 image: alpine
 command: >
 sh -c "apk update &&
 echo $$HostDNSLine >> /etc/hosts &&
 bash"
 #network_mode: bridge
 environment:
 - HostDNSLine=${HOSTDNSREC}

Note ampersand might be mistakenly displayed as & in the above. Then we run it with the following:

# export HOSTDNSREC=$(echo 1.2.3.4 host.docker.internal) && docker-compose up

The second problem is harder to address because the service on host may not bind to docker’s interface. Some services such as ssh bind to all interfaces on host and you can telnet to port 22 with any IP address the host is associated with. This is however not the case for most other services, such as Cassandra or Elastic Search. They typically only bind to main interface, such as ens192, or eth0, and not to the docker interface. In order to make the service available to container, we either need to bind these services to the docker interface, or use iptables rules as an alternative.

Suppose it is a named network and Docker’s interface name is br-90ae024d5324, and the service on host listens to port 9042, we will need the following two commands from host:

# sysctl -w net.ipv4.conf.br-90ae024d5324.route_localnet=1

# iptables -t nat -A PREROUTING -p tcp -i br-90ae024d5324 --dport 9042 -j DNAT --to-destination 127.0.0.1:9042

Note that docker compose can configure to run sysctl in container but not from host. If there are multiple ports, we can turn this into a shell script:

#!/bin/bash
tcp_port_list="9200 9042 8302 8303 8304 8305 8306"

if_name=$(basename -a /sys/class/net/* | grep ^br- | head -1)
echo enable route localnet on interface $if_name
sysctl -w net.ipv4.conf.$if_name.route_localnet=1
for tcp_port in $tcp_port_list; do
 echo open host tcp port $tcp_port to interface $if_name
 iptables -t nat -A PREROUTING -p tcp -i $if_name --dport $tcp_port -j DNAT --to-destination 127.0.0.1:$tcp_port
done

echo $(ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}')

On the other hand, binding service to multiple interfaces usually require some re-configuration on the service itself. For example, if it is Elastic Search, we need to update [network.host] entry in elasticsearch.yml to include multiple IP addresses. For Cassandra, we need to update rpc_address to 0.0.0.0 or set rpc_interface in cassandra.yml.

Integration with storage

The application in the container need to store files to storage available to host, whether it is an NFS share or a block disk. We can use volume mapping with Docker compose, to map a path in container to a path presented to host as persistent volume. At this step, we might run into permission issues.

By default, containers initializes as root (uid=1) within the container, and the entrypoint script launches application as root. When application writes to persistent volume, files are written as root user. In the legacy non-container setup, we expect the application to write file as dhunch user. Moreover, NFS volume will not allow writing files as root (if the server has root squash configured). To address this, there are two approaches:

launch container as a regular user
launch container as root user, then have the entrypoint script launch application as regular user (dhunch)

For approach 1, we need to tell Docker to launch container as a regular user by specify the uid and gid for container to run application. We can specify the following envrionment variable in the compose yaml:

user: ${CURRENT_UID}

Then we assign the environment variable before running docker-compose:

# export CURRENT_UID=$(id -u dhunch):$(id -g dhunch) &amp;&amp; docker-compose up

This allows container to initialize as the regular user. However, if the entry point script needs to perform activities that requires root permission within the container, it will fail. For example, a regular user in container will not be able to update /etc/hosts;

With approach 2, we do not specify user in docker compose so container initializes as root. Then the entry point script launches application as regular user. For example, use su command before launch Java:

su dhunch -c "
exec java \
 -Xms512M -Xmx8192M \
 -Djava.io.tmpdir=$APP_HOME/var/tmp \
 -server \
 -XX:CompileCommandFile=$APP_HOME/etc/hotspot_compiler \
 -jar $APP_HOME/lib/jar/jruby-complete-*.jar \
 --1.9 \
 $APP_HOME/lib/rubybin/runapp.rb
"

Before doing this, we need to first create user dhunch within container, and the uid and gid must match those of the host. So that when container picks up dhunch user, it converts it to the correct uid.

groupadd -g 1011 dhunch
useradd -m -c 'regular user' -u 1011 -g 1011 dhunch

To further understand how uid and gid work, here are two posts with more information.

This user ownership setup will also work for NFS. To configure NFS, we need some extra client-side configurations in the container, as well as a special volume driver for NFS. Refer to this post.