RESTful API on Digi Hunch

Host legacy application in Docker 1 of 2

Fri, 04 Sep 2020 16:24:00 -0400

This is my notes from containerizing a legacy application with Docker compose. We have to run multiple instances of our application because we’re unable to secure additional VMs for this single-VM education environment. The application is target of containerization, because it requires mass reconfiguration (around TCP port) to run multiple instances of the application. We want to use the same application configuration file for multiple containers, and map the TCP port to different groups of ports on the host, leveraging port mapping in Docker. On the other hand, the auxiliary services are not being containerized, such as Cassandra database and ElasticSearch because they can be shared for multiple application instances. In other words, we use Docker to isolate processes of the same application.

Prepare environment

The CentOS server needs to have docker-ce (through YUM) as well as docker-compose (direct download). They can be installed this way:

$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum install docker-ce docker-ce-cli containerd.io

$ curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo systemctl start docker

Our Docker registry is not publicly available. So we need to port the Docker image we need to remote server and load it into the local registry. We first examine the registry locally:

$ curl -XGET https://admin:password@docker.digihunch.com/v2/dhunch/tags/list | python -m json.tool

Once we identify the image, we export it to a tar file:

$ docker save docker.digihunch.com/dhunch > dhunch_image.tar

SCP the file to remote server and load it locally:

$ docker load -i /home/dhunch/dhunch_image.tar
$ docker image ls

We need to distinguish these commands:

docker save: saves an (non-running) image with all layers to file
docker export: saves a running or paused container to file
docker import: import the contents from a tarball to create a filesystem image, most used with docker export
docker load: load an image from a tar archive or STDIN, most used with docker save

Build docker-compose file

I need to cater to the customer environment with a newly create docker-compose file. The customer environment includes specific storage and networking configurations. Docker compose’s official documentation is here. We repeat the following commands for our troubleshooting:

$ docker-compose up -d
$ docker-compose exec -it dhunch1 bash
$ docker container ls

Once we start the container, the status might go unhealthy after it starts. The documentation explains two reasons you’re seeing an unhealthy container:

a single run of the command takes longer than the specified timeout
health check fails; the health check command will retry a number of times before it declares the container as unhealthy.

In our case, It is most likely because it does not pass a built-in health check mechanism. We need to understand where the health check was defined. There are four ways to enable health check:

Dockerfile instruction when building the image
Docker run command
Docker-compose or docker stack yaml file
Docker service

With #1, unfortunately, you can’t reverse engineer an image and view the Dockerfile that were used to built it and review the health check statement. What you can do is check docker events, or inspect the container, and go to the log files as specified under logPath section in the inspection result and look for HealthCheck section. We determined it is the case, then we can disable, or override the built-in healthcheck command from image, with a statement in docker compose.

For network interface, docker compose also allows us to specify MAC address for each container with mac_address keyword (for license key). MAC address generator are available on the internet.

EntryPoint vs CMD

The difference between EntryPoint and CMD is very important when launching container. Some literature also mentions RUN, which is only used when building a new layer of images so it is not relevant here (in the context of launching a container from image). EntryPoint and CMD has similar functionalities both allowing you to specify a command to run. The difference is whether they can be overwritten by command line arguments that user provide to docker-compose or docker run in an ad-hoc manner. As their names suggests, EntryPOINT means what is specified under it must be executed as it launches into the container, regardless of any adhoc commands. On the other hand, CMD is just an entry to save users from typing in a command every time they run docker compose or docker run. Should user prefer a different command, it can be provided as an explicit argument and it will be respected overwriting the pre-defined CMDentry in Dockerfile or command entry in docker-compose.yml.

Both CMD and EntryPoint supports shell and exec forms. More details here.

Choice of Networking

With single-host deployment, the containerized application needs to communicate with other existing, non-containerized service on host, such as database or elastic search. If docker uses host network, the container shares interface with the host and it does not have its own IP address. Host network removes isolation between container and host. This allows container to run the application that was licensed to the host based on MAC address. There is also no port mapping from container to host network. Container simply uses port on host, and is subject to the availability of TCP/UDP port on host.

We will have to use bridge network here. We can force MAC address the app container, and pre-generate license. For container to communicate with a service on host, through bridge network, there are two problems to address:

Container knows the IP of the host (layer-3 connectivity, ping);
Making host service available to container (layer-4 connectivity, telnet);

Docker creates its own interface for bridge network. If it’s an unnamed network, i.e. not explicitly declared under networks section in docker compose, then interface docker0 is used. If it’s a named network, then an interface name starting with br- is used.

The first problem is easier to address, we simply needs to IP address of the host on the interface. We can validate by pinging from container to host. Docker can also use host.docker.internal to reference the host. Unfortunately, this stopped working for linux since 18.09.3.

It is reportedly to be fixed in 20.04 and until it is available, we may add it to manual dns. The following command outputs the entry to add to /etc/hosts in container.

# ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}'

To do this automatically in docker compose, we need some tricks:

Store the Host IP in host environment variable ( use an export command)
Use compose to pass host environment variable to container environment variable
Have the container write its environment variable to /etc/hosts

The compose file will contain a line like this:

services:
 myenv1:
 image: alpine
 command: >
 sh -c "apk update &&
 echo $$HostDNSLine >> /etc/hosts &&
 bash"
 #network_mode: bridge
 environment:
 - HostDNSLine=${HOSTDNSREC}

Note ampersand might be mistakenly displayed as & in the above. Then we run it with the following:

# export HOSTDNSREC=$(echo 1.2.3.4 host.docker.internal) && docker-compose up

The second problem is harder to address because the service on host may not bind to docker’s interface. Some services such as ssh bind to all interfaces on host and you can telnet to port 22 with any IP address the host is associated with. This is however not the case for most other services, such as Cassandra or Elastic Search. They typically only bind to main interface, such as ens192, or eth0, and not to the docker interface. In order to make the service available to container, we either need to bind these services to the docker interface, or use iptables rules as an alternative.

Suppose it is a named network and Docker’s interface name is br-90ae024d5324, and the service on host listens to port 9042, we will need the following two commands from host:

# sysctl -w net.ipv4.conf.br-90ae024d5324.route_localnet=1

# iptables -t nat -A PREROUTING -p tcp -i br-90ae024d5324 --dport 9042 -j DNAT --to-destination 127.0.0.1:9042

Note that docker compose can configure to run sysctl in container but not from host. If there are multiple ports, we can turn this into a shell script:

#!/bin/bash
tcp_port_list="9200 9042 8302 8303 8304 8305 8306"

if_name=$(basename -a /sys/class/net/* | grep ^br- | head -1)
echo enable route localnet on interface $if_name
sysctl -w net.ipv4.conf.$if_name.route_localnet=1
for tcp_port in $tcp_port_list; do
 echo open host tcp port $tcp_port to interface $if_name
 iptables -t nat -A PREROUTING -p tcp -i $if_name --dport $tcp_port -j DNAT --to-destination 127.0.0.1:$tcp_port
done

echo $(ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}')

On the other hand, binding service to multiple interfaces usually require some re-configuration on the service itself. For example, if it is Elastic Search, we need to update [network.host] entry in elasticsearch.yml to include multiple IP addresses. For Cassandra, we need to update rpc_address to 0.0.0.0 or set rpc_interface in cassandra.yml.

Integration with storage

The application in the container need to store files to storage available to host, whether it is an NFS share or a block disk. We can use volume mapping with Docker compose, to map a path in container to a path presented to host as persistent volume. At this step, we might run into permission issues.

By default, containers initializes as root (uid=1) within the container, and the entrypoint script launches application as root. When application writes to persistent volume, files are written as root user. In the legacy non-container setup, we expect the application to write file as dhunch user. Moreover, NFS volume will not allow writing files as root (if the server has root squash configured). To address this, there are two approaches:

launch container as a regular user
launch container as root user, then have the entrypoint script launch application as regular user (dhunch)

For approach 1, we need to tell Docker to launch container as a regular user by specify the uid and gid for container to run application. We can specify the following envrionment variable in the compose yaml:

user: ${CURRENT_UID}

Then we assign the environment variable before running docker-compose:

# export CURRENT_UID=$(id -u dhunch):$(id -g dhunch) &amp;&amp; docker-compose up

This allows container to initialize as the regular user. However, if the entry point script needs to perform activities that requires root permission within the container, it will fail. For example, a regular user in container will not be able to update /etc/hosts;

With approach 2, we do not specify user in docker compose so container initializes as root. Then the entry point script launches application as regular user. For example, use su command before launch Java:

su dhunch -c "
exec java \
 -Xms512M -Xmx8192M \
 -Djava.io.tmpdir=$APP_HOME/var/tmp \
 -server \
 -XX:CompileCommandFile=$APP_HOME/etc/hotspot_compiler \
 -jar $APP_HOME/lib/jar/jruby-complete-*.jar \
 --1.9 \
 $APP_HOME/lib/rubybin/runapp.rb
"

Before doing this, we need to first create user dhunch within container, and the uid and gid must match those of the host. So that when container picks up dhunch user, it converts it to the correct uid.

groupadd -g 1011 dhunch
useradd -m -c 'regular user' -u 1011 -g 1011 dhunch

To further understand how uid and gid work, here are two posts with more information.

This user ownership setup will also work for NFS. To configure NFS, we need some extra client-side configurations in the container, as well as a special volume driver for NFS. Refer to this post.

OAuth 2.0 and OIDC 1 of 2

Sat, 14 Mar 2020 21:10:00 -0400

OAuth 2.0 and OpenID Connect (OIDC 1.0) are different but highly related protocols and they are often confused. When we talk about IAM (identity and access management), we should first distinguish between Authentication (AuthN) and Authorization (AuthZ):

AutheNtication (AuthN, aka Identity Management) is about validating user’s identity by verifying that the user trying to connect is actually who it claims itself to be;
AuthoriZation (AuthZ, aka Access Management) refers to granting or denying access to specific resources based on the requesting user’s identity. It is usually performed after a user is identified through authentication. The most common approach is Role-Based Access Control (RBAC).

In a nutshell, OAuth 2.0 deals with authorization. OIDC is a layer later developed on top of OAuth 2.0, to deal with authentication. This post is greatly influenced by a presentation (1 hour) delivered by Nate Barbettini from Okta, with the slides available here. There is also an abridged and illustrated video (16 min) by Okta available here.

OAuth 2.0

OAuth was originally developed by Twitter and Google in 2006 as an open standard for API authorization. OAuth 2.0 is published in 2012. It allows user to delegate authorization. The original scenario is a user signed up to a new application and allows it to automatically import her Gmail contact. The technical problem to solve is: how can a user (Resource Owner) let an app (Client) to access his contact list stored in Google server (Resource Server)? The proposal is that it redirects user to Google Account page (Authorization Server) for user to log in. Then Google Account issues token to the application (Client) with user’s approval. Note that the user did NOT log in to the application itself with her Google account. From the application’s standpoint, the user had been authenticated already, and was simply importing contact after logging in. The roles involved in OAuth 2.0 are:

Resource: the contact list of the user
Resource owner: the user
Client: the application
Resource Server: contact.google.com
Authorization Server: accounts.google.com

The diagram below illustrates the interactions:

OAuth 2.0 Authorization Code Flow

This page has further details for each step. Note that at step 3 to 5 may seem unnecessary because Auth Server could have send Access Token Grant to Application via User-Agent at step 3, which could have eliminate the need for step 4 and 5. In fact this design is to avoid sending critical information (Access Token Grant) to User-Agent (browser) which is considered in secure. In other words, it avoids front channel (User-agent to auth server) and prefers back channel (Client to Auth server) for security. This is the difference between Authorization code flow and the implicit flow.

With OAuth 2.0, there are a number of flows:

Authorization code (front channel and back channel)
Implicit (front channel only, token returned to user agent directly)
Resource owner password credentials (back channel only)
Client credentials (back channel only)

In Authorization Code Flow, the Application (client) needs a one-time registration with the Auth Server and is given a client ID and client secret, which are sent to Auth Server at step 4 along with Access Token Request, to prove the identity of the client application.

Note that OAuth 2.0 is an inherently insecure protocol since it does not support signature, encryption, channel binding or client verification. The protocol relies entirely on the underlying transport layer security (TLS) to provide confidentiality and integrity.

Also note that throughout the process (Authorization Code Flow as an example), the Client application eventually is granted access to user’s data. However, it does not know anything about the user itself. Neither the authorization code grant, nor the access token grant is obligated to present information about the user itself. Therefore, OAuth 2.0 is designed strictly for permission purpose without the intent to address identity issue. In the flow, the auth server does the authentication (for the purpose of granting access to resources, but none of the authentication. A user logs in to client application as Bob, when he requests to imports contact, he is redirected to account.google.com and there he could put in the credential of Alice and therefore load Alice’s Google contacts into Bob’s App account!

Pseudo-authentication with OAuth 2.0

In many real life OAuth 2.0 implementations, at step 3, the Auth server chooses to include a field about the user’s identity. This makes user’s identity visible to the client, and the client is therefore able to confirm user’s identity in its own code. This also allows client application to use OAuth 2.0 as an authentication method, which is referred to as pseudo-authentication. The access token acts as a kind of “valet key” that the application can include with its request to the auth server, as a proof that it has user’s permission to access the resources (or APIs).

Because the identity provider (auth server) typically (but not always) authenticates the user as part of the process of granting an OAuth access token, it’s tempting to view a successful OAuth access token request as an authentication method itself. However, because OAuth was not designed with this use case in mind, making this assumption can lead to major security flaws.

Nate’s presentation outlined some scenarios where OAuth 2.0 is applied up to 2012, and which ones are misuses:

Simple login – pseudo authentication with OAuth 2.0
Single sign-on across sites – pseudo authentication with OAuth 2.0
Mobile app login – pseudo authentication with OAuth 2.0
Delegated authorization – the only intended use case for OAuth 2.0

To address the authentication issue properly, and in a standard approach, we need OpenID Connect.

OpenID Connect (OIDC)

OpenID Connect is an open standard for authentication, promoted by the non-profit OpenID Foundation. It allows user to be authenticated using a third-party service called identity providers. User may choose to use their preferred OpenID Connect providers to log in to websites that accept the OpenID Connect authentication scheme. For example, a user uses her Facebook to login to an online application.

OpenID Connect is an extension to OAuth 2.0 with a just few additions:

In addition to access token, an ID token is returned by the authorization server;
Userinfo end point is provided in case Id token is not sufficient and more user information is needed;
“openid” is passed as a parameter in the Scope during the initial call to the authorization server;

Therefore OpenID Connect is considered an identity layer on top of OAuth 2.0. Many application supports OpenID Connect such as Apache Nifi. OIDC is comparable with SAML in the sense that both provide SSO feature (federated identity). Here is a comparison table:

	OpenID Connect	SAML
Main Purpose	SSO for consumer/mobile applications	SSO for enterprise applications
Load	Relatively light weight	Heavy weight due to the size of XML messages
Use case	Satisfies both authentication and authorization use cases, often combined with OAuth 2.0	Generally not used for API security
Transport	HTTP GET and HTTP POST	HTTP Redirect (GET) binding, SAML SOAP binding, HTTP POST binding, et

Here are more details about their differences. In general SAML is more common in the enterprise world for SSO and it has been around for a while. When developing new applications for enterprise it is advised to consider OIDC first.

OIDC also has authorization code flow, with the additional fields on top of its counterpart in OAuth 2.0. The authorization server returns both access and ID tokens, wrapped in a data structure named JWT (JSON Web Token). The JWT includes a signature field, allowing the client application to verify it with authorization server’s public key. Nate’s presentation proposes the following flows for each application type:

Web application with server backend: authorization code flow
Native mobile app: authorization code flow with PKCE
Java Script app (SPA) with API backend: implicit flow
Microservices and APIs: client credential flow

In addition, this page from Okta developer has a good summary of how to select flow type (grant) based on each use case.

Summary

OpenID Connect is an authentication protocol for the purpose of validating user’s identity. OAuth 2.0 is an authorization protocol. You should use OAuth 2.0 for granting access to your API, or access to user data in other systems. If you need to log user in, or make your accounts available in other systems, you need OIDC.

The next post about OAuth and OIDC was posted in 2023.