Cloud services on Digi Hunch

Computing services: from PaaS to Serverless

Fri, 21 Oct 2022 19:31:00 -0400

Silicon Valley startups in mid-2000s likely do not run their own IT operations (i.e. renting their own data centre spaces, purchasing their own rack-mounted servers). Since the launch of EC2, AWS has been renting extra computing capacity to those startups, in the IaaS model. The leased infrastructure requires maintenance work, and AWS realized that many of these customers cannot afford specialized database admins, network admins, storage admins, or even server admins. As a result, they created a handful of managed services aiming to cut out admin overhead and let their customer focus on coding. This is how Platform-as-a-service (PaaS) came about.

Let’s take a look at what are exactly operation activities.

Ops activities

IT operation team manages server provisioning, installation of operating system, tuning performance, configuring auto scaling and load balancing, configure networking and storage systems, etc. Networking can be so complex that many infrastructure teams have a dedicated Network Operation Center (NOC), who along with security team, manages key aspects of networking, such as segmentation, router configuration, load balancing, firewall configuration.

For client-server application, the client-side code will make outgoing connections, utilizing the TCP/IP stack on the host through an ephemeral port. The server-side code has to be wrapped as a service. A daemon ensures the process running this service stays up and listens to a TCP port in order to respond to request by invoking the functions. Application team usually assumes these activities.

If database is involved, then the patching, upgrade, replication, data protection are all Ops problems. If storage is involved, then Ops has to manage mass data accumulated over years, the integration between storage and database and applications, performance, replication, etc. Some larger organizations have full-time database administrator and storage administrators.

Then comes container. Containers have their benefits but it increases the operation overhead by an order of magnitude. Running container application at scale warrants its own platform, most likely a Kubernetes platform, to address all of the problems above again at the cluster level. Some organization created platform team to manage container and VM platforms.

It is the Ops, that turns functional code into a running business. It is also the Ops, that becomes a pain point as a startup scales. With IaaS and PaaS models, AWS managed to convince many small businesses to delegate their IT operations to AWS. This is the humble start of cloud computing.

Elastic Beanstalk

At first, I wasn’t too impressed with Elastic Beanstalk, since it abstracts away too many details. However, I later realized that it has been surprisingly popular in the developer community, especially with individual developers and SMBs. It simplifies deployment to the point that their users don’t need to know other AWS services, allowing them to focus on coding application logic.

You configure Applications and Environments (one application may have multiple environments). In the Environment layer, you can specify code platform (e.g. Python 3.8 on 64bit Amazon Linux 2, Java, Go, PHP, Ruby) and even container platform (Docker on EC2 or ECS). Behind the scene, Elastic Beanstalk configures EC2 instances, Elastic Load Balancers, etc on the selected VPC and integrate with logging and monitoring services. In the console, Elastic Beanstalk exposes a list of configurations options (e.g. AMI, instance type). This centralized configuration page is dummied down for those who don’t want to deal with Ops.

The downside of Elastic Beanstalk is it takes away a lot of flexibility. Many developers find Elastic Beanstalk limit their choices of deployment, as their applications scale. Elastic Beanstalk does not suit for applications that demand extensive operation efforts. Its niche market is individual developers and SMB. Few enterprise applications run on Elastic Beanstalk.

Containerization with ECS and EKS

When we containerize an application, we build container images. Then we run these images with container runtimes, is a core feature of container platform. Container platform also provides orchestration engine since we frequently take containers up and down. In addition, container platform provides mechanisms for container networking and storage.

AWS has a couple options for container platform. ECS (Elastic Container Service) came out earlier. It organizes a group of EC2 instances as a cluster. You can manage autoscaling, networking, and persistent storage (EFS, FSx etc) on ECS. EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. Just like AKS, it provides a managed control plane along with computing nodes.

I see ECS as a proprietary and simplified container platform, and Kubernetes as an open-source standard for full-fledged container platform with an entire ecosystem. EKS includes an upstream-certified Kubernetes distribution with a set of tools specific to AWS. Since Kubernetes is the de-facto standard container platform, I prefer EKS by default, unless I can justify the use of ECS. In fact, ECS and Kubernetes have many concepts in common. For example, a “Task” in ECS is equivalent to a Pod in Kubernetes.

Whether it is ECS or EKS, right-sizing the computing node group is always challenging especially when the application traffic load is irregular. AWS Fargate is a technology that provides on-demand, right-sized compute capacities. It works with ECS and EKS. When integrated with EKS, we delegate the node management (e.g. scaling) to Fargate and forget about sizing the node pool.

Using ECS and Fargate involves quite a bit of configurations. To simplify that, we can use App Runner, which builds ECS cluster and uses Fargate to execute the container behind the scenes. App Runner helps client in a way similar to Elastic Beanstalk, but concentrate on Container workload.

Serverless with Lambda and API Gateway

The services above have their limitations when it comes to scaling capability. First, they cannot scale to zero. You still pay for idling resources. Also, it is not easy to find the optimal autoscaling setting. Lambda and API Gateway together solves these challenges. AWS refers to it as serverless, which has since become a buzzword. To understand what it is, let’s examine two concepts:

Function as a Service: service with the ability to execute code on demand. Users only pay for code execution time and do not care where the underlying runtime is
Backend as a Service: service with the ability to listen to a port and respond to web request

Lambda itself is a function as a service. Triggered by events, it only incurs a charge when it’s invoked. It does not stay up and listening to a TCP port for incoming web request, as does a backend service. In order to act as a backend, Lambda needs to pair up with API gateway. In this configuration, API gateway listens to a web request, and it fires an event to trigger the execution of Lambda function. Lambda and API gateway together makes a backend as a service. In AWS, the coupling of API gateway and Lambda function ensures an idle service does not incur computing cost.

Since Lambda supports many types of events as trigger, it is also used in event-driven architecture, either standalone or from a VPC. Under the hood, Lambda runs code in a container (with a quick startup time relative to a VM).

Developers can release Lambda code by uploading zip package to S3 bucket, or just packaging code into container image. For deployment, apart from AWS console and CLI, one can leverage CloudFormation, SAM (serverless application model), or CDK.

Lambda vs Fargate

Both Lambda and Fargate are serverless capabilities, at least from a marketing perspective. Both can be used to back web service but there are differences. They provision computing resource at different granularity.

In a web service, the execution duration of a Lambda function is the response duration to an API request, in terms of seconds. While the server is waiting for a request, there is no usage of the computing resource so you’re not paying for waiting for a request. However, this also creates the delay of cold-start, especially when the code size is large. There are several ways to optimize the cold start (e.g. SnapStart for Java), but none of those can completely get rid of the cold-start delay with a once-after-a-while request. A light GET call could take 5 seconds with cold start.

On the Fargate side, the resource provisioning is based on container lifecycle, instead of request lifecycle. As a result, you’re still paying for wait time, and it is not per-request billing. Since the container remains up, your request is not going to experience the cold-start if it’s been idle for a while. Although, Fargate saves you from the effort to right-sizing the computing nodes for container execution, it is not quite the idea of “scale-to-zero when idle” by itself.

Serverless Architecture

In the white paper AWS Serverless Multi-Tier Architectures with Amazon API Gateway and AWS Lambda, AWS advocates the serverless architecture as a modern alternative to the traditional widely adopted three-tier architecture (presentation, logic and data tiers). In the three tier architecture, the scalability of three tier are managed separately. The modern serverless architecture that AWS whitepaper proposes uses API Gateway and Lambda function in place of Load Balancer and EC2 instances (e.g. in an Auto Scaling Group), as illustrated below:

Both API Gateway and Lambda scale automatically to support the need of application workload. It assumes the role of logic tier in three-tier architecture but requires minimal maintenance work. For presentation tier, AWS has serverless alternatives such as CloudFront, S3. For data tier, AWS has serverless alternatives such as Amazon Aurora for relational database and DynamoDB for NoSQL. However, the “no request, no pay” model for Lambda does not apply to the data tier in serverless architecture.

Whilst this paradigm benefits small shop IT who wants to minimize infrastructure cost, it has downsides. There is no ability for infrastructure optimization. Since you do not manage where the code runs, client may have concerns over security (e.g. multi-tenant runtime). As business grows, keep using Lambda can result in technology lock-in. Also, a less used application usually requires warm-up time. A code start (downloading the code and preparing the environment behind the scene) can take 100ms to over a second.

Conclusion

PaaS attempts to help startups simplify the “grunt work” of IT operation. Serverless takes it even further. The semantics of serverless computing is confusing and the Wikipedia page acknowledges it as a misnomer. The nature of serverless model, is the cloud users delegate server capacity management to cloud platforms. The users don’t need to manage servers, VMs, instances, containers, etc on their own. In a previous post, I discussed the ability to scale to zero, which is just one of the many enabling technologies of serverless. Also, “no request, no pay” is neither an inherent nature of serverless model. Serverless service may involve storage (e.g. data service, S3, Aurora serverless) which incurs storage cost. There is a whitepaper on choosing the right AWS service to deploy your website or web application, with a decision tree.

AWS pioneered serverless with Lambda release in 2014 but competitor follows. In the Azure landscape, there is an entire suite of computing services from virtual machine to serverless (also with a decision tree in documentation). Azure’s counterpart for serverless architecture is Azure Function (released in 2016 for GA) with API Management. As for GCP, the serverless suite includes the event-driven Cloud Function (introduced in 2017) and Knative-based FaaS Cloud Run (introduced in 2019).

There are voices in advocacy of standardization of serverless model, and CNCF had since made minuscule efforts such as CloudEvents. The status quo, unfortunately, is anything but standardized.

Lastly, here is a table that summarizes the pros and cons of each computing service model.

Computing Service Model	Pro	Con
EC2	– Most straightforward and widespread legacy model – Legacy	– Ops tasks can be heavy (e.g. patch and vulnerability management of OS) – Utilization can be low
ECS	– Container orchestration is managed – Convenient to scale – Well integrated with other AWS services	– Limited advanced features – Vendor lock-in
EKS	– Highly scalable and flexible – Advanced, platform-neutral deployment tools available (e.g. Helm, ArgoCD, etc) – Custom configurations (e.g. operators)	– Significant operation overhead – Steep learning curve (especially for teams)
Lambda	– automatically scale – low ops overhead – pay per use	– limited choices of runtime – subject to latency due to cold start; yet warm start incurs cost – not suitable for long running tasks (batch processing jobs etc)

In summary, the Lambda-based serverless model is good for stateless server-side workload with short response time (<15s), tolerance of cold-start, no need for portability across platforms, and no complex package dependency.

Infrastructure deployment in Terraform 1/2

Wed, 11 Aug 2021 21:44:00 -0400

Terraform is an excellent Infrastructure-as-Code (IaC) tool based on Hashicorp Configuration Language (HCL). Compared to JSON or YAML based declarative templates (e.g. CloudFormation and ARM), HCL is more concise, thanks to the flexibility of HCL. On the other hand, HCL is not as flexible as general purpose languages. For that sake, I see HCL as semi-declarative IaC. This post is my notes about best practices with Terraform development, from the context of AWS, but also applies to other cloud platforms.

Complex Types

There are three primitive types (string, number and bool) that forms collection types and structural types. Here are some common ones:

list: element may repeat, and order is maintained:

["orange", "banana", "orange", "apple"]

set: elements are unique and unordered

["apple", "banana", "orange"]

tuple: each element has its own type

["a", 15, true]

object: defined by a schema with named attributes each with its own type

{
 name = "John"
 age = 52
}

list of object

[
 {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 },
 {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
]

map of string

{
 property = "foo"
 attribute = "bar"
}

map of object

{
 objkey1 = {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 }
 objkey2 = {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
}

Whenever applicable, Terraform converts types implicitly or explicitly. For example, when a list or tuple is converted to set, all elements are converted to string and duplicates are removed. Object and map are very similar. Map of string can be converted to object if the attributes comply with the schema. Additional attributes not in the schema are discarded.

HCL Types is similar to Python

Although being totally different beasts, the complex types between HCL and Python are similar, to the point I suspect the HCL design is influenced by Python. I summarize the similarities as such:

Python Types	list []	tuple ()	set {}	dict {}
Terraform Types	list []	tuple []	set []	map {}
ordered	Y You can access item by index	Y	N you cannot access an item by index or key; however you can loop over all itmes	N key-value pair that allows you to access item by key
changeable (mutable)	Y	N You cannot update, add or remove items	Y Add or remove only. no change to existing elements	Keys must remain unique or the values get overwritten; Values are mutable
allow duplicate elements	Y	Y	N	Keys must be unique; values don’t have to

In Python, list and tuple allow elements of mixed types but in IaC like Terraform we mostly don’t need mixed types. In Terraform, an object is a map without a defined type. In most situations, lists and tuples behave identically, as do maps and objects.

Another area of similarity is with the comprehension of list and dict/maps. In Python for example,

## Supposed you need to create a list:
lst=[]
for x in range(10):
 lst.append(x**2)

print(lst)

### that can be simplified as the following to create the list:
lst = [x**2 for x in range(10)]

## You can even add contidion
even_numbers=[num for num in range(10) if num%2==0]

## You can introduce function calls:
words = ["hello", "world", "python", "list", "comprehension"]
lengths = [len(word) for word in words]

## You can even combine two lists
lst1=[1,2,3,4]
lst2=['a','b','c','d']

pair=[[i,j] for i in lst1 for j in lst2]

print(pair)

## With dict, it's similar
evens={x:x**2 for x in range(10) if x%2==0}
print(evens)

In Terraform, we use similar techniques:

[for s in var.list : upper(s)] # build a tuple/list from a list

[for k, v in var.map : length(k) + length(v)] # build a list from a map

{for s in var.list : s => upper(s)} # build a map from a list

[for s in var.list : upper(s) if s != ""] # build a tuple/list from a list with condition

Note that the documentation of Terraform doesn’t explicitly call them out as comprehensions. However, it’s exactly the same idea as comprehensions in Python. Even the range() function exists both in Python and Terraform.

Modularization

Modules allows you to group related resources together. They can also be re-used and called by other modules. It is fairly straightforward to create a module:

put the resource declarations into a sub-directory
define input and output in the directory

However, the introduction of module complicates the directory structure and variable referencing, which is important to take into account before starting creating modules. This guideline has further discussion about when to create a module. I re-wrote the terraform templates in Orthweb project to leverage modularization wherever possible, but there is still some stand-alone resource (e.g. random_id) not belonging to any module.

To reference resources across modules, you need to import those resources (using data source) from within the module. There are a couple of ways. You may pass the argument of data source as input variable, or you can leverage the filter capability of data source. Let’s look at one example of each mechanism.

In the example below, we import a subnet by subnet id:

data "aws_subnet" "private_subnet" {
 id = var.private_subnet_id
}

In the example below, we import a subnet by filtering from all subnets in the VPC by tag:

data "aws_subnet" "private_subnet" {
 vpc_id = var.vpc_id
 filter {
 name = "tag:Name"
 values = ["Private"]
 }
}

There are pros and cons of each approach. A module with mechanism 1 is more transferrable across different environment, because the ID of subnet is explicitly provided. However, authors needs to manage those explicit variables with code. Mechanism 2 fetches target resources with filter. It depends on a well-implemented tagging policy in the resource farm.

The Terraform Registry (since 2017) contains a lot of pre-built modules for each backend platform (e.g. AWS). If you find any module that can be used in your project, the module repo can be referenced directly by Git repository URL. You should be aware of the risk of this practice though. Many platforms are keen to publish modules for their platform. Anyone can publish their own modules to the community as well.

You can quickly generate module documentation with terraform-docs.

Local Execution

Local execution is the basic workflow mode which is mostly seen with very small collaboration team. In this mode, the developer executes terraform binary (Terraform CLI) from their workstation (e.g. Laptop). The Terraform CLI converts code into API calls to interface cloud provider. The most frequently used commands (from terraform directory) are:

terraform init
terraform plan
terraform apply

The init command initializes the working directory. The plan command figures out the delta between code and infrastructure. It outlines the changes it is about to make. The apply command commits the change. The documentation of Terraform CLI commands is here.

Terraform keeps track of the infrastructure it manages in state file. This article explains the purpose of state. State management collaboration difficult with local execution because the state file by default is created in the working directory on user’s workstation. Although the state file can be configured to be stored in a shared location such as S3, it still requires a mechanism to lock the state in a multi-developer collaboration.

In large operations, the same code base in Terraform, is usually used to created several different sets of infrastructures, for example, in different geographic regions. So it is a 1-to-many relationship between the code repo and the infrastructure state. To further complicate things, each state might have been deployed using different revisions of the code. To overcome that challenge, Terraform introduced the concept of workspace, which is essentially an instance of state describing a particular group of infrastructure being managed by the same source code. When there are many workspaces, it becomes tricky to manage them with CLI commands.

State management is a major challenge that needs to be solved for team collaboration in local execution workflow. Each state must use the same revision of Terraform code. You can use Git in combination as a workaround to that limitation but the point is you cannot tie a workspace to a commit with the workspace commands.

In some enterprise environment, the execution is from a VM (e.g. ADO agent on-premise) without Internet access, which poses another challenge. First, we need to pre-load required providers manually. The enterprise needs a proxy solution to safely download packages from Hashicorp website. One good option is Nexus Repository, with both open-source and pro supports. It is a full-function artifactory repo that can host helm repo, apt repo, yum repo, etc. Second, we also need to configure Terraform so it picks up providers locally. Managing plugins without Internet access requires understanding of the order in which Terraform tries to load plugins during initialization.

Remote Execution

In remote execution, the code is executed in Terraform Enterprise or Terraform Cloud. Both are remote web servers. The difference is that Terraform Enterprise is self-hosted service, requiring IT specialist to install and maintain Terraform Enterprise. Terraform Cloud on the other hand, is a managed SaaS service. The pricing model includes a free plan for small number of users.

Terraform workspace configuration

In Terraform Enterprise or Cloud, the remote execution is organized in workspaces. You need to create an organization, and then create workspace under the organization in order to execute code. With each workflow, you can specify version control system (VCS) and subdirectory, to tell the workspace where to fetch Terraform code from. The workspace also allows you to define secrets and variables specific to the workspace. When you execute a workspace plan, the secrets and variables are passed from workspace to the execution logic.

Terraform Workspace Variable configuration

You will also need to design the Terraform code in a way to work seamlessly with the secrets and variables loaded from the workspace. The variable declaration in code should match the definition in workspace. There are already a number of variables that came in handy. Check out this guide.

Each execution is referred to as a “run”, with its own run id. A workspace involves may runs, which may succeed or fail. Each run pulls a specific commit of the source repository, and goes through stages such as plan, and apply. The UI from each run result list out the status of each result, in a very easy to read format.

The state data is persisted in the web server as they were generated. Therefore the collaborator do not need to worry about managing state with CLI tools. If there are files that you do not want picked up by the execution engine, their locations can be added to a file .terraformignore. Refer to this guide.

AWS profile

Local execution still has a lot of use cases in enterprises such as testing with temporary resources. A common challenge is authentication. As discussed, Terraform CLI picks up identity information from AWS CLI and authenticates its way into the backend to run API calls against. So AWS CLI must be configured correctly with the sufficient permission to provision resources. On the other hand, enterprises usually offload IAM to an identity store, such as AzureAD, Okta, etc. Putting those together, the pattern of authentication and authorization usually looks like this:

User logs on via SSO (e.g. SAML). The validation response gives a name of an IAM role.
Upon successful authentication, user takes the IAM role. The role does not have any capability, except for assuming a second IAM role.
The second IAM role (the functional role) grants user the permission to do its business.

The steps above, can be carried out in AWS console, or with AWS cli using assume-role command. However, when we put Terraform in the picture, it becomes a little involving because the credential information is updated whenever the functional role is assumed, and the assume-role command takes a pretty long argument.

To skip typing the long command every time, there are some handy tools, such as aws-azure-login. An even better tool that works with a variety of identity stores is saml2aws. The tool allows you to configure identity backend, assume the functional role, and update credential information in aws credential file, all with a single command. The AWS CLI configuration reads:

[default]
region = us-east-1
output = json
cli_history = enabled
cli_pager =
role_session_name = functional_operation

[profile function_user]
source_profile = default
role_session_name = functional_operation
role_arn = arn:aws:iam::9998887766:role/admin-access
region = us-east-1

In Terraform provider, we need to tell it to assume that role as well:

provider "aws" {
 region = "us-east-1"
 assume_role {
 role_arn = "arn:aws:iam::9998887766:role/admin-access"
 session_name = "terraform"
 }
}

This will ensure Terraform assumes appropriate role before doing its job.

AWS EC2 SSH Key Pair

RSA key authentication for SSH should be used for Linux Instances. When creating an EC2 instance, we give it our public key so we can then later authenticate through SSH. If the key is already stored in AWS, we just need to tell EC2 the name of the key, in the key_name property. If the code is likely to be executed from several different places by different users, then we can write the code so it picks up public key from user’s workstation (~/.ssh/id_rsa.pub). Here is an example:

variable "local_pubkey_file" {
 type = string
 default = "~/.ssh/id_rsa.pub"
}
data "local_file" "pubkey" {
 filename = pathexpand(var.local_pubkey_file)
}
resource "aws_key_pair" "user-pubkey" {
 key_name = "runner-pubkey"
 public_key = data.local_file.pubkey.content
}
resource "aws_instance" "bastion" {
 instance_type = "t2.micro"
 key_name = aws_key_pair.user-pubkey.key_name
 ......
}

For remote execution, we can even add an option to pass public key in as variable, to override the key file variable. For an example, check out my orthweb project.

To upload files to EC2 instance from Terraform execution environment, we can use the file provisioner with ssh as connection type.

Docker components

Wed, 28 Oct 2020 20:23:00 -0400

The previous post about virtualization and containerization brought up some underlying technologies which Docker build containers on, including:

namespaces – a Linux kernel mechanism to isolate resources. It allows a process to run within an isolated environment (mnt, pid, net, ipt, uts, user, cgroup)
cgroups – a Linux kernel mechanism to limit resource usage of a process or process group
unionFS (this will be further discussed under Docker storage)

In this post we further discuss the components in Docker, the dominant and popular player in container technology, as shown in the diagram below:

The component names can be seen under docker install directory. It consists of three groups:

Docker related: docker, dockerd, docker-init and docker-proxy
Containerd related: containerd, containerd-shim and ctr
Container runtime: runc

Now we discuss each group:

Docker-related components

docker is just an implementation of docker client, it supports commands to achieve all functions between client and server. Alternatively, user may use REST API, or Docker SDK to communicate with Docker server.

dockerd is the server process, to receive requests from docker (client), SDK library or REST API caller. It executes the request and returns status to client. There are three ways for docker (client) to communicate with dockerd.

By Unix Socket (unix://socket_path). The default socket path used by dockerd is /var/run/docker.sock, which is why only root can use docker after installation.
TCP request (tcp://host:port). It is recommended to configure TLS communication in production environment.
By file descriptor (fd://) used in systemd service.

Unix socket is the default communication method. To allow remote access to dockerd, use -H to specify HOST and PORT when starting dockerd.

docker-init is used by Docker as PID 1 process for containers, in case it needs to recycle zombie containers. To use this, specify –init when running container.

docker-proxy is used for port mapping. When you use -p switch with docker run, this docker-proxy is the service that maps the container port to host port. It does so by modifying the iptables nat rule.

Containerd related components

containerd component was separated from dockerd since Docker 1.11, in compliance with OCI standard. It is responsible for life cycle management of containers, it also manages images (e.g. pulling from repo), request from dockerd to call runc, storage and network resources.

dockerd uses UNIX socket to send request to containerd. The default socket path for containerd is /run/containerd/containerd.sock. containerd execute the task and return status to dockerd. You may also directly use containerd to manage containers.

ctr (containderd-ctr) is the client of containerd, mostly used only in development and testing. If the environment does not have dockerd, then you can use ctr as client, to send request directly to containerd.

containerd-shim is used to decouple containerd from the containers. containerd-shim is the parent process of containers. This is so that restarting containerd does not impact the running containers.

Container runtime

runc is a standard implementation of OCI container runtime. It is a command-line tool to create and run containers.

Host legacy application in Docker 2 of 2

Thu, 22 Oct 2020 17:54:00 -0400

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

Application service	TCP port to bind
Business service A	8030
Business service B	8040
Business service C	8050
……	…

TCP port requirement

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
The OS needs to host 6 processes of the same application all running at the same time;
The OS does not allow multiple processes to bind to a single TCP port (duh!);
It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

OS	Container ID	Application Service	container port	published port
Host CentOS	Container 1 (Instance #1)	Business Service A	8030	9301
		Business Service B	8040	9401
		Business Service C	8050	9501
	Container 2 (Instance #2)	Business Service A	8030	9302
		Business Service B	8040	9402
		Business Service C	8050	9502
	Container 3 (Instance #3)	Business Service A	8030	9601
		Business Service B	8040	9602
		Business Service C	8050	9603

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
 dapp1:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp1
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9301:8030 # BUSINESS SERVICE A
 - 9401:8040 # BUSINESS SERVICE B
 - 9501:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:cf
 environment:
 - INSTANCE_TAG=dapp1 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp2:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp2
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9302:8030 # BUSINESS SERVICE A
 - 9402:8040 # BUSINESS SERVICE B
 - 9502:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d0 
 environment:
 - INSTANCE_TAG=dapp2 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp3:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp3
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9601:8030 # BUSINESS SERVICE A
 - 9602:8040 # BUSINESS SERVICE B
 - 9603:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d1 
 environment:
 - INSTANCE_TAG=dapp3 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
networks:
 vcnet:
 driver: bridge
 driver_opts:
 com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
- Initialization work that cannot be done in Dockerfile, such as setting environment variable
- Launch the application, including pointing log file to stdout
- Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
build_image.sh: this script makes the image build process smoother
- check if image to build already exist, and ask permission to delete the existing image if so;
- build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

	Dockerfile instruction for container	Permission issue during mount by docker-compose
Bad config	RUN mkdir -p /var/lib/dapp && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.
Good config	RUN mkdir -p /var/lib/dapp/dcontainer/archive && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue.

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.