Software Development on Digi Hunch

GitHub Action Gotchas

Fri, 27 Jan 2023 01:15:02 -0400

I started with GitHub Actions a couple years ago. Recently I came across a few interesting use cases while I was trying to setup Terraform workflow with GitHub actions. These use cases prompted me to make use some new features in GitHub Action. So I put them in a post here.

Runners can assume IAM Role in AWS

In many scenarios we want to execute AWS CLI command from GitHub action. Also, executables such as terraform inherits credential from AWS CLI. The credential should be a temporary role-based credential instead of an IAM user based on access keys.

There is a GitHub Action called configure-aws-credentials-for-github-actions that can help configure GitHub runner using OIDC identity provider (since Nov 2021 v1.6.0). With the action, the GitHub runner can assume an IAM role as an IAM user (with access key), or using a web identity.

For a GitHub runner to have a web identity thereby assume an IAM role, we should configure OIDC provider in AWS. We can do that from AWS console (i.e. under IAM), or using CloudFormation code. Below is a snippet as an example:

Resources:
 GitHubOIDC:
 Type: AWS::IAM::OIDCProvider
 Properties:
 Url: https://token.actions.githubusercontent.com
 ClientIdList: 
 - sts.amazonaws.com
 ThumbprintList:
 - 6938fd4d98bab03faadb97b34396831e3780aea1

Then from the configured OIDC provider, we can obtain a thumbprint. GitHub action gives the thumbprint here. In AWS, we configure an IAM role whose AssumeRolePolicyDocument will reference the thumbprint. Here is an example. In the condition section of AssumeRolePolicyDocument, we can also specify a specific GitHub repository so that only Actions from that repository can assume the IAM role with their web identities.

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1-node16
 with:
 role-to-assume: ${{ vars.IAM_ROLE_ARN }}
 aws-region: ${{ vars.AWS_REGION }}

This way, we map a GitHub runner’s web identity to an IAM role with a step using the Action above. We also filter what GitHub org and what repo can trigger actions that assumes the role, by the condition clause in the role statement. If the step fails, we can look at CloudTrail on the AWS side for causes. Look for entries with AssumeRoleWithWebIdentity as Event Name.

Reusable workflows

For better reusability of Action steps, GitHub introduced reusable workflows (generally available since Nov 2021). It is particularly helpful when we need to run a workflow for different environments. The reusable workflows files (YML) can be placed in separate repositories, and be reference as such. This allows enterprises to centralize the management of reusable workflows.

I have been using the act project to emulate GitHub action locally on MacBook. As of Jan 2023, act does not support reusable workflow.

With the split between caller and reusable workflows, we have a new challenge of passing secrets and variables between them. It is not straightforward and GitHub documentation needs improvement to get the documentation clear. Also because the word “environment” is used in different contexts, it is ambiguous and therefore difficult to Google relevant information.

Passing variables

First, there are several types of variables in GitHub action:

Environment variable: declared under env keyword in a workflow. To use environment variable, use the env context. For example: ${{ env.MY_VARIABLE }}
Configuration variable: introduced in Jan 2023, configuration variables are defined at repository, environment and organization levels. To use configuration variable, use vars context, and ensure the workflow job specifies a value for environment attribute.
Secrets: GitHub also calls it Environment secret when defined at environment level. It works the same way as a configuration variable because it is also specific to an environment. The content is not viewable once set.

The reason GitHub action makes this so confusing, is that on one page, its documentation distinguishes between environment variable and configuration variable:

On another page, the document refers to configuration variables at environment level as environment variable:

It seems that “configuration variable” is too new for GitHub to refine its documentation as of January. This semantical confusion gave me a hard time investigating how to pass “Environment variable” to reusable workflows. I will stick to the meaning on the first page to distinguish environment variable and configuration variable at environment level.

Passing environment variable isn’t straightforward. In this discussion thread, people discussed how inconvenient it is. I used the workaround in this comment, where I had to create a job for the sake of storing variable values to output.

Pass secret is easier. This is an insightful blog post (Dec 2021) about passing secret to reusable workflow. The attempt 3 in the post works for me. First, we pass the value of environment to the reusable workflow as an input, then at job level specify the environment with the value. Then in the jobs we can reference secrets as ${{ secrets.NAME }}. The job will pick up the secret based on the correct environment.

It appears that since May 2022, GitHub introduced secrets: inherit keyword to address this. However, the method above still works for configuration variable.

Authentication of GitHub Actions

By default, a GitHub action can access the code repository that triggers the action and no other repositories (with GITHUB_TOKEN). However, in many cases we need to access external repositories. For example, terraform init command from a GitHub action implicitly calls git clone to pull module code from external repositories. A GitHub workflow may also reference a workflow file from external repositories.

The question is how to authenticate GitHub workflow to access external repo. This post has a thorough discussion. We may create a Personal Access Token and pass it to set-git-credentials action. We are essentially sharing a personal credential (and repo access) with a GitHub action, which is not a good practice.

The proper way to solve this problem, is to create a separate GitHub App and grant the access only the repo that the workflow needs to access. The GitHub App will generate a private key. Then we supply the private key to workflow-application-token-action so the workflow can act as the GitHub App, thereby access the external repos. The post has more details in the GitHub App section. Suppose we have terraform get command to clone external repo, the actions may look like this:

 - name: HashiCorp - Setup Terraform
 uses: hashicorp/setup-terraform@v2

 - name: Get RepoReader App Token
 id: get_repo_reader_token
 uses: peter-murray/workflow-application-token-action@v2
 with:
 application_id: ${{ vars.REPO_READER_APPLICATION_ID }}
 application_private_key: ${{ secrets.REPO_READER_PRIVATE_KEY }}

 - name: Cache Git Creds
 uses: de-vri-es/setup-git-credentials@v2
 with:
 credentials: https://x-access-token:${{ steps.get_repo_reader_token.outputs.token }}@github.com/

 # Terraform Get implicitly calls git clone which uses the credential cached as above
 - name: Terraform Get 
 run: terraform get

Another benefit of using GitHub App is that the token is a short-lived credential that expires as the job is finished, whereas a PAT will expire on a preset date. In this use case we can think of GitHub App as a service account with minimized privilege to read a short list of repos.

Final words

I came across this reddit post against pipeline use and there are some legit points. For example, the user is frustrated with the limitation with variable passing and unclear documentation. The post wasn’t specific to GitHub action but I do share some frustration while working with GitHub Actions.

On a positive note, since I first used GitHub actions, it has evolved quite a bit with new features, although the documentation is somewhat lagging. It is still very helpful as all of these are free to personal use. I look forward to more interesting features.

Git Branching Strategy

Sun, 07 Mar 2021 19:20:42 -0400

I have been in two discussions about Git branching strategy in different organizations. Too many concepts! So I open this post to jot down the lineage of common branching strategies to help organizations develop their branching policies. In terms of reference, there is a lot from Atlassian documentation. In addition, I also find this one article a good resource.

Centralized workflow (no branching)

In Centralized Workflow, the team uses a central repository to serve as the single-point-of-entry for all changes to the project. The default branch is master, and all changes are committed to this branch. This workflow does not require any other branches beside master. Local changes may conflict with upstream commits, and conflict needs to be resolved.

This workflow is usually seen in teams transitioning from SVN, with very basic skill level. This workflow may also be adopted in teams working on configuration management instead of source code. Centralized workflow is great for small teams. The conflict resolution process can form a bottleneck as the team scales in size.

Feature Branch Workflow

Instead of directly committing on their local master branch, developers create a new branch every time they start work on a new feature. Feature branches should have descriptive names (e.g. issue#112). Feature branches are pushed to the central repository so that they can be shared to other developers without touching any official (master) code.

To get feedback on the new feature branch, create a pull request in a repository management solution (e.g. Bitbucket Cloud, Bitbucket Server). Before merge, you may have to resolve merge conflicts if others have made changes to the master branch of repo. This is to make sure your local master is synchronized with the upstream master. When your pull request is approved and conflict free, you can merge your branch to master branch.

feature branch workflow

The Git Feature Branch Workflow is a composable workflow that can be leveraged by other high-level Git workflows. Git Feature Branch Workflow is branching model focused, instead of release focused. The Git Feature Branch Workflow can be incorporated into other workflows. The Gitflow, and Git Forking Workflows traditionally use a Git Feature Branch Workflow in regards to their branching models.

Gitflow Workflow

First published in 2010 by Vincent Driessen. Gitflow defines a strict branching model designed around the project release. This provides a robust framework for managing larger projects. In addition to Feature Branch Workflow, Gitflow workflow assigns very specific roles to different branches and defines how and when they should interact.

Instead of a single master branch, this workflow uses two branches to record the history of the project. The master branch stores the official release history, and the develop branch serves as an integration branch for features. It is also convenient to tag all commits in the master branch with a version number.

This workflow is operated in the following ways:

A develop branch is created from master
Feature branches are created from develop. When a feature is complete, with PR reviewed, it is merged into the develop branch. Features branches
Once develop has acquired enough features for a release (or a predetermined release date is approaching), we fork a release branch off of develop. Creating this branch starts the next release cycle, so new features can be added to develop after this point. On the release branch itself, only bug fixes, documentation generation, and other release-oriented tasks should go in this branch. Once ready to ship, the release branch gets merged into master and tagged with a version number. In addition, it should also be merged back into develop, which may have progressed since the release was initiated.
Maintenance or hotfix branches are used to quicly patch production releases. Hotfix branches are a lot like release branchs and feature branches except they’re based on master instead of develop. As soon as the fix is complete, it should be merged into both master and develop (or the current release branch), and master should be tagged with an updated version number.

Gitflow workflow

There is a Git extension named git-flow to provide high-level repository operations for this Workflow, such as start a release, finish a release, start a hotfix, finish a hotfix.

Gitflow is ideally suited for projects that have a scheduled release cycle and for the DevOps best practice of continuous delivery. It ensures that the master branch reflects what is deployed (e.g. in production). However, it is quite complex and have a steep learning curve for organizations. It also runs long-lived branches, which is considered bad from CI/CD perspective. Branches are by definition to isolate and hide changes, whereas continuous integration is about exposing changes early on and frequently. In that sense, the Gitflow branching model and CI/CD are mutually exclusive ideas.

In 2020, Vincent Driessen added a note at the beginning of his article on Gitflow:
If your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team.
If, however, you are building software that is explicitly versioned, or if you need to support multiple versions of your software in the wild, then git-flow may still be as good of a fit to your team as it has been to people in the last 10 years.

Driessen’s notes also points to some simple alternatives. On the other hand, the Atlassian tutorial on Gitflow has described Gitflow workflow as a legacy (since Aug 2021 based on web archive). It points out at the beginning that Gitflow has fallen in popularity in favor of trunk-based workflows.

GitHub flow, Trunk-based development (TBD) and GitLab flow

It is operationally expensive to manage multiple mainlines in Git flow workflow, with both source control and release in the picture. Some simple alternatives have been evolved, with single mainline, for example, GitHub flow and trunk-based development. They differ in where the release is performed from.

In the GitHub flow, release is performed from branch before being merged back to master (trunk).

GitHub flow

In trunk-based development, release is not performed until the feature branch has been merged to the trunk (master). In trunk-based development, feature branches are supposed to be short-lived. It is a common practice among DevOps teams, since it streamlines merging and integration phases.

Trunk-based development

Trunk-based development has gained some momentum in recent years, due to its DevOps friendliness. This is a website that advocates it and here’s a DZone article about it.

Gitlab Flow

In response to the shortcomings of GitHub flow and Gitflow, Gitlab introduced its own proposal of branching strategy, known as Gitlab flow. The most distinctive aspect is the environment branches. In Gitlab flow, you run multiple long-lived branches, each of them representing an environment. The typical steps are as follows:

You create short-lived feature branches, and merge them often to the master.
Every developer starts from master and targets master. Other branches are merged from previous lower environment branches.
You can deploy a new version to production, by merging master into the production branch.
If you need to know what code is in production, you can check out the production branch to see.

Gitlab flow

You only need to work with release branches, if you need to release software code to the outside world. Here are some best practices in GitLab flow.

Forking Workflow

Forking workflow is fundamentally different. The key steps are as follows:

A developer ‘forks’ an ‘official’ server-side repository. This creates their own server-side copy. This is their personal public repository, and no other developer are allowed to push to it.
The new server-side copy is cloned to their local system. This forms an environment dedicated to this developer.
With the local clone, developer needs to create the upstream remote manually using “git remote add upstream” command. This allows the developer keep the local repository up-to-date as the official project progresses.
A new local feature branch is created. Developer commits to the new local branch, and pushes to their own copy of repository on server.
Developer files a pull request from the new branch (in own copy of repository) to the ‘official’ repository. The project maintainer knows that an update is ready to be integrated. The PR also serves as a discussion thread.
The PR gets approved for merge and is merged into the original server-side repository.

This workflow has other names, such as fork-and-branch workflow, and is commonly used in GitHub for managing open-source projects. However, this should not be confused with the aforementioned GitHub flow.

Conclusion

In order to fully support distributed source control, Git abstract version control problems into concepts such as commit, branch, etc. This makes discussion about Git workflow and branching strategy difficult due to the conceptual hurdles and organization differences. We covered choices of Git branching strategy in this post. As Vincent Driessen commented in his original Gitflow posting, panaceas don’t exist. We should consider the context (e.g. team size, Git skill level, etc) to determine the best Git branching strategy.

Docker components

Wed, 28 Oct 2020 20:23:00 -0400

The previous post about virtualization and containerization brought up some underlying technologies which Docker build containers on, including:

namespaces – a Linux kernel mechanism to isolate resources. It allows a process to run within an isolated environment (mnt, pid, net, ipt, uts, user, cgroup)
cgroups – a Linux kernel mechanism to limit resource usage of a process or process group
unionFS (this will be further discussed under Docker storage)

In this post we further discuss the components in Docker, the dominant and popular player in container technology, as shown in the diagram below:

The component names can be seen under docker install directory. It consists of three groups:

Docker related: docker, dockerd, docker-init and docker-proxy
Containerd related: containerd, containerd-shim and ctr
Container runtime: runc

Now we discuss each group:

Docker-related components

docker is just an implementation of docker client, it supports commands to achieve all functions between client and server. Alternatively, user may use REST API, or Docker SDK to communicate with Docker server.

dockerd is the server process, to receive requests from docker (client), SDK library or REST API caller. It executes the request and returns status to client. There are three ways for docker (client) to communicate with dockerd.

By Unix Socket (unix://socket_path). The default socket path used by dockerd is /var/run/docker.sock, which is why only root can use docker after installation.
TCP request (tcp://host:port). It is recommended to configure TLS communication in production environment.
By file descriptor (fd://) used in systemd service.

Unix socket is the default communication method. To allow remote access to dockerd, use -H to specify HOST and PORT when starting dockerd.

docker-init is used by Docker as PID 1 process for containers, in case it needs to recycle zombie containers. To use this, specify –init when running container.

docker-proxy is used for port mapping. When you use -p switch with docker run, this docker-proxy is the service that maps the container port to host port. It does so by modifying the iptables nat rule.

Containerd related components

containerd component was separated from dockerd since Docker 1.11, in compliance with OCI standard. It is responsible for life cycle management of containers, it also manages images (e.g. pulling from repo), request from dockerd to call runc, storage and network resources.

dockerd uses UNIX socket to send request to containerd. The default socket path for containerd is /run/containerd/containerd.sock. containerd execute the task and return status to dockerd. You may also directly use containerd to manage containers.

ctr (containderd-ctr) is the client of containerd, mostly used only in development and testing. If the environment does not have dockerd, then you can use ctr as client, to send request directly to containerd.

containerd-shim is used to decouple containerd from the containers. containerd-shim is the parent process of containers. This is so that restarting containerd does not impact the running containers.

Container runtime

runc is a standard implementation of OCI container runtime. It is a command-line tool to create and run containers.

Host legacy application in Docker 2 of 2

Thu, 22 Oct 2020 17:54:00 -0400

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

Application service	TCP port to bind
Business service A	8030
Business service B	8040
Business service C	8050
……	…

TCP port requirement

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
The OS needs to host 6 processes of the same application all running at the same time;
The OS does not allow multiple processes to bind to a single TCP port (duh!);
It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

OS	Container ID	Application Service	container port	published port
Host CentOS	Container 1 (Instance #1)	Business Service A	8030	9301
		Business Service B	8040	9401
		Business Service C	8050	9501
	Container 2 (Instance #2)	Business Service A	8030	9302
		Business Service B	8040	9402
		Business Service C	8050	9502
	Container 3 (Instance #3)	Business Service A	8030	9601
		Business Service B	8040	9602
		Business Service C	8050	9603

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
 dapp1:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp1
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9301:8030 # BUSINESS SERVICE A
 - 9401:8040 # BUSINESS SERVICE B
 - 9501:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:cf
 environment:
 - INSTANCE_TAG=dapp1 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp2:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp2
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9302:8030 # BUSINESS SERVICE A
 - 9402:8040 # BUSINESS SERVICE B
 - 9502:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d0 
 environment:
 - INSTANCE_TAG=dapp2 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp3:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp3
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9601:8030 # BUSINESS SERVICE A
 - 9602:8040 # BUSINESS SERVICE B
 - 9603:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d1 
 environment:
 - INSTANCE_TAG=dapp3 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
networks:
 vcnet:
 driver: bridge
 driver_opts:
 com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
- Initialization work that cannot be done in Dockerfile, such as setting environment variable
- Launch the application, including pointing log file to stdout
- Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
build_image.sh: this script makes the image build process smoother
- check if image to build already exist, and ask permission to delete the existing image if so;
- build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

	Dockerfile instruction for container	Permission issue during mount by docker-compose
Bad config	RUN mkdir -p /var/lib/dapp && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.
Good config	RUN mkdir -p /var/lib/dapp/dcontainer/archive && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue.

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.

Automated Deployment Pipeline 3 of 3

Wed, 14 Oct 2020 17:27:00 -0400

Background

We have previously covered a pipeline example with Jenkins calling Ansible to leverage OpenSSH configuration and Ansible inventory. We also discussed a use case with declarative pipeline.

In this posting, I will provide another advanced example, built on declarative pipeline. The pipeline file will be pulled from Git repository. Also, the script is executed on a remote agent, instead of the Jenkins master server. The reason this example is important, is that:

Jenkinsfile is now version controlled (pipeline as code);
Service script (e.g. python) is also version controlled from a central repository;
Computing resource is provided by a remote agent. Since the script is pulled before running, the agent is still fungible;
The result from service script execution is archived (similar to the way build artifact is stored in Jenkins) in master.

The architecture of this pipeline now becomes the followings:

Configure Agent

Jenkins has a plugin called SSH Build Agent, that allows you to configure a Linux agent, communicating with Jenkins master in SSH. For Window agent, it uses JNLP to communicate with master, which is outside of our scope of discussion. As I touched on in previous post, Jenkins uses its own implementation of SSH protocol to achieve this. This means that it cannot re-use the configurations in ~/.ssh/config and thus the ability to do SSH chaining is eliminated. This is incompatible with our architecture so I have to register Jenkins agent using a different launch method “Launch agent via execution of command on the master”. The execution of command on master can still leverage OpenSSH config file. In order to do so, we must copy the agent.jar file to the remote agent first (URL is ${JENKINS_URL}/jnlpJars/agent.jar). Then use SSH command to call the jar file from agent (aka slave) machine. You may add some java argument for troubleshooting. Below is an example configuration for the node.

I shall also note that if the remote agent is a different operating system where the path of bash might be different, then you need to include the directory of bash executable in PATH environment variable. That can be done as in the screenshot above (PATH=${PATH:/usr/bin}). If this is incorrect, you might run into issues when running sh step in Jenkins pipeline. Here is an article about this. To translate that page, the symptom of this issue includes:

Error from pipeline execution that says the following, which is very generic:

process apparently never started in /home/dhunch/jenkins/workspace/site-remote-job@tmp/durable-b997d26c 
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

Job status shows:

hudson.AbortException: script returned exit code -2
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:659)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:605)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:549)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Threaddump (only available during execution) shows:

Thread #6
 at DSL.sh(awaiting process completion in /home/dhunch/jenkins/workspace/site-remote-job/durable-b997d26c; recurrence period: 9543ms; check task scheduled; cancelled? false done? false)
 at WorkflowScript.run(WorkflowScript:9)
 at DSL.script(Native Method)

This issue is tricky because none of the symptoms above make you think about the environment variable defined for the agent, and that the issue does not occur until you actually execute a Jenkins pipeline, usually well after the node agent is registered, and only impacts shell step (sh). It is recommended to test this with sh steps in pipeline.

Configure Repository

Our example job executes service script pulled from SCM, on a remote agent. Then the master pulls the result file to itself. The git repository can be set in Jenkins job, where you specify that the script named Jenkinsfile (from the repo) is the pipeline file that needs to be executed.

This is fairly simple, but what makes the situation more complex is the following few requirements:

it’s the agent node that needs git (installed) and clone to repo;
the master does not need (and should not attempt to) clone from repo;
the master needs to do its job without having to pull SCM;

We need to be able to specify whether each step needs to pull from SCM. The following pipeline syntax shows how this is done:

def agent_dir = 'initial_value'
pipeline {
 agent none
 options { skipDefaultCheckout(true) }
 stages {
 stage('Execute Job') {
 agent {
 label 'remote-agent-customer1'
 }
 options { skipDefaultCheckout(false) }
 steps {
 echo 'Executing job on node'
 sh 'whoami && pwd'
 script {
 agent_dir = sh(returnStdout: true, script: 'echo -n ${WORKSPACE}')
 // the variable should not include carriage return
 }
 sh "echo ${agent_dir}"
 }
 }
 stage('Pull Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 echo 'Pulling job below'
 sh "echo ${agent_dir}"
 sh "scp dhunch@site1:\"${agent_dir}\"/result.csv ./"
 }
 }
 }
}

In this example pipeline file, we declare agent none for the pipeline, then an agent for each specific stage. we also specify the option skipDefaultCheckout as true at the step where pulling from SCM is not needed. This allows us to finish job with multiple agent, and only pull from SCM as needed. This snippet also exemplifies how to declare a variable, assign it from stdout from one agent, and persist the value across ensuing stages.

Store Result

The reason we run pipeline jobs on this remote agent is because it is sitting in customer network and has local direct access to data. So this is perfect for situation such as data analytical jobs, which access to database on local network and store result. We need to pull the result file back to agent and make it available on Jenkins. This is so similar to “archive artifact” task (commonly seen in CI process) that we can simply use its plugin to achieve what we need. Before archive artifact, we need to pull it to local (master), as shown in the example code above. After that, we need another stage to archive the result.

 stage('Archive Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 archiveArtifacts artifacts: '*.csv',onlyIfSuccessful: true,fingerprint: true
 }
 }

I believe there is plugins to compress artifacts as well.

Conclusion

At the end I’d like to reiterate my perception about Jenkins. It is a very generic and adaptive automation platform that originally evolved from use cases in build automation. Due to this original root, many components in Jenkins are named around Continuous Integration use cases, such as the “build” button, and the “archiveArtifacts” step. These misnomers underplays what Jenkins can potentially do in continuous deployment or other automation scenarios. It is important for automation engineers to understand Jenkins components and plugins, through their functionalities and not by the name, and therefore make creative use of Jenkins as automation engine in all scenarios.

An alternative to this proposed pipeline would be Ansible Tower, a commercial project based on open-source Ansible, but with nice UI support. Ansible Tower is Ansible oriented, and it does not have everything that Jenkins can do. It should still be a decent alternative given the proposes pipeline uses Ansible a lot.

Automated Deployment Pipeline 2 of 3

Tue, 06 Oct 2020 22:05:00 -0400

In this posting, we continue to discuss Jenkins’ ability to automate deployment routines. Jenkins supports freestyle project out of the box, as well as Pipeline with several plugins. Freestyle project allows user to specify multiple steps on UI. This does not scale well when your entire process involves many steps. As explained on Jenkins’ website:

While standard Jenkins “freestyle” jobs support simple continuous integration by allowing you to define sequential tasks in an application lifecycle, they do not create a persistent record of execution, enable one script to address all the steps in a complex workflow, or confer the other advantages of pipelines.
In contrast to freestyle jobs, pipelines enable you to define the whole application lifecycle. Pipeline functionality helps Jenkins to support continuous delivery (CD). The Pipeline plugin was built with requirements for a flexible, extensible, and script-based CD workflow capability in mind.

So although freestyle projects are easy to set up, and can do technically everything that Jenkins pipeline can do, the major advantage of Jenkins Pipeline is the ability to manage multiple-step as code, and version control the pipeline-as-code. Here is some more information.

We will start with a freestyle project to understand Jenkins’ ability and then advance to building pipelines.

Freestyle projects

As mentioned, Jenkins calls a task a “build”, and the build can be triggered in a variety of ways:

Time schedule (with or without parameter)
Remotely via API
On completion of other projects
Poll SCM for changes
Commit to SCM

There are also several ways to execute a job:

Execute Shell command, batch command or groovy script
Invoke Ansible adhoc command, playbook or vault
Conditional on specified boolean value, file existence, etc

After the job one can specify post-build jobs, for example:

start other build projects
notification of various means
publishing result file

In deployment, it is common task to execute a task over SSH, using SSH command or invoke Ansible command with Ansible plugin. The former fits simple command line tasks. For example:

The Ansible plugin is good for more steps and more complicated inventory hierarchies. This post includes an example of an inventory involving multiple layers. The other limitation with SSH command is lack of a straightforward configuration to escalate privilege and run remote command. On the other hand, Ansible addressed this with become method. Below is a screenshot

In order to use Ansible command, you will also need to specify where the Ansible binary in Global Tool Configuration. You will also need to store vault credential in Jenkins credential store so it’s not being prompted.

Jenkins Pipeline

Jenkins pipeline allows one to describe actions in a pipeline in groovy Domain Specific Language (DSL). There are two styles of pipelines:

Declarative pipeline is identified by a block named ‘pipeline’, it is relatively new and supports the pipeline-as-code concept. It can be stored as Jenkinsfile in code repository or edited in Jenkins’ UI.
Scripted pipeline is identified by a block named ‘node’, it is the conventional format and can only be edited in Jenkins’ UI.

To fully understand the two styles of pipelines you also need to know declarative programming and imperative programming. Read this instruction for a better explanation. Here is another good one that focus on their differences.

Jenkins’ plugin provides UI components for user to input information (e.g. Invoke Ansible Command), now with Jenkins pipelines, plugins can provide step functions in order to help Jenkins user. This page from Jenkins lists the most common ones, many of which requires plugin installation.

One example of step function is SSH Pipeline Steps. It allows one to issue SSH connection from Jenkinsfile. However, in our deployment scenarios, it has some limitations:

Unable to match a group of host by specific pattern
Cannot use host name aliases
SSH tunneling is not supported
No means of privilege escalation

Out of these 1 through 3 are due to the fact that SSH step function does not use the OpenSSH configuration on the machine. The jenkins-ssh-slaves plugin uses trilead SSH2 implementation written in Java. Only OpenSSH implementation uses ~/.ssh/config file. This creates problem whenever SSH tunneling is needed. For example, to register a remote slave node via SSH tunneling, we need to select “Launch agent via execution of command on the master” instead of “Launch agent via SSH” as launch method. Here is an instruction and below is what it looks like:

Similarly, if we use ssh-agent plugin as tool for deployment, we cannot use any configuration made by OpenSSH. Therefore Ansible in Jenkins Pipeline is a better tool for deployment because it can use OpenSSH.

Pipelines with Ansible

In this section I demonstrate the use of Ansible playbook and adhoc command in pipeline through two examples.

In example 1, the job pulls authorized key file from SCM, and then use a playbook from SCM, to push the key file to all servers in the specified inventory.

Here is the playbook file for example 1:

---
# Example:
# ansible-playbook -l all -i ~/ansible/inventories/bh.yml push-key.yml --ask-vault-pass
- name: push key to target
 hosts: all
 tasks:
 - name: sync key to host
 copy:
 src: "{{authorized_keys_src}}"
 dest: "/home/dhunch/.ssh/authorized_keys"
 force: yes
 mode: 0600

Here is the pipeline script (note ampersand is mistakenly displayed as & in the box below):

import java.net.URLEncoder;
 
pipeline {
 agent any
 
 options {
 skipDefaultCheckout(true)
 }
 environment {
 BITBUCKET_CREDS = credentials('bitbucket') 
 //BITBUCKET_CREDS_USR and BITBUCKET_CREDS_PSW are set
 BITBUCKET_CREDS_USR = "${BITBUCKET_CREDS_USR}"
 BITBUCKET_CREDS_PSW = URLEncoder.encode("${BITBUCKET_CREDS_PSW}", "UTF-8") 
 // if password contains special character we need to url encode it. e.g. @->%40
 }
 stages {
 stage('Start') {
 steps {
 echo "Starting pipeline ..."
 deleteDir()
 }
 }
 stage('Prep') {
 steps{
 wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: "${BITBUCKET_CREDS_PSW}", var: 'RANDOM']]]) {
 // MaskPasswordsBuildWrapper requires Mask Passwords Plugin and is to mask specific string in console output. 
 // Otherwise BITBUCKET_CREDS_PSW will display in the clear
 sh "git init && git config core.sparsecheckout true"
 sh "git remote add origin https://$BITBUCKET_CREDS_USR:${BITBUCKET_CREDS_PSW}@bitbucket.org/vendorcompoany/configmanagerepo.git"
 sh "echo 'public_keys/*' >> .git/info/sparse-checkout"
 echo "Downloading key file"
 sh "git pull --depth=1 origin master" 
 }
 }
 }
 
 stage('Deploy to Site 1') {
 steps {
 echo '> Deploying to Site 1 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site1.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 1 ...'
 }
 }
 
 stage('Deploy to Site 2') {
 steps {
 echo '> Deploying to Site 2 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site2.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 2 ...'
 }
 }
 
 stage('Deploy to Site 3') {
 steps {
 echo '> Deploying to Site 3 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site3.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 3 ...'
 }
 }
 }
 post {
 always {
 cleanWs()
 }
 }
}

In this example, we pull a sub-directory from git repo. We use two tricks to minimize amount of traffic. First, we use git pull with depth=1 so only the required recent commits are pulled, not the entire history. Second we use sparse checkout to get result from a sub-directory, not the entire repo.

When connecting to repo, password is required. We pull the credential from Jenkins’ credential store and they are masked by default. However, the password must be converted to URL string when used in git remote add origin. Otherwise if the password contains special character the URL will not work.

This modification to password brings about another challenge, password masking during the job execution. We will have to build our own wrapper function using class MaskPasswordsBuildWrapper in order to mask any variable.

In example 2, we wrap an Ansible step function to call adhoc command to check version on all hosts. Here is the pipeline script:

def HunchVersionQuery(pattern, siteinventory){
 step([
 $class: 'AnsibleAdHocCommandBuilder',
 ansibleName: 'Ansible on Mac',
 inventory: [$class: 'InventoryPath',path: siteinventory],
 hostPattern: pattern,
 module: 'shell',
 command: 'cat /etc/*release',
 forks: 1,
 vaultCredentialsId: 'ansible-vault-pass'
 ]);
}

pipeline {
 agent any

 stages {
 stage('Start') {
 steps {
 echo 'Staring Pipeline'
 }
 }
 stage('Query Site1') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site1.yml')
 }
 }
 stage('Query Site2') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site2.yml')
 }
 }
 stage('Query Site3') {
 steps {
HunchVersionQuery('*app','~/ansible/inventories/site3.yml')
 }
 }
 }
}

Because Ansible plugin does not provide a warpper function for adhoc command step, we will have to build our own wrapper function in the above code.