git on Digi Hunch

GitHub Action Gotchas

Fri, 27 Jan 2023 01:15:02 -0400

I started with GitHub Actions a couple years ago. Recently I came across a few interesting use cases while I was trying to setup Terraform workflow with GitHub actions. These use cases prompted me to make use some new features in GitHub Action. So I put them in a post here.

Runners can assume IAM Role in AWS

In many scenarios we want to execute AWS CLI command from GitHub action. Also, executables such as terraform inherits credential from AWS CLI. The credential should be a temporary role-based credential instead of an IAM user based on access keys.

There is a GitHub Action called configure-aws-credentials-for-github-actions that can help configure GitHub runner using OIDC identity provider (since Nov 2021 v1.6.0). With the action, the GitHub runner can assume an IAM role as an IAM user (with access key), or using a web identity.

For a GitHub runner to have a web identity thereby assume an IAM role, we should configure OIDC provider in AWS. We can do that from AWS console (i.e. under IAM), or using CloudFormation code. Below is a snippet as an example:

Resources:
 GitHubOIDC:
 Type: AWS::IAM::OIDCProvider
 Properties:
 Url: https://token.actions.githubusercontent.com
 ClientIdList: 
 - sts.amazonaws.com
 ThumbprintList:
 - 6938fd4d98bab03faadb97b34396831e3780aea1

Then from the configured OIDC provider, we can obtain a thumbprint. GitHub action gives the thumbprint here. In AWS, we configure an IAM role whose AssumeRolePolicyDocument will reference the thumbprint. Here is an example. In the condition section of AssumeRolePolicyDocument, we can also specify a specific GitHub repository so that only Actions from that repository can assume the IAM role with their web identities.

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1-node16
 with:
 role-to-assume: ${{ vars.IAM_ROLE_ARN }}
 aws-region: ${{ vars.AWS_REGION }}

This way, we map a GitHub runner’s web identity to an IAM role with a step using the Action above. We also filter what GitHub org and what repo can trigger actions that assumes the role, by the condition clause in the role statement. If the step fails, we can look at CloudTrail on the AWS side for causes. Look for entries with AssumeRoleWithWebIdentity as Event Name.

Reusable workflows

For better reusability of Action steps, GitHub introduced reusable workflows (generally available since Nov 2021). It is particularly helpful when we need to run a workflow for different environments. The reusable workflows files (YML) can be placed in separate repositories, and be reference as such. This allows enterprises to centralize the management of reusable workflows.

I have been using the act project to emulate GitHub action locally on MacBook. As of Jan 2023, act does not support reusable workflow.

With the split between caller and reusable workflows, we have a new challenge of passing secrets and variables between them. It is not straightforward and GitHub documentation needs improvement to get the documentation clear. Also because the word “environment” is used in different contexts, it is ambiguous and therefore difficult to Google relevant information.

Passing variables

First, there are several types of variables in GitHub action:

Environment variable: declared under env keyword in a workflow. To use environment variable, use the env context. For example: ${{ env.MY_VARIABLE }}
Configuration variable: introduced in Jan 2023, configuration variables are defined at repository, environment and organization levels. To use configuration variable, use vars context, and ensure the workflow job specifies a value for environment attribute.
Secrets: GitHub also calls it Environment secret when defined at environment level. It works the same way as a configuration variable because it is also specific to an environment. The content is not viewable once set.

The reason GitHub action makes this so confusing, is that on one page, its documentation distinguishes between environment variable and configuration variable:

On another page, the document refers to configuration variables at environment level as environment variable:

It seems that “configuration variable” is too new for GitHub to refine its documentation as of January. This semantical confusion gave me a hard time investigating how to pass “Environment variable” to reusable workflows. I will stick to the meaning on the first page to distinguish environment variable and configuration variable at environment level.

Passing environment variable isn’t straightforward. In this discussion thread, people discussed how inconvenient it is. I used the workaround in this comment, where I had to create a job for the sake of storing variable values to output.

Pass secret is easier. This is an insightful blog post (Dec 2021) about passing secret to reusable workflow. The attempt 3 in the post works for me. First, we pass the value of environment to the reusable workflow as an input, then at job level specify the environment with the value. Then in the jobs we can reference secrets as ${{ secrets.NAME }}. The job will pick up the secret based on the correct environment.

It appears that since May 2022, GitHub introduced secrets: inherit keyword to address this. However, the method above still works for configuration variable.

Authentication of GitHub Actions

By default, a GitHub action can access the code repository that triggers the action and no other repositories (with GITHUB_TOKEN). However, in many cases we need to access external repositories. For example, terraform init command from a GitHub action implicitly calls git clone to pull module code from external repositories. A GitHub workflow may also reference a workflow file from external repositories.

The question is how to authenticate GitHub workflow to access external repo. This post has a thorough discussion. We may create a Personal Access Token and pass it to set-git-credentials action. We are essentially sharing a personal credential (and repo access) with a GitHub action, which is not a good practice.

The proper way to solve this problem, is to create a separate GitHub App and grant the access only the repo that the workflow needs to access. The GitHub App will generate a private key. Then we supply the private key to workflow-application-token-action so the workflow can act as the GitHub App, thereby access the external repos. The post has more details in the GitHub App section. Suppose we have terraform get command to clone external repo, the actions may look like this:

 - name: HashiCorp - Setup Terraform
 uses: hashicorp/setup-terraform@v2

 - name: Get RepoReader App Token
 id: get_repo_reader_token
 uses: peter-murray/workflow-application-token-action@v2
 with:
 application_id: ${{ vars.REPO_READER_APPLICATION_ID }}
 application_private_key: ${{ secrets.REPO_READER_PRIVATE_KEY }}

 - name: Cache Git Creds
 uses: de-vri-es/setup-git-credentials@v2
 with:
 credentials: https://x-access-token:${{ steps.get_repo_reader_token.outputs.token }}@github.com/

 # Terraform Get implicitly calls git clone which uses the credential cached as above
 - name: Terraform Get 
 run: terraform get

Another benefit of using GitHub App is that the token is a short-lived credential that expires as the job is finished, whereas a PAT will expire on a preset date. In this use case we can think of GitHub App as a service account with minimized privilege to read a short list of repos.

Final words

I came across this reddit post against pipeline use and there are some legit points. For example, the user is frustrated with the limitation with variable passing and unclear documentation. The post wasn’t specific to GitHub action but I do share some frustration while working with GitHub Actions.

On a positive note, since I first used GitHub actions, it has evolved quite a bit with new features, although the documentation is somewhat lagging. It is still very helpful as all of these are free to personal use. I look forward to more interesting features.

Getting started with GitHub Actions

Thu, 27 May 2021 13:52:31 -0400

In my orthweb project, I had to compile a library on my own. In search for free computing resources I realized that GitHub action can meet all my needs.

CI/CD pipeline

As a development project grows, there are many operational tasks demanding automation. Prior to pipeline technology, developers used to use Makefile to organize command execution locally. Today, its role has declined, but Makefile is a good choice in certain situations. In most cases though, to offload the build command execution to a shared system, automation engines like Jenkins came around. Then Jenkins evolved into pipelines.

In strict terms, CI pipeline is the build pipeline; and CD pipeline is release pipeline. The two types of pipelines use pretty much the same pool of building blocks, with different purposes. The build pipeline focuses on producing quality artifact in a consistent manner. The release pipelines focus on system stability while deploying an artifact across different environments. Because release pipelines may connect to different environment, it has to deal with various situations. It is very common to have multiple stages in release pipeline, each stage pointing to a different environment (e.g. DEV, TEST and PROD). At workplace both could be loosely referred to as CI/CD pipeline, or even simply pipeline.

A lot of projects provide pipeline capability: BitBucket, Bamboo, TeamCity, Jenkins, Azure DevOps, AWS CodePipeline, TravisCI etc. Since late 2018, GitHub also joined the game with GitHub actions. It is openly free for public repositories, and has a free tier for private repositories. It executes task as defined in .github/workflow/action.yaml in the code project. I will take my own project as an example.

Runners

You can run jobs in self-hosted runners or GitHub managed runners, similar to other pipeline solutions (e.g. self-hosted agent vs managed agent from Azure DevOps). The GitHub hosted runners only have three operating systems to support: Windows, Ubuntu and MacOS. The Ubuntu and Windows runners are built from Standard_DS2_v2 VMs in Microsoft Azure. They are pre-installed with a virtual environment with packages required for common build tasks. The same virtual environment is also used in hosted agents by Azure DevOps. While they are free and you can elevate privilege on the runner, you cannot SSH or RDP to it for further troubleshooting.

The self-hosted runners require users to manage the instance on their own, including configuring virtual environment, installing GitHub Action Runner, etc.

Workflow file

Most pipeline declaration uses YAML or JSON, such as Jenkinsfile, AWS CodePipeline. GitHub refers to an automation process as a “workflow” and you can program the workflow in YAML (.github/workflow/action.yaml). Here is the reference and an example with environmental variable and versioning:

The handling of environment is documented here. There are a lot of custom actions available in GitHub Marketplace. For example, the versioning in the above example, uses an action by Einar Egilsson, which is open source itself.

Example pipeline

My example pipeline consists of two phases:

Build Library: spin up a docker container to build source code, and publish the artifact
Publish Image: add the artifact to an existing Docker image, and publish the result as my own image.

The status of the pipeline is also open, and can be found here. The retention period of artifact is 90 days by default but can be customized. To persist the artifact, I add it to my own Docker image and publish it to DockerHub, hence the second phase.

When building the second phase, I need to generate secret from my DockerHub account and store that encrypted secrets in GitHub settings, so that the secret value can be referenced in workflow file.

Failures in Actions are displayed in error steps and by default the rest of the steps are skipped.

I use the Docker build & push plugin to build and push my own docker image to DockerHub. Apart from DockerHub as my choice, GitHub also has its own artifactory GitHub Packages with a small free tier. It supports NPM, Docker, Maven, Gradle, etc.

Triggers of Action

Most of the times, GitHub action are triggered upon commit to main branch of the repo. In GitHub, this is known as a workflow_dispatch event. This is not the only event that can trigger GitHub action. All the available events are listed here on its documentation. This makes it very flexible to trigger action at many points in the workflow.

One example is to trigger GitHub action during PR review. When a developer opens a PR with a few commits in the proposed branch, the PR can preemptively check linting, style, etc and even build the application. These activities can also be defined in a GitHub action manifest with pull_request as triggering event.

Troubleshooting

In general, it is painful to troubleshoot activities happening inside of runners. I often had to write a few steps for the sake of printing variables, and trigger a run to see what their value is. This requires a lot of time especially when I have to wait for available runners.

To help troubleshooting pipeline runs there is an open-source utility called act. You can run GitHub actions locally from a Docker container on your MacBook. You can deliver environment variables and secrets via files. If you ever need to troubleshoot the runner environment, you have the option to connect to the Shell environment inside of the runner container. This tool is extremely helpful.

Closing remarks

GitHub action really makes the CI/CD pipeline capability available to any developers who stores their code on GitHub. GitHub expands from a code repository solution to a full CI/CD solution with a free tier sufficient for personal projects.

Git Branching Strategy

Sun, 07 Mar 2021 19:20:42 -0400

I have been in two discussions about Git branching strategy in different organizations. Too many concepts! So I open this post to jot down the lineage of common branching strategies to help organizations develop their branching policies. In terms of reference, there is a lot from Atlassian documentation. In addition, I also find this one article a good resource.

Centralized workflow (no branching)

In Centralized Workflow, the team uses a central repository to serve as the single-point-of-entry for all changes to the project. The default branch is master, and all changes are committed to this branch. This workflow does not require any other branches beside master. Local changes may conflict with upstream commits, and conflict needs to be resolved.

This workflow is usually seen in teams transitioning from SVN, with very basic skill level. This workflow may also be adopted in teams working on configuration management instead of source code. Centralized workflow is great for small teams. The conflict resolution process can form a bottleneck as the team scales in size.

Feature Branch Workflow

Instead of directly committing on their local master branch, developers create a new branch every time they start work on a new feature. Feature branches should have descriptive names (e.g. issue#112). Feature branches are pushed to the central repository so that they can be shared to other developers without touching any official (master) code.

To get feedback on the new feature branch, create a pull request in a repository management solution (e.g. Bitbucket Cloud, Bitbucket Server). Before merge, you may have to resolve merge conflicts if others have made changes to the master branch of repo. This is to make sure your local master is synchronized with the upstream master. When your pull request is approved and conflict free, you can merge your branch to master branch.

feature branch workflow

The Git Feature Branch Workflow is a composable workflow that can be leveraged by other high-level Git workflows. Git Feature Branch Workflow is branching model focused, instead of release focused. The Git Feature Branch Workflow can be incorporated into other workflows. The Gitflow, and Git Forking Workflows traditionally use a Git Feature Branch Workflow in regards to their branching models.

Gitflow Workflow

First published in 2010 by Vincent Driessen. Gitflow defines a strict branching model designed around the project release. This provides a robust framework for managing larger projects. In addition to Feature Branch Workflow, Gitflow workflow assigns very specific roles to different branches and defines how and when they should interact.

Instead of a single master branch, this workflow uses two branches to record the history of the project. The master branch stores the official release history, and the develop branch serves as an integration branch for features. It is also convenient to tag all commits in the master branch with a version number.

This workflow is operated in the following ways:

A develop branch is created from master
Feature branches are created from develop. When a feature is complete, with PR reviewed, it is merged into the develop branch. Features branches
Once develop has acquired enough features for a release (or a predetermined release date is approaching), we fork a release branch off of develop. Creating this branch starts the next release cycle, so new features can be added to develop after this point. On the release branch itself, only bug fixes, documentation generation, and other release-oriented tasks should go in this branch. Once ready to ship, the release branch gets merged into master and tagged with a version number. In addition, it should also be merged back into develop, which may have progressed since the release was initiated.
Maintenance or hotfix branches are used to quicly patch production releases. Hotfix branches are a lot like release branchs and feature branches except they’re based on master instead of develop. As soon as the fix is complete, it should be merged into both master and develop (or the current release branch), and master should be tagged with an updated version number.

Gitflow workflow

There is a Git extension named git-flow to provide high-level repository operations for this Workflow, such as start a release, finish a release, start a hotfix, finish a hotfix.

Gitflow is ideally suited for projects that have a scheduled release cycle and for the DevOps best practice of continuous delivery. It ensures that the master branch reflects what is deployed (e.g. in production). However, it is quite complex and have a steep learning curve for organizations. It also runs long-lived branches, which is considered bad from CI/CD perspective. Branches are by definition to isolate and hide changes, whereas continuous integration is about exposing changes early on and frequently. In that sense, the Gitflow branching model and CI/CD are mutually exclusive ideas.

In 2020, Vincent Driessen added a note at the beginning of his article on Gitflow:
If your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team.
If, however, you are building software that is explicitly versioned, or if you need to support multiple versions of your software in the wild, then git-flow may still be as good of a fit to your team as it has been to people in the last 10 years.

Driessen’s notes also points to some simple alternatives. On the other hand, the Atlassian tutorial on Gitflow has described Gitflow workflow as a legacy (since Aug 2021 based on web archive). It points out at the beginning that Gitflow has fallen in popularity in favor of trunk-based workflows.

GitHub flow, Trunk-based development (TBD) and GitLab flow

It is operationally expensive to manage multiple mainlines in Git flow workflow, with both source control and release in the picture. Some simple alternatives have been evolved, with single mainline, for example, GitHub flow and trunk-based development. They differ in where the release is performed from.

In the GitHub flow, release is performed from branch before being merged back to master (trunk).

GitHub flow

In trunk-based development, release is not performed until the feature branch has been merged to the trunk (master). In trunk-based development, feature branches are supposed to be short-lived. It is a common practice among DevOps teams, since it streamlines merging and integration phases.

Trunk-based development

Trunk-based development has gained some momentum in recent years, due to its DevOps friendliness. This is a website that advocates it and here’s a DZone article about it.

Gitlab Flow

In response to the shortcomings of GitHub flow and Gitflow, Gitlab introduced its own proposal of branching strategy, known as Gitlab flow. The most distinctive aspect is the environment branches. In Gitlab flow, you run multiple long-lived branches, each of them representing an environment. The typical steps are as follows:

You create short-lived feature branches, and merge them often to the master.
Every developer starts from master and targets master. Other branches are merged from previous lower environment branches.
You can deploy a new version to production, by merging master into the production branch.
If you need to know what code is in production, you can check out the production branch to see.

Gitlab flow

You only need to work with release branches, if you need to release software code to the outside world. Here are some best practices in GitLab flow.

Forking Workflow

Forking workflow is fundamentally different. The key steps are as follows:

A developer ‘forks’ an ‘official’ server-side repository. This creates their own server-side copy. This is their personal public repository, and no other developer are allowed to push to it.
The new server-side copy is cloned to their local system. This forms an environment dedicated to this developer.
With the local clone, developer needs to create the upstream remote manually using “git remote add upstream” command. This allows the developer keep the local repository up-to-date as the official project progresses.
A new local feature branch is created. Developer commits to the new local branch, and pushes to their own copy of repository on server.
Developer files a pull request from the new branch (in own copy of repository) to the ‘official’ repository. The project maintainer knows that an update is ready to be integrated. The PR also serves as a discussion thread.
The PR gets approved for merge and is merged into the original server-side repository.

This workflow has other names, such as fork-and-branch workflow, and is commonly used in GitHub for managing open-source projects. However, this should not be confused with the aforementioned GitHub flow.

Conclusion

In order to fully support distributed source control, Git abstract version control problems into concepts such as commit, branch, etc. This makes discussion about Git workflow and branching strategy difficult due to the conceptual hurdles and organization differences. We covered choices of Git branching strategy in this post. As Vincent Driessen commented in his original Gitflow posting, panaceas don’t exist. We should consider the context (e.g. team size, Git skill level, etc) to determine the best Git branching strategy.

Automatic deployment of Orthanc on AWS

Sun, 08 Nov 2020 00:54:06 -0400

[Update] I changed reverse proxy from Nginx to Envoy. Here‘s the detail.

[Update] Some security improvement was introduced in may 2021. Here‘s detail.

[Update] Here’s the link to the orthweb repository.

In this project we introduce a medical imaging web service based on Orthanc, an open-source project of DICOM server, and a pipeline to deploy such server automatically and consistently. We deploy Orthanc on AWS automatically. This little project involves a number of technical deets in DevOps, to deliver a web application prototype with an automated deployment pipeline.

A brief on imaging

In medical imaging, scanning devices are the data collectors. It consists of various categories of scanners, such as Computed Tomography (CT), and Ultrasound (US). They are collectively referred to as modality, but vary significantly in terms of image generation and hardware manufacturing. The challenges to exchange data between these heterogeneous scanning devices and centralized computers came around as early as the 1980s, which brought about ACR-NEMA standard in 1985, under the initiative between American College Radiology (ACR) and National Electrical Manufacturers Association (NEMA). The standard lately evolved into DICOM (Digital Imaging Communication in Medicine), a comprehensive set of standard in the ISO framework that governs modern imaging data storage and exchange across several disciplines (radiology, cardiology, pathology, etc) that operate around images in medicine.

In addition to defining a file format to store imaging data, DICOM also includes an upper layer protocol that dictates how two compliant devices (referred as application entity, each identified by AE title) can negotiate a common syntax to transfer objects (e.g. an image, a report or a discovery). Upper layer refers to layer 5-7 in OSI model, or application layer in TCP/IP model.

Imaging server

Once scanner acquires images from patient, they stores the exams to imaging server for persistent storage. The functionalities of such server expands overtime since 1990s and hence go by different names in different eras, such as PACS (Picture Archive and Communication Systems), VNA (Vendor Neutral Archive) and EI (enterprise imaging) archive. Regardless of naming, they can be generally seen as a highly specialized variation of enterprise content management system. They are usually hosted with a centralized database to index clinical information at patient, exam and image levels. The other key component is the persistent storage devices, usually in the form of a NAS.

Orthanc is an open-source initiative for such imaging servers. It provides a DICOM endpoint, allowing scanning devices to store medical images. It also provides a web viewer allowing users to see the images stored. It is released for many platforms, including Docker images.

Infrastructure as code

We use Amazon Web Service (AWS) for infrastructure as service, and Terraform as the tool to provision resources off AWS, in a reliable and consistent mechanism, known as Infrastructure-as-Code. Terraform is an alternative to CloudFormation, AWS’s proprietary infrastructure-as-code technology. Terraform is developed by Hashicorp as an open-source project, and therefore is vendor neutral. It supports multiple public cloud vendor through different providers. Each provider accesses the vendor specific SDK. For example, the AWS provider integrates with AWS SDK. As a result, the code used in one vendor cannot just be applied to a different vendor without a major overhaul. Terraform’s current version is 0.13 as of Oct 2020, and has gone through some syntax changes since version 0.11. Terraform also produces files for state management locally in the working directory.

When executing, Terraform combines all files in the working directory to assess variables, and create required resources. It is compatible with the most of AWS resources. For example, you can specify user data with templates when creating EC2 instances. You can also create managed service instance as long as it is supported by the provider.

Architecture

Orthanc web server stores data in sqlite by default, but also has a plugin to support PostgreSQL, an open-source relational database. AWS has managed service (RDS) based on PostgreSQL. In this project, we create an RDS instance that span across two availability zones for minimum high availability. Orthanc also supports storing imaging data including pixels in PostgreSQL, which obviates the need for a dedicated file storage system.

We deploy the application in Docker’s containers for compatibility and portability. The Orthanc server is shipped in Docker images, available in Docker hub registry. The docker environment is configured as part of EC2 instance bootstrapping, including installing packages with YUM, initializing and customizing environment variables. The docker-compose file, and the auxiliary configuration files are provided in the repo. The bootstrapping script installs git and pulls required files from this GitHub repo.

This demo project does not include load balancing, DNS management, or container orchestration.

Security

Orthanc’s web browser natively supports HTTPS. However, the DICOM port does not support TLS natively, as their development has made clear in the FAQ. This leaves a severe security vulnerability because all patient data (protected health information in HIPPA context) would be sent across the Internet in the clear, visible to every network interface along the route. To address this issue we brought in Nginx as a reverse proxy to work at TCP layer to terminate encrypted traffic for Orthanc’s DICOM end point. DICOM upper layer works on top of TCP layer.

In Nginx literature, this use case is referred to as SSL Termination for TCP Upstream Servers. Note that Nginx is providing layer 4 capability in this use case so the certificate and key configuration should not be placed under http section of the configuration file. This layer 4 capability in fact enables security configurations of all protocol that operates in upper layers and can be used in a broad range of situations. It is also noteworthy that Nginx can re-encrypt the traffic on the way out to upstream, for even tighter security control measure as outlined in this use case.

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
 worker_connections 1024;
}

stream {
 upstream dicom_backend {
 server orthanc-backend:4242;
 }

 server {
 listen 11112 ssl;
 proxy_pass dicom_backend;

 ssl_certificate conf.d/site.pem;
 ssl_certificate_key conf.d/site.pem;
 ssl_protocols SSLv3 TLSv1 TLSv1.2 TLSv1.3;
 ssl_ciphers HIGH:!aNULL:!MD5:ECDH+AESGCM;
 ssl_session_cache shared:SSL:20m;
 ssl_session_timeout 4h;
 ssl_handshake_timeout 30s;
 }
}

It is also helpful to use Nginx to terminate HTTPS traffic, using a pair of certificate and key. When testing with self-signed certificate I realized that Chrome browser has specific requirement on self-signed certificate, or it won’t load the page. So the certificate has to be created as instructed here.

For better security, it is advisable that the RDS instance is provisioned in private subnet, with its data encrypted both in-transit and at-rest. Docker service should also manage sensitive information as secrets.

Summary

The deliverable is stored in this Github repo. The docker part of it can be executed on MacBook with PostgreSQL. The entire hardware stack represented by terraform code, can be executed against AWS to create required resources. Checkout README for further instruction. To emulate a modality, one will need a TLS supported DICOM application entity, Horos is a great project on MacOS to serve this purpose, both as DICOM-compliant sender and a viewer. Alternatively, consider some command-line based DICOM toolkit such as dcmtk, or grassroot dicom.

Automated Deployment Pipeline 3 of 3

Wed, 14 Oct 2020 17:27:00 -0400

Background

We have previously covered a pipeline example with Jenkins calling Ansible to leverage OpenSSH configuration and Ansible inventory. We also discussed a use case with declarative pipeline.

In this posting, I will provide another advanced example, built on declarative pipeline. The pipeline file will be pulled from Git repository. Also, the script is executed on a remote agent, instead of the Jenkins master server. The reason this example is important, is that:

Jenkinsfile is now version controlled (pipeline as code);
Service script (e.g. python) is also version controlled from a central repository;
Computing resource is provided by a remote agent. Since the script is pulled before running, the agent is still fungible;
The result from service script execution is archived (similar to the way build artifact is stored in Jenkins) in master.

The architecture of this pipeline now becomes the followings:

Configure Agent

Jenkins has a plugin called SSH Build Agent, that allows you to configure a Linux agent, communicating with Jenkins master in SSH. For Window agent, it uses JNLP to communicate with master, which is outside of our scope of discussion. As I touched on in previous post, Jenkins uses its own implementation of SSH protocol to achieve this. This means that it cannot re-use the configurations in ~/.ssh/config and thus the ability to do SSH chaining is eliminated. This is incompatible with our architecture so I have to register Jenkins agent using a different launch method “Launch agent via execution of command on the master”. The execution of command on master can still leverage OpenSSH config file. In order to do so, we must copy the agent.jar file to the remote agent first (URL is ${JENKINS_URL}/jnlpJars/agent.jar). Then use SSH command to call the jar file from agent (aka slave) machine. You may add some java argument for troubleshooting. Below is an example configuration for the node.

I shall also note that if the remote agent is a different operating system where the path of bash might be different, then you need to include the directory of bash executable in PATH environment variable. That can be done as in the screenshot above (PATH=${PATH:/usr/bin}). If this is incorrect, you might run into issues when running sh step in Jenkins pipeline. Here is an article about this. To translate that page, the symptom of this issue includes:

Error from pipeline execution that says the following, which is very generic:

process apparently never started in /home/dhunch/jenkins/workspace/site-remote-job@tmp/durable-b997d26c 
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

Job status shows:

hudson.AbortException: script returned exit code -2
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:659)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:605)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:549)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Threaddump (only available during execution) shows:

Thread #6
 at DSL.sh(awaiting process completion in /home/dhunch/jenkins/workspace/site-remote-job/durable-b997d26c; recurrence period: 9543ms; check task scheduled; cancelled? false done? false)
 at WorkflowScript.run(WorkflowScript:9)
 at DSL.script(Native Method)

This issue is tricky because none of the symptoms above make you think about the environment variable defined for the agent, and that the issue does not occur until you actually execute a Jenkins pipeline, usually well after the node agent is registered, and only impacts shell step (sh). It is recommended to test this with sh steps in pipeline.

Configure Repository

Our example job executes service script pulled from SCM, on a remote agent. Then the master pulls the result file to itself. The git repository can be set in Jenkins job, where you specify that the script named Jenkinsfile (from the repo) is the pipeline file that needs to be executed.

This is fairly simple, but what makes the situation more complex is the following few requirements:

it’s the agent node that needs git (installed) and clone to repo;
the master does not need (and should not attempt to) clone from repo;
the master needs to do its job without having to pull SCM;

We need to be able to specify whether each step needs to pull from SCM. The following pipeline syntax shows how this is done:

def agent_dir = 'initial_value'
pipeline {
 agent none
 options { skipDefaultCheckout(true) }
 stages {
 stage('Execute Job') {
 agent {
 label 'remote-agent-customer1'
 }
 options { skipDefaultCheckout(false) }
 steps {
 echo 'Executing job on node'
 sh 'whoami && pwd'
 script {
 agent_dir = sh(returnStdout: true, script: 'echo -n ${WORKSPACE}')
 // the variable should not include carriage return
 }
 sh "echo ${agent_dir}"
 }
 }
 stage('Pull Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 echo 'Pulling job below'
 sh "echo ${agent_dir}"
 sh "scp dhunch@site1:\"${agent_dir}\"/result.csv ./"
 }
 }
 }
}

In this example pipeline file, we declare agent none for the pipeline, then an agent for each specific stage. we also specify the option skipDefaultCheckout as true at the step where pulling from SCM is not needed. This allows us to finish job with multiple agent, and only pull from SCM as needed. This snippet also exemplifies how to declare a variable, assign it from stdout from one agent, and persist the value across ensuing stages.

Store Result

The reason we run pipeline jobs on this remote agent is because it is sitting in customer network and has local direct access to data. So this is perfect for situation such as data analytical jobs, which access to database on local network and store result. We need to pull the result file back to agent and make it available on Jenkins. This is so similar to “archive artifact” task (commonly seen in CI process) that we can simply use its plugin to achieve what we need. Before archive artifact, we need to pull it to local (master), as shown in the example code above. After that, we need another stage to archive the result.

 stage('Archive Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 archiveArtifacts artifacts: '*.csv',onlyIfSuccessful: true,fingerprint: true
 }
 }

I believe there is plugins to compress artifacts as well.

Conclusion

At the end I’d like to reiterate my perception about Jenkins. It is a very generic and adaptive automation platform that originally evolved from use cases in build automation. Due to this original root, many components in Jenkins are named around Continuous Integration use cases, such as the “build” button, and the “archiveArtifacts” step. These misnomers underplays what Jenkins can potentially do in continuous deployment or other automation scenarios. It is important for automation engineers to understand Jenkins components and plugins, through their functionalities and not by the name, and therefore make creative use of Jenkins as automation engine in all scenarios.

An alternative to this proposed pipeline would be Ansible Tower, a commercial project based on open-source Ansible, but with nice UI support. Ansible Tower is Ansible oriented, and it does not have everything that Jenkins can do. It should still be a decent alternative given the proposes pipeline uses Ansible a lot.

Automated Deployment Pipeline 2 of 3

Tue, 06 Oct 2020 22:05:00 -0400

In this posting, we continue to discuss Jenkins’ ability to automate deployment routines. Jenkins supports freestyle project out of the box, as well as Pipeline with several plugins. Freestyle project allows user to specify multiple steps on UI. This does not scale well when your entire process involves many steps. As explained on Jenkins’ website:

While standard Jenkins “freestyle” jobs support simple continuous integration by allowing you to define sequential tasks in an application lifecycle, they do not create a persistent record of execution, enable one script to address all the steps in a complex workflow, or confer the other advantages of pipelines.
In contrast to freestyle jobs, pipelines enable you to define the whole application lifecycle. Pipeline functionality helps Jenkins to support continuous delivery (CD). The Pipeline plugin was built with requirements for a flexible, extensible, and script-based CD workflow capability in mind.

So although freestyle projects are easy to set up, and can do technically everything that Jenkins pipeline can do, the major advantage of Jenkins Pipeline is the ability to manage multiple-step as code, and version control the pipeline-as-code. Here is some more information.

We will start with a freestyle project to understand Jenkins’ ability and then advance to building pipelines.

Freestyle projects

As mentioned, Jenkins calls a task a “build”, and the build can be triggered in a variety of ways:

Time schedule (with or without parameter)
Remotely via API
On completion of other projects
Poll SCM for changes
Commit to SCM

There are also several ways to execute a job:

Execute Shell command, batch command or groovy script
Invoke Ansible adhoc command, playbook or vault
Conditional on specified boolean value, file existence, etc

After the job one can specify post-build jobs, for example:

start other build projects
notification of various means
publishing result file

In deployment, it is common task to execute a task over SSH, using SSH command or invoke Ansible command with Ansible plugin. The former fits simple command line tasks. For example:

The Ansible plugin is good for more steps and more complicated inventory hierarchies. This post includes an example of an inventory involving multiple layers. The other limitation with SSH command is lack of a straightforward configuration to escalate privilege and run remote command. On the other hand, Ansible addressed this with become method. Below is a screenshot

In order to use Ansible command, you will also need to specify where the Ansible binary in Global Tool Configuration. You will also need to store vault credential in Jenkins credential store so it’s not being prompted.

Jenkins Pipeline

Jenkins pipeline allows one to describe actions in a pipeline in groovy Domain Specific Language (DSL). There are two styles of pipelines:

Declarative pipeline is identified by a block named ‘pipeline’, it is relatively new and supports the pipeline-as-code concept. It can be stored as Jenkinsfile in code repository or edited in Jenkins’ UI.
Scripted pipeline is identified by a block named ‘node’, it is the conventional format and can only be edited in Jenkins’ UI.

To fully understand the two styles of pipelines you also need to know declarative programming and imperative programming. Read this instruction for a better explanation. Here is another good one that focus on their differences.

Jenkins’ plugin provides UI components for user to input information (e.g. Invoke Ansible Command), now with Jenkins pipelines, plugins can provide step functions in order to help Jenkins user. This page from Jenkins lists the most common ones, many of which requires plugin installation.

One example of step function is SSH Pipeline Steps. It allows one to issue SSH connection from Jenkinsfile. However, in our deployment scenarios, it has some limitations:

Unable to match a group of host by specific pattern
Cannot use host name aliases
SSH tunneling is not supported
No means of privilege escalation

Out of these 1 through 3 are due to the fact that SSH step function does not use the OpenSSH configuration on the machine. The jenkins-ssh-slaves plugin uses trilead SSH2 implementation written in Java. Only OpenSSH implementation uses ~/.ssh/config file. This creates problem whenever SSH tunneling is needed. For example, to register a remote slave node via SSH tunneling, we need to select “Launch agent via execution of command on the master” instead of “Launch agent via SSH” as launch method. Here is an instruction and below is what it looks like:

Similarly, if we use ssh-agent plugin as tool for deployment, we cannot use any configuration made by OpenSSH. Therefore Ansible in Jenkins Pipeline is a better tool for deployment because it can use OpenSSH.

Pipelines with Ansible

In this section I demonstrate the use of Ansible playbook and adhoc command in pipeline through two examples.

In example 1, the job pulls authorized key file from SCM, and then use a playbook from SCM, to push the key file to all servers in the specified inventory.

Here is the playbook file for example 1:

---
# Example:
# ansible-playbook -l all -i ~/ansible/inventories/bh.yml push-key.yml --ask-vault-pass
- name: push key to target
 hosts: all
 tasks:
 - name: sync key to host
 copy:
 src: "{{authorized_keys_src}}"
 dest: "/home/dhunch/.ssh/authorized_keys"
 force: yes
 mode: 0600

Here is the pipeline script (note ampersand is mistakenly displayed as & in the box below):

import java.net.URLEncoder;
 
pipeline {
 agent any
 
 options {
 skipDefaultCheckout(true)
 }
 environment {
 BITBUCKET_CREDS = credentials('bitbucket') 
 //BITBUCKET_CREDS_USR and BITBUCKET_CREDS_PSW are set
 BITBUCKET_CREDS_USR = "${BITBUCKET_CREDS_USR}"
 BITBUCKET_CREDS_PSW = URLEncoder.encode("${BITBUCKET_CREDS_PSW}", "UTF-8") 
 // if password contains special character we need to url encode it. e.g. @->%40
 }
 stages {
 stage('Start') {
 steps {
 echo "Starting pipeline ..."
 deleteDir()
 }
 }
 stage('Prep') {
 steps{
 wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: "${BITBUCKET_CREDS_PSW}", var: 'RANDOM']]]) {
 // MaskPasswordsBuildWrapper requires Mask Passwords Plugin and is to mask specific string in console output. 
 // Otherwise BITBUCKET_CREDS_PSW will display in the clear
 sh "git init && git config core.sparsecheckout true"
 sh "git remote add origin https://$BITBUCKET_CREDS_USR:${BITBUCKET_CREDS_PSW}@bitbucket.org/vendorcompoany/configmanagerepo.git"
 sh "echo 'public_keys/*' >> .git/info/sparse-checkout"
 echo "Downloading key file"
 sh "git pull --depth=1 origin master" 
 }
 }
 }
 
 stage('Deploy to Site 1') {
 steps {
 echo '> Deploying to Site 1 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site1.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 1 ...'
 }
 }
 
 stage('Deploy to Site 2') {
 steps {
 echo '> Deploying to Site 2 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site2.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 2 ...'
 }
 }
 
 stage('Deploy to Site 3') {
 steps {
 echo '> Deploying to Site 3 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site3.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 3 ...'
 }
 }
 }
 post {
 always {
 cleanWs()
 }
 }
}

In this example, we pull a sub-directory from git repo. We use two tricks to minimize amount of traffic. First, we use git pull with depth=1 so only the required recent commits are pulled, not the entire history. Second we use sparse checkout to get result from a sub-directory, not the entire repo.

When connecting to repo, password is required. We pull the credential from Jenkins’ credential store and they are masked by default. However, the password must be converted to URL string when used in git remote add origin. Otherwise if the password contains special character the URL will not work.

This modification to password brings about another challenge, password masking during the job execution. We will have to build our own wrapper function using class MaskPasswordsBuildWrapper in order to mask any variable.

In example 2, we wrap an Ansible step function to call adhoc command to check version on all hosts. Here is the pipeline script:

def HunchVersionQuery(pattern, siteinventory){
 step([
 $class: 'AnsibleAdHocCommandBuilder',
 ansibleName: 'Ansible on Mac',
 inventory: [$class: 'InventoryPath',path: siteinventory],
 hostPattern: pattern,
 module: 'shell',
 command: 'cat /etc/*release',
 forks: 1,
 vaultCredentialsId: 'ansible-vault-pass'
 ]);
}

pipeline {
 agent any

 stages {
 stage('Start') {
 steps {
 echo 'Staring Pipeline'
 }
 }
 stage('Query Site1') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site1.yml')
 }
 }
 stage('Query Site2') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site2.yml')
 }
 }
 stage('Query Site3') {
 steps {
HunchVersionQuery('*app','~/ansible/inventories/site3.yml')
 }
 }
 }
}

Because Ansible plugin does not provide a warpper function for adhoc command step, we will have to build our own wrapper function in the above code.

Automated Deployment Pipeline 1 of 3

Wed, 30 Sep 2020 22:04:00 -0400

The business case

You launched a software application. You installed it on two customer sites. You support the application mostly by SSH to customer server and run Bash commands, or slightly better, Bash scripts.

The product is a hit to the market. You hired 20 support specialists in a customer service department. The dream client came through: an enterprise giving you a fleet of 100 servers to deploy your application on.

More staff, more business, more installations, more incidents, but the same old command driven steps. Problems:

Non-standard support procedures. Every one takes notes and everyone’s notes are slightly different.
Information sharing among team members are ad hoc, and at high level.
Post-mortem discussion is driven by memory and command fragments, instead of evidence end-to-end

If that looks like your organization, chances are you also suffer from some secondary damages over the long term, such as:

Downtime resolutions rely on the knowledgeable few
Documentation helps. But it never catches up to the latest version of application
Lack of auditing of commands during support

I propose an automation scheme to existing support and deployment practice. This automation scheme combines a suite of common technologies, such as Bash, Python, Ansible, OpenSSH and Jenkins. The automation allows the department to, either fully or partially, operationalize the steps in support and deployment, and eventually shift towards agile practice.

Bash, Python and Ansible

Bash script is based on shell command, perfect for running critical system tasks such as volume management. When it turns into a script, it can be cumbersome, especially with complex data structure. Python, as a tool for system administration, is a good complement to that.

Python 2 comes with most Linux distributions, and is also a dependency of other built-in tools such as YUM. Python3 can be installed easily from default YUM repositories. Both Python2 and Python3 can exist on the same operating system, although new module development are now shifted to Python3. Python’s syntax is very simple and offers object-oriented programming ability. Moreover, there is an entire open-source community behind Python, which offers modules in every aspect of IT (for example, Datastax has a driver module for connecting to Cassandra). Those modules are installed with PIP tool, or PIP3 for python3.

Both bash and Python executes on local machine. To run them on remote servers over SSH. You want to have a list of target hosts, and specify which one to execute the script against. This is where Ansible comes in handy. Ansible is superior in the following aspects:

Free and open-source, with commercial alternative (Towers);
Inventory management (inventory);
Desire state engine (roles)

Ansible is built on Python and is agent-less. Connectivity to remote host is done via secure shell so it can take advantage of existing SSH configurations. Job execution on the target machine is done through Python. With Python you can also develop custom module in Ansible. For some use cases in customer support with Ansible, refer to my two previous postings about Ansible at scale.

Jenkins

The tools above forms a package for automation. The issue is that all of them are command-line based. Any task that requires Ansible requires the IT professional craft up long command, such as running playbook, executing a role, or ad-hoc command. This is inconvenient when a task needs to be done during an incident. Such tasks also require trained professional with the relevant skills.

These tasks can be stored in, or initiated by Jenkins. Although Jenkins is well known for build automation in continuous integration, it is automation engine for any command-line based IT tasks. The button to start such tasks in Jenkins UI is called “Build”, which is also a misnomer that underplays Jenkins’ versatility: building application from source code is just one of the many IT tasks that involves multiple long running commands. In this and next article we introduce Jenkins as an engine for deployment automation.

The infrastructure architecture is diagramed as above, and with the connection across Internet, the target hosts must be hardened properly in the following aspects:

Connectivity to remote host is via SSH chaining, through an SSH proxy;
Root login must be disabled for remote session or by password;
Service user may be shared, but must be authenticated by individual RSA key pair;
Service user connected remotely needs to escalate privilege by su if needed;

I want to make a theoretical distinction between our topic here and continuous deployment. We simply focus on the technical side of deployment automation. Essentially automating a few bash scripts. On the other hand, a continuous deployment process is an extension to an existing continuous integration pipeline, with the vision to streamline the process end-to-end from code commit to production rollout. Implementing CI/CD pipelines should be approached as an organizational program rather than an individual technical initiative. Here is a good technical overview on CI/CD pipeline with Jenkins and Ansible.

Security

The security mechanism of this system is based on OpenSSH because the connectivity between servers are through SSH chaining. RSA key authentication must be used in order to encrypt traffic with password-less login. Connection to an SSH host can be done through a proxy server. Below is an example of SSH configuration:

Include customer1.config
Include customer2.config

Host *
 IdentityFile ~/.ssh/id_rsa
 ServerAliveInterval 60
 ServerAliveCountMax 3
 Compression yes
 ControlPersist 3h
 ControlPath ~/.ssh/sockets/%r@%h-%p

Host gateway
 Hostname support.digihunch.com
 User jdoe
 Port 2223

Host customer-server-0
 Hostname 192.168.201.12
 User support
 ProxyCommand ssh -W %h:%p gateway

Open SSH configuration file (~/.ssh/config) needs to be configured properly with useful host names and aliases. To prevent the config files from growing too long, include statement can be used to reference other configuration file (available with OpenSSH v7.3 sp1 and up).

The host names (as well as aliases) listed in SSH configuration can be directly referenced in Ansible inventory, allowing Ansible (and Jenkins) to reference site by alias and connect to target host through proxy.

Plugins

Jenkins has a community that develops a variety of plugins, which makes Jenkins the most powerful automation platform. Here are some examples of useful plugins:

Audit Trail: output job execution history to file or Elasticsearch;
Credentials: stores credentials in Jenkins;
Pipeline: build declarative (new) or scripted (old) pipeline for Jenkins jobs;
Simple Theme: just a theme but allows console output to be dark (using CSS);
Job Configuration History: job configuration audit;
Mask password: mask variables (including password) from console output
Ansible: invokes ad-hoc commands and playbooks
SSH agent, SSH pipeline steps, SSH credentials: features related in SSH in Jenkins pipelines.
Purge Job History: purge all of build history, or purge by time and number of old builds.
Parameterized Scheduler: schedule to run a job and provide parameter
Workspace cleanup: clean up workspace when invoked.

In the next article, we will go over some common job configurations.

Common local Git operations

Wed, 08 Apr 2020 11:54:00 -0400

This is a summary of concepts in common Git operations. We will discuss brach, merge, rebase, cherrypick, stash and reset. Then we’ll discuss pull, fetch, and push.

Commit, Branch and HEAD

When you run “git commit”, the following happens:

Git checksums each subdirectory, and stores them as a tree object (file path and name) and blob object (file content) in Git repository;
Git creates a commit object that has the metadata and a pointer to the root project tree; or if this is not the first commit, the pointer will point to the commit immediately before it

The operations above should form a chain of commit. It can be a long chain and may diverge into branches. In Git semantics however, a branch is simply a lightweight, movable pointer to one of the commits. The default branch name in Git is master. A Git repository may contain multiple branches and the name master itself does not suggest any privilege. There is also a special pointer called HEAD, which indicates the branch you are currently working on. So branch is essentially a pointer to a commit; HEAD is essentially a pointer to a branch. “git checkout” can switch branch that HEAD points to.

Basic Merge

One type of basic merge simply moves branch pointer from one commit to another (along the same chain) without creating any commit. Here is a diagram before basic merge:

Before basic merge, Hotfix branch is based on master

The following command performs basic merge:

$ git checkout master
$ git merge hotfix
Updating f42c576..3a0874c
Fast-forward
 index.html | 2 ++
 1 file changed, 2 insertions(+)

Then Git simply moves the pointer (named master) forward. There is no divergent work to move together, hence no chance of merge conflict. This type of basic merge is also called “fast-forward” merge.

After basic merge, master is fast-forwarded to hotfix

The other type of merge involves reconciling divergent work together, which may or may not involve conflict. Suppose this is the commit tree to start with:

Three snapshots used in a typical merge

The following commands perform the merge:

$ git checkout master
Switched to branch 'master'
$ git merge iss53
Merge made by the 'recursive' strategy.
index.html | 1 +
1 file changed, 1 insertion(+)

Instead of just moving the branch pointer forward, Git creates a new snapshot that results from this three-way merge and automatically creates a new commit that points to it. This is referred to as a merge commit, and is special in that it has more than one parent.

A merge commit

Now that your work is merged in, you have no further need for the iss53 branch. You can close the issue in your issue-tracking system, and delete the branch:

$ git branch -d iss53

If commits from two respective branches changes the same file in different ways, then there is a merge conflict. In this case, Git cannot just create a merge commit. Instead it asks the user to resolve the conflict first. You have to choose either side of the change, or just merge the content yourself. At this point, if you introduce a change that does not appear in any parent, it is referred to as an evil merge.

Beyond the basic merge, there are more sophisticated merge conflict resolution tools covered in advanced merging.

Merge and Rebase

There are two ways to integrate changes from one branch to another. Merge and rebase. Suppose your commit chain diverge into a master branch and a feature branch. Merging (from feature to master) takes the content of feature branch and integrate it with master branch.

$ git checkout master
$ git merge feature

When you rebase a feature branch onto master, you move the base of the feature branch to master branch’s ending point.

$ git checkout feature
$ git rebase master

After merge, you are still on the same branch. The commits from other branch are integrated into the branch that you are already on. There is no change in any existing commits (history). After rebase, your base will be moved to a different branch, along with the commits that you have made in the previous branch (since the diverge). In other words, by re-playing those commits on a different branch, it changed history.

merge vs rebase

The chart above is stolen from this article, which does a better job explain in detail the difference, pros and cons of merge and rebase. Merge does create a “merge commit”, and a git history full of merges can be cluttered. Rebase does not create an extra commit but since it changes the history of a branch, it has impact to other collaborators. It can be done in an interactive way (with -i switch). The golden rules of rebasing is covered in this article. One of the principles is that never perform a rebase on a public branch.

git operations

Cherrypick

In a cherrypick operation, the current branch does not change. You simply pick interested commits from other branches to re-apply to your current branch. You may pick a single or a series of commits from other branch. These commits are not “moved” to your current branch. They remain intact. They are just re-played as new commit to current branch. Unlike rebase, there is no re-writing of history, hence not as dangerous.

Reset and Stash

Suppose you are working on a part of a project and it starts getting messy. There has been an urgent bug that needs your immediate attention. It is time to save your changes and switch branches. If you are okay to give up your uncommitted work, you may perform a reset, in one of the three modes covered in a previous article.

But most likely, you don’t want to do a commit of half-done work. The solution is git stash. Stashing is handy if you need to quickly switch context and work on something else but you’re mid-way through a code change and aren’t quite ready to commit. In the most basic workflow, you need to run this command to save your uncommitted (but staged) work. As soon as you stash your change, the working directory is clean with all uncommitted local changes saved elsewhere. You can perform any other Git operations, such as change branch. When you’re ready to resume, you may pop the stash. Here is an example:

$ git add .
$ git stash
$ git checkout correctbranch
$ git stash pop

Instead of pop, you can also use apply to keep the changes in working directory.

$ git stash apply

More details are on this page from Bitbutket.

Fetch and Pull

A git fetch simply downloads blob data from remote so the .git directory comes in sync with the server. It does not attempt to update the local working directory. If there is staged or uncommitted local changes, fetch will not impact them. A git pull is essentially git fetch followed by git merge. In addition to downloading blob data, it also updates local working directory. Therefore, there is a chance of merge conflict when the same file has been modified locally. Git will usually guide you through the merge conflict by flagging the conflict area in the file and let you decide the survival changes. For example:

#! /usr/bin/env ruby

def hello
<<<<<<< HEAD
 puts 'hola world'
=======
 puts 'hello mundo'
>>>>>>> mundo
end

hello()

You will be prompted in an editor session to reconcile the conflict. Once the file is saved, you will also need to do a “merge commit”, before you can pull again.

Push

Git push is the opposite of pull, where you merge local branch to the remote. (There is no opposite of fetch because there is no point to merge to remote without updating working directory, no collaborator works on the working directory on the server after all). If the local branch has fallen out of sync with the remote, there is a chance of merge error during git push. To minimize the chance of a merge during push, we can run a git pull before and reconcile any potential conflict locally. This is known as a pre-merge.

Visualizer

I came across a great visualizer of commit chain here. In the command panel type some git command and it will print the commit graph for you

Clean up Git repository

Sat, 26 Oct 2019 20:33:00 -0400

A BitBucket repo has a hard limit of 2GB in size, and soft limit of 1GB. This is not expandable as per Bitbucket and contributors will start receiving warnings once soft limit is reached. We can tell the usage of a repo from the landing page of the repo in BitBucket.

Git is a distributed version control system for source code management, which implies the followings:

It is intended for source code, or configuration code; but not for storing build artifacts, or installers;
Git remembers every single commit, including the ones associated with large files;
Even a contributor deletes a large file (“git rm filename”) after commit, the large file is only removed from the HEAD. The historical commit still stores the file. After all, the whole point of version control is to survive crazy deletion.
distributed means that those large files will be pulled down to contributors laptop (waste everybody’s space although up to 2G:);

With all these implications, shrinking the size of a repo isn’t as straightforward as just removing large files from current commit. We’d have to rewrite the commit history. Here are the steps we should take once repo size grows over the soft limit.

Clean up remote orphaned branches

Removing these branches (remotes/origin/branchname) per se does not free up space. It simplifies the branch structure, leaving /remote/origin/HEAD the only branch left to cleanse for the rest of the steps.

# git push origin --delete branchname

Remove useless files in current commit (HEAD)

In this step we remove useless files in current commit. Again we should not expect much space freed because all file committed previously, even deleted, are still stored. They are just now showing up in the working directory. For this step, we can create a separate local dir on Mac:

mkdir -p /Users/digihunch/repo-cleanup
cd /Users/digihunch/repo-cleanup

Now within the new directory, we create a bare repo and then the full repo:

git clone --mirror https://gh@bitbucket.org/digihunch/source.git
git clone https://gh@bitbucket.org/digihunch/source.git

Then we dive into the full repo and identify the large files:

find . -type f -size +1000k -exec ls -lh {} \; |awk '{print $9":" $5}'

We can run “git rm ” against the files identified as too large or deletable. Then commit and push to remote repo. This removes large files from current commit.

Remove large file and the relevant commits in the history

As previously mentioned, we have to re-write the history so history forget about the large files. After this step, the historical commits that large files are associated with will all be deleted. Compare the two charts below to understand what the effect is:

We can use the bare repo created in the last step, with “git filter-branch” tool to cleanse the branch tree. Some advocate as a faster third party tool BFG Repo-Cleaner as a faster, third-party alternative but I usually lean towards native tool. This article explains the command switches.

git filter-branch -f --tree-filter "rm -rf \large_file.zip" --prune-empty -- --all

After this steps the repo should be cleansed. According to this guide from BitBucket, we still need to contact their support to run a garbage collection for us in order to see the size change. It even takes time for the size to be reflected after garbage collection. This reference also does great job explaining what we need to do.

Other contributors re-sync history

It is important to understand that the step above modifies history. Although the commit hash did not change, they are assigned with different commit-ids and you can tell from the commit history where it displays former commit id.

This activity only affects remote repository. Each contributor’s local repository still stores the old commits and should be sync’ed with the remote origin by deleting the entire repo and run “git clone” again. Although not welcomed by every individual contributors, but it is a necessary evil and better approached with explicit instruction.

Because this activity takes higher risks, changes each commit, involves vendor support and requires activities by each contributor, the support team should focus on preventing this from happening instead of fixing it.

Configure pre-commit hook as a preventive measure

As we have more Ansible tasks related, working directory becomes complicated and sometimes contributors accidentally committed large unwanted files (and pushed into the remote repo). Down the road, the best practice is to prevent contributors from committing junks.

The best spot to detect this should be a pre-receive hook on the server side, which is only available with self-hosted Bitbucket Server. Unfortunately, this is not a viable option for Bitbucket cloud. Our best bet is client-side pre-commit hook, in which a script performs size check when contributors run “git commit”. The purpose is to fail the commit if total file size is over the limit (20M), and the hook itself should be version controlled as well. Compared to (server side) pre-receive hook, the drawback of (client side) pre-commit hook is it requires initial client configuration. The upside is it captures large files before commit.

This hook can be a shell script as simple as this:

commitsizelimit=20
stagedfilelist=$(git diff --name-only --cached)
stagedfilecnt=`echo "$stagedfilelist"| sed '/^\s*$/d' |wc -l`
if [[ $stagedfilecnt -gt 0 ]]; then
 totalcommitsize=$(du -cm $stagedfilelist | tail -1 | cut -f 1)
 # Redirect output to stderr.
 exec 1>&2

 if [[ $totalcommitsize > $commitsizelimit ]]; then
 echo "Warning: Total size of all files in staging area is "$totalcommitsize"MB, exceeding the limit of "$commitsizelimit"MB."
 echo " To list files by size, run 'du -ch \$(git diff --name-only --cached)'"
 echo " To drop large ones from staging area with 'git rm -f filename'"
 echo " To bypass this limit, use 'git commit --no-verify'"
 exit 1
 fi
fi

In the repo we will have a .githook directory to store hooks (e.g. ~/source/.githooks/pre-commit) and point to the hooks directory using the following command:

git config core.hooksPath .githooks

Git Explained 2 of 2

Tue, 18 Jun 2019 17:06:00 -0400

This is a continuation from Git Explained 1 of 2 where the fundamental concepts are covered. In this article we introduce some tools for customization and maintenance.

As for Git configuration, there are two files to dictate your Git configuration. ~/.gitconfig and .git/config in project directory. Running `git config –list –show-origin` shows all config entries and where they are from. For example you can custom your Git hooks location. Neither of the two files are being version controlled, so the configuration is only effective in the client environment,

Server side Git hooks

Git implementation supports server side hooks (pre-receive, update, post-receive). They are bash scripts placed in .git/hooks with specific names, fired upon event occurrence. Exit code of 1 from the scripts fails the event. Since server side Git hook consumes server resources, many repository hosting vendors (e.g. BitBucket Cloud) do not support it. You will need to enable it in self-hosted servers (e.g. BitBucketServer).

Client side Git hooks

Since server side hooks are not widely supported in every vendor, client-side Git hooks is good alternative places to implement functions such as code style check, commit size check, etc

The default directory for hooks is .git/hooks/ under the project directory is not version controlled and not easy to share with the team. If the hook needs to be shared among project contributors, we can place hooks files in .githooks/ under project directory. This will make the hook files version controlled. In addition, we need to point the hooks to this directory in configuration, by running `git config core.hooksPath .githooks` from project directory.

Web hooks

Web hooks can be thought of as an event notification mechanism. It is a common feature provided by VCS repository hosting providers. If a certain type of event occurs to the repo, web hook will fire an RESTful API call. The HTTP Endpoint, authentication secret and event payload are pre-configured in the repo settings. Web hooks are commonly supported by Git-based repo implementation, such as BitBucket cloud, GitHub, GitLab or AWS CodeCommit. It can also be enabled in self hosted Git repo. Web hook is a powerful tool to drive downstream event, such as Jenkins to start building the code. The major difference between server side hook and web hook is web hook is RESTful API driven, whereas server side hook is executing a script.

Pipelines

Some repo hosting vendor also provide a feature named pipeline. BitBucket has Pipeline as a CI/CD tool, AWS has AWS Code Pipeline, and GitLab offers CI/CD pipeline as well. These pipelines are usually in the form of a YAML file in the repo with a special name. The YAML spell out the steps to perform along the pipeline.

Squash Commits

For small projects I was in the habit of committing to main branch. I often need to squash a number of commits into one to “clean up”. Usually a Pull Request (e.g. in GitHub) or Merge Request (e.g. in GitLab) have such option during approval. We can squash a few commits with git command as well (suppose we want to squash the most recent 16 commits):

git rebase -i HEAD~16
git push origin +main

The command will open text editor to allow you to mark what to do with each commit. You can mark all except one commit as squash. For the commit to keep, mark it as pick. Then save the text editor. Git rebase will perform the squash for you. However, since this is a rebase, do not do this if there are other collaborators working on the same branch.

Cleanse a repository

Take BitBucket cloud for example, the size of a remote repo has a non-expandable hard limit of 2GB, and a soft limit of 1GB. Once the soft limit is reached, a warning will be displayed on Bitbucket’s landing page as well as when contributors pushes changes. Once the hard limit is reached, the entire repo will turn read-only mode.

Space consumption can be caused by accidental committing of large file. As covered in the previous post, files are stored as blob objects in .git directory. If a file was deleted by `git rm` command, it simply means it is de-referenced from the next commit and on. After all, Git as a distributed version control system, has the ability to magically restore the deleted file when we want. The cost of that magic, is that deleted file is permanently stored in the repo, in the form of blob object, although not present in the working directory. It consumes space not only in remote repo, but also in the local repo of each contributor.

The purpose of Git repository is to store source code which are fairly small texts. However if a contributor pushed in large files, it can be tricky to cleanse a Git repo on the remote side. Here is some guidelines:

Before cleansing, identify large unwanted files in current working directory, delete them with `git rm` and then commit this change in master branch.
Consolidate branches (e.g. delete useless remote branches with `git push origin –delete branch_name`). This step itself does not free up space in remote repo but it simplifies the branches;
Remove large blob objects and commit objects that reference them. This step essentially is re-writing the commit history of repo. Given the risk, it is recommended to perform this step from a separate local project directory with bare repo only without working directory. This article presents some good advices and illustrations. The command recommended is `git filter-branch -f –tree-filter “rm -rf \large_file.zip” –prune-empty — –all` ; and if that is too slow here is an alternative named BFG Repo-Cleaner.
After this step, each commit will have a new hash id. For the repo size to reduce, either wait until the next garbage collection on the server side, or request cloud vendor to run a manual garbage collection.

Git Explained 1 of 2

Sat, 08 Jun 2019 22:39:00 -0400

In a nutshell, Git is a distributed version control system, commonly used as source control management. It places files in one of three logical areas (working dir, staging, repo) below depending where it is in its lifecycle.

There are many cheetsheets out there but this article will just sort through some concepts unique to Git. To understand how Git works it is crucial to think in terms of Git data model.

Working directory a single checkout of one version of the project. These files are pulled out of the object database in the Git directory (upon checkout) and placed in the project directory on disk, for you to use or modify;

Index a file contained in your Git directory (stored as binary data in file .git/index) that keeps information about what will go into your next commit. To display what’s in the index, run `git ls-files –stage`. Read this post for further details

Repository: where Git stores the metadata and object database for your project. The local repository is in .git/ under the project directory.

Add – register one or more modified files to staging area. You may edit several files with only a few needed registered for future commit. Add activity ensures the file edited are recorded in the index (as a preview of next commit). You technically need to run add against each file. But the command syntax with * or . allows you to capture all edits in the same directory or under.

Commit – persist the staged file edits to the repository (so they are stored in Git object database). A commit represents all the file edits that were staged by add command in previous steps.

Branch – a branch is simply a movable pointer to a commit. Default branch name created by git init is called “master”. Other than the name, there is nothing special about master branch. Everytime you commit, the master branch pointer moves forward automatically. Branch pointers are kept in .git/refs directory. Read this post for further details.

HEAD – the pointer to the current branch reference, which is in turn a pointer to the last commit made on that branch. Git use HEAD pointer to know what branch you’re currently on. HEAD will be the parent of the upcoming commit.

Tag – an annotated tag contains the SHA of the commit being tagged. Alias of a commit.

Merge – choose current commit of other branch and apply it onto your branch.

Rebase – copy all commits from other branch to your branch. Compared to merge, rebasing forms a cleaner commit history.

Cherrypick – choose a previous commit from other branch and apply it onto your branch.

Stash – temporarily stashes changes you’ve made to working tree so you can work on something else, and then come back and re-apply them later on.

Reset – at a high level, reset is to revert some operations. After pulling code, developer usually follow three steps: editing->add->commit. reset is to reverse these steps, based on different modes. The Pro Git reference has further details on the three different modes:

soft mode (reverse operation of commit): based on what branch HEAD points to, move where that branch points to (e.g. from latest commit, to a different commit several steps up the link);
mixed mode (default; reverse operation of commit and add) – in addition to soft mode, also update index;
hard mode (reverse operation of commit, add and file editing)- in addition to mixed mode, also update working directory. Edit on files are discarded.

Git object model – In Git database, files, commits and directories are stored as objects, In Git object model, there are three types (to tell object type, run `git cat-file -t`). Read this post for further details:

blob object – stores file data with metadata; use `git show` to examine blob object;
tree object – represents a directory. It references other tree objects (sub-directories) or blob objects (files under the directory, of a certain version); use `git ls-tree` to examine tree object;
commit object – represents a commit. It references its parent commit, as well as a tree object that represents the entire project directory. use `git cat-file -p` to inspect commit object;

This diagram from from Git Pro outlines the interactions amongst these types of objects.

For more details, the official documentation is actually the most helpful reference with illustrations. In addition, I find on Hakcermoon three excellent articles with thorough explanation on data model, branching and index.