pipeline on Digi Hunch

Getting started with GitHub Actions

Thu, 27 May 2021 13:52:31 -0400

In my orthweb project, I had to compile a library on my own. In search for free computing resources I realized that GitHub action can meet all my needs.

CI/CD pipeline

As a development project grows, there are many operational tasks demanding automation. Prior to pipeline technology, developers used to use Makefile to organize command execution locally. Today, its role has declined, but Makefile is a good choice in certain situations. In most cases though, to offload the build command execution to a shared system, automation engines like Jenkins came around. Then Jenkins evolved into pipelines.

In strict terms, CI pipeline is the build pipeline; and CD pipeline is release pipeline. The two types of pipelines use pretty much the same pool of building blocks, with different purposes. The build pipeline focuses on producing quality artifact in a consistent manner. The release pipelines focus on system stability while deploying an artifact across different environments. Because release pipelines may connect to different environment, it has to deal with various situations. It is very common to have multiple stages in release pipeline, each stage pointing to a different environment (e.g. DEV, TEST and PROD). At workplace both could be loosely referred to as CI/CD pipeline, or even simply pipeline.

A lot of projects provide pipeline capability: BitBucket, Bamboo, TeamCity, Jenkins, Azure DevOps, AWS CodePipeline, TravisCI etc. Since late 2018, GitHub also joined the game with GitHub actions. It is openly free for public repositories, and has a free tier for private repositories. It executes task as defined in .github/workflow/action.yaml in the code project. I will take my own project as an example.

Runners

You can run jobs in self-hosted runners or GitHub managed runners, similar to other pipeline solutions (e.g. self-hosted agent vs managed agent from Azure DevOps). The GitHub hosted runners only have three operating systems to support: Windows, Ubuntu and MacOS. The Ubuntu and Windows runners are built from Standard_DS2_v2 VMs in Microsoft Azure. They are pre-installed with a virtual environment with packages required for common build tasks. The same virtual environment is also used in hosted agents by Azure DevOps. While they are free and you can elevate privilege on the runner, you cannot SSH or RDP to it for further troubleshooting.

The self-hosted runners require users to manage the instance on their own, including configuring virtual environment, installing GitHub Action Runner, etc.

Workflow file

Most pipeline declaration uses YAML or JSON, such as Jenkinsfile, AWS CodePipeline. GitHub refers to an automation process as a “workflow” and you can program the workflow in YAML (.github/workflow/action.yaml). Here is the reference and an example with environmental variable and versioning:

The handling of environment is documented here. There are a lot of custom actions available in GitHub Marketplace. For example, the versioning in the above example, uses an action by Einar Egilsson, which is open source itself.

Example pipeline

My example pipeline consists of two phases:

Build Library: spin up a docker container to build source code, and publish the artifact
Publish Image: add the artifact to an existing Docker image, and publish the result as my own image.

The status of the pipeline is also open, and can be found here. The retention period of artifact is 90 days by default but can be customized. To persist the artifact, I add it to my own Docker image and publish it to DockerHub, hence the second phase.

When building the second phase, I need to generate secret from my DockerHub account and store that encrypted secrets in GitHub settings, so that the secret value can be referenced in workflow file.

Failures in Actions are displayed in error steps and by default the rest of the steps are skipped.

I use the Docker build & push plugin to build and push my own docker image to DockerHub. Apart from DockerHub as my choice, GitHub also has its own artifactory GitHub Packages with a small free tier. It supports NPM, Docker, Maven, Gradle, etc.

Triggers of Action

Most of the times, GitHub action are triggered upon commit to main branch of the repo. In GitHub, this is known as a workflow_dispatch event. This is not the only event that can trigger GitHub action. All the available events are listed here on its documentation. This makes it very flexible to trigger action at many points in the workflow.

One example is to trigger GitHub action during PR review. When a developer opens a PR with a few commits in the proposed branch, the PR can preemptively check linting, style, etc and even build the application. These activities can also be defined in a GitHub action manifest with pull_request as triggering event.

Troubleshooting

In general, it is painful to troubleshoot activities happening inside of runners. I often had to write a few steps for the sake of printing variables, and trigger a run to see what their value is. This requires a lot of time especially when I have to wait for available runners.

To help troubleshooting pipeline runs there is an open-source utility called act. You can run GitHub actions locally from a Docker container on your MacBook. You can deliver environment variables and secrets via files. If you ever need to troubleshoot the runner environment, you have the option to connect to the Shell environment inside of the runner container. This tool is extremely helpful.

Closing remarks

GitHub action really makes the CI/CD pipeline capability available to any developers who stores their code on GitHub. GitHub expands from a code repository solution to a full CI/CD solution with a free tier sufficient for personal projects.

Git Explained 2 of 2

Tue, 18 Jun 2019 17:06:00 -0400

This is a continuation from Git Explained 1 of 2 where the fundamental concepts are covered. In this article we introduce some tools for customization and maintenance.

As for Git configuration, there are two files to dictate your Git configuration. ~/.gitconfig and .git/config in project directory. Running `git config –list –show-origin` shows all config entries and where they are from. For example you can custom your Git hooks location. Neither of the two files are being version controlled, so the configuration is only effective in the client environment,

Server side Git hooks

Git implementation supports server side hooks (pre-receive, update, post-receive). They are bash scripts placed in .git/hooks with specific names, fired upon event occurrence. Exit code of 1 from the scripts fails the event. Since server side Git hook consumes server resources, many repository hosting vendors (e.g. BitBucket Cloud) do not support it. You will need to enable it in self-hosted servers (e.g. BitBucketServer).

Client side Git hooks

Since server side hooks are not widely supported in every vendor, client-side Git hooks is good alternative places to implement functions such as code style check, commit size check, etc

The default directory for hooks is .git/hooks/ under the project directory is not version controlled and not easy to share with the team. If the hook needs to be shared among project contributors, we can place hooks files in .githooks/ under project directory. This will make the hook files version controlled. In addition, we need to point the hooks to this directory in configuration, by running `git config core.hooksPath .githooks` from project directory.

Web hooks

Web hooks can be thought of as an event notification mechanism. It is a common feature provided by VCS repository hosting providers. If a certain type of event occurs to the repo, web hook will fire an RESTful API call. The HTTP Endpoint, authentication secret and event payload are pre-configured in the repo settings. Web hooks are commonly supported by Git-based repo implementation, such as BitBucket cloud, GitHub, GitLab or AWS CodeCommit. It can also be enabled in self hosted Git repo. Web hook is a powerful tool to drive downstream event, such as Jenkins to start building the code. The major difference between server side hook and web hook is web hook is RESTful API driven, whereas server side hook is executing a script.

Pipelines

Some repo hosting vendor also provide a feature named pipeline. BitBucket has Pipeline as a CI/CD tool, AWS has AWS Code Pipeline, and GitLab offers CI/CD pipeline as well. These pipelines are usually in the form of a YAML file in the repo with a special name. The YAML spell out the steps to perform along the pipeline.

Squash Commits

For small projects I was in the habit of committing to main branch. I often need to squash a number of commits into one to “clean up”. Usually a Pull Request (e.g. in GitHub) or Merge Request (e.g. in GitLab) have such option during approval. We can squash a few commits with git command as well (suppose we want to squash the most recent 16 commits):

git rebase -i HEAD~16
git push origin +main

The command will open text editor to allow you to mark what to do with each commit. You can mark all except one commit as squash. For the commit to keep, mark it as pick. Then save the text editor. Git rebase will perform the squash for you. However, since this is a rebase, do not do this if there are other collaborators working on the same branch.

Cleanse a repository

Take BitBucket cloud for example, the size of a remote repo has a non-expandable hard limit of 2GB, and a soft limit of 1GB. Once the soft limit is reached, a warning will be displayed on Bitbucket’s landing page as well as when contributors pushes changes. Once the hard limit is reached, the entire repo will turn read-only mode.

Space consumption can be caused by accidental committing of large file. As covered in the previous post, files are stored as blob objects in .git directory. If a file was deleted by `git rm` command, it simply means it is de-referenced from the next commit and on. After all, Git as a distributed version control system, has the ability to magically restore the deleted file when we want. The cost of that magic, is that deleted file is permanently stored in the repo, in the form of blob object, although not present in the working directory. It consumes space not only in remote repo, but also in the local repo of each contributor.

The purpose of Git repository is to store source code which are fairly small texts. However if a contributor pushed in large files, it can be tricky to cleanse a Git repo on the remote side. Here is some guidelines:

Before cleansing, identify large unwanted files in current working directory, delete them with `git rm` and then commit this change in master branch.
Consolidate branches (e.g. delete useless remote branches with `git push origin –delete branch_name`). This step itself does not free up space in remote repo but it simplifies the branches;
Remove large blob objects and commit objects that reference them. This step essentially is re-writing the commit history of repo. Given the risk, it is recommended to perform this step from a separate local project directory with bare repo only without working directory. This article presents some good advices and illustrations. The command recommended is `git filter-branch -f –tree-filter “rm -rf \large_file.zip” –prune-empty — –all` ; and if that is too slow here is an alternative named BFG Repo-Cleaner.
After this step, each commit will have a new hash id. For the repo size to reduce, either wait until the next garbage collection on the server side, or request cloud vendor to run a manual garbage collection.