Jobs on Digi Hunch

GitHub Action Gotchas

Fri, 27 Jan 2023 01:15:02 -0400

I started with GitHub Actions a couple years ago. Recently I came across a few interesting use cases while I was trying to setup Terraform workflow with GitHub actions. These use cases prompted me to make use some new features in GitHub Action. So I put them in a post here.

Runners can assume IAM Role in AWS

In many scenarios we want to execute AWS CLI command from GitHub action. Also, executables such as terraform inherits credential from AWS CLI. The credential should be a temporary role-based credential instead of an IAM user based on access keys.

There is a GitHub Action called configure-aws-credentials-for-github-actions that can help configure GitHub runner using OIDC identity provider (since Nov 2021 v1.6.0). With the action, the GitHub runner can assume an IAM role as an IAM user (with access key), or using a web identity.

For a GitHub runner to have a web identity thereby assume an IAM role, we should configure OIDC provider in AWS. We can do that from AWS console (i.e. under IAM), or using CloudFormation code. Below is a snippet as an example:

Resources:
 GitHubOIDC:
 Type: AWS::IAM::OIDCProvider
 Properties:
 Url: https://token.actions.githubusercontent.com
 ClientIdList: 
 - sts.amazonaws.com
 ThumbprintList:
 - 6938fd4d98bab03faadb97b34396831e3780aea1

Then from the configured OIDC provider, we can obtain a thumbprint. GitHub action gives the thumbprint here. In AWS, we configure an IAM role whose AssumeRolePolicyDocument will reference the thumbprint. Here is an example. In the condition section of AssumeRolePolicyDocument, we can also specify a specific GitHub repository so that only Actions from that repository can assume the IAM role with their web identities.

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1-node16
 with:
 role-to-assume: ${{ vars.IAM_ROLE_ARN }}
 aws-region: ${{ vars.AWS_REGION }}

This way, we map a GitHub runner’s web identity to an IAM role with a step using the Action above. We also filter what GitHub org and what repo can trigger actions that assumes the role, by the condition clause in the role statement. If the step fails, we can look at CloudTrail on the AWS side for causes. Look for entries with AssumeRoleWithWebIdentity as Event Name.

Reusable workflows

For better reusability of Action steps, GitHub introduced reusable workflows (generally available since Nov 2021). It is particularly helpful when we need to run a workflow for different environments. The reusable workflows files (YML) can be placed in separate repositories, and be reference as such. This allows enterprises to centralize the management of reusable workflows.

I have been using the act project to emulate GitHub action locally on MacBook. As of Jan 2023, act does not support reusable workflow.

With the split between caller and reusable workflows, we have a new challenge of passing secrets and variables between them. It is not straightforward and GitHub documentation needs improvement to get the documentation clear. Also because the word “environment” is used in different contexts, it is ambiguous and therefore difficult to Google relevant information.

Passing variables

First, there are several types of variables in GitHub action:

Environment variable: declared under env keyword in a workflow. To use environment variable, use the env context. For example: ${{ env.MY_VARIABLE }}
Configuration variable: introduced in Jan 2023, configuration variables are defined at repository, environment and organization levels. To use configuration variable, use vars context, and ensure the workflow job specifies a value for environment attribute.
Secrets: GitHub also calls it Environment secret when defined at environment level. It works the same way as a configuration variable because it is also specific to an environment. The content is not viewable once set.

The reason GitHub action makes this so confusing, is that on one page, its documentation distinguishes between environment variable and configuration variable:

On another page, the document refers to configuration variables at environment level as environment variable:

It seems that “configuration variable” is too new for GitHub to refine its documentation as of January. This semantical confusion gave me a hard time investigating how to pass “Environment variable” to reusable workflows. I will stick to the meaning on the first page to distinguish environment variable and configuration variable at environment level.

Passing environment variable isn’t straightforward. In this discussion thread, people discussed how inconvenient it is. I used the workaround in this comment, where I had to create a job for the sake of storing variable values to output.

Pass secret is easier. This is an insightful blog post (Dec 2021) about passing secret to reusable workflow. The attempt 3 in the post works for me. First, we pass the value of environment to the reusable workflow as an input, then at job level specify the environment with the value. Then in the jobs we can reference secrets as ${{ secrets.NAME }}. The job will pick up the secret based on the correct environment.

It appears that since May 2022, GitHub introduced secrets: inherit keyword to address this. However, the method above still works for configuration variable.

Authentication of GitHub Actions

By default, a GitHub action can access the code repository that triggers the action and no other repositories (with GITHUB_TOKEN). However, in many cases we need to access external repositories. For example, terraform init command from a GitHub action implicitly calls git clone to pull module code from external repositories. A GitHub workflow may also reference a workflow file from external repositories.

The question is how to authenticate GitHub workflow to access external repo. This post has a thorough discussion. We may create a Personal Access Token and pass it to set-git-credentials action. We are essentially sharing a personal credential (and repo access) with a GitHub action, which is not a good practice.

The proper way to solve this problem, is to create a separate GitHub App and grant the access only the repo that the workflow needs to access. The GitHub App will generate a private key. Then we supply the private key to workflow-application-token-action so the workflow can act as the GitHub App, thereby access the external repos. The post has more details in the GitHub App section. Suppose we have terraform get command to clone external repo, the actions may look like this:

 - name: HashiCorp - Setup Terraform
 uses: hashicorp/setup-terraform@v2

 - name: Get RepoReader App Token
 id: get_repo_reader_token
 uses: peter-murray/workflow-application-token-action@v2
 with:
 application_id: ${{ vars.REPO_READER_APPLICATION_ID }}
 application_private_key: ${{ secrets.REPO_READER_PRIVATE_KEY }}

 - name: Cache Git Creds
 uses: de-vri-es/setup-git-credentials@v2
 with:
 credentials: https://x-access-token:${{ steps.get_repo_reader_token.outputs.token }}@github.com/

 # Terraform Get implicitly calls git clone which uses the credential cached as above
 - name: Terraform Get 
 run: terraform get

Another benefit of using GitHub App is that the token is a short-lived credential that expires as the job is finished, whereas a PAT will expire on a preset date. In this use case we can think of GitHub App as a service account with minimized privilege to read a short list of repos.

Final words

I came across this reddit post against pipeline use and there are some legit points. For example, the user is frustrated with the limitation with variable passing and unclear documentation. The post wasn’t specific to GitHub action but I do share some frustration while working with GitHub Actions.

On a positive note, since I first used GitHub actions, it has evolved quite a bit with new features, although the documentation is somewhat lagging. It is still very helpful as all of these are free to personal use. I look forward to more interesting features.

Getting started with GitHub Actions

Thu, 27 May 2021 13:52:31 -0400

In my orthweb project, I had to compile a library on my own. In search for free computing resources I realized that GitHub action can meet all my needs.

CI/CD pipeline

As a development project grows, there are many operational tasks demanding automation. Prior to pipeline technology, developers used to use Makefile to organize command execution locally. Today, its role has declined, but Makefile is a good choice in certain situations. In most cases though, to offload the build command execution to a shared system, automation engines like Jenkins came around. Then Jenkins evolved into pipelines.

In strict terms, CI pipeline is the build pipeline; and CD pipeline is release pipeline. The two types of pipelines use pretty much the same pool of building blocks, with different purposes. The build pipeline focuses on producing quality artifact in a consistent manner. The release pipelines focus on system stability while deploying an artifact across different environments. Because release pipelines may connect to different environment, it has to deal with various situations. It is very common to have multiple stages in release pipeline, each stage pointing to a different environment (e.g. DEV, TEST and PROD). At workplace both could be loosely referred to as CI/CD pipeline, or even simply pipeline.

A lot of projects provide pipeline capability: BitBucket, Bamboo, TeamCity, Jenkins, Azure DevOps, AWS CodePipeline, TravisCI etc. Since late 2018, GitHub also joined the game with GitHub actions. It is openly free for public repositories, and has a free tier for private repositories. It executes task as defined in .github/workflow/action.yaml in the code project. I will take my own project as an example.

Runners

You can run jobs in self-hosted runners or GitHub managed runners, similar to other pipeline solutions (e.g. self-hosted agent vs managed agent from Azure DevOps). The GitHub hosted runners only have three operating systems to support: Windows, Ubuntu and MacOS. The Ubuntu and Windows runners are built from Standard_DS2_v2 VMs in Microsoft Azure. They are pre-installed with a virtual environment with packages required for common build tasks. The same virtual environment is also used in hosted agents by Azure DevOps. While they are free and you can elevate privilege on the runner, you cannot SSH or RDP to it for further troubleshooting.

The self-hosted runners require users to manage the instance on their own, including configuring virtual environment, installing GitHub Action Runner, etc.

Workflow file

Most pipeline declaration uses YAML or JSON, such as Jenkinsfile, AWS CodePipeline. GitHub refers to an automation process as a “workflow” and you can program the workflow in YAML (.github/workflow/action.yaml). Here is the reference and an example with environmental variable and versioning:

The handling of environment is documented here. There are a lot of custom actions available in GitHub Marketplace. For example, the versioning in the above example, uses an action by Einar Egilsson, which is open source itself.

Example pipeline

My example pipeline consists of two phases:

Build Library: spin up a docker container to build source code, and publish the artifact
Publish Image: add the artifact to an existing Docker image, and publish the result as my own image.

The status of the pipeline is also open, and can be found here. The retention period of artifact is 90 days by default but can be customized. To persist the artifact, I add it to my own Docker image and publish it to DockerHub, hence the second phase.

When building the second phase, I need to generate secret from my DockerHub account and store that encrypted secrets in GitHub settings, so that the secret value can be referenced in workflow file.

Failures in Actions are displayed in error steps and by default the rest of the steps are skipped.

I use the Docker build & push plugin to build and push my own docker image to DockerHub. Apart from DockerHub as my choice, GitHub also has its own artifactory GitHub Packages with a small free tier. It supports NPM, Docker, Maven, Gradle, etc.

Triggers of Action

Most of the times, GitHub action are triggered upon commit to main branch of the repo. In GitHub, this is known as a workflow_dispatch event. This is not the only event that can trigger GitHub action. All the available events are listed here on its documentation. This makes it very flexible to trigger action at many points in the workflow.

One example is to trigger GitHub action during PR review. When a developer opens a PR with a few commits in the proposed branch, the PR can preemptively check linting, style, etc and even build the application. These activities can also be defined in a GitHub action manifest with pull_request as triggering event.

Troubleshooting

In general, it is painful to troubleshoot activities happening inside of runners. I often had to write a few steps for the sake of printing variables, and trigger a run to see what their value is. This requires a lot of time especially when I have to wait for available runners.

To help troubleshooting pipeline runs there is an open-source utility called act. You can run GitHub actions locally from a Docker container on your MacBook. You can deliver environment variables and secrets via files. If you ever need to troubleshoot the runner environment, you have the option to connect to the Shell environment inside of the runner container. This tool is extremely helpful.

Closing remarks

GitHub action really makes the CI/CD pipeline capability available to any developers who stores their code on GitHub. GitHub expands from a code repository solution to a full CI/CD solution with a free tier sufficient for personal projects.

Basic Resource Object in Kubernetes 2 of 2

Mon, 08 Feb 2021 21:02:16 -0400

We continued from previous posting about resource object, starting from storage related ones.

Volume

In Kubernetes, we use the term volume to refer to a section storage device. There are many plugins, compliant to Container Storage Interface (CSI), to allow heterogeneous storage resources to be surfaced as volumes in Kubernetes. CSI allows storage driver to operate in parallel to the main Kubernetes code tree. Any driver that complies with CSI would work with any orchestration platform that requires CSI, such as Docker Swarm, Kubernetes. Three main resources in the storage system are: PV (persistent volumes), PVC (persistent volume claims), and SC (storage classes).

Persistent Volume

Persistent Volumes (PV) allows you to map external storage onto the Kubernetes cluster. It is a representation of the external storage on the cluster. A single external storage volume can only be represented by a single PV. For example, you cannot have a 50GB external volume that has two 25GB PVs each representing half of it.

PV can be mounted in three options:

RWO (ReadWriteOnce): allows single PVC to mount. This is common for block device.
RWM (ReadWriteMany): allows multiple PVCs to bind as read and write. This is common for file and object level access.
ROM (ReadOnlyMany): allows multiple PVCs to bind as read only. Think of it along the lines of ISO media.

Note that a PV can only be opened in one of the modes above. All connecting PVC (if multiple are allowed) will use that mode.

Persistent Volume Claim

Persistent Volume Claims (PVC) act like tickets that authorize applications (Pods) to use a PV. Once a Pod has the PVC, it can bind the respective PV as a volume. You need to specify PV name when declaring a PVC to associate them. Pods do not act directly on PVs, they always act on the PVC object that is bound to the PV. When a PVC is released, two actions can be configured in the policy: Delete and Retain. The delete policy will delete the PV as well as associated storage resource on the external storage system. The retain policy will keep the associated PV object on the cluster as well as any data stored on the associated external assets.

The spec section of PVC object declaration must match the fields in the corresponding PV it binds to. For example access modes, capacity and storage class name.

Storage Class

Storage classes allow you to define different classes (or tiers) of storage using an external provisioner such as aws-ebs. This works well with cloud storage provider. As long as the plugin for storage backend is available, you can configure as many StorageClass object as you need, and even specify to encrypt them. Storage classes create PV dynamically, so you will need to create PVC object that reference the newly created storage class, in order to use cloud storage.

The whole purpose of storage class is to create PVs dynamically, for various storage backend/plugin. You just create the StorageClass object and use a plugin to tie it to a particular type of storage on a particular storage back-end. When matching PVCs appear, the StorageClass dynamically creates the required volume on the back-end storage system.

If a cluster has a default storage class, you can deploy a Pod using just PVC with PodSpec, without explicitly declare storage class per Pod. However, this is not recommended in production.

ConfigMaps

With modern application it is a good practice to decouple configurations from application execution environment. They are stored separately but brought together at runtime. ConfigMap (CM) allows you to store configuration data outside of a Pod, and dynamically inject the configuration data into a Pod at runtime. ConfigMaps are essentially key/value pairs, and each key/value pair is called an entry.

Once data is stored in a ConfigMap, it can be injected into containers at run-time via one of the three methods:

environment variables: updates to ConfigMap is not updated
arguments to the container’s startup command (very limited)
files in a volume (most flexible): requires creating a ConfigMap volume in the Pod template and mounting. Eateries in the ConfigMap will appear in the container as individual files. You can make changes to entries after a container is deployed, and the change is seen in the file.

The application is unaware that the data originally came from a ConfigMap. Also note that ConfigMap is not to store sensitive data.

Secret

Kubernetes Secrets let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Storing confidential information in a Secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image.

The name of a Secret object must be a valid DNS subdomain name. A Secret can be used with a Pod in three ways:

As files in a volume mounted on one or more of its containers.
As container environment variable.
By the kubelet when pulling images for the Pod.

Ingress

Ingress manages manages external access to the services in a cluster, typically HTTP. It may provide load balancing, SSL termination and name-based virtual hosting. Also, you must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.You can choose from a number of Ingress controllers. Nginx is a common flavour.

By default, containers run with unbounded compute resources on a Kubernetes cluster. With resource quotas, cluster administrators can restrict resource consumption and creation on a namespace basis. Within a namespace, a Pod or Container can consume as much CPU and memory as defined by the namespace’s resource quota. There is a concern that one Pod or Container could monopolize all available resources. A LimitRange is a policy to constrain resource allocations (to Pods or Containers) in a namespace.

A LimitRange provides constraints that can:

Enforce minimum and maximum compute resources usage per Pod or Container in a namespace.
Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace.
Enforce a ratio between request and limit for a resource in a namespace.
Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime.

Resource Quotas

When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources. Resource quotas are a tool for administrators to address this concern.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.

High Performance Computing

Fri, 11 Dec 2020 23:42:00 -0400

Overview

High Performance Computing (HPC) has recently been commoditized with the advent of commodity server hardware (x86 server), virtualization technology and cloud delivery model. It is common in specialized industries where intensive computing tasks are required, for example:

HCL (healthcare and life science): drug discovery, computer aided diagnosis (CAD), genome engineering;
CAD, CAE, CAM (computer aided design, engineering, and manufacturing): 3D modeling, computational fluid dynamics (CFD), finite element analysis (FEA), structural mechanical design, etc
Finance: portfolio management, automated trading, risk analysis
Geoscience and geo-engineering: oil and gas exploration, geographic data, weather forecasting;
Scientific computation

Computing performance is measured in FLOPS (floating point operations per second) and is usually delivered in a cluster to aggregate the computing power from a number of networked nodes. This is referred to as an HPC cluster.

Hardware stack

An HPC cluster features the following components:

Head node (aka master node or login node): a gateway and coordinator; head node may be broken into several nodes
Compute node (worker node): the executor of jobs; the compute node can either be homogenous or heterogeneous, for different purposes. the number of compute nodes can be quite large

There are four common form factors for server: tower, rack-mount, blade, mainframe. Traditionally, the nodes are rack-mount 1U “pizza box” servers. Bladed systems started to replace due to the increased node density, thanks to the shared/redundant power and cooling management. In the past, the HPC cluster is operated in data centres, which is an expensive operation item. In the last decades, many organizations extends their compute workload to the cloud, forming a hybrid model.

HPC typically has specialized storage system because HPC applications notoriously create large amounts of data. NFS traditionally does not scale well as number of node increases. Some proprietary storage system such as Isilon provides good performance via NFS protocol. There are also open-source parallel file system such as Lustre and HDFS.

HPC networking handles three types of traffic:

computation traffic between compute nodes (if the compute nodes interact with each other)
file system traffic: for compute nodes to read and write on file system (e.g. NFS)
administrative traffic: fairly light compared to the two above

For that, many HPC runs two networks, a private (backend) network and a public (frontend) network. Backend network must be high speed and low latency, typically in the form of 10Gig Ethernet, or InfiniBand.

Software stack

On the software layer, the core functionality is Message Passing Interface (MPI), a specification for the developers and users of message passing libraries. MPI constitutes a standardized and portable message-passing system which consists of a library and a protocol to support parallel computing. MPI enables passing information between various nodes of a HPC cluster or between particular clusters, and has different implementations that provide the libraries to run HPC applications in a distributed manner across different physical nodes.

In the operation, user submits a job through head node in order to request the resource. User needs to specify the resources for the job (e.g. how many CPU cores, how much memory, etc). The head node runs a scheduler to allocate computing resource based on pre-defined policies, based on priority of jobs, availability of resources, distribution of load, etc. Depending on the nature of the computing jobs, the nodes participating in the task may or may not communicate with one another. If they do need to talk to each other, the program must support it. Such program can be called a cluster program, and the MPI (message passing interface) library greatly facilitates the development of such program. The sub-jobs communicating with each other also creates a considerable amount of network traffic within the cluster.

Cluster software ties all nodes in the cluster together. It turns raw hardware into a functioning cluster by provisioning (installing and configuring) the head nodes. Compute nodes can usually be added or removed dynamically therefore the head nodes should be able to provision compute nodes, and administer cluster, leaving the programming as the job for the user to complete. As mentioned, in parallel programming, the most important HPC tool is MPI (Message Passing Interface), which allows programs to talk to one another over cluster networks. There are both open (e.g. Open MPI) and commercial MPI (e.g. Microsoft MPI) versions. Cluster software should also provide compilers, debuggers, and profilers in addition to MPI.

There are cluster software in both Linux and Windows operating systems: Rocks Clusters, Oscar (Open Source Clusters Application Resources), Red Hat HPC solution, Microsoft HPC pack and AWS Parallel Cluster.

Implementation

Here is an example of setting up HPC cluster with CentOS. Despite of the well documented steps, note that the author of the document refers to HPC cluster simply as cluster, which is ambiguous. There are three basic motivators for creating a cluster: high performance computing (HPC), network traffic load balancing, and service resilience in the form of high availability (HA). The author should be specific in the document about the HPC cluster. If RDMA (Infiniband) network is involved, a configuration guide is provided in RedHat literature.

Here is an example of deploying HPC cluster in AWS. Here is the guide to deploy HPC pack in Microsoft technologies.

HPC and Big Data

HPC and Big Data are two distinctive computing paradigmes. Although there is some signs of convergence and blurred boundaries, it is still a long way before one can treat HPC and Big Data interchangeably. This paper does a phenomenal job in comparing the two paradigms.

The fundamental difference lies in the respective problems they intend to address. HPC focuses on the large computational loads, whereas Big Data targets applications that need to handle very large and complex data sets (usually in the order of multi-terabytes or exabytes). Many scientific data analytics applications are becoming I/O bound in modern systems, such as seismic algorithms, Big Data applications are thus very demanding in terms of storage, to accommodate such a masive amount of data, while HPC is usualy thought more in inters of sheer computational needs. The open-source projects in Big Data also aims to run on conventional hardware to make it easier and less expensive to scale. This is not the main focus of HPC.

So, you can run Big Data (e.g. Hadoop) analytics jobs on HPC gear. On the other hand, you can’t run HPC jobs on commodity hardware as commonly seen in the Big Data stack. Both HPC and Hadoop analytics use parallel processing of data. In a Hadoop/analytics environment, data is stored on commodity hardware and distributed across multiple nodes of hardware. In HPC, where the size of data file is much greater, data storage in centralized. Also, because of the sheer volume of its files, HPC also requires more expensive networking communications such as Infiniband, because the size of the file it processes require high throughput and low latency.

In BigData job, each query in Hadoop reads data from disk and runs as a separate MapReduce job. Spark enables in-memory iterative processing (through the RDD abstraction), allowing the user to query repeatedly on a dataset without having to perform intermediate disk operations. RDD are exposed in the Spark API where each dataset is represented as a read-only object, and transformations are invoked using methods on these objects. For an example project, check out this post.

The underlying software stacks for HPC and Big Data are fundamentally different, mainly due to the differences represent in their target class of applications, as outlined in the diagram below:

software stack difference between HPC and Big Data

As to which one is for me, the over-simplified advice is: if you can avoid HPC and just use Hadoop for your analytics, do it. It is cheaper, easier, and more cloud friendly. However, bear in mind that an all-Hadoop shop is not possible for many industries such as life sciences, weather, pharmaceutical, and academic applications.