git hook on Digi Hunch

Clean up Git repository

Sat, 26 Oct 2019 20:33:00 -0400

A BitBucket repo has a hard limit of 2GB in size, and soft limit of 1GB. This is not expandable as per Bitbucket and contributors will start receiving warnings once soft limit is reached. We can tell the usage of a repo from the landing page of the repo in BitBucket.

Git is a distributed version control system for source code management, which implies the followings:

It is intended for source code, or configuration code; but not for storing build artifacts, or installers;
Git remembers every single commit, including the ones associated with large files;
Even a contributor deletes a large file (“git rm filename”) after commit, the large file is only removed from the HEAD. The historical commit still stores the file. After all, the whole point of version control is to survive crazy deletion.
distributed means that those large files will be pulled down to contributors laptop (waste everybody’s space although up to 2G:);

With all these implications, shrinking the size of a repo isn’t as straightforward as just removing large files from current commit. We’d have to rewrite the commit history. Here are the steps we should take once repo size grows over the soft limit.

Clean up remote orphaned branches

Removing these branches (remotes/origin/branchname) per se does not free up space. It simplifies the branch structure, leaving /remote/origin/HEAD the only branch left to cleanse for the rest of the steps.

# git push origin --delete branchname

Remove useless files in current commit (HEAD)

In this step we remove useless files in current commit. Again we should not expect much space freed because all file committed previously, even deleted, are still stored. They are just now showing up in the working directory. For this step, we can create a separate local dir on Mac:

mkdir -p /Users/digihunch/repo-cleanup
cd /Users/digihunch/repo-cleanup

Now within the new directory, we create a bare repo and then the full repo:

git clone --mirror https://gh@bitbucket.org/digihunch/source.git
git clone https://gh@bitbucket.org/digihunch/source.git

Then we dive into the full repo and identify the large files:

find . -type f -size +1000k -exec ls -lh {} \; |awk '{print $9":" $5}'

We can run “git rm ” against the files identified as too large or deletable. Then commit and push to remote repo. This removes large files from current commit.

Remove large file and the relevant commits in the history

As previously mentioned, we have to re-write the history so history forget about the large files. After this step, the historical commits that large files are associated with will all be deleted. Compare the two charts below to understand what the effect is:

We can use the bare repo created in the last step, with “git filter-branch” tool to cleanse the branch tree. Some advocate as a faster third party tool BFG Repo-Cleaner as a faster, third-party alternative but I usually lean towards native tool. This article explains the command switches.

git filter-branch -f --tree-filter "rm -rf \large_file.zip" --prune-empty -- --all

After this steps the repo should be cleansed. According to this guide from BitBucket, we still need to contact their support to run a garbage collection for us in order to see the size change. It even takes time for the size to be reflected after garbage collection. This reference also does great job explaining what we need to do.

Other contributors re-sync history

It is important to understand that the step above modifies history. Although the commit hash did not change, they are assigned with different commit-ids and you can tell from the commit history where it displays former commit id.

This activity only affects remote repository. Each contributor’s local repository still stores the old commits and should be sync’ed with the remote origin by deleting the entire repo and run “git clone” again. Although not welcomed by every individual contributors, but it is a necessary evil and better approached with explicit instruction.

Because this activity takes higher risks, changes each commit, involves vendor support and requires activities by each contributor, the support team should focus on preventing this from happening instead of fixing it.

Configure pre-commit hook as a preventive measure

As we have more Ansible tasks related, working directory becomes complicated and sometimes contributors accidentally committed large unwanted files (and pushed into the remote repo). Down the road, the best practice is to prevent contributors from committing junks.

The best spot to detect this should be a pre-receive hook on the server side, which is only available with self-hosted Bitbucket Server. Unfortunately, this is not a viable option for Bitbucket cloud. Our best bet is client-side pre-commit hook, in which a script performs size check when contributors run “git commit”. The purpose is to fail the commit if total file size is over the limit (20M), and the hook itself should be version controlled as well. Compared to (server side) pre-receive hook, the drawback of (client side) pre-commit hook is it requires initial client configuration. The upside is it captures large files before commit.

This hook can be a shell script as simple as this:

commitsizelimit=20
stagedfilelist=$(git diff --name-only --cached)
stagedfilecnt=`echo "$stagedfilelist"| sed '/^\s*$/d' |wc -l`
if [[ $stagedfilecnt -gt 0 ]]; then
 totalcommitsize=$(du -cm $stagedfilelist | tail -1 | cut -f 1)
 # Redirect output to stderr.
 exec 1>&2

 if [[ $totalcommitsize > $commitsizelimit ]]; then
 echo "Warning: Total size of all files in staging area is "$totalcommitsize"MB, exceeding the limit of "$commitsizelimit"MB."
 echo " To list files by size, run 'du -ch \$(git diff --name-only --cached)'"
 echo " To drop large ones from staging area with 'git rm -f filename'"
 echo " To bypass this limit, use 'git commit --no-verify'"
 exit 1
 fi
fi

In the repo we will have a .githook directory to store hooks (e.g. ~/source/.githooks/pre-commit) and point to the hooks directory using the following command:

git config core.hooksPath .githooks

Git Explained 2 of 2

Tue, 18 Jun 2019 17:06:00 -0400

This is a continuation from Git Explained 1 of 2 where the fundamental concepts are covered. In this article we introduce some tools for customization and maintenance.

As for Git configuration, there are two files to dictate your Git configuration. ~/.gitconfig and .git/config in project directory. Running `git config –list –show-origin` shows all config entries and where they are from. For example you can custom your Git hooks location. Neither of the two files are being version controlled, so the configuration is only effective in the client environment,

Server side Git hooks

Git implementation supports server side hooks (pre-receive, update, post-receive). They are bash scripts placed in .git/hooks with specific names, fired upon event occurrence. Exit code of 1 from the scripts fails the event. Since server side Git hook consumes server resources, many repository hosting vendors (e.g. BitBucket Cloud) do not support it. You will need to enable it in self-hosted servers (e.g. BitBucketServer).

Client side Git hooks

Since server side hooks are not widely supported in every vendor, client-side Git hooks is good alternative places to implement functions such as code style check, commit size check, etc

The default directory for hooks is .git/hooks/ under the project directory is not version controlled and not easy to share with the team. If the hook needs to be shared among project contributors, we can place hooks files in .githooks/ under project directory. This will make the hook files version controlled. In addition, we need to point the hooks to this directory in configuration, by running `git config core.hooksPath .githooks` from project directory.

Web hooks

Web hooks can be thought of as an event notification mechanism. It is a common feature provided by VCS repository hosting providers. If a certain type of event occurs to the repo, web hook will fire an RESTful API call. The HTTP Endpoint, authentication secret and event payload are pre-configured in the repo settings. Web hooks are commonly supported by Git-based repo implementation, such as BitBucket cloud, GitHub, GitLab or AWS CodeCommit. It can also be enabled in self hosted Git repo. Web hook is a powerful tool to drive downstream event, such as Jenkins to start building the code. The major difference between server side hook and web hook is web hook is RESTful API driven, whereas server side hook is executing a script.

Pipelines

Some repo hosting vendor also provide a feature named pipeline. BitBucket has Pipeline as a CI/CD tool, AWS has AWS Code Pipeline, and GitLab offers CI/CD pipeline as well. These pipelines are usually in the form of a YAML file in the repo with a special name. The YAML spell out the steps to perform along the pipeline.

Squash Commits

For small projects I was in the habit of committing to main branch. I often need to squash a number of commits into one to “clean up”. Usually a Pull Request (e.g. in GitHub) or Merge Request (e.g. in GitLab) have such option during approval. We can squash a few commits with git command as well (suppose we want to squash the most recent 16 commits):

git rebase -i HEAD~16
git push origin +main

The command will open text editor to allow you to mark what to do with each commit. You can mark all except one commit as squash. For the commit to keep, mark it as pick. Then save the text editor. Git rebase will perform the squash for you. However, since this is a rebase, do not do this if there are other collaborators working on the same branch.

Cleanse a repository

Take BitBucket cloud for example, the size of a remote repo has a non-expandable hard limit of 2GB, and a soft limit of 1GB. Once the soft limit is reached, a warning will be displayed on Bitbucket’s landing page as well as when contributors pushes changes. Once the hard limit is reached, the entire repo will turn read-only mode.

Space consumption can be caused by accidental committing of large file. As covered in the previous post, files are stored as blob objects in .git directory. If a file was deleted by `git rm` command, it simply means it is de-referenced from the next commit and on. After all, Git as a distributed version control system, has the ability to magically restore the deleted file when we want. The cost of that magic, is that deleted file is permanently stored in the repo, in the form of blob object, although not present in the working directory. It consumes space not only in remote repo, but also in the local repo of each contributor.

The purpose of Git repository is to store source code which are fairly small texts. However if a contributor pushed in large files, it can be tricky to cleanse a Git repo on the remote side. Here is some guidelines:

Before cleansing, identify large unwanted files in current working directory, delete them with `git rm` and then commit this change in master branch.
Consolidate branches (e.g. delete useless remote branches with `git push origin –delete branch_name`). This step itself does not free up space in remote repo but it simplifies the branches;
Remove large blob objects and commit objects that reference them. This step essentially is re-writing the commit history of repo. Given the risk, it is recommended to perform this step from a separate local project directory with bare repo only without working directory. This article presents some good advices and illustrations. The command recommended is `git filter-branch -f –tree-filter “rm -rf \large_file.zip” –prune-empty — –all` ; and if that is too slow here is an alternative named BFG Repo-Cleaner.
After this step, each commit will have a new hash id. For the repo size to reduce, either wait until the next garbage collection on the server side, or request cloud vendor to run a manual garbage collection.