bash on Digi Hunch

Automated Deployment Pipeline 3 of 3

Wed, 14 Oct 2020 17:27:00 -0400

Background

We have previously covered a pipeline example with Jenkins calling Ansible to leverage OpenSSH configuration and Ansible inventory. We also discussed a use case with declarative pipeline.

In this posting, I will provide another advanced example, built on declarative pipeline. The pipeline file will be pulled from Git repository. Also, the script is executed on a remote agent, instead of the Jenkins master server. The reason this example is important, is that:

Jenkinsfile is now version controlled (pipeline as code);
Service script (e.g. python) is also version controlled from a central repository;
Computing resource is provided by a remote agent. Since the script is pulled before running, the agent is still fungible;
The result from service script execution is archived (similar to the way build artifact is stored in Jenkins) in master.

The architecture of this pipeline now becomes the followings:

Configure Agent

Jenkins has a plugin called SSH Build Agent, that allows you to configure a Linux agent, communicating with Jenkins master in SSH. For Window agent, it uses JNLP to communicate with master, which is outside of our scope of discussion. As I touched on in previous post, Jenkins uses its own implementation of SSH protocol to achieve this. This means that it cannot re-use the configurations in ~/.ssh/config and thus the ability to do SSH chaining is eliminated. This is incompatible with our architecture so I have to register Jenkins agent using a different launch method “Launch agent via execution of command on the master”. The execution of command on master can still leverage OpenSSH config file. In order to do so, we must copy the agent.jar file to the remote agent first (URL is ${JENKINS_URL}/jnlpJars/agent.jar). Then use SSH command to call the jar file from agent (aka slave) machine. You may add some java argument for troubleshooting. Below is an example configuration for the node.

I shall also note that if the remote agent is a different operating system where the path of bash might be different, then you need to include the directory of bash executable in PATH environment variable. That can be done as in the screenshot above (PATH=${PATH:/usr/bin}). If this is incorrect, you might run into issues when running sh step in Jenkins pipeline. Here is an article about this. To translate that page, the symptom of this issue includes:

Error from pipeline execution that says the following, which is very generic:

process apparently never started in /home/dhunch/jenkins/workspace/site-remote-job@tmp/durable-b997d26c 
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

Job status shows:

hudson.AbortException: script returned exit code -2
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:659)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:605)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:549)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Threaddump (only available during execution) shows:

Thread #6
 at DSL.sh(awaiting process completion in /home/dhunch/jenkins/workspace/site-remote-job/durable-b997d26c; recurrence period: 9543ms; check task scheduled; cancelled? false done? false)
 at WorkflowScript.run(WorkflowScript:9)
 at DSL.script(Native Method)

This issue is tricky because none of the symptoms above make you think about the environment variable defined for the agent, and that the issue does not occur until you actually execute a Jenkins pipeline, usually well after the node agent is registered, and only impacts shell step (sh). It is recommended to test this with sh steps in pipeline.

Configure Repository

Our example job executes service script pulled from SCM, on a remote agent. Then the master pulls the result file to itself. The git repository can be set in Jenkins job, where you specify that the script named Jenkinsfile (from the repo) is the pipeline file that needs to be executed.

This is fairly simple, but what makes the situation more complex is the following few requirements:

it’s the agent node that needs git (installed) and clone to repo;
the master does not need (and should not attempt to) clone from repo;
the master needs to do its job without having to pull SCM;

We need to be able to specify whether each step needs to pull from SCM. The following pipeline syntax shows how this is done:

def agent_dir = 'initial_value'
pipeline {
 agent none
 options { skipDefaultCheckout(true) }
 stages {
 stage('Execute Job') {
 agent {
 label 'remote-agent-customer1'
 }
 options { skipDefaultCheckout(false) }
 steps {
 echo 'Executing job on node'
 sh 'whoami && pwd'
 script {
 agent_dir = sh(returnStdout: true, script: 'echo -n ${WORKSPACE}')
 // the variable should not include carriage return
 }
 sh "echo ${agent_dir}"
 }
 }
 stage('Pull Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 echo 'Pulling job below'
 sh "echo ${agent_dir}"
 sh "scp dhunch@site1:\"${agent_dir}\"/result.csv ./"
 }
 }
 }
}

In this example pipeline file, we declare agent none for the pipeline, then an agent for each specific stage. we also specify the option skipDefaultCheckout as true at the step where pulling from SCM is not needed. This allows us to finish job with multiple agent, and only pull from SCM as needed. This snippet also exemplifies how to declare a variable, assign it from stdout from one agent, and persist the value across ensuing stages.

Store Result

The reason we run pipeline jobs on this remote agent is because it is sitting in customer network and has local direct access to data. So this is perfect for situation such as data analytical jobs, which access to database on local network and store result. We need to pull the result file back to agent and make it available on Jenkins. This is so similar to “archive artifact” task (commonly seen in CI process) that we can simply use its plugin to achieve what we need. Before archive artifact, we need to pull it to local (master), as shown in the example code above. After that, we need another stage to archive the result.

 stage('Archive Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 archiveArtifacts artifacts: '*.csv',onlyIfSuccessful: true,fingerprint: true
 }
 }

I believe there is plugins to compress artifacts as well.

Conclusion

At the end I’d like to reiterate my perception about Jenkins. It is a very generic and adaptive automation platform that originally evolved from use cases in build automation. Due to this original root, many components in Jenkins are named around Continuous Integration use cases, such as the “build” button, and the “archiveArtifacts” step. These misnomers underplays what Jenkins can potentially do in continuous deployment or other automation scenarios. It is important for automation engineers to understand Jenkins components and plugins, through their functionalities and not by the name, and therefore make creative use of Jenkins as automation engine in all scenarios.

An alternative to this proposed pipeline would be Ansible Tower, a commercial project based on open-source Ansible, but with nice UI support. Ansible Tower is Ansible oriented, and it does not have everything that Jenkins can do. It should still be a decent alternative given the proposes pipeline uses Ansible a lot.

Automated Deployment Pipeline 2 of 3

Tue, 06 Oct 2020 22:05:00 -0400

In this posting, we continue to discuss Jenkins’ ability to automate deployment routines. Jenkins supports freestyle project out of the box, as well as Pipeline with several plugins. Freestyle project allows user to specify multiple steps on UI. This does not scale well when your entire process involves many steps. As explained on Jenkins’ website:

While standard Jenkins “freestyle” jobs support simple continuous integration by allowing you to define sequential tasks in an application lifecycle, they do not create a persistent record of execution, enable one script to address all the steps in a complex workflow, or confer the other advantages of pipelines.
In contrast to freestyle jobs, pipelines enable you to define the whole application lifecycle. Pipeline functionality helps Jenkins to support continuous delivery (CD). The Pipeline plugin was built with requirements for a flexible, extensible, and script-based CD workflow capability in mind.

So although freestyle projects are easy to set up, and can do technically everything that Jenkins pipeline can do, the major advantage of Jenkins Pipeline is the ability to manage multiple-step as code, and version control the pipeline-as-code. Here is some more information.

We will start with a freestyle project to understand Jenkins’ ability and then advance to building pipelines.

Freestyle projects

As mentioned, Jenkins calls a task a “build”, and the build can be triggered in a variety of ways:

Time schedule (with or without parameter)
Remotely via API
On completion of other projects
Poll SCM for changes
Commit to SCM

There are also several ways to execute a job:

Execute Shell command, batch command or groovy script
Invoke Ansible adhoc command, playbook or vault
Conditional on specified boolean value, file existence, etc

After the job one can specify post-build jobs, for example:

start other build projects
notification of various means
publishing result file

In deployment, it is common task to execute a task over SSH, using SSH command or invoke Ansible command with Ansible plugin. The former fits simple command line tasks. For example:

The Ansible plugin is good for more steps and more complicated inventory hierarchies. This post includes an example of an inventory involving multiple layers. The other limitation with SSH command is lack of a straightforward configuration to escalate privilege and run remote command. On the other hand, Ansible addressed this with become method. Below is a screenshot

In order to use Ansible command, you will also need to specify where the Ansible binary in Global Tool Configuration. You will also need to store vault credential in Jenkins credential store so it’s not being prompted.

Jenkins Pipeline

Jenkins pipeline allows one to describe actions in a pipeline in groovy Domain Specific Language (DSL). There are two styles of pipelines:

Declarative pipeline is identified by a block named ‘pipeline’, it is relatively new and supports the pipeline-as-code concept. It can be stored as Jenkinsfile in code repository or edited in Jenkins’ UI.
Scripted pipeline is identified by a block named ‘node’, it is the conventional format and can only be edited in Jenkins’ UI.

To fully understand the two styles of pipelines you also need to know declarative programming and imperative programming. Read this instruction for a better explanation. Here is another good one that focus on their differences.

Jenkins’ plugin provides UI components for user to input information (e.g. Invoke Ansible Command), now with Jenkins pipelines, plugins can provide step functions in order to help Jenkins user. This page from Jenkins lists the most common ones, many of which requires plugin installation.

One example of step function is SSH Pipeline Steps. It allows one to issue SSH connection from Jenkinsfile. However, in our deployment scenarios, it has some limitations:

Unable to match a group of host by specific pattern
Cannot use host name aliases
SSH tunneling is not supported
No means of privilege escalation

Out of these 1 through 3 are due to the fact that SSH step function does not use the OpenSSH configuration on the machine. The jenkins-ssh-slaves plugin uses trilead SSH2 implementation written in Java. Only OpenSSH implementation uses ~/.ssh/config file. This creates problem whenever SSH tunneling is needed. For example, to register a remote slave node via SSH tunneling, we need to select “Launch agent via execution of command on the master” instead of “Launch agent via SSH” as launch method. Here is an instruction and below is what it looks like:

Similarly, if we use ssh-agent plugin as tool for deployment, we cannot use any configuration made by OpenSSH. Therefore Ansible in Jenkins Pipeline is a better tool for deployment because it can use OpenSSH.

Pipelines with Ansible

In this section I demonstrate the use of Ansible playbook and adhoc command in pipeline through two examples.

In example 1, the job pulls authorized key file from SCM, and then use a playbook from SCM, to push the key file to all servers in the specified inventory.

Here is the playbook file for example 1:

---
# Example:
# ansible-playbook -l all -i ~/ansible/inventories/bh.yml push-key.yml --ask-vault-pass
- name: push key to target
 hosts: all
 tasks:
 - name: sync key to host
 copy:
 src: "{{authorized_keys_src}}"
 dest: "/home/dhunch/.ssh/authorized_keys"
 force: yes
 mode: 0600

Here is the pipeline script (note ampersand is mistakenly displayed as & in the box below):

import java.net.URLEncoder;
 
pipeline {
 agent any
 
 options {
 skipDefaultCheckout(true)
 }
 environment {
 BITBUCKET_CREDS = credentials('bitbucket') 
 //BITBUCKET_CREDS_USR and BITBUCKET_CREDS_PSW are set
 BITBUCKET_CREDS_USR = "${BITBUCKET_CREDS_USR}"
 BITBUCKET_CREDS_PSW = URLEncoder.encode("${BITBUCKET_CREDS_PSW}", "UTF-8") 
 // if password contains special character we need to url encode it. e.g. @->%40
 }
 stages {
 stage('Start') {
 steps {
 echo "Starting pipeline ..."
 deleteDir()
 }
 }
 stage('Prep') {
 steps{
 wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: "${BITBUCKET_CREDS_PSW}", var: 'RANDOM']]]) {
 // MaskPasswordsBuildWrapper requires Mask Passwords Plugin and is to mask specific string in console output. 
 // Otherwise BITBUCKET_CREDS_PSW will display in the clear
 sh "git init && git config core.sparsecheckout true"
 sh "git remote add origin https://$BITBUCKET_CREDS_USR:${BITBUCKET_CREDS_PSW}@bitbucket.org/vendorcompoany/configmanagerepo.git"
 sh "echo 'public_keys/*' >> .git/info/sparse-checkout"
 echo "Downloading key file"
 sh "git pull --depth=1 origin master" 
 }
 }
 }
 
 stage('Deploy to Site 1') {
 steps {
 echo '> Deploying to Site 1 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site1.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 1 ...'
 }
 }
 
 stage('Deploy to Site 2') {
 steps {
 echo '> Deploying to Site 2 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site2.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 2 ...'
 }
 }
 
 stage('Deploy to Site 3') {
 steps {
 echo '> Deploying to Site 3 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site3.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 3 ...'
 }
 }
 }
 post {
 always {
 cleanWs()
 }
 }
}

In this example, we pull a sub-directory from git repo. We use two tricks to minimize amount of traffic. First, we use git pull with depth=1 so only the required recent commits are pulled, not the entire history. Second we use sparse checkout to get result from a sub-directory, not the entire repo.

When connecting to repo, password is required. We pull the credential from Jenkins’ credential store and they are masked by default. However, the password must be converted to URL string when used in git remote add origin. Otherwise if the password contains special character the URL will not work.

This modification to password brings about another challenge, password masking during the job execution. We will have to build our own wrapper function using class MaskPasswordsBuildWrapper in order to mask any variable.

In example 2, we wrap an Ansible step function to call adhoc command to check version on all hosts. Here is the pipeline script:

def HunchVersionQuery(pattern, siteinventory){
 step([
 $class: 'AnsibleAdHocCommandBuilder',
 ansibleName: 'Ansible on Mac',
 inventory: [$class: 'InventoryPath',path: siteinventory],
 hostPattern: pattern,
 module: 'shell',
 command: 'cat /etc/*release',
 forks: 1,
 vaultCredentialsId: 'ansible-vault-pass'
 ]);
}

pipeline {
 agent any

 stages {
 stage('Start') {
 steps {
 echo 'Staring Pipeline'
 }
 }
 stage('Query Site1') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site1.yml')
 }
 }
 stage('Query Site2') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site2.yml')
 }
 }
 stage('Query Site3') {
 steps {
HunchVersionQuery('*app','~/ansible/inventories/site3.yml')
 }
 }
 }
}

Because Ansible plugin does not provide a warpper function for adhoc command step, we will have to build our own wrapper function in the above code.

Automated Deployment Pipeline 1 of 3

Wed, 30 Sep 2020 22:04:00 -0400

The business case

You launched a software application. You installed it on two customer sites. You support the application mostly by SSH to customer server and run Bash commands, or slightly better, Bash scripts.

The product is a hit to the market. You hired 20 support specialists in a customer service department. The dream client came through: an enterprise giving you a fleet of 100 servers to deploy your application on.

More staff, more business, more installations, more incidents, but the same old command driven steps. Problems:

Non-standard support procedures. Every one takes notes and everyone’s notes are slightly different.
Information sharing among team members are ad hoc, and at high level.
Post-mortem discussion is driven by memory and command fragments, instead of evidence end-to-end

If that looks like your organization, chances are you also suffer from some secondary damages over the long term, such as:

Downtime resolutions rely on the knowledgeable few
Documentation helps. But it never catches up to the latest version of application
Lack of auditing of commands during support

I propose an automation scheme to existing support and deployment practice. This automation scheme combines a suite of common technologies, such as Bash, Python, Ansible, OpenSSH and Jenkins. The automation allows the department to, either fully or partially, operationalize the steps in support and deployment, and eventually shift towards agile practice.

Bash, Python and Ansible

Bash script is based on shell command, perfect for running critical system tasks such as volume management. When it turns into a script, it can be cumbersome, especially with complex data structure. Python, as a tool for system administration, is a good complement to that.

Python 2 comes with most Linux distributions, and is also a dependency of other built-in tools such as YUM. Python3 can be installed easily from default YUM repositories. Both Python2 and Python3 can exist on the same operating system, although new module development are now shifted to Python3. Python’s syntax is very simple and offers object-oriented programming ability. Moreover, there is an entire open-source community behind Python, which offers modules in every aspect of IT (for example, Datastax has a driver module for connecting to Cassandra). Those modules are installed with PIP tool, or PIP3 for python3.

Both bash and Python executes on local machine. To run them on remote servers over SSH. You want to have a list of target hosts, and specify which one to execute the script against. This is where Ansible comes in handy. Ansible is superior in the following aspects:

Free and open-source, with commercial alternative (Towers);
Inventory management (inventory);
Desire state engine (roles)

Ansible is built on Python and is agent-less. Connectivity to remote host is done via secure shell so it can take advantage of existing SSH configurations. Job execution on the target machine is done through Python. With Python you can also develop custom module in Ansible. For some use cases in customer support with Ansible, refer to my two previous postings about Ansible at scale.

Jenkins

The tools above forms a package for automation. The issue is that all of them are command-line based. Any task that requires Ansible requires the IT professional craft up long command, such as running playbook, executing a role, or ad-hoc command. This is inconvenient when a task needs to be done during an incident. Such tasks also require trained professional with the relevant skills.

These tasks can be stored in, or initiated by Jenkins. Although Jenkins is well known for build automation in continuous integration, it is automation engine for any command-line based IT tasks. The button to start such tasks in Jenkins UI is called “Build”, which is also a misnomer that underplays Jenkins’ versatility: building application from source code is just one of the many IT tasks that involves multiple long running commands. In this and next article we introduce Jenkins as an engine for deployment automation.

The infrastructure architecture is diagramed as above, and with the connection across Internet, the target hosts must be hardened properly in the following aspects:

Connectivity to remote host is via SSH chaining, through an SSH proxy;
Root login must be disabled for remote session or by password;
Service user may be shared, but must be authenticated by individual RSA key pair;
Service user connected remotely needs to escalate privilege by su if needed;

I want to make a theoretical distinction between our topic here and continuous deployment. We simply focus on the technical side of deployment automation. Essentially automating a few bash scripts. On the other hand, a continuous deployment process is an extension to an existing continuous integration pipeline, with the vision to streamline the process end-to-end from code commit to production rollout. Implementing CI/CD pipelines should be approached as an organizational program rather than an individual technical initiative. Here is a good technical overview on CI/CD pipeline with Jenkins and Ansible.

Security

The security mechanism of this system is based on OpenSSH because the connectivity between servers are through SSH chaining. RSA key authentication must be used in order to encrypt traffic with password-less login. Connection to an SSH host can be done through a proxy server. Below is an example of SSH configuration:

Include customer1.config
Include customer2.config

Host *
 IdentityFile ~/.ssh/id_rsa
 ServerAliveInterval 60
 ServerAliveCountMax 3
 Compression yes
 ControlPersist 3h
 ControlPath ~/.ssh/sockets/%r@%h-%p

Host gateway
 Hostname support.digihunch.com
 User jdoe
 Port 2223

Host customer-server-0
 Hostname 192.168.201.12
 User support
 ProxyCommand ssh -W %h:%p gateway

Open SSH configuration file (~/.ssh/config) needs to be configured properly with useful host names and aliases. To prevent the config files from growing too long, include statement can be used to reference other configuration file (available with OpenSSH v7.3 sp1 and up).

The host names (as well as aliases) listed in SSH configuration can be directly referenced in Ansible inventory, allowing Ansible (and Jenkins) to reference site by alias and connect to target host through proxy.

Plugins

Jenkins has a community that develops a variety of plugins, which makes Jenkins the most powerful automation platform. Here are some examples of useful plugins:

Audit Trail: output job execution history to file or Elasticsearch;
Credentials: stores credentials in Jenkins;
Pipeline: build declarative (new) or scripted (old) pipeline for Jenkins jobs;
Simple Theme: just a theme but allows console output to be dark (using CSS);
Job Configuration History: job configuration audit;
Mask password: mask variables (including password) from console output
Ansible: invokes ad-hoc commands and playbooks
SSH agent, SSH pipeline steps, SSH credentials: features related in SSH in Jenkins pipelines.
Purge Job History: purge all of build history, or purge by time and number of old builds.
Parameterized Scheduler: schedule to run a job and provide parameter
Workspace cleanup: clean up workspace when invoked.

In the next article, we will go over some common job configurations.

Personal Vim cheatsheet

Mon, 07 Oct 2019 17:52:00 -0400

This is my personal cheatsheet as intermediate Vim user so I skipped the ones that I consider basic. All the commands listed are used in command mode for fast editing.

Command execution

In command mode, use colon to start ex command. Here are some examples of ex commands:

e: edit file
g: global command
q: quit
w: write
s: substitute

The commands (g)lobal and (s)ubstitute are heavily used in string manipulation. The rest are summarized here:

:e!	reload current file discarding all unsaved changes
:e newfile.txt	open file newfile.txt for editing
:e .	load current directory
:w!	force write (if permission allows)
shift + zz	equivalent to :wq!
shift + zq	equivalent to :q!

Operator

.	repeat last operation
fx	find next character x on the same line

Text Editing

y for yank(copy), i for inside, a for around, d for delete, w for word, p for paragraph or paste. Examples:

yyp	copy current line and insert after
diw	delete the entire word where the cursor sits in (dw deletes from cursor to end of word; db deletes from cursor to beginning of word)
shift + V	select entire line in visual mode (v selects character in visual mode)
2>	visual mode: indent twice on all selected lines edit mode: indent once for 2 lines
3<	visual mode: outdent three times on all selected lines edit mode: outdent once for 3 lines
di”	delete everything between the double quotes surrounding the cursor (exclusive); use c instead of d to finish the same effect with insert mode
di>	delete everything between < and > surrounding the cursor (exclusive); use (c)hange instead of (d)elete to finish the same effect with insert mode
dit	delete everything between tags. e.g. <xml>contenttodelete</xml>
dip	delete the entire paragraph
da’	delete everything between the single quote surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert mode
da}	delete everything between { and } surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert mode
dt.	delete all characters until the next .
i	insert at cursor location
shift + I	move cursor to first non-blank character of line and start in insert mode
a	insert at the location next to cursor
shift + A	move cursor to last non-blank character of line and start in insert mode
0	move to beginning of line. ^ moves to first non-blank character in the line. $ moves to the end of line

Note: wherever d is used in this table, c can be used instead for the same effect but switch to editing mode at the end.

String Manipulation

The general patterns are:

[range]g/pattern/cmd
[range]s/match/replacement/option

If range is not specified, it applies to current line only! To specify the whole file, use % range. You may also specify line range such as “10,20”. Here are some examples:

:%s/bacon/lettuce	For every line of the file, replace the first occurrence of bacon in each line to lettuce
:%s/bacon/lettuce/g	For every line of the file, replace all occurrences of bacon to lettuce
:s/bacon/lettuce	For current line, replace the first occurrence of bacon to lettuce
:s/bacon/lettuce/gi	For current line, replace all occurrences of bacon to lettuce, case insensitive
:g/bacon/d	delete all lines that contain pattern ‘bacon’
:g!/lettuce/d	delete all lines that do not contain pattern ‘lettice’; or use :v/lettuce/d instead
:g/^\s*$/d	delete all blank lines. \s* represents zero or more white spaces

Bookmarking

ma	mark cursor line as bookmark a
`a	jump to cursor position at line a
‘a	jump to beginning of line a
`.	jump to last line where change occurred
“	jump back

Insert Mode

Ctrl + N

Auto complete

Edit and run

While tmux and screen can help split screen in Bash, we sometimes need to split a bash screen to run a quick command when we’re already in vim. This can be done with some simple commands.

:term	Open up a terminal above vim. You can also spell :ter or :terminal
Ctrl+W; Ctrl+W	Press Ctrl+W twice can help you toggle between the terminal and vim buffer
Ctrl+D	Close the terminal

This can be very helpful when you are debugging code and need it run repeatedly. You don’t need to exit vim just to run a command and come back. Note that while you’re in terminal, you can’t use Ctrl+W as a shortcut key to backspace a word. Use Alt + Delete instead.

Linux Admin Basics 3 of 3 – text processing, regex, sed & awk

Wed, 06 Jun 2018 19:56:00 -0400

Most of the text processing can be processed by awk and sed. Sed is non-interactive stream editor that allows you to specify all editing instructions in one place and execute them on a single pass through the file. Awk is a pattern-matching programming language.

Using sed and awk requires some understanding of regular expressions. Here’s the basics of regular expression signs:

sign	operation
^	matches beginning of line
$	matches end of line
.	matches any single character (wildcard)
*	repeat previous token zero, one or more times.
+	repeat previous token one or more times
?	repeat previous token zero or one time
[…]	matches any one of the class of characters enclosed between the classes. ^ as first character reverses the match – is used to ndicate a range of characters
()	groups regex
\|	either preceding or following regex can be matched
{n,m}	matches a range of occurrences of the single character that immediately precedes it. {n} will match exactly n occurrences {n,} will match at least n occurrences {n,m} will match any number of occurrences between n and m

Common expressions

exp	interpretation
[^0-9]	excluding number
[15]00*	matches “10”, “50”, “100”, “500”, “1000”, “5000”. Here the first 0 is literal, the second is modified by *, see the table above
.*	any number (including 0) of any character
<.*>	any html tags
book	matches book with preceding and following spaces
books*	matches books, or book, but not “book.” “book?” etc
book.*	matches book, followed by any number of characters, or none followed by a space

Note that regular expression comes in several different flavours, which can be confusing and frustrating. This is a good summary. There are DFA (Deterministic Finite Automata) based engines and NFA (Non-Deterministic Finite Automata) based engines:

NFA based engines can “go back” in the regex, used in Perl, Python, vim, sed and GNU grep.
DFA based engines cannot “go back” in the regex, used in awk and BSD grep.

Standard	IEEE POSIX BRE	IEEE POSIX ERE	PCRE
Detail	Basic Regular Expressions	Extended Regular Expressions that add repetition, alternation on top of BRE	Perl Compatible regular expression.
Engine	DFA	DFA	NFA
GNU grep	grep by default, or grep -G	egrep grep -E	grep -P
BSD grep	grep	egrep
GNU sed	sed	sed -r	NA. Just use perl
BSD sed	sed	sed -E	NA. Just use perl
awk		awk

The best way to check isn on BSD manual and GNU.

Here are several examples of sed and awk I came across at work:

Find and remove duplicate lines:

awk '!x[$0]++' input_file.txt > output_file.txt

Remove white spaces at the beginning and end of each line:

awk '{$1=$1}1' input_file.txt > output_file.txt

Print with multiple dilimiters (;, , , and |)

awk -F '[;,|]' '{print $1, $3, $5}'

Print with calculation between columns

awk '{res=$1-$2;print res,$0}'

Print rows conditionally

awk '$1>20{print;}'

Prefix each line of a file

awk '$0="PREFIX|"$0' input.txt > prefix.input.txt

Replace string original to new in file

sed -i 's/original/new/g' file.txt

Remove multiple patterns

sed 's/pattern1\|pattern2\|pattern3//g'

Delete the first matching pattern only

sed 's/pattern//'

Remove blank lines:

sed -i '/^$/d' input_file.txt > output_file.txt

Copy from line 100 to line 500 of input file to output file

sed -n 100,500p input.log>output.log

Merge every three lines:

sed 'N;N;N; s/\n/ /g'

Linux Admin Basics 2 of 3 – shell scripting

Sun, 18 Mar 2018 15:18:00 -0400

Bash Options

You can set bash option in two ways, with shopt or with set command. For example:

shopt -s extglob
set -o nounset

Here is a list of options. I often use “set -e” right after shebang to tell the script to exit upon failed command, because the rest of the script will be error-prone after the failed command. I can use “set +e” to negate the flag.

Also, we can use the unset command to delete the variables during program execution, or the export command to export a variable or function to the environment of all the child processes running in the current shell.

User menu and argument processing

It is a common task in bash scripting to display user menu or process argument. Both use case structure. Here are the examples:

User menu:

#! /bin/bash
select car in BMW TOYOTA TESLA
do
 case $car in
 BMW)
 echo "X3";;
 TOYOTA)
 echo "Camry";;
 TESLA)
 echo "Some";;
 *)
 echo "Unknown";;
 esac
done

Argument processiong example:

#! /usr/bin/bash
while getopts ":ht" opt; do
 case ${opt} in
 h ) echo "option h"
 ;;
 t ) echo "option t"
 ;;
 \? ) echo "Usage: cmd [-h] [-t]"
 ;;
 esac
done

Try to run the script with the following options:

$ test.sh -t
$ test.sh -h
$ test.sh -ht
$ test.sh -th
$ test.sh -a

Build-in commands declare, local, let, eval

The declare command allow you to assign an attribute to a variable. For example:

-a indexed array
-A associative array
-i integer
-r read only

The local descriptor declare a variable that is only effective in the code block. For example, within the function.

With command let, each arg is taken as arithmetic expression:

$ A=1
$ B=2
$ let sum=$A+$B
$ echo "A=$A, B=$B, C=$C"
$ A=1, B=2, C=3

With command eval, each arg is taken as a string:

$ A=1
$ B=2
$ eval sum=$A+$B
$ echo "A=$A, B=$B, C=$C"
A=1, B=2, C=1+2

Commands trap and inotifywait

The trap command that you can use to catch signals from script and execute code. For example:

trap "info 'caught interrupt, will stop'; exit 2" INT

Inotify is a file system monitoring mechansim. The inotifywait can be used in many complex scenarios. It requires inotify-tools package. As an example, the following snippet use inotifywait to montor a directory, and trigger rsync upon changes

#!/bin/bash

DESTHOST=172.17.23.132
DESTHOSTDIR=/www/htdocs/
SRCDIR=/www/htdocs/

inotifywait -mr --timefmt '%d/%m/%y %H:%M' --format '%T %w %f' \
-e create,delete,modify,attrib $SRCDIR | while read DATE TIME DIR FILE; do
 $FILECHANGE=${DIR}${FILE}
 rsync -avze 'ssh' $SRCDIR root@${DESTHOST}:${DESTHOSTDIR} &amp;>/dev/null &amp;&amp; \
 echo "At ${TIME} on ${DATE}, file $FILECHANGE was backed up via rsync" >> /var/log/filesync.log
done

The example above shows how to copy a file to a remote server upon change. There is a simpler alternative to it, a tool called lsyncd, which combines rsync and inotify. lsyncd is a linux package (can be installed with yum) and remains an open source project.

The configuration is done in LUA language. Here is a simple example:

settings {
 logfile = "/var/log/lsyncd/lsyncd.log",
 statusFile = "/var/log/lsyncd/lsyncd.stat",
 statusInterval = 1
}

sync{
 default.rsyncssh,
 source="/etc/dhunch/",
 host="remote-host",
 targetdir="/etc/dhunch/",
 delay = 10,
 exclude={'*.bak',
 '*.tmp',
 'deploy.log',
 'themes/'},
 rsync={
 checksum=true,
 times=true,
 chown="hunch:hunch",
 chmod="755"
 }
}

In this example, we tell lsyncd to use rsyncssh mechanism and it can use hosts defined in openssh configuration (e.g. ~/.ssh/config). For more than one hosts, we can declare the sync section more than once. Full documentation is here.

Linux Admin Basics 1 of 3 – Bash

Fri, 02 Feb 2018 21:35:00 -0400

This article summarized my time-saving Linux tips, mostly with CentOS environment.

Bash Shortcuts

Bash shortcut illustration

These shortcuts save a lot of arrow keystrokes. Some of them may require some tweaking to work on MacOS.

ctrl – w	delete word
ctrl – a	cursor to end of line
ctrl – e	cursor to beginning of line
ctrl – k	delete to end of line
ctrl – u	delete to beginning of line
ctrl – l	clear screen
cd –	change to previous directory
alt – b	move cursor back by a word
alt – f	move cursor forward by a word
alt – .	type last parameter of previous command

Job Control

We need to first understand the following job states:

	Actively running	Displayed in Active Session
foreground job	Yes	Yes
suspended job	No	No
background job	Yes	No

These states can be managed by the following shortcut keys:

ctrl – z	send active job to suspended
ctrl – c	send SIGINT to active job to kill it
jobs	list jobs with id
bg	bring a job to background. the current shell still “owns” the job
fg	bring a job to foreground (by id)
disown	remove job from current shell’s job table. the job is still running and can be found by ps command
kill	kill a job by id. the job is no longer running and won’t be found by ps command

Note 1: running a command with ampersand(&) at the end starts the process and pushes it to the background, so you can continue typing;
Note 2: an example is to start vim with a file, ctrl-z to push to background, jobs to view, fg + job id to bring it back

Different persons may share screen on Linux shell and interact with each other. To do so, everyone need to log on the same server as the same linux user. Then the first person runs:

screen -S screen_name

Then the second (and third, etc) person runs:

screen -x screen_name

Now everyone can collaborate by seeing what each other is doing and type at the same time.

I/O redirection

	file descriptor	input or output expression
stdin	0	to read file: 0< or < for short to read a single line: <<< to read multiple lines: <<
stdout	1	to overwrite: 1> or > for short to append: 1>> or >> for short
stderr	2	to overwrite: 2> to append: 2>>

For example, here is how you display all lines between “ANYWORD” (aka “here document”):

cat <<ANYWORD
paragraph
after
paragraph
ANDWORD

In addition, to redirect between commands, pipe (|) is used. You may also redirect stdout into stderr or the other way round. Below are several examples:

command1 \| command2	redirect stdout of command1 to stdin of command2
command3 > /dev/null	get rid of stdout from command3
command4 2> file1	redirect stderr of command4 to file1
command5 \| tee file2	redirect stdout of command5 to stdin for tee command, which display stdout and write to file2 at the same time
command6 2>&1	redirect command6’s stderr into stdout.
command7 1>&2	redirect command7’s stdout into stderr

Note: in the last two examples, the ampersand just indicates the following number is file descriptor instead of file name.

For example, in the first command below, stderr is redirected to stdout, and stdout goes to file. The second and third commands are just two forms of shortcut for the first:

command > file 2>&1
command >& file
command &> file

Here is an advanced example to compare two files from two different servers:

diff <(ssh user@host1 'cat /tmp/file1.txt') <(ssh user@host2 'cat /tmp/file2.txt')

With the knowledge of I/O redirection, let’s compare the efficiency of the following two commands in dealing with a huge file:

cat huge.file | mycmd
mycmd < huge.file

In the first command, cat first causes an I/O read of huge.file, then stdout gets written into pipe buffer. Lastly, the mycmd reads it from its stdin. This is an example of inefficiency.

In the second command, huge.file is provided as stdin to my cmd. Only one read is involved as compared with two read and one write in the previous command. The second command is expected to be three times as fast as the first.

Here is a diagram to illustrate command and I/O redirection:

Check out the page for more scenarios.

Operators

First, let’s introduce some special shell variable and operators:

IFS	A special variable indicating internal field separator
;	The ; token just separates commands to run. Use it when you’d like to combine multiple lines of commands into a single line
&&	Logical AND operator. When you run command1 && command2, command2 ONLY runs if command1 returns true (success, or exit status = 0).
\|\|	Logical OR operator. When you run command1 && command2, command2 ONLY runs if command1 does not return false (fail, or exit status != 0)
{}	command combination operator. e.g. {command1;command2}
()	precedence operator

Note the use of && and || are fairly common. && is used when you want to use the first command to provide a good status for the second. || is used when you want to report error about the first command. For example:

test -d "/tmp/newdir" || mkdir -p "/tmp/newdir"
test -f "/var/run/app/app.pid" || echo >&2 "ERROR: cannot detect pid"

Attributes

I’ve been using chown and chmod to manipulate owners and file permissions against owners, groups and other users. I recently realized another command chattr (change attribute) which controls file attributes, regardless of users or groups. In other words, if you change the attribute of a file to immutable, then even the owner isn’t able to change it. Its manual has full list of attributes, but common ones are:

a -> Append only
i -> Immutable
c -> File automatically compressed in kernel.

For example, the append only attribute is commonly used for log file to prevent any other users from modifying the file. Sometimes we want to keep system files from being changed as well:

chattr -i /etc/resolv.conf
echo "nameserver 8.8.8.8" > /etc/resolv.conf 
chattr +i /etc/resolv.conf

Whatever attribute that we set with chattr, we can use lsattr to list them. These two commands are in Linux not BSD.

Loop

If you can tell how many iterations, then use for loop. For example, you want to execute a command for each file in a directory:

for file in /tmp/*.hl7
do
 echo "Picking File $file" >> output.txt
 hl7snd -f "$file" -d destionatiohost:2398 >> output.txt
 sleep 2
done

or you may specify range and steps for a for loop:

for i in {1..11..2} {13..23..2}
do
 echo "hostmachine"$i
 scp local.file "hostmachine"$i:/tmp/
done

Note that the way we use step here is only available on Bash 4.x (check version through variable $BASH_VERSION, reference)

If it’s hard to tell how many iterations, then use while loop. Here are some examples

Example 1. You want to execute the same command multiple times, each with a line in a file as parameter:

while IFS= read -r var
do
 echo $(date +%Y%m%d:%H%M%S) displaying line "$var" >> resend.log
 sleep 2
done < list.txt

Example 2. One step further, if the input file is a CSV format like this:

acc1,pid1
acc2,pid2
acc3,pid3

And you want to repeat a command multiple times, each execution with a column 1 as parameter 1, column 2 as parameter 2, etc. Then you can run something like this:

IFS=,$(echo -en "\r\n\b"); while read -r f1 f2; 
do 
echo processing $f1, $f2
curl -i -X PUT -H "Content-Type:application/json" -d '{"userId":"dhunch"}' --url http://localhost:9876/monitoring/document/$f1/$f2/publish;
done < input.txt >>out.txt

In this example IFS assignment is done before while loop. So it will take effect outside of while loop. This is why \r, \n and \b are specified in addition to comma. If this is not something you like, you may put IFS assignment after while and only specify comma as field separator.

Example 3. If you just want to repeat the same command every 5 seconds, apart from watch command, you can leverage while loop doing something like this:

while true; do df -Ph | grep "sda1"; sleep 10; done

In fact in all these examples while loop can be expressed in a single line, with semicolon to indicate where you would have typed enter.

find, with xargs and -exec {}

The find command offers flexibility searching files with certain conditions.

Example 1. Find files from within last 5 * 24 hours in current directory:

find -maxdepth 1 -type f -mtime -5

Example 2. Find tar files older than 6 * 24 hours:

find . -type f -name "*.tar" -mtime +6

Notes:

the switch -iname is similiar to -name but case insensitive
the switch -mtime goes by modified datetime; -atime goes by access time
the switch -mmin measures in minutes
the swtich -daystart makes it measure time from the beginning of current day (instead of 24 hours from current time of current day)

Here are more information about find command, and more examples.

To execute command per result from find, we have the options of xargs and -exec {}. Both build command based on parameter input, instead of I/O redirection. xargs is considered more efficient and it also works with commands other than find. -exec {} only works with find command.

Here’s a basic example of -exec {}

find . -iname '*.dcm' -exec dcm2txt {} \;

On the other hand, xargs works with any command followed by a pipe. It takes input from previous command, split it by space or carriage return into a list, then build command with each item on the list. To help understand parameter passing with xargs, we examine two examples. The first example is a directory with three files in it: x.a, y.a, z.a

$ ls
x.a x.b x.c

If we want to prefix each file with pre, we can do the following:

ls | xargs -I aa echo "mv aa prefix_aa"

In the second example, suppose find command produces the following result:

firstdir
seconddir

Compare the following two commands:

find . -type d | xargs ls -l
find . -type d | xargs -n 1 ls -l

In the first command, xargs invokes “ls -l firstdir seconddir” whereas in the second command, xargs invokes “ls -l firstdir” and then “ls -l seconddir”. The first command requires that the utility takes multiple parameters (in this case, ls does. Other commands such as wc, grep also do). The second command is particular helpful when the utility only takes one parameter. This is because the switch -n sets the maximum number of arguments taken from standard input for each invocation of utility.

Last but not least, operators are quite useful in Bash scripting. Here is a good reference.