ansible role on Digi Hunch

Ansible at scale 2 of 2

Mon, 25 May 2020 22:08:54 -0400

Template (with Jinja2) and files

In an Ansible role, we can use files or templates to achieve similar results for configuration files. If the configuration file is the same across all targets then we can place it in files directory to push out. If the content of configuration file varies depending on the cluster size, we use Jinja2 template. For example, when you configure zookeeper configuration, a first entry may require total number of nodes in the cluster, a second entry may require the hostname of the server itself; and a third entry may require a comma separated line with hostnames of all nodes in the cluster. This is a typical use case of Jinja template.

We need to make sure Jinjas version is above 2.11.2 (as of May 2020) because older version such as 2.7.2 has known issues with namespaces. To check version and then upgrade Jinja2, we need to use pip:

pip show Jinja2
pip install -U Jinja2

The Ansible template module takes Jinja2 file as input and delivers result file on target host. Note that if the template references host variables from Ansible playbook, then you need to gather facts about host. This means you will have to use a basic playbook like below instead of adhoc command.

A basic playbook to test Jinja2 template is:

- hosts: '{{ansible_limit}}'
 gather_facts: yes
 tasks:
 - template:
 src: cassandra_xml.j2
 dest: /tmp/cassandra.xml

Although Jinja2 offers a lot of flexibility with loop and if-else statement, it is just a templating language and not a programming language. It requires some tricks to achieve what you may otherwise easily do with programming language. One example is persisting a variable outside of a loop. As per the document, it is not possible to set variables inside a block and have them show up outside of it. This also applies to loops. The only exception to that rule are if statements which do not introduce a scope. To achieve that, you would have to use namespace, for each loop where you need to access the variable afterwards from outside of the loop.

{% block db_cluster_config_nobackup %}
{% set ns=namespace(nodeid=0) %}
{% for host in groups[my_db_group]|sort %}
 <var name="DBHost{{ns.nodeid+1}}" value="{{hostvars[host].inventory_hostname}}" />
{% set ns.nodeid=ns.nodeid+1 %}
{% endfor %}
 <var name="DBClusterHosts" value="{% for i in range(ns.nodeid) %}
${DBHost{{i+1}}}{% if not loop.last %},{% endif %}
{% endfor %}" />
{% endblock cass_cluster_config_nobackup %}

For the same reason, you might as well clearly define the start and end of each block in order to not run into trouble with scoping behaviours of variables. These limitations makes Jinja2 template not easy to read and may take several rounds of playbook runs to troubleshoot.

Handler vs conditional task

Sometimes you only want to run a task when its previous task results a change. There are two ways to achieve this: conditional task and handler.

With conditional task, we register the result of previous task to a variable, and execute the ensuing tasks conditionally based on assessment of the variable. We’d have to specify the condition for each of the subsequent tasks that needs to execute conditionally. These tasks, if condition is met, can execute immediately after the first task that registers the variable.

The alternative is through an Ansible mechanism called handler. Handler implements a series of tasks in a separate yaml file in the handers directory under the role. In the triggering task we need to notify the handler. The tasks in the hander will fire if the triggering task returns “changed” in its result. Handler is a great way to shorten the length of task or Playbook. However, we need to understand several subtleties with regard to handlers:

Although handler is notified during a task run, it is not fired until the end of each block of tasks in a play. They are not immediately fired after triggering task.
A handler will only execute once at the end of play, even if it was notified multiple times by different tasks during the play run.
Handler tasks are executed in the order of declaration, not in the order of notification.

In summary, Ansible’s notification handling mechanism is asynchronous, once-only, and out of sequence. The points above are illustrated in the following playbook:

---
- hosts: ghdocker
 tasks:
 - name: CopyFile3
 copy:
 src: ~/ansible/file3.txt
 dest: /tmp/file3.txt
 notify:
 - handler3
 - handlergeneral
 - name: CopyFile2
 copy:
 src: ~/ansible/file2.txt
 dest: /tmp/file2.txt
 notify:
 - handler2
 - handlergeneral
 - name: CopyFile1
 copy:
 src: ~/ansible/file1.txt
 dest: /tmp/file1.txt
 notify:
 - handler1
 - handlergeneral
 - debug: msg="end of play!"
 handlers:
 - name: handler1
 debug: msg="file1.txt has been copied."
 - name: handler2
 debug: msg="file2.txt has been copied."
 - name: handler3
 debug: msg="file3.txt has been copied."
 - name: handlergeneral
 debug: msg="A file has been copied"

Here is the output of the playbook run:

PLAY [ghdocker] ******************************************************************************

TASK [Gathering Facts] ******************************************************************************
ok: [ghdocker]

TASK [CopyFile3] ******************************************************************************
changed: [ghdocker]

TASK [CopyFile2] ******************************************************************************
changed: [ghdocker]

TASK [CopyFile1] ******************************************************************************
changed: [ghdocker]

TASK [debug] ******************************************************************************
ok: [ghdocker] => {
 "msg": "end of play!"
}
RUNNING HANDLER [handler1] ******************************************************************************
ok: [ghdocker] => {
 "msg": "file1.txt has been copied."
}

RUNNING HANDLER [handler2] ******************************************************************************
ok: [ghdocker] => {
 "msg": "file2.txt has been copied."
}

RUNNING HANDLER [handler3] ******************************************************************************
ok: [ghdocker] => {
 "msg": "file3.txt has been copied."
}

RUNNING HANDLER [handlergeneral] ******************************************************************************
ok: [ghdocker] => {
 "msg": "A file has been copied"
}
PLAY RECAP ******************************************************************************
ghdocker : ok=9 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Handler is a good way to keep idempotency. For example, Ansible does not have a way to import a yum .repo file to create a repo. We have to take two steps:

use get_url module to download the repo file (e.g. to /tmp),
use shell module to call yum-config-manager.

The problem is these two steps are not idempotent. If you repeat them, it will attempt to import the same repo file again. A little trick here is to use force=no option on get_url so it does not attempt to download if the file is already present in target directory. Then notify a handler to import repo file so the shell command is only called if there is a change.

The task looks like this:

- name: download repo file
 get_url:
 url: https://download.docker.com/linux/centos/docker-ce.repo
 dest: /tmp/docker-ce.repo
 mode: '0755'
 force: no
 notify:
 - Add docker repository

The handler looks like this:

- name: Add docker repository
 shell: yum-config-manager --add-repo=/tmp/docker-ce.repo

The handler is only fired when it is notified after get_url module returns changed in its result. Running the task again will not cause it to attempt to add the same repo again.

Note that when you use command or shell module, Ansible typically reports changed status. If this is not desired (e.g. you don’t want it to notify handler all the time), this behaviour can be overridden with changed_when parameter. You can specify conditions to meet in order to consider the shell/command module to have a changed result. Here is an example.

Ansible commands

In operation, our engineer needs to run a command on a group of servers. I encourage the use of Ansible adhoc command whenever possible. I recommend start with the following two commands:

ansible-inventory --graph
ansible all -m ping

The ping module triggers an “Ansible ping” to targets in the specified group. Over the years, Ansible community developed many helpful modules, such as yum, yum_repository, apt_rpm, uri, synchronize, fine, copy, etc and many can be used instead of bash command. However, sometimes, the expected Ansible module is either unavailable or missing function. For example, Ansible’s uri module cannot replace curl command with the following switches:

curl -s -XGET http://{{inventory_hostname}}:8080/objects/{{object_id}}/binary/all -o /dev/null -w '%{response_code} %{size_download} %{time_total} %{speed_download}\n' | awk '{if ($1==200) print "size="$2/1048576"MB,time="$3"s,speed="$4/1048576"MB/s"; else if($1==404) print "Cannot find object {{object_id}}"; else print "Unknown error. Code "$1 " when retrieving object {{object_id}}";}'

To leverage all these curl options, we still need to use the shell module in Ansible to call the command in shell.

Other helpful Ansible commands include ansible-pull for pulling playbooks from VCS repo, and ansible-console for interactive adhoc command execution.

Tags and extra variables

Both tags(-t) and extra variables (-e) are great ways to achieve flow control in playbooks. You can specify to run tasks with certain tags or skip tasks with certain tags. Extra variables can override the default variables from the host or the group. Both are great tools to improve re-usability of a Playbook.

Speed up execution

To speed up execution of Ansible tasks, there are several ways. For example, we can disable fact gathering by default so it only gathers fact if explicitly specified. This can be set in gathering=explicit under defaults section of ansible configuration file. If you have to gather facts, you may cache the facts using the following:

[defaults]
gathering = smart
fact_caching_timeout = 86400
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_fact_cache

Other than caching, Ansible allows you to select from several execution strategies for playbook. The linear strategy introduces configurable parallelization per task. The free strategy introduces parallelization per play.

linear (by default): Up to the fork limit of hosts will execute each task at the same time and then the next series of hosts until the batch is done, before going on to the next task. This mode ensures the progress is synchronized at each task.
free: as specified above, this is preferred when there is no need to coordinate the progress between each host target. It is a “free run” for each host all the way till the end of the playbook.
debug: essentially linear strategy except that the progress is controlled by an interactive debug session

The fork limit, with a conservative default of 5, can be adjusted in Ansible configuration. The execution strategy can be either specified in Ansible configuration, or specified per play. For example, the following snippet sets the strategy to free for the current play:

---
- hosts: all
 strategy: free
 tasks:
...

Ansible documentation also mentions some play-level keywords to control execution. The serial keyword, is one of them. It can be set along with any strategy above, and it introduces the effect of hosts batching. The value can be a single number, a percentage, or even a list of numbers (if size for each batch is different). Note that the batch size should not exceed the fork limit. This is particularly useful in rolling upgrades. For example:

---
- name: test play
 hosts: webservers
 serial: "30%"

With the parallelization capacity outlined above, a potential concern is some heavy-lifting task may consume a lot of resources, if being executed for all hosts at the same time. Luckily, Ansible has a task/block level keyword throttle, which “de-parallelize” the multi-host progress at a particular task, or block. Here is an example provided by Ansible documentation:

tasks:
- command: /path/to/cpu_intensive_command
 throttle: 1

If there are long running tasks, we can specify async and poll values so Ansible leaves a task running and check back later. For example, the following task allows Ansible to move on and check back every 5 seconds, if the task takes longer than 45 seconds, it is considered failed:

---
 - hosts: all
 remote_user: root
 tasks:
 - name: simulate long running task for 15 sec, wait for up to 45 sec, poll every 5 sec
 command: /bin/sleep 15
 async: 45
 poll: 5

Python Version

The recommendation is to use Python3 for any new development because there is no dependency. If there is no preference specified, Ansible tries to find out the appropriate interpreter and it can be seen in the response of ansible ping module. You can also force the interpreter by providing additional parameter ansible_python_interpreter. To change default interpreter, specify interpreter_python in ansible.cfg. For example:

[defaults]
inventory=~/ansible/inventories/site.yml
library=~/ansible/library/
vault_password_file = ~/ansible/.vault_key
host_key_checking = False
display_skipped_hosts = False
retry_files_enabled = False
interpreter_python=/usr/bin/python3

[privilege_escalation]
become_method=sudo

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=1h
pipelining = True

My open issues

I have some minor details that I have not been able to address, after a lot of time googling around. So I have to leave them for future reference.

If an Ansible playbook involves multiple plays (i.e. each with their own host), there is no way to persist a variable across different plays. A dumb alternative is to make all the variables to use available for every single host (under all directory).

In Jinja2 template, if I need to access the group of a target host (as defined in inventory), and the target belongs to multiple groups, I cannot filter to match the group I need.

Ansible at scale 1 of 2

Sun, 17 May 2020 19:38:34 -0400

The Ansible In Depth white paper outlines Ansible’s use cases in four categories:

Configuration management
Application deployment
Orchestration: for coordinating a multi-machine process such as interacting with load balancer and rolling cluster upgrade
As-needed task execution: ad-hoc tasks on large number of hosts

At work, my original automation scheme involves several Ansible Playbooks that started off simple but have been sprawling ever since. I have to spend some time to revamp the Ansible code base, following the best practices from the official documentation. The goal is to:

Reduce the number of Playbooks;
Improve code re-usability (by Ansible roles);
Increase portability across different customer environment;
Improve security;
Re-organize the directory so that more team members can contribute to different parts of it.

The Ansible code base is used by our customer service engineers, some of whom are ingrained with the established way they have been using certain Playbooks in their daily tasks. This requires me, throughout the development initiative, to ensure a consistent interaction between them and their Playbook commands.

Inventory and variables

We cannot guarantee that every custom environment are identical, but what we can do is make sure that for a new environment, the only change to make is inventory and variables. This is where we can strike a balance between portability and customization. No changes should be made to tasks, roles or Playbooks when the Ansible directory is deployed at a different customer environment.

If the total number of servers to manage are under 100, all can be listed in a single inventory file, and specify the only inventory file as default so the -i switch is not required for every Ansible command run. If there are more than 100 servers, it is advisable to separate them out into several inventory files in YAML, each less than 200 lines. You will have to specify inventory file with -i each time you run Ansible command.

The inventory may contain a hierarchy of groups, in order to facilitate command calls to specific groups of servers. For example:

all:
 children:
 prod:
 children:
 prod_app:
 children:
 prod_app_dc1:
 hosts:
 apphost01:
 apphost03:
 zk_id: 1
 apphost05:
 apphost07:
 zk_id: 2
 apphost09:
 apphost11:
 zk_id: 3
 apphost13:
 prod_app_dc2:
 hosts:
 apphost02:
 apphost04:
 zk_id: 1
 apphost06:
 apphost08:
 zk_id: 2
 apphost10:
 apphost12:
 zk_id: 3
 apphost14:
 vars:
 has_app: yes
 has_nginx: yes
 has_db: no
 prod_db:
 children:
 prod_db_dc1:
 hosts:
 dbhost01:
 dbhost03:
 dbhost05:
 dbhost07:
 dbhost09:
 dbhost11:
 vars:
 clustered: yes
 bk_node: dbhost11
 prod_db_dc2:
 hosts:
 dbhost02:
 dbhost04:
 dbhost06:
 dbhost08:
 dbhost10:
 dbhost11:
 vars: 
 clustered: yes
 bk_node: dbhost12
 vars:
 has_app: no
 has_nginx: no
 has_db: yes
 vars:
 ansible_become_pass: '{{site_prod_root_pw}}'
 test:
 children:
 test_dc1:
 hosts:
 tapphost01:
 test_dc2:
 hosts:
 tapphost02:
 vars:
 ansible_become_pass: '{{site_test_root_pw}}'
 has_app: yes
 has_nginx: yes
 has_db: yes
 vars:
 ansible_become: yes
 ansible_become_method su
 ansible_become_user: root

User of Ansible Playbook can use -l to specify a pattern that matches a single or multiple groups, such as prod_db_dc*. In the above example, the password is not stored in clear text. They should reference a variable from a separate file encrypted by ansible-vault.

From Playbooks to roles

As a refresher from the white paper, a single “task” in Ansible is essentially a module call with parameters. A “play” consists of a series of tasks (defined under “tasks” section) all to execute on a specified host (defined under “hosts” section). A Playbook consist of one or several plays, as shown in this example. In reality though, a Playbook usually contains only one play. Even that one play can grow to an unmanageable length, as complexity increases over time. This is where we need to change our approach towards scalability and manageability.

The Ansible community advocates the use of roles in place of Playbooks. The concept of Ansible “role” seems fairly abstract and confusing at the beginning. The word “role” pictures a static server state, whereas our existing Playbooks are full of actions (think of shell scripts). How would one convert an action list into static states? After some thought, I came to the understanding that roles should be thought of as desired end state. Yes, the end state is static, but that’s all we care about. This is essentially the whole idea of Ansible’s desired state configuration: you start from the end state and leave it to modules to complete what needs to be done to reach that state. The concept of role perfectly reflects how Ansible wants you to think about solving an infrastructure problem – stop thinking about what you need to do. Instead, think about what you ultimately want, start from the desired state and work backwards.

In our own setup, the best practice turned out to be: if the Playbook involves a single play with less than 5 tasks, just stick to Playbook. We don’t get rid of Playbooks just for the sake of it. Otherwise, if a Playbook has grown to more than 5 tasks, we need to think about our desired state, and either implement a new role, or incorporate it into an existing role. This is the time we have to transition from the Playbook oriented thinking to the role oriented thinking. Each role directory can include a task sub-directory with main.yml that references the rest of the tasks. Each role can define its own role-related variables. If there’s a lot in common between two roles, we can even have a common role with or without its main.yml.

A simplified version of our Ansible directory structure looks like this:

├── deploy-app.yml
├── deploy-db.yml
├── inventories
│   ├── group_vars
│   │   ├── all
│   │   │   ├── all.yml
│   │   │   └── vault_all.yml
│   │   ├── prod_dc1_db.yml
│   │   ├── prod_dc2_db.yml
│   │   ├── test_dc1_db.yml
│   │   └── test_dc2_db.yml
│   ├── host_vars
│   └── site_inventory.yml
├── roles
│   ├── common
│   │   ├── files
│   │   └── tasks
│   │   ├── log.yml
│   │   ├── skip_self.yml
│   │   └── validate_path.yml
│   ├── db_conf
│   │   ├── defaults
│   │   │   └── main.yml
│   │   ├── files
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   │   ├── main.yml
│   │   │   ├── start_db.yml
│   │   │   ├── stop_db.yml
│   │   │   └── update_cluster_var.yml
│   │   ├── templates
│   │   │   ├── myid.j2
│   │   │   ├── db_properties.j2
│   │   │   └── zookeeper_properties.j2
│   │   └── vars
│   │   └── main.yml
│   └── app_conf
│   ├── defaults
│   │   └── main.yml
│   ├── files
│   ├── handlers
│   │   └── main.yml
│   ├── meta
│   │   └── main.yml
│   ├── README.md
│   ├── tasks
│   │   ├── bk_app_conf.yml
│   │   ├── empty_app_conf.yml
│   │   ├── main.yml
│   │   ├── push_app_conf.yml
│   │   ├── start_app.yml
│   │   ├── stop_app.yml
│   │   ├── tar_app_conf.yml
│   │   ├── untar_app_conf.yml
│   │   └── update_cluster_var.yml
│   ├── templates
│   │   └── dbref_xml.j2
│   └── vars
│   └── main.yml
├── service-app.yml
└── service-db.yml

Variables specific to a group of hosts or individual hosts can be included in different yml files. When the entire directory is moved to a different customer environment, our engineers will need to update the inventory and variable files. The task, roles and Playbooks should build their logics using those variables.

Vault

Our previous implementation of Ansible Playbook stores sudo password base64 encoded and use no_log to avoid displaying values. Now we move those to encrypted variable yml file using ansible-vault. We reference the value to encrypt as regular variable:

ansible_become_pass: '{{passtoencrypt}}'
ansible_become_method: sudo
ansible_become: yes

Then we run the following:

ansible-vault create vault_all.yml

This prompt for a key, and once you type in the key it opens a text editor where we can store the real password. For example:

passtoencrypt: MyP@ssw0rd4real!

Use the text editor to save file. The file is now saved encrypted and must be open with correct key (aka vault password). If we call Ansible Playbook with –ask-vault-pass switch then the Playbook will prompt for key input, or use include_vars to include variable from vault file (example). If we want to even skip this, we can store the key in a file and reference them from vault_password_file in ansible.cfg

Ansible Vault has more commands to edit or view the encrypted variables in the documentation.

Optimize connection

OpenSSH 5.6 and later supports multiplexing where multiple SSH sessions share a TCP connection. This can be turned on so that the following SSH connections save the time of TCP handshake. This can be configured in ansible configuration file under ssh_connection. Below is an example of this option with ControlPersist=1h. So the TCP connection is torn down after 1 hour.

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=1h

The other option we can leverage is pipelining. Ansible takes three steps to execute a task:

build a python script based on module used
copy the python script to remote host
execute the python script on the remote host

If pipelining is turned on, the python script is passed in along with the SSH session, this would save a roundtrip and increase performance. Pipelining can be configured under ssh_connection in Ansible configuration file:

[ssh_connection]
pipelining = True

In the example below we can see by pipelining we cut the number of connection in half:

# with pipelining
[ghunch@control-host ~]$ ansible remote-host -vvvv -m ping | grep EST
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch


# without pipelining
[ghunch@control-host ~]$ ansible remote-host -vvvv -m ping | grep EST
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch

Note that if we use sudo command, then we need to disable requiretty in /etc/sudoers on the remote host.

Custom Module

It’s fairly straightforward to build a custom module in Ansible. Just place the module file (modulename.py) in inventory directory and use it as you would with regular Ansible module. The module file needs to be completed in Python with certain return value. Before creating custom module, you should look for existing modules to avoid re-inventing the wheel. You may also need to determine whether you simply need to run a python script on target host (with Ansible’s script module), or you really need an Ansible module. The former is procedural, and the latter focus on desired state. Custom module is more used in proprietary development.