Logs on Digi Hunch

AKS Lessons Learned 1 of 2

Sat, 04 Dec 2021 02:11:06 -0400

In general, troubleshooting Kubernetes is tricky. That is because one has to get in and out of pods. I took two days to troubleshoot some networking issues with private AKS cluster. For the amount of of tricks I had to employ, I need to take some notes.

The issue

After writing the Terraform code, I used the following dummy service to test the private AKS cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: aks-helloworld-one
spec:
 replicas: 1
 selector:
 matchLabels:
 app: aks-helloworld-one
 template:
 metadata:
 labels:
 app: aks-helloworld-one
 spec:
 containers:
 - name: aks-helloworld-one
 image: neilpeterson/aks-helloworld:v1
 ports:
 - containerPort: 80
 env:
 - name: TITLE
 value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
 name: aks-helloworld-one
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
 app: aks-helloworld-one

The expected behaviour, is that the service object will tell cloud API to provision a load balancer, with public IP listing at port 80. I should be able to curl to the IP address and connect to the site in the Pod. However, I was not able to. On the bastion host, I was able to curl to nodePort of the node address. But anything on public IP does not work, no matter where I ran curl from. This feels like a basic issue, but is is quite annoying because the native troubleshooting tool for Azure Load Balancer is horrible. In and out of a bunch of components named “insights”, “diagnostic log”, or “Metrics”, I can’t simply find a way to trace whether it received an HTTP request. Most of the information I was able to see was irrelevant or useless.

The approach

The hard way to troubleshooting infrastructure as code, is configuration comparison approach: revert to a baseline configuration, and see if the expected function works. Then from the baseline, change one configuration at a time and see where it starts to break. This approach is very time consuming, and AKS cluster as a relatively large resource, with numerous attributes, takes this effort to extreme.

The baseline configuration I started with is:

az aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner

With this baseline, I simply use kubectl to apply the YAML file above. Then I can tell that the port is working. With a good start point, I started to apply one change at a time and repeat the test. I ran into a snug when I’m using the following configuration:

az aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner --enable-private-cluster --network-plugin azure --network-policy calico

With the cluster created from the command above, the variable introduced is –enable-private-cluster. This puts the cluster on a private network. I cannot connect to the cluster via a public endpoint anymore, and thus have to figure out some tricks to run the kubectl commands. I had to play with the Command Run feature of AKS cluster because I don’t have a bastion host when using AZ CLI command. The Command Run feature would not allow me to use any file from bastion host. So i had to create my test objects, the Deploy and the Service objects all by imperative commands. The equivalent commands I worked out is:

kubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1 --replicas=1 --port=80
kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type='LoadBalancer'

Then I realized a limitation with Command Run feature: it only supports basic command switches and doesn’t like switches such as –replicas. So I used the following commands:

az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get no"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get deploy"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type=LoadBalancer"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get svc"

This trick allows me to continue with the testing eliminate Azure CNI and Calico policy as the cause. Testing after each cluster creation is painful because the cluster creation can take 10 minutes.I had to temporarily minimize the size of the cluster to speed up provisioning.

I finally came to the point that I can reproduce the issue using TF template. I realized that when I set the vnet_subnet_id attribute of azurerm_kubernetes_cluster’s default_node_pool, the problem came back. That’s the smoking gun that the node subnet is the issue.

The Network Security Group on Node Subnet

The node subnet has an associated network security group. I discovered that once I add an allow rule for port 80 to the security group, the curl test will work. I also noticed the security group rule change will take 60 sec to come to effect and load balancer will also take 60 sec to warm up.

This confuses me because port 80 is only listened by the load balancer and not by any of the nodes. It’s most likely when public load balancer is used the load balancer is placed on the node subnet. According to this note:

Inbound, external traffic flows from the load balancer to the virtual network for your AKS cluster. The virtual network has a Network Security Group (NSG) which allows all inbound traffic from the load balancer. This NSG uses a service tag of type LoadBalancer to allow traffic from the load balancer.

The packet coming from external source can travel up to the VNet, but it was blocked at the NSG of node subnet.

Lessons Learned

We always need to have some dummy service ready to test what we need. We can use nginx dummy service like:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-nginx
spec:
 selector:
 matchLabels:
 run: my-nginx
 replicas: 2
 template:
 metadata:
 labels:
 run: my-nginx
 spec:
 containers:
 - name: my-nginx
 image: nginx
 ports:
 - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: my-nginx
 labels:
 run: my-nginx
spec:
 type: LoadBalancer
 ports:
 - port: 80
 protocol: TCP
 selector:
 run: my-nginx

As discussed above, it’s also important to have a Bastion host able to access the control plane when the AKS cluster is private. Azure touts about CloudShell (and its ability to run in specified V-Net) but it’s pretty useless in troubleshooting. CloudShell sessions run inside of Kubernetes cluster and lacks common network troubleshooting tool such as nc. Azure has a managed service for Bastion but it requires a subnet with the exact name of AzureBastionSubnet.

We will explore more issues in the next post.

Automated Deployment Pipeline 1 of 3

Wed, 30 Sep 2020 22:04:00 -0400

The business case

You launched a software application. You installed it on two customer sites. You support the application mostly by SSH to customer server and run Bash commands, or slightly better, Bash scripts.

The product is a hit to the market. You hired 20 support specialists in a customer service department. The dream client came through: an enterprise giving you a fleet of 100 servers to deploy your application on.

More staff, more business, more installations, more incidents, but the same old command driven steps. Problems:

Non-standard support procedures. Every one takes notes and everyone’s notes are slightly different.
Information sharing among team members are ad hoc, and at high level.
Post-mortem discussion is driven by memory and command fragments, instead of evidence end-to-end

If that looks like your organization, chances are you also suffer from some secondary damages over the long term, such as:

Downtime resolutions rely on the knowledgeable few
Documentation helps. But it never catches up to the latest version of application
Lack of auditing of commands during support

I propose an automation scheme to existing support and deployment practice. This automation scheme combines a suite of common technologies, such as Bash, Python, Ansible, OpenSSH and Jenkins. The automation allows the department to, either fully or partially, operationalize the steps in support and deployment, and eventually shift towards agile practice.

Bash, Python and Ansible

Bash script is based on shell command, perfect for running critical system tasks such as volume management. When it turns into a script, it can be cumbersome, especially with complex data structure. Python, as a tool for system administration, is a good complement to that.

Python 2 comes with most Linux distributions, and is also a dependency of other built-in tools such as YUM. Python3 can be installed easily from default YUM repositories. Both Python2 and Python3 can exist on the same operating system, although new module development are now shifted to Python3. Python’s syntax is very simple and offers object-oriented programming ability. Moreover, there is an entire open-source community behind Python, which offers modules in every aspect of IT (for example, Datastax has a driver module for connecting to Cassandra). Those modules are installed with PIP tool, or PIP3 for python3.

Both bash and Python executes on local machine. To run them on remote servers over SSH. You want to have a list of target hosts, and specify which one to execute the script against. This is where Ansible comes in handy. Ansible is superior in the following aspects:

Free and open-source, with commercial alternative (Towers);
Inventory management (inventory);
Desire state engine (roles)

Ansible is built on Python and is agent-less. Connectivity to remote host is done via secure shell so it can take advantage of existing SSH configurations. Job execution on the target machine is done through Python. With Python you can also develop custom module in Ansible. For some use cases in customer support with Ansible, refer to my two previous postings about Ansible at scale.

Jenkins

The tools above forms a package for automation. The issue is that all of them are command-line based. Any task that requires Ansible requires the IT professional craft up long command, such as running playbook, executing a role, or ad-hoc command. This is inconvenient when a task needs to be done during an incident. Such tasks also require trained professional with the relevant skills.

These tasks can be stored in, or initiated by Jenkins. Although Jenkins is well known for build automation in continuous integration, it is automation engine for any command-line based IT tasks. The button to start such tasks in Jenkins UI is called “Build”, which is also a misnomer that underplays Jenkins’ versatility: building application from source code is just one of the many IT tasks that involves multiple long running commands. In this and next article we introduce Jenkins as an engine for deployment automation.

The infrastructure architecture is diagramed as above, and with the connection across Internet, the target hosts must be hardened properly in the following aspects:

Connectivity to remote host is via SSH chaining, through an SSH proxy;
Root login must be disabled for remote session or by password;
Service user may be shared, but must be authenticated by individual RSA key pair;
Service user connected remotely needs to escalate privilege by su if needed;

I want to make a theoretical distinction between our topic here and continuous deployment. We simply focus on the technical side of deployment automation. Essentially automating a few bash scripts. On the other hand, a continuous deployment process is an extension to an existing continuous integration pipeline, with the vision to streamline the process end-to-end from code commit to production rollout. Implementing CI/CD pipelines should be approached as an organizational program rather than an individual technical initiative. Here is a good technical overview on CI/CD pipeline with Jenkins and Ansible.

Security

The security mechanism of this system is based on OpenSSH because the connectivity between servers are through SSH chaining. RSA key authentication must be used in order to encrypt traffic with password-less login. Connection to an SSH host can be done through a proxy server. Below is an example of SSH configuration:

Include customer1.config
Include customer2.config

Host *
 IdentityFile ~/.ssh/id_rsa
 ServerAliveInterval 60
 ServerAliveCountMax 3
 Compression yes
 ControlPersist 3h
 ControlPath ~/.ssh/sockets/%r@%h-%p

Host gateway
 Hostname support.digihunch.com
 User jdoe
 Port 2223

Host customer-server-0
 Hostname 192.168.201.12
 User support
 ProxyCommand ssh -W %h:%p gateway

Open SSH configuration file (~/.ssh/config) needs to be configured properly with useful host names and aliases. To prevent the config files from growing too long, include statement can be used to reference other configuration file (available with OpenSSH v7.3 sp1 and up).

The host names (as well as aliases) listed in SSH configuration can be directly referenced in Ansible inventory, allowing Ansible (and Jenkins) to reference site by alias and connect to target host through proxy.

Plugins

Jenkins has a community that develops a variety of plugins, which makes Jenkins the most powerful automation platform. Here are some examples of useful plugins:

Audit Trail: output job execution history to file or Elasticsearch;
Credentials: stores credentials in Jenkins;
Pipeline: build declarative (new) or scripted (old) pipeline for Jenkins jobs;
Simple Theme: just a theme but allows console output to be dark (using CSS);
Job Configuration History: job configuration audit;
Mask password: mask variables (including password) from console output
Ansible: invokes ad-hoc commands and playbooks
SSH agent, SSH pipeline steps, SSH credentials: features related in SSH in Jenkins pipelines.
Purge Job History: purge all of build history, or purge by time and number of old builds.
Parameterized Scheduler: schedule to run a job and provide parameter
Workspace cleanup: clean up workspace when invoked.

In the next article, we will go over some common job configurations.

Log file navigator (lnav)

Wed, 23 Sep 2020 21:03:00 -0400

I’ve used a number of log viewers in command terminal, on MacOS and Linux server. I read system logs, log4j formats, as well as json formats. Unfortunately, I have not found an ideal (free) log viewer, either on UI or in command terminal.

Out of those I tried lnav is one of the better ones. It’s been around for more than 10 years and is configurable for a variety of formats. It is available in EPEL-repository for Linux and home brew for Mac.

For example, if our log (produced by log4j) looks like this:

2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): start ARTIM 5000ms
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): Client closed connecti
on without sending data
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550) enter state: Sta1
2020-10-06 20:59:25,471,INFO,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): close Socket[addr=/10.100.101.10,port=24976,localport=44104]
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.AssociationReaper - [platform-dicomServer-44104-574917] Stop check for idle Association(552550)

We can introduce custom formatting, for example:

{
 "dapp" : {
 "title" : "dapp log4j format",
 "description" : "dapp log4j format",
 "regex" : {
 "dapp" : {
 "pattern": "^(?<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}),(?<level>\\w+),(?<component>[\\w-.]+) - \\[(?<thread>[^ ]+)\\] (?<body>.*)$"
 }
 },
 "level-field" : "level",
 "level" : {
 "error" : "ERROR",
 "warning" : "WARN",
 "info" : "INFO",
 "debug" : "DEBUG"
 },
 "value" : {
 "level" : { "kind" : "string", "identifier" : true },
 "component" : { "kind" : "string", "identifier" : true },
 "thread" : { "kind" : "string", "identifier" : true },
 "body" : { "kind" : "string" }
 },
 "highlights" : {
 "DIMSE" : {
 "pattern" : "A-(ASSOCIATE-(RQ|AC)|RELEASE-(RQ|RP)|ABORT)|C-(STORE|MOVE|FIND|ECHO)-(RQ|RSP)",
 "color" : "Red"
 }
 },
 "sample" : [
 {
 "line" : "2020-10-06 12:00:28,500,INFO,dicom.dicom-main - [main] Start listening on port 44104"
 }
 ]
 }
}

Save the content above as ~/.lnav/formats/installed/dapp.json, then load the log file with lnav, lnav will display the log by presenting columns in different colours.

One of the default behaviours is highlighting the IPv4 address, as shown above. This behaviour is however, not optional and currently cannot be turned off, which is reported here.