Azure Portal on Digi Hunch

AKS Lessons Learned 2 of 2

Sat, 18 Dec 2021 01:18:00 -0400

Even though Azure Kubernetes Service (AKS) is a managed service, building a cluster is not trivial. For help resources, I would start with the webinar “Configure Your AKS cluster with Confidence” from April 2021, which focuses on a set of working best practices (convention over configuration) but obviously not every recommendation suits every use case. For a deeper technical tour, the John Savill’s Technical Training channel has good videos (from 2020) on AKS overview, high availability and networking. Lastly, there is also an AKS checklist to remind you of the implementation details to consider.

All the references aside, I need to write down some gotchas from my implementation experience in the last two month.

Identity and Access Management

AKS is a special type of Azure resource in the sense that it manages other Azure services on user’s behalf. Therefore the access management needs to consider several aspects:

Access Type	Mechanisms involved	Example
User access Kubernetes API	Azure AD, Azure RBAC and Kubernetes RBAC. – Azure AD is for authentication – Azure RBAC for Kubernetes – Kubernetes RBAC	A user connects to Kube-API server using kubectl
AKS access other Azure resource	There are several identities that represents different components of AKS. For example, the AKS cluster, the node agent pool, and each add-on. The AKS cluster can be represented as a service principal, or managed identity (system assigned or user assigned). The node agent pool can be represented as a managed identity	AKS cluster connects to a VNet in a different resource group. (requiring cluster’s identity to have network contributor role on the target network resource group) AKS node agent pulls images from ACR (requiring the node agent pool’s identity to have ArcPull role on the target ACR)
Pod access other Azure resource	AAD-Pod Managed Identity	Business workload connects to managed database service such as PostgreSQL on Azure.
Pod access Kubernetes API	Access Kubernetes API using Service Account. This issue is solved completely by Kubernetes native mechanisms. Roles and ClusterRoles defines permissions. RoleBindings and ClusterRoleBindings associates Service Accounts with permissions	Workload access ConfigMap, Secret etc.

In the first access type, for RBAC with user to access Kubernetes API, there is an overlap between Azure RBAC and Kubnernetes RBAC. Azure RBAC has four built-in roles and three of them (reader, writer, admin) are namespaced. When you use Azure CLI to manage to assign one of those roles, the rolebinding and cluster rolebinding record stored in etcd will be updated accordingly.

RBAC mechansim	Use case
Azure RBAC for Kubernetes	Manage RBAC programmatically using Azure CLI, or infrastructure as code
Kubernetes RBAC	Manage RBAC declaratively with more granularity for all types of Kubernetes resources including CRD

For ease of operation it is advised to use Kubernetes RBAC whenever possible. Azure RBAC is still used for RBAC at the level of Azure resource but not at the level of Kubernetes resource.

In the the second access type, AKS cluster may use managed identity or service principal. Azure’s recommendation is managed identity over service principal. Managed Identity is a wrapper around Service Principal with less overhead. Managed Identity can be system assigned (created at the time of cluster creation), or user assigned (can be created ahead of time by Azure administrator and imported to the cluster’s context).

The second access type can be further broken down because there are several components in AKS that uses their own identities. I list the related managed identities as below:

Name	Purpose	BYO identity with Terraform
cluster identity	This identity represents the cluster	Specify in identity block in kubernetes_cluster resource
agent pool identity	This identity represents kubelet running in the agent pool	Specify in kubelet_identity block.
addon: azurepolicy	This identity represents azure policy addon to access the policy	N/A
addon: omsagent	This identity represents OMS agent to access monitoring etc	Specify in oms_agent_identity block
addon: secret	This identity represents to the secret addon, to access AKV	Specify in secret_identity block
addon: ingress gateway	This identity represents the ingress application gateway	ingress_application_gateway_identity block

By default, the system creates a new managed identity for each of the required identity above. For simplicity with identity management, we may create a managed identity and use it for all the occasions where an identity is needed and user assigned (BYO) identity is supported.

In the “az aks show” command return (a JSON document), the identity section (root level) reports the cluster identity, the identityProfile section (root level) reports the agent pool (kubelet) identity. Other identities such as omsagent, are reported in their own child document.

Node Networking

In Azure, a subnet can span across multiple availability zones. Therefore an AKS cluster can put its nodes on a single subnet with nodes evenly distributed across three AZs for high availability. The AZ of each node is indicated in the node label, and can be displayed with kubectl command.

Within a single AZ, a good practice to minimize latency between nodes is to place the nodes in a proximity placement group (PPG). However, only a single PPG can be associated with a node group. You can’t have three PPGs, one in each AZ, for a single subnet.

Pod Networking

The default Pod networking model is kubenet, which involves overlay network. Pod-to-Pod traffic across nodes requires Network Address Translation (NAT). To overcome this performance tax, Azure introduces Azure CNI which gives each Pod an routable IP address from the VNet’s CIDR. This requires advanced IP planning to prevent IP exhaustion. A risk introduced in Azure CNI is that all Pods are exposed on the V-net, which needs to be protected by Network Security Group and/or outbound firewall.

DNS

On the DNS side, when AKS cluster integrate with an external node network, it may create weird issues that are hard to troubleshoot. Another example is with DNS. If the V-Net uses an external DNS server (which is common for enterprises with hybrid network to use an on-premise DNS server), then the cluster creation failed with time-out with misleading error messages (for example, this comment). This is because the DNS name of the newly created cluster is not resolvable within the V-NET, which points to the on-prem DNS server. The fix to that is:

Use a BYO DNS zone (in Azure) for AKS cluster creation;
The AKS cluster will publish the A-record to the zone. To allow this to happen, the AKS cluster’s managed identity needs to have DNS contributor permission for the zone;
Configure the on-prem DNS for conditional forwarding to the DNS zone

This fix will allow AKS to resolve its name and therefore confirm its own creation. Here is a good blog about the DNS zone options.

Another potential issue introduced with the use of on-prem DNS server, is the resolution of single-label hostname of the nodes. This is not just an issue in the context of AKS. It is a generic issue with VMs running on a V-Net pointing to on-prem DNS, as explained in detail here.

In this situation, we should use the fully qualified hostname instead of single-label hostname. The fully qualified hostname with DNS suffix can help the on-prem server to configure conditional forwarding. For example, when the DNS suffix is *.internal.cloudapp.net, then forward it to Azure’s virtual internal DNS server 168.63.129.16 which can resolve the hostname.

If only the Pods need to resolve those FQDNs, then we can configure Core-DNS with conditional forwarding, which will take effect only at the cluster level without the need for changing the on-prem DNS. The Core-DNS configuration looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
 name: coredns-custom # this is the name of the configmap you can overwrite with your changes
 namespace: kube-system
data:
 cloudapp.override: | # you may select any name here, but it must end with the .override file extension
 log
 rewrite continue {
 name regex ^(.*[0-9]{7}-vmss[0-9]{6})$ {1}.internal.cloudapp.net
 answer name ^(.*)\.internal\.cloudapp\.net$ {1}
 }
 forward internal.cloudapp.net 168.63.129.16
 cloudapp.server: |
 internal.cloudapp.net:53 {
 errors
 log
 cache 10
 forward . 168.63.129.16
 }

Alternatively, use Pod DNS policy so that the Pod can use an external DNS server.

Initial Service Account

When a cluster is created, an Azure AD user or group can be assigned as cluster administrator. For a CI/CD pipeline to interact with the newly created cluster, a service account in Kubernetes is needed. Suppose we use Terraform to create the AKS cluster, we can create such service account automatically with the Kubernetes provider. This requires that the Terraform execution environment to have network access to the cluster. If the AKS cluster is located in a private network, then the agent where Terraform CLI runs should also be on the network. Alternatively, use Terraform Enterprise hosted in an environment with access to the cluster’s network.

Integration with Azure KeyVault

Azure Key Vault can store several types of secrets, key value pair, X509 keys and certificate. When AKV is integrated with an AKS cluster, the Kubernetes workload can access the secrets as mounted volumes, using CRD named SecretProviderClass. Further, they can be presented as Kubernetes Secret, using a Pod to sync between mounted content and Secret. AKV has three types of entries: key, certificate and secret (key-value). The certificate entry requires both key and certificate are stored, with optional certificate chain. In my opinion this is an over design. Unless we need Azure to manage the certificate (e.g. rotation) I would simply use secret to store my own X509 key and certificate.

Azure Deets

Mon, 25 Oct 2021 23:13:52 -0400

Both Azure and AWS are leading players in public cloud. AWS developed a lot of SMB customer in technology. Azure attracted many enterprises from their on-prem customers. In addition to clienteles, their models to manage resources in the cloud are also different in several aspects. While this post is by no means a comprehensive comparison, it serves as a refresher on how Azure is different from AWS in cloud engineering.

Subscription and Resource Group

Both subscription and resource group are means to manage resources in group. Subscription is associated with a credit card and groups resource financially. Resource group groups resources logically. A subscription can have multiple resource groups, as illustrated here in the hierarchy map.

Azure CLI

There are two CLI tools: Azure PowerShell for PowerShell users, and Azure CLI for Linux users. The distinction is not clear, because PowerShell can also run on multiple platforms such as MacOS and Linux. On the other hand, Linux Bash can run on Windows (e.g. using WSL2). The different lies more in the command nomenclature. For example, to list subscriptions, the PowerShell Cmdlets reads:

Get-AzSubscription

The Azure CLI comes much cleaner:

az account list

The command and argument naming in Azure CLI align with Linux commands. I prefer Azure CLI for its succinctness. When looking up VM skus, we can use this command:

az vm list-skus -l eastus2

The return is a JSON document. In advanced use cases, we can filter the result by using jq utility. Alternatively, we can filter the result and groom the output with jmespath query, just like AWS CLI. For example, in eastus2 region, we look for instances that:

Has virtualMachines as resource Type
Has AcceleratedNetworking enabled
Has EncryptionAtHostSupported enabled
Has PremiumIO enabled

For each of the result, we print out:

number of vCPUs
Memory size

For this use case we will have to provide a JMESPath query as below, then output the result as a table:

az vm list-skus -l eastus2 --query "[?resourceType=='virtualMachines' && capabilities[?name=='AcceleratedNetworkingEnabled' && value=='True'] && capabilities[?name=='EncryptionAtHostSupported' && value=='True'] && capabilities[?name=='PremiumIO' && value=='True']].{Name:name,vCPUs:capabilities[?name=='vCPUs'].value|[0],MemoryGB:capabilities[?name=='MemoryGB'].value|[0]}" --output table

The result looks like this:

I use variations of the command above very often to find out the best instance for AKS nodes.

Infrastructure as Code

The native infrastructure as code option is ARM (Azure Resource Manager) template in JSON format. It is extremely wordy and perhaps why Azure later developed Bicep as the second generation of IaC tool. Terraform has a provider for Azure as well. For comparison among ARM, Terraform and Bicep, I have written a blog post for Slalom build covering more details.

Networking

Here is a great post comparing Azure Network with AWS.

At a high level, Azure Virtual Network (or VNet) is the equivalent of VPC in Amazon. Likewise, peering can be configured between VNets. As to subnet, Azure is different because there is no conceptual distinction between public subnet and private subnet. In AWS, public subnet is subnet attached with an Internet Gateway via a network route. So “private” or “public” are in terms of outbound traffic. On the other side, Azure does not distinguish between private or public subnet. Resources connected to a VNet have access out to the Internet by default. As to inbound traffic, you can make a VM available on Internet by giving it a public IP (same as AWS). You can make it available to other VNet, by configuring a service endpoint. Customers typically need custom routes to redirect outbound traffic (e.g. through firewall). In VPC, subnets are mapped to availability zones one-to-one, whereas in Azure, a subnet may traverse multiple availability zones.

With Azure, it is also important to understand difference between Azure service endpoint and Azure private endpoint:

Azure service endpoint: provides connectivity to Azure services over n optimized route over the Azure backbone network. Traffic will leave your VNet.
Azure private endpoint: a NIC that uses private IP from your VNet. This NIC connects you privately and securely to a service powered by Azure Private Link. By enabling a private endpoint, you’re bringing the service into your VNet.

On security group, we can associated a network security group with a network interface, or with a subnet. In contrast, in AWS, a security group can only be associated with an instance’s network interface.

Azure Bastion and Jump Box

There is a managed service called Azure Bastion. It is a SSH/RDP proxy fully managed as PaaS. However, its use case is virtual machines. It cannot be used to access other services. For example, if you create a private AKS cluster, then you need a command terminal to access the API server. This is not what Azure Bastion can do. Instead, you either need a virtual machine in the AKS network as jump box.

The alternative is an Azure Cloud Shell, which will require storage but can be configured to be placed inside of a V-Net. However, Azure Cloud Shell is not running inside of a full-fledged Linux operating system. You cannot install commands.

To create a bastion host, e.g. without public IP address, use the following CLI command:

az vm create -n MyBastion -g AutomationTest --image UbuntuLTS --subnet suitable-porpoise-node-subnet --vnet-name suitable-porpoise-vnet --ssh-key-values ~/.ssh/id_rsa.pub --authentication-type ssh

If the bastion host is needed with a public Ip, configure the network security group accordingly. The AZ CLI command above will create a VM, with an OS user named after the command line terminal user.

IAM

Azure AD is a managed identity service. Here is the difference between Active Directory and Azure AD.

Azure RBAC is a mechanism for authorization. Just like IAM policies, Azure RBAC enforces permissions using role assignment, which consists of:

security principal ( user, group, service principal, or managed idenity)
role definition: defines what actions is allowed and what is not allowed
scope: the object of the action

To enforce RBAC, one needs to create role assignment objects, each specifying principal, role, and scope.

Service Principle and Managed Identity

The two concepts may appear confusing. I find this article a great reference to demystify them. The takeaway is: service principle is the equivalent of service account in old Active Directory. Managed identity is a service principle automatically managed by a resource. Managed identity can be user assigned or system assigned.

Managed Identity is a “wrapper” around a service principal. It is automatically created and automatically rotated.

Azure DevOps

I think of Azure DevOps (ADO) of a managed pipeline implementation, with a repository (just like BitBucket), a board to manage tickets (similar to JIRA), Wiki (just like Confluence), Artifactory. The Pipelines is the part that’s similar to Jenkins. ADO calls a build pipeline a Pipeline, and a release/deployment pipeline a Release. A pipeline and a release are fundamentally the same but they are used in different ways. A pipeline’s input is usually the code repository, and the output is artifact. A release’s input is usually an artifact, and it connects to infrastructure in different environments.

ADO has its own ecosystem for plugins, managed under Visual Studio marketplace. Many extensions are open-source. If you are not happy with an extension, you can publish your own extension to market place.

Logging and Monitoring

Azure Monitor manages metrics, logs and alerts. To further analyze logs, create a log analytics workspaces, where you can run Kusto queries. You can create a workbook and embed Kusto queries into visual objects on the workbook.

Storage

Azure manages storage resources under storage account. The resource classes include Blob (object), File, Queue, Table and Disk (block). They have a few acronyms on redundancy levels.

Redundancy Option	Data Copy	Access level
LRS – Locally redundant	synchronously copy your data three times within the AZ in the primary region.	Write is acknowledged after three synchronous writes.
ZRS – Zone-redundant	synchronously copy your data across three AZs in the primary region. your data is still accessible for both read and write even if one AZ becomes unavailable.	Write is acknowledged after three synchronous writes. If an AZ becomes unavailable, Azure undertakes networking updates (e.g. DNS re-pointing). Application may perceive a blip where re-try policies may help.
GRS – Geo-redundant	LRS in primary region + asynchronously copy your data to a single AZ in the secondary region + LRS in secondary region	your data in the secondary region isn’t available for read or write access unless there is a failover to the secondary region. for read access to the secondary region, configure your storage account to use RA-GRS (read-access geo-redundant storage) If the primary region becomes unavailable, you can choose to fail over to the secondary region. After the failover has completed, the secondary region becomes the primary region, and you can again read and write data.
GZRS – Geo-zone-redundant	ZRS in primary region + asynchronously copy your data to a single AZ in the secondary region + LRS in secondary region	your data in the secondary region isn’t available for read or write access unless there is a failover to the secondary region. for read access to the secondary region, configure your storage account to use RA-GZRS (read-access geo-zone-redundant storage) If the primary region becomes unavailable, you can choose to fail over to the secondary region. After the failover has completed, the secondary region becomes the primary region, and you can again read and write data.

The disaster recovery and failover happens at storage account level.