Security Group on Digi Hunch

Infrastructure deployment in Terraform 2/2

Fri, 05 Nov 2021 01:05:00 -0400

In a previous post, I introduced Terraform Cloud and covered how to use AWS profiles with Terraform. This time I explored some alternatives to Terraform Cloud, in the context of Azure. I use Scalr as an example of multi-cloud management platform. I will also discuss some issues I’ve came across while managing permissions and variables for Terraform.

Scalr

Scalr is a multi-cloud management platform. I first used it in January but since then it seemed to focus on being a collaboration platform for Terraform. It organizes deployment by environments and workspaces. Accounts in the free tiers is allowed to have one Environment. You will also need to configure (cloud) providers and VCS providers. Once configured, it is important to link a cloud provider with an Environment. Each workspace inside of an Environment can be associated with a VCS provider. In the case of Terraform, this limits a workspace with a single cloud provider.

Permission with Azure

I have a resource group (e.g. named AutomationTest) under a subscription. My account has Contributor role of this resource group. To run Terraform, I could login to Azure as my own account on my environment using AWS CLI. Terraform will pick up the session from Azure CLI and execute as my user. However, it is recommended to run Terraform as a separate own entity. This would allow me to run Terraform template from Scalr, or Terraform Cloud. It is also a good practice for Terraform to use a separate account than a regular user account. There are a number of ways to do this as suggested on the guides for Terraform azurerm provider, including:

Authenticating via a Service Principal and a Client Secret
Authenticating via a Service Principal and a Client Certificate
Authenticating via Managed Identity
Authenticating via the Azure CLI, only recommended when running Terraform locally.

I chose the first option and followed the instruction, using the following CLI command to create the service principal:

az ad sp create-for-rbac -n tf-sp --role="Contributor" --scopes="/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest"

The command returns a few attributes (client ID, tenant ID, secret) that I needed to configure cloud providers in Scalr. The service principal will also be visible under App Registrations in Azure. Once configured I needed to link the provider to an Environment, for Scalr to make an connection to Azure. Otherwise, the Scalr run will return the following Error:

Terraform error

Once provider linking is completed, Scalr automatically populate required environment variables in the workspace. They show up as “Shell” variables under VARIABLES tab.

Under the Terraform tab are input variables that you wish to put in for Terraform template to pick up. Then you can run the template. This works well until I came across a permission issue when I added azurerm_role_assignment resource in Terraform template. What I was trying to do is something like this:

resource "azurerm_role_assignment" "admin_assignment" {
 scope = var.rbac_aks_id
 role_definition_name = "Azure Kubernetes Service RBAC Admin"
 principal_id = var.rbac_principal_object_id
}

And whenever at this line, the following error returned:

Apparently the code 403 indicates Azure doesn’t think the Terraform Service Principal has the privilege to perform Microsoft.Authorization/roleAssignments action. The reason dates back to the way I created service principle above, where I specified contributor role for resource group. However, contributor as a built-in role does not include the permission to assign roles in Azure RBAC. To address this issue, I needed a custom role, named TerraformContributor, with the following definition:

 {
 "assignableScopes": [
 "/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest"
 ],
 "description": "Grants full access to manage all resources, but does not allow you to manage assignments in Azure Blueprints, or share image galleries.",
 "id": "/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/providers/Microsoft.Authorization/roleDefinitions/637824aa-52ae-42f6-a24e-26b2a443afdf",
 "name": "637824aa-52ae-42f6-a24e-26b2a443afdf",
 "permissions": [
 {
 "actions": [
 "*"
 ],
 "dataActions": [],
 "notActions": [
 "Microsoft.Blueprint/blueprintAssignments/delete",
 "Microsoft.Compute/galleries/share/action",
 "Microsoft.Blueprint/blueprintAssignments/write"
 ],
 "notDataActions": []
 }
 ],
 "roleName": "TerraformContributor",
 "roleType": "CustomRole",
 "type": "Microsoft.Authorization/roleDefinitions"
 }

Compare this with the JSON statement of built-in contributor role, the exclusion of actions for Microsoft.Authorization are removed. The creation can be completed with CLI command “az role definition create” in the subscription, or use Azure portal. Once the role is created, create a new service principal using the

az ad sp create-for-rbac -n tf-sp --role="TerraformContributor" --scopes="/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest"

This solution is suggested on this thread.

Composite type for Input variable

Sometimes we want to define an input variable that describes a hierarchy of attributes on a resource. A good example would be Azure Kubernetes service. We can use either object or map as the variable type. The example below has a variable of each type.

variable "cluster_detail" {
 description = "AKS cluster"
 type = object({
 resource_group = string,
 cluster_name = string,
 kubernetes_version = string,
 node_subnet = object({
 subnet_name = string
 vnet_name = string
 resource_group = string
 }),
 pod_subnet = object({
 subnet_name = string
 vnet_name = string
 resource_group = string
 }),
 lb_subnet = object({
 subnet_name = string
 vnet_name = string
 resource_group = string
 }),
 ad_admin_group_object_ids = list(string)
 })
}

variable "common_tags" {
 description = "common tags"
 type = map(any)
 default = {
 tagA = "valueA"
 tagB = "valueB"
 }
}

When using object as the type, the default value needs to define all fields.

In this example where node pool configuration is exposed, we can see how using variables with hierarchy helps template user customize infrastructure specification.

Parsing Map as Input variable

To find out the best multi-cloud management platform, I tested a few of them. I find it inconsistent when I have an input variable of the map type. With Terraform Cloud, I have to specify the variable to be parsed as HCL, and the value has to be:

{"Environment" = "Dev", "Owner" = "info@digihunchtest.com"}

With Scalr, I also have to specify the variable to be parsed as HCL, and the value can be either the one above, or the one below:

{"Environment":"Dev","Owner":"info@digihunchtest.com"}

So Scalr is more flexible in parsing maps. Apart from Scalr and Terraform, I also tested env0 but I gave up after an hour. I could not specify to parse a variable as HCL. They need to work harder on this. Neither was I able to figure out the right syntax as plain variable. I did not test Atlantis or SpaceLift.

Enterprise Deployment

When deploying code to enterprises with their own network environment, Scalr supports running a self-hosted agent inside of the Enterprise network. This is also supported by Terraform Cloud (manage in cloud, execution in enterprise network). This is very useful when the execution machine needs to access the resource created in the enterprise environment.

A good example, is using Terraform’s Azure provider to provision an AKS cluster on the corporate network. Then use Terraform’s kubernetes provider to connect to the newly created cluster and create some Kubernetes object such as service account, as illustrated in this blog post.

AWS CDK example in Python – provision Kubernetes Nodes

Thu, 03 Dec 2020 21:14:00 -0400

There are two mechanisms to initialize instances in AWS. Cloud init and CloudFormation Init. Both are widely used and we discuss each of them in this posting. Then we will give an example of using AWS CDK in Python.

Cloud-Init

Cloud-Init is a service originally built for Ubuntu, as a bootstrapping utility to customize a Linux VM as it boots for the first time. It has evolved to be an industry standard multi-distribution method for cross-platform (public or private) cloud instance initialization, or even bare-metal installation. In cloud-init you can install packages and write files, or configure users and security. Because cloud-init is called during the initial boot process, there are no additional steps or required agents to apply your configuration.

Cloud-Init uses UserData, which is part of instance metadata. With AWS, you can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.

The cloud-config files are text files encoded in base64, with more details covered in the documentation here. cloud-init also works across distributions. For example, you don’t use apt-get install or yum install to install a package. Instead you can define a list of packages to install. cloud-init automatically uses the native package management tool for the distro you select.

CloudFormation Init

The cloudformation init mechanism does not only initialize instance, it also provides a mechanism for the resource being created to communicate with other resources. It allows an instance to emit signal to a different resource (via cfn-signal). It can also monitor changes to external resource and invoke local action (using cfn-hup with hooks). CloudFormation Init requires several components to work together:

The cloudformation resource should have metadata. The metadata must have a key AWS::CloudFormation::Init in which configsets are declared.
The UserData must use helper script (cfn-init) to invoke configuration jobs
The UserData can use helper script (cfn-signal) to signal with a CreationPolicy or WaitCondition (of the same or different resource), so you can synchronize other resources in the stack when the prerequisite resource or application is ready.
The cfn-hup service on the instance can be configured, to check for updates to metadata and execute custom hooks when changes are detected.

Comparison

While there are overlaps between the functionalities of Cloud Init and CloudFormation Init, the major difference is the latter support extended features (signal, update, etc); whereas the former is vendor neutral. The table below summarized some the differences:

	Cloud Init	CloudFormation Init
Works on	Linux OS distribution	CloudFormation resource, in combination with cfn helper scripts
Usecase	Initialization only	Both initialization and resource update
Trigger	cloud-init systemd service	Initial: from UserData Update: by cfn hook
Adoption	Multiple cloud vendors and bare-metal system	AWS cloud instances
Action Playbook	/var/lib/cloud/ Instance Metadata -> User Data, encoded in base 64	CloudFormation Resource -> Metadata section -> AWS::CloudFormation::Init -> configSets and configs
Log file and stdout	/var/log/cloud-init.log /var/log/cloud-init-output.log	/var/log/cfn-init.log /var/log/cfn-init-cmd.log

Comparison between cloud-init and cfn-init

AWS Cloud Development Toolkit (CDK)

Traditionally, AWS CloudFormation uses template in YAML or JSON for resource declaration. As the size of system grows, the amount of resource involved grows quickly and the size of such declaration file may grow beyond manageable. Nested stacks and export of output are mechanisms designed to combat the template sprawling, but to a very limited extent. Two reasons it is hard to control template size are:

In declarative statements, each line carries very small piece of information. Without flow controls such as if-else, loops, object oriented structure, the level of code reusability is very low;
Some auxiliary resources (such as AWS::EC2::VPCGatewayAttachment) must be declared explicitly, even though they are insignificant to the stack functionality

To address these challenges, AWS introduced AWS CDK (cloud development tookkit), which supports multiple languages (JavaScript, TypeScript, Python, Java, and C#). The CDK was natively developed in TypeScript, which is supposed to be the preferred development language. A tutorial is provided here, with detailed API documentation here.

To install aws cdk and create a hello world project, follow this example.

An Example in Python

I have created an example for AWS CDK in Python. The purpose is to create some EC2 instance to complete a lab for Kubernetes (without using managed EKS service). The example provisions the followings:

VPC, a public and private subnets, Internet and NAT gateways;
Relevant security groups and permissions
Bastion host, public instances in public subnet
Private instances in private subnet, with public route through NAT gateway

The private instances forms a cluster for Kubernetes lab. We will use kubespray to initialize these instances. During the bootstraping, we download kubespray, install ansible, etc.

Here is the code repo for this example. With CloudFormation only, the single template could go well beyond 1000 lines. With CDK, the code are organized into several different python files, each representing a stack. The stacks can be stood up with command:

cdk deploy vpc-stack
cdk deploy security-stack
cdk deploy bastion-stack
cdk deploy private-stack

Although the documentation in Python is available, there are generally not a lot of examples built out on the Internet. The pypi site provides some Python specific examples for each module (e.g. core and aws-ec2). Given these libraries are available for only 2 years (since 2018), many advocates TypeScript as the language. However, I have implemented some CloudFormation init, used helper script, and UserData in this example, without running into any language specific issues.It should be noted that the EC2 instance by default will call cfn-init. So there is no need to explicitly run cfn-signal or cfn-init from user data in python code (example). This can be verified in file /var/lib/cloud/instances/<instance-id>/user-data.txt which automatically includes the following lines:

# fingerprint: e1b32ead13878deb
(
 set +e
 /opt/aws/bin/cfn-init -v --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 -c config_set_1,config_set_2
 /opt/aws/bin/cfn-signal -e $? --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456
 cat /var/log/cfn-init.log >&2
)

In addition to Python, AWS CDK also supports other languages. In the next post, we will discuss use of CDK in Typescript.