Amazon Web Services on Digi Hunch

AWS Systems Manager is an Omnipotent Hodgepodge

Sun, 29 Oct 2023 21:32:49 -0400

Introduction to Systems Manager

AWS Systems Manager addresses a lot of SysOps requirements for configuration management, including server automation. In this domain, there is another AWS service called OpsWorks. However, with OpsWorks Stack, OpsWorks Chef and OpsWorks Puppet all coming EOL in 2024, the entire OpsWorks service is mostly deprecated. By partnering with leaders such as Chef and Puppet, OpsWork services represent the era when AWS needed to mirror the configuration management capability on premise, in an effort to convince customers migrating to the cloud. Today, AWS Systems Manager has evolved to fill a lot of gaps around configuration management for servers in the cloud.

Although AWS Systems Manager sounds like a single service. It consists a collection of many seemingly disparate capabilities that serves similar requirements around configuration management. In fact, many of the Systems Manager capabilities are built on top of a couple of what I call core capabilities, such as Session Manager, RunCommand, Automation. This post will review these core capabilities and how Systems Manager employs them to expand with other capabilities.

SSM Agent and Session Manager

What enables all other capabilities is the SSM agent installed on the EC2 instances. The agent running as a systemctl task by ssm-user on EC2 instances. Most of AMIs come with this agent pre-installed. It stores the logs in /var/log/amazon/ssm/. This agent works with an instance profile with a role with the AmazonSSMManagedInstanceCore managed policy, in order to communicate with AWS Systems Manager (ssm.<region>.amazonaws.com) backend. Because of that, you also need to provide a network path to the backend endpoint, either via Internet, or interface endpoint.

This communication also allows an IAM user to connect to an instance’s shell. A common use case is for private instance that do not have Internet access but do have access to SSM backend endpoint. In a previous post I discussed using Session Manager to replace a bastion host to connect to EKS nodes.

When launching an instance using an AMI with SSM pre-installed, the SSM agent should launch after all the config sets from Cloudformation Init are finished. As a result, the Cloudformation Init script is not able to communicate with SSM backend via the agent, unless you install and start SSM agent first on your own, in CloudFormation Init. To troubleshoot SSM, it is important to review its logs.

Through Systems Manager Hybrid Activation, the SSM agent can also work on virtual machines out of AWS and report back to with SSM backend. This gives on-prem servers the identities (instance tags, instance profiles) required for Systems Manager to manage them as if they were EC2 instances. As a result, extend Systems Manager capabilities to on-prem fleet (requiring advanced instances tier).

Types of SSM Documents

There are several types of document that SSM uses, including:

Command Document
Automation Document
Package Document
Session Document
Policy Document
Change Calendar Document

The AWS documentation has a table on what they each are for. Here I’ll focus on three types of documents: Command Document, Automation Document and Session Document.

The Command Document is for the RunCommand capability. It executes on EC2 instances usually performing tasks relating to the operating system or application. I think of a Command Document as an Ansible Playbook that consists of Ansible tasks. We can author Command document that runs configuration steps using plugins, such as aws:downloadContents, aws:runShellScript, etc. This feature directly competes with Ansible. To troubleshoot why a command fail on an instance, check the file ssm-document-worker.log in the ssm agent log directory. Each log entry should have a command ID as reference.

The Automation Document (aka runbooks) is for the Automation Capability. You can define sequence of actions for automation. There are many pre-defined actions such as executing AWS API calls (aws:executeAwsApi), run commands (aws:runCommand), or executing a Lambda function. Therefore a runbook requires an IAM role (Automation Role). The schema of action sequence (YAML or JSON) looks very similar to an Ansible playbook. The web console comes with an UI to visualize the action sequence but most of the time I’d rather .

Session Document is for Session Manager capability. AWS Systems Manager Session Manager uses Session documents to determine which type of session to start, such as a standard session, a port forwarding session, or a session to run an interactive command. In most cases, automation developers do not need to create their own Session document, because the pre-built ones are sufficient:

AWS-PasswordReset
AWS-StartInteractiveCommand
AWS-StartPortForwardingSession
AWS-StartPortForwardingSessionToSocket
AWS-StartSSHSession

In my experience, I use the AWS-StartSSHSession and AWS-StartPortForwardingSession documents most often. To establish SSH connection for forward port to connecting host for Remote Desktop session.

To author your own document, reference the schema correctly and use the latest SSM agent. However, I would explore if any existing shared document in the library already covers what you need. For example, the command document AWS-JoinDirectoryServiceDomain help join a Windows server to a managed Active Directory domain. The command document AWS-RunPatchBaseline is used by Systems Manager Patch Manager capability to check and apply operating system patches. They include steps for Windows, MacOS and Linux instances. The automation runbook AWS-AttachIAMToInstance helps you add IAM role to an EC2 instance.

RunCommand and Automation

The Run Command capability run on top of SSM agent. You can specify one or more target instances. You also specify other other options such as command parameters, rate control and where the output goes. This capability allows an IAM user to run command directly on the OS of an instance (using an OS user ssm-agent) and centrally keep track of those command runs on the AWS side. The most common commands to run on the OS is packaged into Command Documents. There is even a Command Document that allows you to run a pre-built Ansible playbook.

Another way this capability is extremely helpful, is that we can reduce the load of cloud init process. Traditionally, we put a log of logics in the user data script for the cloud init process to execute. The cloud-init mechanism comes from Linux OS and the execution of the user data script is not very transparent to troubleshoot. You have to check the cloud-init-output log from the OS. The use of the UserData script should be reserved for establishing communication with CloudFormation endpoint and SSM endpoint. From there, other automation tasks should be done using SSM capabilities (e.g. State Manager) for better manageabilities.

Take an example of joining a newly provisioned Windows server to a domain. If we do this in user data script, we will have a few problems. First, we can only tell success/fail state from logs in the OS. Second, if an OS user inadvertently removed the instance from domain, there is no mechanism to capture that. If we use Systems Manager’s RunCommand capability, along with State Manager association, the AWS management console will be able to tell whether domain joining is successful, and the association can detect when the instance is removed from domain, report this finding as out of compliance, and remediate the issue. We’ll discuss State Manager in more detail in the next section.

As part of automation, we often have to invoke AWS API calls, which happens outside of any target VMs. The Automation capability of Systems Manager is for this scenario. You can orchestrate your API calls using Automation runbooks. These automation steps do not execute on any target EC2 instance, so they do not rely on SSM agent. However, it needs its own IAM role to perform API tasks. This capability saves you from having to run API calls by creating a new Shell environment to run AWS CLI, or from your own Lambda function using the boto3 SDK library.

When we combine Automation and RunCommand capabilities, we can perform most of the automation orchestration steps. They are the core capabilities that further enable a variety of other Systems Manager capabilities.

Maintenance Window and State Manager

Maintenance window is a very straight forward capability to schedule RunCommand activity with a cron or rate expression. You can specify target by instance tags, define one or more tasks, and define a window of activity and at what point prior to the end of Windows should the agent stop performing more activities (cutoff). Each task can be a type of a RunCommand command, Step Function, Lambda function and automation runbooks.

State Manager is a similar capabilities with a lot of feature overlap with Maintenance Window. State Manager operates on the concept of associations. An association connects target instances to command document or automation runbook to execute. Similar to Maintenance Window, you can specify a schedule expression, document parameters and instance tags. State Manager was brought in to combat configuration drift. The associated document should consist of idempotent scripts so that a State Manager association can repeatedly execute these documents to ensure compliance.

Maintenance Window is more about scheduling one or more tasks. On the State Manager side however, association failure by default will be reported as out of compliance compliance. This is useful in scenarios such as keeping a Window instance in the domain, or keeping SSM agent up to date. You can choose either capability for many common setups but they have subtle differences. For example, for Patch management, you can use State Manager to detect missing patches and report compliance, and Maintenance Window to actually apply the missing patches. In fact, there is a document page on choosing between State Manager and Maintenance Windows to distinguish their best use cases.

Fleet Manager and Inventory

Fleet Manager presents a centralized view for all instances for users to perform common administration tasks, such as exploring file systems and logs, admin users and groups, manage registry and events on Windows instances, check processes and performance metrics. It also gives shortcuts to patch nodes, run commands, start session, etc. I think of Fleet Manager as a minimalist configuration management UI. It is not as sophisticated as those from Ansible Tower or Puppet but it comes at no additional cost.

A very useful feature of Fleet Manager is to run a web-based remote desktop to connect to Windows Instances. This saves the need for a bastion host as long as the instances have SSM connection. You will need the RSA private key to decrypt the Administrator password, which I would not recommend. If the Windows server is on a domain, you can enter your domain credential via Fleet Manager. If the users logged in via IAM identity center, Fleet Manager also has the login option for them via SSO using IAM Identity Center identity. When a user logs in this way, Fleet Manager uses RunCommand capability to execute AWSSSO-CreateSSOUser document against the server to create a local admin user.

Another aspect of configuration management is the inventory management. Unlike in Ansible, the term inventory in the context of Systems Manager refers to the metadata of instances, which includes installed applications, AWS components, network configurations, instance details, services, Windows registry and roles, etc. The full list of what is part of metadata is in the document and you can even define your own inventory item. To gather inventory data, we can makes use of a State Manager association to execute the AWS-GatherSoftwareInventory document. Once we set up the association, the agents will report inventory data back to Systems Manager. More importantly, we can create Resource Data Sync objects to write inventory data (along with compliance data) to S3 buckets, allowing downstream applications to consume. A common use case is to run Athena query against those bucket and produce QuickSight dashboard.

Patch Manager and Compliance

The Patch Manager also operates on State Manager associations. The automation runbook is AWSRunPatchBaseline, where you can just scan for missing patches or install them as well. The SSM document can run on all three platforms (Windows, Linux and MacOS) and determines which patches are missing relative to a the Patch Baseline. There should be at least one default Patch baseline. Each OS (e.g. Ubuntu, Debian, Amazon Linux, etc) classifies patches differently, and a patch baseline is a configuration that defines whether a patch is approved based on operating system and their classifications. The automation document also allows you to override the patch baseline. When executing the document to scan for patches, it records patch compliance information using the PutInventory API command. When using the document to install patches, you can run the document from a Maintenance Window and specify whether you need to reboot the target instance if required.

The compliance capability reports compliance status for instances. By default there are two types of compliance: association and patch. The association compliance detects whether a state manager association is failed on certain instances. The patch compliance, as just mentioned, checks whether patches are up to date relative to the specified patch baseline. You can also define custom compliance item (with put-compliance-items API) but the documentation isn’t clear on what exactly it can achieve and where on the instance does it pull the compliance status. From the example in put-compliance-items, custom compliance type seems to check the installation of additional software package in the inventory.

Other capabilities

Amongst the other capabilities, the one I use the most often is parameter store, which is a way to store a variable for different services to consume.

In the domain of change management, the change manager is a mini change management system. Organization can use it to manage their change process such as approvals. More importantly, you can fire automation runbook from change manager and tie it back to the change control item. Change calendar allows you to block changes during specific period. Both of them are organization level capabilities.

When it comes to operations management, the Incident Manager capability allows you to create response plan for incidents. Response plan can execute runbook actions once an incident is logged. It also helps you notify the on-call incident response team. On the other hand, OpsCenter capability allows you to create OpsItem, which also includes a way to execute runbook. The OpsData can aggregate to Explorer, which is a centralized dashboard for operations data. The Explorer, OpsCenter and Incident Manager capabilities can operate at organization level.

These capabilities around change management and operations management come nowhere close to full-fledged ITSM solutions such as ServiceNow or SMAX. However, they have the ability to trigger runbooks and natively integrate with other AWS services.

There is also a quick setup capability which uses pre-baked CloudFormation template to configure other services. For Patch manager the current recommendation is to use quick setup to configure patch policy.

Summary

Systems Manager has so many capabilities that I cannot cover everything in a single post. Here is a good walk-through. Some capabilities like session manager, fleet manager and state manager, are extremely helpful. However, in my opinion, there are two problems with grouping all these capabilities under Systems Manager. First, With too many different capabilities, this service lacks focus, which makes it difficult to learn. Second, some capabilities have overlap with other capabilities, or another AWS services, which also makes it confusing. I try to sort out how these capabilities enable each other in the diagram below:

This diagram may not be 100% accurate but it demonstrate the dependencies and can assist troubleshooting. For example, when compliance is missing data, check the execution history of run command. It also illustrates the key role of SSM agent as the underlying enabler of most of the other capabilities.

Overall, Systems Manager is extremely powerful. You can try to replace your server management solutions (e.g. Ansible, Chef and Puppet) with Systems Manager configurations. With a good understanding of its capabilities, you can build your fleet automation in an efficient and scalable way.

Orchestrate Landing Zone with Landing Zone Accelerator on AWS

Fri, 22 Sep 2023 23:05:04 -0400

As a continuation to the last post, we explore the Landing Zone Accelerator on AWS (LZA) as an orchestration tool in this post. LZA borrows a lot from the ASEA, an accelerator project to deploy the security reference architecture (SRA). LZA is a multi-purpose project that consists of both the orchestration engine (the accelerator itself) and a few reference architectures (as configuration files).

Comparison with Control Tower

First, let’s sort out how LZA is related to Control Tower. Control Tower’s main functionalities are available as an AWS service, with some customization capabilities available as a standalone solution on top of the service, as I discussed in the last post. Unlike Control Tower, LZA as a whole is a standalone solution. Luckily, the installation of the solution itself is highly automated.

I see LZA both as an extension of Control Tower, and as a complement to Control Tower. It is an extension of Control Tower because LZA can co-exist with Control Tower. We can configure LZA to enable Control Tower and use its Account Factory to provision new accounts (alternatively but not recommended, we can opt out of Control Tower and manage account creation on our own). I also see LZA as a complement to Control Tower because it comes with full end to end automation scheme for networking infrastructure and most of the services involved. This is missing in Control Tower, which leaves it with users to provision networking infrastructure in the customization.

Thanks to the infrastructure automation capability, even if you do not have a strong regulatory requirement, there are still good reason to go with LZA for its low-code automation capability. Below is a table that summarizes the differences:

Control Tower	Landing Zone Accelerator
– Multi-account management tool – Governance layer – Customization Framework to bring your own infrastructure automation	– can manage Control Tower – low-code automation engine for infrastructure automation and service deployment based on CDK – reference configurations based on common industry profiles and regulatory requirements

Comparison between Control Tower and Landing Zone Accelerator

As the name suggests, LZA is an accelerator so there is no expectation of its user knowing how to program infrastructure as code. However, it still expects its users to know YAML very well. Knowing how CloudFormation and CDK works can greatly help the users troubleshoot deployment.

Reference architectures in LZA

The input of LZA is configuration as code in YAML format. The LZA repository comes with a number of sample configurations to implement some industry-based best practices. The reference architectures currently include:

General best practices reference configuration: for clients other than the categories below;
Government customers: US Gov Cloud (FedRAMP compliant, on aws-us-gov partition), US State and Local Government, China (on aws-cn partition), Canada Federal (CCCS compliant) TSE-SE (Highly Trusted Secure Enclave Sensitive Edition) on commercial partition for governments, national security, defence, and law enforcement customers reference architecture;
Election: for election customers including elections agencies, committees and campaigns;
Healthcare: for healthcare customers. However, the document does not mention HIPAA compliance or anything related to the HIPAA Reference Architecture;
Finance and Taxation: for tax workload to secure Federal Tax Information (FTI) data;
Education: for education industry customers.

Many of these reference architecture shares a few common traits in the networking design. Take the CCCS reference as an example, the networking involves the followings:

Workload VPCs: consisting of a number VPCs for production and test environments;
Shared services VPC: hosting common services such as pipelines, Active Directories, etc
Endpoint VPCs: centrally hosting interface endpoints
Perimeter VPCs: acting as ingress, egress and inspection VPCs.

The Perimeter VPC hosts firewalls (either AWS Network Firewall or NGFW appliances behind Gateway Load Balancers). All the VPCs are centrally managed in an AWS network account, and are shared to other accounts using Resource Access Manager. The reference architecture document keeps the details of this architecture, which was derived from the security reference architecture (SRA).

Special Purpose VPCs

I consider the non-workload VPCs as special purpose VPCs. The shared services VPC is the most straight-forward. The Endpoint VPC is the most standardized. It is used to centrally host VPC interface endpoints for security and cost reasons. Unlike Gateway endpoint which is only available for S3 and DynamoDB, interface endpoint carries a standing charge and therefore should be consolidated. In addition, since interface endpoints are based on interfaces, we can centrally control the security group and interface policy.

To integrate the endpoint VPC, not only do we need to create those interface endpoint. We also need to account for routing (using Transit Gateway route tables) and name resolution. For name resolution, we need to create a Route53 private hosted zone for each DNS name, such as ec2.us-east-1.amazonaws.com and associate them with each workload VPC. Note that the interface endpoints DNS name may not always follow the same format. See the exceptions in my old post. Also note that this would create a lot of associations (between Private Hosted Zone for each Interface endpoint and each workload VPC). For example, 20 workload VPC with 30 private hosted zones will create 600 associations. To overcome this, use Route53 profile (introduced in April 2024).

Another special purpose VPC is the perimeter VPC. This VPC vary greatly between customers because of different requirement and historical preferences. One of the key design areas is the placement of NGFW, which is discussed in this post.

LZA Orchestration Engine

The installation process may feel complex at the beginning because we have to first install the pipeline to that installs the pipeline. The initial setup consists the following steps:

CloudFormation installs the installer. We start with a CloudFormation template to deploy the LZA installer itself. It deploys resources such as CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject). These resources are in the INSTALLER circle in the diagram below;
The installer installs the accelerator core. In the LZA installer, the CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject) drive the installation of the LZA. The input is the official LZA GitHub and we need a GitHub token for this step. The output is the actual LZA orchestration engine, including CodePipeline (AWSAccelerator-Pipeline) and CodeBuild (AWSAccelerator-BuildProject and AWSAccelerator-ToolkitProject). The user may specify their own GitHub repo as the configuration repo. Otherwise, a CodeCommit repo will be created. The LZA resources are shown in the CORE circle in the diagram below;
The acceleration core configures the landing zone. The LZA orchestration engine deploys actual resources in the landing zone, with the CodeCommit repo (aws-accelerator-config) or the specified GitHub repo as input.

If we enable Control Tower with LZA, we should first log in to management account and configure Landing Zone with Control Tower. we can also create (and register) the required OUs and accounts from Control Tower. Then we can deploy Landing Zone Accelerator with default configuration.

After the initial setup, we will need to iterate over the aws-accelerator-config repo. We implement our landing zone design in YAML configuration following the schema documentation. Changes in the configuration repo will trigger the pipeline (aka LZA’s orchestration engine) to redo step 3, whereas step 1 and step 2 are performed only once. The duration of step 3 is significantly longer than the first two steps.

Pitfalls

If LZA manages Control Tower, it expects existing OUs registered in Control Tower or it will report error. For account, LZA can create accounts listed in the manifest but not yet created. However, with the lengthy account vendor process for multiple account we run the risk of task time out in the pipeline.

During the installation, some account may run into quota limit. For example, the Networking Account usually have more than five VPCs whereas the quota is 5 VPCs per region per account. We need to increase the quota on those accounts.

The full deployment usually creates some SCPs. However, if we ever need to re-deploy a configuration, some steps steps might be blocked by certain SCPs. Attempts to temporarily detach SCPs from OUs, or modify SCPs often get reverted. The cause is an EventBridgeRule in us-east-1 region named RevertScpChangesModifySc. The rule should be disabled temporarily to perform the troubleshooting activity. We can do this with the following steps:

Disable the EventBridgeRule RevertScpChangesModifySc , which is only present in us-east-1 region;
Detach SCPs and note down what are detached, one OU at a time;
Go to the failed CF stack in the region, delete the failed stacks (after turning off termination protection);
Rerun the pipeline step from where it failed. This time it should go past the failure to the end, if SCP is the cause as per our assumption;
Re-attach SCPs. Suppose Security and Infrastructure OUs share one group of SCPs, and Dev, Test, and Prod OUs share a different group of SCPs;
Re-enable the EventBridgeRule;

Even with the EventBridgeRule RevertScpChangesModifySc disabled, when you re-run LZA deployment pipeline, the Accounts step will re-attach the SCPs using the AWSAccelerator-AccountsStack in the management account in us-east-1 region.

In general, how SCP works with organization structure is something to be very careful about, especially when the hierarchy consists of multiple layers of OUs. It is important to keep in mind, that deny statements in SCP take effect down the hierarchy, where as allow statements only affects the immediate child account of the OU where the SCP is attached to, as per the evaluation logic. As a result, an SCP with allow * statement (in the LZA-AWSFullAWSAccess managed policy) must be applied to Root, every OU at each level, and every account, for LZA to function.

In addition, there are some hard limits for SCP. Each SCP has a size limit of 5120 characters, and each OU can attach a limit of 5 SCPs.

Challenges

Powered by CDK, LZA automates the creation of a lot of resources. The configuration files uses the deploymentTargets attribute to allow users to specify to which accounts or OUs the declared resources will be deployed to.

Supporting many resources is a double-edge sword. Because the accelerator needs to go through every aspect of a landing zone, it is very slow to run. The accelerator pipeline may take as long as 40 minutes without any change to the configuration code. This is extremely slow if you just want to make some small changes in the configuration (e.g. update route table, add IAM role).

Even though LZA supports many resources, it’s not flexible with every resource. For example, today we can deploy IAM roles using RoleSet. However, in the trust policy of the IAM role you can only specify a two types of principals under the assumedBy attribute: account and service types. On the other hand a trust policy can support many other types of principals such as another IAM role.

AZ Mapping

Another important ability that LZA does not support is consistent AZ mapping across accounts. (Correction: this is now supported in LZA v1.5 as of Oct 2023). In some example LZA configurations, we deploy VPCs across multiple accounts using two or three availability zones referenced by their logical ID, such as us-east-1a and us-east-1b. However, AWS maps logical ID to physical ID and the mapping may be different in each AWS account. Using the same logical ID cannot guarantee the physical AZ are the same across account. As of LZA 1.5, the ability to reference physical ID in availabilityZone is supported in LZA configuration file.

Summary

I’ve spent a lot of time on LZA recently. It is extremely powerful. LZA streamlined the landing zone deployment process with configuration as code. It also allows users to customize their landing zone towards their own architectural needs and compliance requirement. For example, you can declare arbitrary SSM parameters in each account.

On the down side, the LZA deployment is time consuming through the pipelines. It tries to automate too many aspects of the infrastructure, which makes itself quite a complex project. Expect lots of changes in each new version.

The idea of being a low-code solution is to make it simple for end users but it often sacrifices flexibility. For example, if you want to create an IAM role in each new account that references the Management account ID, it is not possible until such feature is implemented in the accelerator. When the accelerator pipeline fails, it still requires deep CloudFormation knowledge to troubleshoot.

Orchestrate Landing Zone with AWS Control Tower

Sat, 19 Aug 2023 17:25:00 -0400

Following an introduction to AWS Landing Zone, I’ll dive deeper into Control Tower as an orchestration tool in this post.

More on Landing Zone

In data center operation, there are numerous tasks that other teams have to complete before the the deployment of an application. For example, the 42U server cabinet must be in place with dual powers. The cabinet comes with a network switch in the middle and each ethernet port must be provisioned. Once the server is connected to the central network, the NOC team assigns IP address, configure dynamic VLAN on the switches, and configures firewalls etc according to the connectivity requirement. The new server also needs to report to centralized monitoring solutions such as SolarWinds. In cloud operations, the scope and target of a landing zone is similar to those data center operations work, with networking being the core. The idea is that the landing zone ensures security, compliance and governance, so that applications (analogous to paratroopers) can focus on their primary responsibility.

Although Landing Zone is a general concept for any cloud service provider (CSP), each CSP has some prescriptive guidances on setting up landing zone in their particular environment. For example, an AWS prescriptive landing zone typically covers the following apsects:

Multi-account structure
Identity and Access Management
Governance (controls and guardrails)
Networking
Additional Security Services

The multi-account best practice is an important aspect in AWS as the OU structure dictates how effective Service Control Policies (SCPs) can govern the entire footprint. A landing zone orchestration solution should also apply guardrails and controls based on the organization’s security and compliance requirement. In addition, it is common expectation that landing zone orchestration solutions create required networking resources such that applications are ready to deploy securely.

This post is about Control Tower.

Intro to Control Tower

Control Tower is a landing zone orchestration solution available as an AWS service. I have three impressions over Control Tower. First, it is good with managing multi-account structure. It integrates closely with AWS Organizations and requires client to have a dedicated log archive account and a dedicated security tooling account, which also serves as audit account.

The second impression is Control Tower makes governance more straightforward. AWS seems to use the term control and guardrail interchangeably. Below is an illustration:

The detective controls only captures and reports violations. The preventive controls stops the violating API request. The proactive controls remediates the detected violations. Another way to look at the controls are the priority of the controls:

Mandatory controls: there are about 23 mandatory controls that are enforced on each account. As soon as a new workload account becomes part of Control Tower, these controls will come enforced.
Optional controls including strongly recommended controls and Elective controls.

There are many existing controls and AWS is still releasing new one (example). AWS Config service uses conformance pack to organize relevant controls. For example, there are sample conformance packs such as “Operational Best Practices for “NIST 800 53 rev4” and “Security Best Practices for EKS”. In addition, you can even build your own conformance pack. Once you deploy a conformance pack, AWS config deploys additional rules against the current environment.

The third impression is that Control Tower falls short with infrastructure automation. It provides a number of customization mechanism to leave infrastructure (mostly networking) automation with users. I will explain later.

Mandatory Accounts

The function of Control Tower does not depend on a specific account structure that aligns with the multi-account best practice. However it does require three mandatory accounts to set up automatically when creating control tower. The account where Control Tower is configured is the management account. The log archive account is the owner of S3 buckets that hosts buckets for loggings. The audit account is a restricted account that’s designed to give your security and compliance teams read and write access to all accounts in your landing zone. We also use this account as delegated administrator account for several security services at organization level. I’ll elaborate in the next section.

The management account is directly under root OU and there is no SCPs applied to the management account. Also Control Tower does not turn on AWS Config recorder and configure delivery channel on the management account itself.

The log archive account and audit account are under the Security OU (or otherwise named during the setup). At the end of Control Tower setup a set of mandatory controls will be applied at the OU level and thus effective to each account.

Apart from the mandatory accounts, oftentimes there is an Infrastructure OU. Typically we place shared services account under this OU. The account is for common services such as central networking, managed Active Directory, DevOps pipeline etc. Another optional OU is workload OU, where we can create new workload accounts or enrol existing workload accounts.

Each enrolled workload account have its CloudTrail configured with an organization-level trail, which sends log to the central bucket that log archive account owns. Each enrolled account also has AWS Config recorder configured with a delivery channel pointing to the configuration log bucket also owned by log archive account. Note that Control Tower does not turn on recorder on the management account itself.

Account Enrolment

Most likely a customer is already in use of AWS Organization. They can delegate one account as management account, and use Control Tower console to create a landing zone for the intended regions. During the creation, they will have to create two additional account, one as log archive account, and the other as security tooling or audit account. This is in alignment with multi-account best practices. After the creation of landing zone, the two new account are automatically enrolled in Control Tower.

Going forward, the client should create new accounts using Control Tower’s Account Factory to save a separate enrolment step. For existing accounts and OUs we’ll need to enroll them into Control Tower. At the beginning, the Control Tower Landing Zone (CTLZ) only has the mandatory controls, which are the bare minimum governance. When we enroll existing OUs and accounts, these controls (mostly detective and preventive) will extend to the newly enrolled OUs or accounts. Therefore, it is important to not introduce numerous controls prior to having all accounts and OUs enrolled. It is important to have a proper OU hierarchy upfront because both detective and preventive controls (Config Rules and SCPs) are typically applied at OU level and effective to all the children of the the OU.

To enroll an existing account, the account must meet some prerequisites. An IAM role with cross-account trust must be manually created. Since each account can have only one AWS Config configuration recorder and delivery channel, if an existing account already has one of them, they must be manually deleted using CLI.

Account Factory defines what happens when we create a new account and it is where we can bake in the customizations. When it comes to infrastructure automation, Control Tower leaves pretty much everything up to the account factory customization.

Delegated Administrator

Many AWS services are available as organization wide service. For example, CloudTrail, GuardDuty and even AWS Organization. Most of these organization level services allows you to specify one account as delegated administrator for the entire organization. For example, for System Manager and Service Catalog, the best practice is to designate an operation account or shared services account as the delegated administrator.

There are several security-related, organization-wide services, such as GuardDuty, Detective, Macie, Security Hub, Inspector, Audit Manager, and Firewall Manager. The best practices for these services is to designate the audit account as the delegated administrator. We think of the audit account as an aggregation point (or points for organizations that split the functionality across multiple accounts) for these AWS services. You can find this recommendation from the documentation of each service. However, currently Control Tower does not enforce this best practice. Even though Control Tower dashboard gives you visibility to security services such as GuardDuty and SecurityHub, you are still on your own to configure these services outside of Control Tower.

Whether you had never used these services, or you had previously designated another account as administrator for these services, it is good time to align with the best practice for delegated administrator right after Control Tower setup.

If you had previously designated other accounts, you need to take additional steps to revoke the old delegation and designate new delegated administrator account. These general steps can be performed from console or CLI and apply to all these services:

From the old administrator account, remove all the members
From the management account, revoke delegation
From the management account, delegate new administrator account (audit account)
From the new administrator account, invite all the members, and make sure to tell the service to automatically include new accounts going forward

Note that when you invite an account, that account must not have created, invited or associated with other accounts (as a member). In other words, we should never let a non-delegated administrator account create, invite or associate other account as member.

Control Tower Customization

Control Tower by itself does not play a big role in networking automation. You can create a VPC when provisioning a new account in Control Tower. That’s about what you can do. It does not address how the new VPC connects to existing networking space. It does not distinguish the existing VPC topology. To address this, Control Tower has a few customization options:

	AFC Account Factory Customization	AFT Account Factory Terraform	CfCT Customizations for Control Tower
Supported IaC language	Anything that Service Catalog Support (e.g. CloudFormation and Terraform HCL)	Terraform HCL	CloudFormation
Readiness	AFC is a native mechanism. You can specify the Service Catalog product during creation of an Account in Control Tower console.	Users need to first bootstrap the solution, which creates Terraform pipelines and other components.	Users need to first bootstrap the solution which creates a pipeline and other components.
How it works	Control Tower launches the specified Service Catalog product (aka blueprint) during account creation.	Quite complex. Explained in the section below	Control Tower posts a lifecycle events to Amazon EventBridge. A lambda function will process the event using pre-baked CloudFormation templates and Step Functions.
Prebuilt customizations	No. However, for certain products, there are some partner-maintained blueprints	Yes	No

AFC is fairly straightforward to use based on this post, as long as you know Service Catalog. It is a native capability of Control Tower. The other two ways for CT landing zone customization are not native capability and each require a separate bootstrapping process to deploy the pipeline-based solution.

Customizations for Control Tower

In CfCT, we first bootstrap the solution from management account, using this CloudFormation template on GitHub. We usually need to version control the custom configuration so we should select “AWS CodeCommit” as the value for the CodePipelineSource parameter. The bootstrapping (execution of this CloudFormation template) takes about 5 minutes.

The solution is based on Control Tower’s lifecycle event to trigger the pipeline with a source code repository (or S3 bucket). The source code repository (AWS CodeCommit) stores the custom configuration. A custom configuration consists of a manifest file in YAML format, made up with a number of resource sections. Each resource section references an artifact, either a CloudFormation template or a policy file to apply. The artifact can either be stored in the same repository, or in a remote S3 bucket.

Account Factory Terraform

Introduced in late 2021, the AFT solution is based on the terraform-aws-control_tower_account_factory repository. It is too complex as a solution in my opinion. At the beginning you need to have a designated OU and account for AFT and install the solution itself with Terraform. Note that, the solution does not address the state storage of the installation of the solution itself (chicken-or-egg). The installation also installed a few IAM roles required on the AFT management account. Having a landing zone management account and a AFT management account is not a neat setup.

Once installed, the solution consists of four repositories. Their purposes are as follows (copied from workshop instruction):

Account requests – handles placing or updating account requests. See example here.
AFT account provisioning customizations – manages non-Terraform customizations that are applied to all accounts. This stage runs before the global customizations stage. Examples available here
Global customizations – Global customizations – manages Terraform-based customizations that are applied to all accounts created by and managed with AFT. Examples available
Account customizations – Account customizations – manages Terraform-based customizations that are applied only to specific accounts created by and managed with AFT. Examples available

With this solution in place, when we create a new account via account request repo, quite a number of steps will happen after that, as the workshop instruction illustrated:

When I first looked into how this work I’m very concerned about the maintainability of this “solution”. The amount of services and pipelines involved in this solution makes it difficult to troubleshoot end to end. To improve that AWS even added an enhancement for request tracing. Nonetheless, I would not go with AFT just with how complex it looks like. For more details, check out the Control Tower workshop.

Summary

Control Tower helps you set up a Landing Zone without network infrastructure automation. It provides a couple of customization mechanisms, allowing you to bake in your own infrastructure automation. To build a landing zone, you can go with Control Tower in the following situations:

you have general regulatory requirement and need to enable governance capabilities
your networking stack isn’t centrally managed or isn’t large enough to warrant infrastructure automation
If you do need networking automation, then you will need the customization capability of Control Tower. In this case, your team should have good handle on infrastructure as code (CloudFormation or Terraform)

Out of the customization options, AFC seems the least complex to me. The other two, especially AFT, is too complex. We use pipelines to trail and error with other stacks, and the pipeline solution itself must be simple.

On the other hand, if you environment has strong regulatory requirement, or you are seeking a prescriptive network architecture, or your team does not have the capacity with CloudFormation or Terraform Templates, consider Landing Zone Accelerator on AWS.