logging on Digi Hunch

Orchestrate Landing Zone with AWS Control Tower

Sat, 19 Aug 2023 17:25:00 -0400

Following an introduction to AWS Landing Zone, I’ll dive deeper into Control Tower as an orchestration tool in this post.

More on Landing Zone

In data center operation, there are numerous tasks that other teams have to complete before the the deployment of an application. For example, the 42U server cabinet must be in place with dual powers. The cabinet comes with a network switch in the middle and each ethernet port must be provisioned. Once the server is connected to the central network, the NOC team assigns IP address, configure dynamic VLAN on the switches, and configures firewalls etc according to the connectivity requirement. The new server also needs to report to centralized monitoring solutions such as SolarWinds. In cloud operations, the scope and target of a landing zone is similar to those data center operations work, with networking being the core. The idea is that the landing zone ensures security, compliance and governance, so that applications (analogous to paratroopers) can focus on their primary responsibility.

Although Landing Zone is a general concept for any cloud service provider (CSP), each CSP has some prescriptive guidances on setting up landing zone in their particular environment. For example, an AWS prescriptive landing zone typically covers the following apsects:

Multi-account structure
Identity and Access Management
Governance (controls and guardrails)
Networking
Additional Security Services

The multi-account best practice is an important aspect in AWS as the OU structure dictates how effective Service Control Policies (SCPs) can govern the entire footprint. A landing zone orchestration solution should also apply guardrails and controls based on the organization’s security and compliance requirement. In addition, it is common expectation that landing zone orchestration solutions create required networking resources such that applications are ready to deploy securely.

This post is about Control Tower.

Intro to Control Tower

Control Tower is a landing zone orchestration solution available as an AWS service. I have three impressions over Control Tower. First, it is good with managing multi-account structure. It integrates closely with AWS Organizations and requires client to have a dedicated log archive account and a dedicated security tooling account, which also serves as audit account.

The second impression is Control Tower makes governance more straightforward. AWS seems to use the term control and guardrail interchangeably. Below is an illustration:

The detective controls only captures and reports violations. The preventive controls stops the violating API request. The proactive controls remediates the detected violations. Another way to look at the controls are the priority of the controls:

Mandatory controls: there are about 23 mandatory controls that are enforced on each account. As soon as a new workload account becomes part of Control Tower, these controls will come enforced.
Optional controls including strongly recommended controls and Elective controls.

There are many existing controls and AWS is still releasing new one (example). AWS Config service uses conformance pack to organize relevant controls. For example, there are sample conformance packs such as “Operational Best Practices for “NIST 800 53 rev4” and “Security Best Practices for EKS”. In addition, you can even build your own conformance pack. Once you deploy a conformance pack, AWS config deploys additional rules against the current environment.

The third impression is that Control Tower falls short with infrastructure automation. It provides a number of customization mechanism to leave infrastructure (mostly networking) automation with users. I will explain later.

Mandatory Accounts

The function of Control Tower does not depend on a specific account structure that aligns with the multi-account best practice. However it does require three mandatory accounts to set up automatically when creating control tower. The account where Control Tower is configured is the management account. The log archive account is the owner of S3 buckets that hosts buckets for loggings. The audit account is a restricted account that’s designed to give your security and compliance teams read and write access to all accounts in your landing zone. We also use this account as delegated administrator account for several security services at organization level. I’ll elaborate in the next section.

The management account is directly under root OU and there is no SCPs applied to the management account. Also Control Tower does not turn on AWS Config recorder and configure delivery channel on the management account itself.

The log archive account and audit account are under the Security OU (or otherwise named during the setup). At the end of Control Tower setup a set of mandatory controls will be applied at the OU level and thus effective to each account.

Apart from the mandatory accounts, oftentimes there is an Infrastructure OU. Typically we place shared services account under this OU. The account is for common services such as central networking, managed Active Directory, DevOps pipeline etc. Another optional OU is workload OU, where we can create new workload accounts or enrol existing workload accounts.

Each enrolled workload account have its CloudTrail configured with an organization-level trail, which sends log to the central bucket that log archive account owns. Each enrolled account also has AWS Config recorder configured with a delivery channel pointing to the configuration log bucket also owned by log archive account. Note that Control Tower does not turn on recorder on the management account itself.

Account Enrolment

Most likely a customer is already in use of AWS Organization. They can delegate one account as management account, and use Control Tower console to create a landing zone for the intended regions. During the creation, they will have to create two additional account, one as log archive account, and the other as security tooling or audit account. This is in alignment with multi-account best practices. After the creation of landing zone, the two new account are automatically enrolled in Control Tower.

Going forward, the client should create new accounts using Control Tower’s Account Factory to save a separate enrolment step. For existing accounts and OUs we’ll need to enroll them into Control Tower. At the beginning, the Control Tower Landing Zone (CTLZ) only has the mandatory controls, which are the bare minimum governance. When we enroll existing OUs and accounts, these controls (mostly detective and preventive) will extend to the newly enrolled OUs or accounts. Therefore, it is important to not introduce numerous controls prior to having all accounts and OUs enrolled. It is important to have a proper OU hierarchy upfront because both detective and preventive controls (Config Rules and SCPs) are typically applied at OU level and effective to all the children of the the OU.

To enroll an existing account, the account must meet some prerequisites. An IAM role with cross-account trust must be manually created. Since each account can have only one AWS Config configuration recorder and delivery channel, if an existing account already has one of them, they must be manually deleted using CLI.

Account Factory defines what happens when we create a new account and it is where we can bake in the customizations. When it comes to infrastructure automation, Control Tower leaves pretty much everything up to the account factory customization.

Delegated Administrator

Many AWS services are available as organization wide service. For example, CloudTrail, GuardDuty and even AWS Organization. Most of these organization level services allows you to specify one account as delegated administrator for the entire organization. For example, for System Manager and Service Catalog, the best practice is to designate an operation account or shared services account as the delegated administrator.

There are several security-related, organization-wide services, such as GuardDuty, Detective, Macie, Security Hub, Inspector, Audit Manager, and Firewall Manager. The best practices for these services is to designate the audit account as the delegated administrator. We think of the audit account as an aggregation point (or points for organizations that split the functionality across multiple accounts) for these AWS services. You can find this recommendation from the documentation of each service. However, currently Control Tower does not enforce this best practice. Even though Control Tower dashboard gives you visibility to security services such as GuardDuty and SecurityHub, you are still on your own to configure these services outside of Control Tower.

Whether you had never used these services, or you had previously designated another account as administrator for these services, it is good time to align with the best practice for delegated administrator right after Control Tower setup.

If you had previously designated other accounts, you need to take additional steps to revoke the old delegation and designate new delegated administrator account. These general steps can be performed from console or CLI and apply to all these services:

From the old administrator account, remove all the members
From the management account, revoke delegation
From the management account, delegate new administrator account (audit account)
From the new administrator account, invite all the members, and make sure to tell the service to automatically include new accounts going forward

Note that when you invite an account, that account must not have created, invited or associated with other accounts (as a member). In other words, we should never let a non-delegated administrator account create, invite or associate other account as member.

Control Tower Customization

Control Tower by itself does not play a big role in networking automation. You can create a VPC when provisioning a new account in Control Tower. That’s about what you can do. It does not address how the new VPC connects to existing networking space. It does not distinguish the existing VPC topology. To address this, Control Tower has a few customization options:

	AFC Account Factory Customization	AFT Account Factory Terraform	CfCT Customizations for Control Tower
Supported IaC language	Anything that Service Catalog Support (e.g. CloudFormation and Terraform HCL)	Terraform HCL	CloudFormation
Readiness	AFC is a native mechanism. You can specify the Service Catalog product during creation of an Account in Control Tower console.	Users need to first bootstrap the solution, which creates Terraform pipelines and other components.	Users need to first bootstrap the solution which creates a pipeline and other components.
How it works	Control Tower launches the specified Service Catalog product (aka blueprint) during account creation.	Quite complex. Explained in the section below	Control Tower posts a lifecycle events to Amazon EventBridge. A lambda function will process the event using pre-baked CloudFormation templates and Step Functions.
Prebuilt customizations	No. However, for certain products, there are some partner-maintained blueprints	Yes	No

AFC is fairly straightforward to use based on this post, as long as you know Service Catalog. It is a native capability of Control Tower. The other two ways for CT landing zone customization are not native capability and each require a separate bootstrapping process to deploy the pipeline-based solution.

Customizations for Control Tower

In CfCT, we first bootstrap the solution from management account, using this CloudFormation template on GitHub. We usually need to version control the custom configuration so we should select “AWS CodeCommit” as the value for the CodePipelineSource parameter. The bootstrapping (execution of this CloudFormation template) takes about 5 minutes.

The solution is based on Control Tower’s lifecycle event to trigger the pipeline with a source code repository (or S3 bucket). The source code repository (AWS CodeCommit) stores the custom configuration. A custom configuration consists of a manifest file in YAML format, made up with a number of resource sections. Each resource section references an artifact, either a CloudFormation template or a policy file to apply. The artifact can either be stored in the same repository, or in a remote S3 bucket.

Account Factory Terraform

Introduced in late 2021, the AFT solution is based on the terraform-aws-control_tower_account_factory repository. It is too complex as a solution in my opinion. At the beginning you need to have a designated OU and account for AFT and install the solution itself with Terraform. Note that, the solution does not address the state storage of the installation of the solution itself (chicken-or-egg). The installation also installed a few IAM roles required on the AFT management account. Having a landing zone management account and a AFT management account is not a neat setup.

Once installed, the solution consists of four repositories. Their purposes are as follows (copied from workshop instruction):

Account requests – handles placing or updating account requests. See example here.
AFT account provisioning customizations – manages non-Terraform customizations that are applied to all accounts. This stage runs before the global customizations stage. Examples available here
Global customizations – Global customizations – manages Terraform-based customizations that are applied to all accounts created by and managed with AFT. Examples available
Account customizations – Account customizations – manages Terraform-based customizations that are applied only to specific accounts created by and managed with AFT. Examples available

With this solution in place, when we create a new account via account request repo, quite a number of steps will happen after that, as the workshop instruction illustrated:

When I first looked into how this work I’m very concerned about the maintainability of this “solution”. The amount of services and pipelines involved in this solution makes it difficult to troubleshoot end to end. To improve that AWS even added an enhancement for request tracing. Nonetheless, I would not go with AFT just with how complex it looks like. For more details, check out the Control Tower workshop.

Summary

Control Tower helps you set up a Landing Zone without network infrastructure automation. It provides a couple of customization mechanisms, allowing you to bake in your own infrastructure automation. To build a landing zone, you can go with Control Tower in the following situations:

you have general regulatory requirement and need to enable governance capabilities
your networking stack isn’t centrally managed or isn’t large enough to warrant infrastructure automation
If you do need networking automation, then you will need the customization capability of Control Tower. In this case, your team should have good handle on infrastructure as code (CloudFormation or Terraform)

Out of the customization options, AFC seems the least complex to me. The other two, especially AFT, is too complex. We use pipelines to trail and error with other stacks, and the pipeline solution itself must be simple.

On the other hand, if you environment has strong regulatory requirement, or you are seeking a prescriptive network architecture, or your team does not have the capacity with CloudFormation or Terraform Templates, consider Landing Zone Accelerator on AWS.

Logging and Monitoring in Kubernetes with PLG stack

Wed, 13 Oct 2021 21:29:00 -0400

We’ve checked out the the actors in PLG stack (Promtail, Loki, Node Exporter, Prometheus, Grafana) and whipped up a quick pipeline on MacOS. Now I’m going a little further to implement the same PLG stack (Prometheus Loki and Grafana) in a Kubernetes cluster. This setup is for demo only, therefore no persistent storage is enabled.

Test Workload

I host a deployment of Flog with three pods running on Minikube. Flog is an open-source emulating log generation behaviour of an application. On the Minikube cluster we start the deployment as below:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2
kubectl create ns obsv
kubectl -n obsv create deployment flog --image=mingrammer/flog --replicas=3 -- flog -f rfc3164 -l -d 300ms
kubectl -n obsv get po

The Pods will come up in a heartbeat.

I will use helm to install the objects required for logging and metrics pipelines. There are multiple Helm charts for each components. Some high-level charts (usually with a name suffix of -stack) contain several other resource as sub-charts. They are created as one-stop-shop for multiple components but I found none of them serve my exact purpose. For example, both loki-stack and kube-prometheus-stack include Grafana. But I only need one instance of Grafana. Therefore I stick to the low-level charts.

Log Shipping

Add a helm repo, and install loki and promtail. Note that we need to specify correct loki address when installing Promtail.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --namespace obsv --install loki grafana/loki
helm upgrade --namespace obsv --install promtail grafana/promtail --set "config.lokiAddress=http://loki.obsv.svc.cluster.local:3100/loki/api/v1/push"

When installing Promtail, a DaemonSet is created on the Node. The default configuration applies appropriate configuration and tagging strategy for Kubernetes Pod and Node. So the only customization I specified is Loki address. We can then check logging with Loki. To do so, first expose port 3100 to host, and then use logcli (e.g. on MacOS) to query for logs:

kubectl -n obsv port-forward service/loki 3100:3100
logcli labels
logcli labels pod
logcli query '{pod="flog-775d5fc5c8-p4rlx"}'

Log lines should be pumped to Loki a minute after Loki comes up. The logcli labels command should display the tags. The logcli query command should return the log lines.

Metrics

I use Premetheus with node exporter. In its architecture, Prometheus contain the server, the pushgateway, and alertmanager. The helm chart for Prometheus contains all of those components. It also has a dependency repo for kube-state-metrics. To install:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm --namespace=obsv install prometheus prometheus-community/prometheus
kubectl -n obsv get svc

Once installed, a DaemonSet for Prometheus Node Exporter is created. The exporter is already configured by default for Kubernetes monitoring. The Prometheus server is also configured, on port 80 by default. it needs to be forwarded in order to access from Browser:

kubectl --namespace obsv port-forward service/prometheus-server 9100:80

To verify installation of Prometheus, browse to localhost:9100 to examine the metrics.

Visualization

Last but not least, I will configure Grafana. The repo has been added already so we’ll just install the chart:

helm upgrade --namespace obsv --install grafana grafana/grafana
kubectl get secret --namespace obsv grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The second command retrieves the credential. To access the web portal, we need port forwarding again:

kubectl port-forward --namespace obsv service/grafana 3000:80

To verify installation, browse to http://localhost:3000 and log in as user admin with the password above. Then add two data sources:

Type: Prometheus, URL: http://prometheus-server.obsv.svc.cluster.local:80
Type: Loki, URL: http://loki.obsv.svc.cluster.local:3100

Then we can explore data using both data sources.

PLG stack

Summary

In the last two posts I reviewed the setups for PLG stack in Kubernetes, from a regular environment to k8s cluster. Fluentd, Prometheus are both CNCF projects. The PLG stack seems to be more adopted than EFK but both have their own advantages. Welcome to the PLG vs EFK debate.

Intro to PLG stack -Prometheus, Loki and Grafana

Sun, 03 Oct 2021 12:59:00 -0400

Last month we discussed log shipping with EFK. This week I spent sometime checking out its alternative Loki. Having been exposed to the ELK stack extensively, I am also interested in exploring the counterparts in this new stack, such as Prometheus, Loki and Grafana. So I need to address the issues of shipping both metrics and logs.

Let’s start by clarifying the terms:

Grafana is a visualizer. It supports many backends such as Prometheus, Loki, Elasticsearch, CloudWatch and Azure Monitor. It is the flagship product of Grafana Labs.

Prometheus is a time-series database and alerting platform. To push metrics to Premethus, you can either integrate your application with client library (in their term, instrumenting), or configure an existing exporters for a third party application such as PostgreSQL. Prometheus collects and stores its metrics as time series data ( i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels) and it comes with basic visualization capability. Premetheus is a CNCF project since 2016 and is maintained by Grafana Labs.

Loki is a log aggregation system, also developed by Grafana Labs. Loki does not index the contents of the logs. Instead it groups entries into streams, and indexes a set of labels for each log stream. You can use grafana or logcli to consume the logs. Loki supports clients such as Fluentd, Fluentbit, Logstash and Promtail.

Promtail is a log collection agent built for Loki.

The roles of the components such as Prometheus, Loki, Grafana and Promtail are similar to the ELK stack. Grafana resembles Kibana. Promtail resembles Filebeat. Premethus exporters resemble Metricbeat. Both Premetheus and Loki resemble Elasticsearch in some aspects. Premetheus keeps metrics and Loki persists log streams.

These tools are heavily used in Kubernetes. For a simple start, I’d like to just configure two minimally working pipelines on my MacBook without any containerization. My example setup is to achieve the followings:

Fake up some log lines and ship them to Loki
Ship OS metrics to Prometheus
Display the metrics and logs with Grafana

Here’s the diagram of what we want to implement:

This is simple enough to build a quick and dirty setup

Configure Node Exporter

We can use home brew to install node_exporter on MacOS. For Linux, use the equivalent package management tool:

brew install node_exporter
brew services start node_exporter
curl http://localhost:9100/metrics | grep "node_"

The node_exporter collect metrics and make it available for scrape using port 9100 as shown above.

Configure Prometheus

Install Prometheus with homebrew and start the service:

brew install prometheus
brew services start prometheus

If curl to port 9090 with GET returns “Found”, then Prometheus is successfully installed. We also want to configure it so it scrapes node exporter for metrics. Edit the configuration file, in my case, /usr/local/etc/prometheus.yml, by adding the followings:

 - job_name: node
 static_configs:
 - targets: ['localhost:9100']

This tells Prometheus to scrape metrics at specified interval. Restart Prometheus and browse to http://localhost:9090/ for Prometheus UI. Click on Status -> Targets and you can see the node as an export. Click on Graph and Execute a query for example “node_memory_free_bytes” and click on Graph. You should see a plot of the metric value.

We can later configure to display the chart in Grafana.

Generate log lines

We use a tool called flog to generate fake logs:

brew tap mingrammer/flog
brew install flog
flog -f rfc3164 -l -d 300ms -t log -o /tmp/test.log -w

The command above produces log in RFC3164 format to /tmp/test.log, one line every 300ms. The log does not rotate.

Configure Loki

Install Loki with homebrew and start the service. We want to configure Loki before Promtail so it is ready to receive logs. You might as well install LogCLI to interact with Loki.

brew install loki
brew install logcli
brew services start loki

If nc to port 3100 returns success, then Loki is successfully installed.

Configure Promtail

Install promtail using homebrew:

brew install promtail

Note that Promtail is not installed as a homebrew service. Although it can be manually configured as a service, I’d rather stay focused and use command line just for the time being. A copy of configuration file is located in /usr/local/etc/promtail-local-config.yaml but it needs to be modified first with a job to tell it where to scrap the log lines. The configuration looks like this:

server:
 http_listen_port: 9080
 grpc_listen_port: 0

positions:
 filename: /tmp/positions.yaml

clients:
 - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: app
 static_configs:
 - targets:
 - localhost
 labels:
 job: applogs
 __path__: /tmp/test.log

Then, use the command as in the documentation to start promtail:

promtail -config.file /usr/local/etc/promtail-local-config.yaml

Port 9080 will be open when Promtail is running. Promtail will push logs to Loki. You can use LogCLI to interact with Loki and see the latest log lines pushed to Loki.

logcli labels job
logcli query '{job="applogs"}'

However, unlike Prometheus, Loki itself does not have any visualization capabilities. We will need Grafana to display the log nicely on the web.

Configure Grafana

We use Grafana to visualize both the logs and metrics. To install Grafana on Mac:

brew install grafana
brew services start grafana

If curl to port 3000 with GET returns “Found”, then Grafana is successfully installed. Browse to localhost:3000, with default credential admin and admin. From the UI, add two data sources. For the first data source, specify Prometheus as type and localhost:9000 as destination. For the second, specify Loki as the destination http://localhost:3100

The steps to see logs in Loki is pretty much the same as on this page of its documentation.

Exploring logs

Exploring metrics data is similar. Click on Explore on the side bar, then select Prometheus from the dropdown as data source. Then execute a query such as “node_memory_free_bytes”.

Explore Metrics

With this quick and dirty configuration, we established a good understanding of what Prometheus, Loki and Grafana do. Next, we will move all these configurations to K8s cluster and understand some specific points of configurations.

Log Shipping in Kubernetes with EFK stack

Sat, 04 Sep 2021 21:50:00 -0400

I first worked on log shipping with ELK stack three years ago. In the context of Kubernetes cluster, log shipping has similar challenges. In this post I will discuss the set up of log shipping with Kubernetes cluster using EFK stack

Logging Architecture

As discussed, if the Kubernetes cluster has a runtime in compliant with CRI (e.g. containerd), then the stdout and stderr of the Pod is stored on the node, in the location /var/log/containers/.

When creating log shipping solution, it is important to use a compliant cluster (e.g. minikube) to ensure what you develop will work across environments.

The Kubernetes document has a section on logging architecture which is a good start point. It outlines several different patterns. Logging at the node level is turned on by default and does not require special configuration, as explained in the section above. The EFK pattern is close to the diagram under using a node logging agent for cluster-level logging.

Node logging agent

The diagram above is stolen from Kubernetes documentation. In EFK stack, the agent is a daemonset running fluentd Pod.

EFK stack

While we can use ELK (Elasticsearch, Logstash, Kibana) stack for log shipping, EFK (Elasticsearch, Fluentd, Kibana) is generally recommended in Kubernetes cluster. We compare the two in the following table:

	ELK	EFK
Development language	All in Java. Logstash and Filebeat both require JVM. Managed as an open-source project by Elastic company	Fluentd in Ruby and does not require JVM to run. Fluentd is a CNCF project built to integrate with Kubernetes.
Typical pattern	Filebeat acts as a lightweight collector to monitor the source log. Logstash as aggregator to receive from filebeat, and push to Elasticsearch	The fluentd Pod can be configured to serve as forwarder and aggregator based on configuration. fluentd-forwarder is deployed as daemonset on node, and ship the result to fluentd-aggregator, which may run in a separate cluster. The fluentd-aggregator pushes processed results to Elasticsearch.
Metrics	use metric beat for data collection	scrape metrics from prometheus server
Event Routing	tag-based	if-then statement

ELK and EFK stacks comparison

The main problems in log shipping are:

column mapping : identify column patterns in each log line and map them to appropriate column in Elasticsearch.
multi-line processing: identify when a logging entry spread across multiple lines and process accordingly.

Suppose we want to congregate the logs from stdout and stderr of PostgreSQL pods. The raw output in /var/log/container on the node, looks like this:

2021-08-28T15:27:46.75370563Z stdout F server stopped
2021-08-28T15:27:46.757173069Z stderr F postgresql-repmgr 15:27:46.75 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:46.883126928Z stderr F postgresql-repmgr 15:27:46.88 INFO ==> Registering Primary...
2021-08-28T15:27:47.017164653Z stderr F postgresql-repmgr 15:27:47.01 INFO ==> Loading custom scripts...
2021-08-28T15:27:47.023334611Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
2021-08-28T15:27:47.026169813Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.343607487Z stderr F postgresql-repmgr 15:27:47.34 INFO ==> Stopping PostgreSQL...
2021-08-28T15:27:47.448111425Z stdout F waiting for server to shut down.... done
2021-08-28T15:27:47.448172479Z stdout F server stopped
2021-08-28T15:27:47.453722807Z stderr F postgresql-repmgr 15:27:47.45 INFO ==> ** PostgreSQL with Replication Manager setup finished! **
2021-08-28T15:27:47.453829953Z stdout F
2021-08-28T15:27:47.503516746Z stderr F postgresql-repmgr 15:27:47.50 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.532558987Z stdout F waiting for server to start....2021-08-28 15:27:47.532 GMT [273] LOG: pgaudit extension initialized
2021-08-28T15:27:47.533307459Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-28T15:27:47.533466407Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv6 address "::", port 5432
2021-08-28T15:27:47.537987947Z stdout F 2021-08-28 15:27:47.537 GMT [273] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-28T15:27:47.547956465Z stdout F 2021-08-28 15:27:47.547 GMT [273] LOG: redirecting log output to logging collector process
2021-08-28T15:27:47.548005463Z stdout F 2021-08-28 15:27:47.547 GMT [273] HINT: Future log output will appear in directory "/opt/bitnami/postgresql/logs".
2021-08-28T15:27:47.551741571Z stdout F 2021-08-28 15:27:47.551 GMT [275] LOG: database system was shut down at 2021-08-28 15:27:47 GMT
2021-08-28T15:27:47.558012894Z stdout F 2021-08-28 15:27:47.557 GMT [273] LOG: database system is ready to accept connections
2021-08-28T15:27:47.618577092Z stdout F done
2021-08-28T15:27:47.618708978Z stdout F server started
2021-08-28T15:27:47.630065958Z stderr F postgresql-repmgr 15:27:47.62 INFO ==> ** Starting repmgrd **
2021-08-28T15:27:47.638116348Z stderr F [2021-08-28 15:27:47] [NOTICE] repmgrd (repmgrd 5.2.1) starting up
2021-08-28T15:27:47.65317627Z stderr F INFO: set_repmgrd_pid(): provided pidfile is /opt/bitnami/repmgr/tmp/repmgr.pid
2021-08-28T15:27:47.653232015Z stderr F [2021-08-28 15:27:47] [NOTICE] starting monitoring of node "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:27:47.681683703Z stderr F [2021-08-28 15:27:47] [NOTICE] monitoring cluster primary "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:28:11.742865958Z stderr F [2021-08-28 15:28:11] [NOTICE] new standby "orthweb-postgresql-ha-postgresql-1" (ID: 1001) has connected

From this snippet of log, we can see each line in stdout or stderr is appended with a timestamp. There are multi-line log entries but each is still appended with a timestamp. This is just how kubernetes keeps the log file for Pod stdout and stderr. To handle that, we need to first take out the real log line, and then process multi-line.

We will go over the installation of EFK stack and the mechanism to address the two challenges above.

Install Elasticsearch and Kibana

To install Elasticsearch, we use the helm chart provided by the official repository:

helm repo add elastic https://helm.elastic.co

If we run multiple pods on the same hosts, then we need some customized values in order to get the installation to work. The values.yaml file looks like this:

---

antiAffinity: "soft"

esJavaOpts: "-Xmx128m -Xms128m"

# Allocate smaller chunks of memory per pod.
resources:
 requests:
 cpu: "100m"
 memory: "512M"
 limits:
 cpu: "1000m"
 memory: "512M"

Then we can “preview” what values are used for installation, with helm’s template command:

helm template elasticsearch elastic/elasticsearch -f values.yaml

The effect of the antiAffinity property allows multiple Elasticsearch Pod to be scheduled on the same node. This is not required in production with multiple nodes. To install Elasticsearch and Kibana, run:

helm install elasticsearch elastic/elasticsearch -f values.yaml
helm install kibana elastic/kibana

The Kibana service is exposed on port 5601 of the cluster. To access the port on cluster, we need to run port-forward command as below:

kubectl port-forward deployment/kibana-kibana 5601

Install Fluentd

There are different ways to configure Fluentd. For example, in the forwarder-aggregator pattern, a forwarder Pod is a Daemonset on each Kubernetes node. The forwarder pushes to the aggregator, at port 24224. This page has some configuration details. This pattern is similar to filebeat -> logstash pattern in ELK stack.

In our case, we use a simplified pattern, with a fluentd daemonset acting as collector and then forward to Elasticsearch. Similarly, in ELK stack we can use filebeat to push to Elasticsearch without Logstash. The only reason is the entire architecture isn’t as complicated as requiring an aggregator.

We need to create configmap as below:

kind: ConfigMap
apiVersion: v1
metadata:
 name: fluentd-cm
 namespace: default
 labels:
 app.kubernetes.io/component: forwarder
 app.kubernetes.io/instance: fluentd
 app.kubernetes.io/managed-by: Helm
 app.kubernetes.io/name: fluentd
 helm.sh/chart: fluentd-1.3.0
 annotations:
 meta.helm.sh/release-name: fluentd
 meta.helm.sh/release-namespace: default
data:
 fluentd.conf: |
 # Ignore fluentd own events
 <match fluent.**>
 @type null
 </match>

 # HTTP input for the liveness and readiness probes
 <source>
 @type http
 port 9880
 </source>

 # Throw the healthcheck to the standard output instead of forwarding it
 <match fluentd.healthcheck>
 @type null
 </match>

 # Get the logs from the containers running in the node
 <source>
 @type tail
 read_from_head true
 tag kubernetes.*
 path /var/log/containers/orthweb-postgresql-ha-postgresql-**.log
 pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
 <parse>
 @type regexp
 expression ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[^Z]*Z)\s(?<stream>[^\s]+)\s(?<character>[^\s])\s(?<message>.*)$
 </parse>
 </source>
 # enrich with kubernetes metadata
 <filter kubernetes.**>
 @type kubernetes_metadata
 @id filter_kube_metadata
 kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
 verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
 ca_file "#{ENV['KUBERNETES_CA_FILE']}"
 skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
 skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
 skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
 skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
 </filter>


 <match kubernetes.var.log.containers.orthweb-postgresql-ha-postgresql-**.log>
 @type elasticsearch
 include_tag_key true
 host "elasticsearch-master.default.svc.cluster.local"
 port "9200"
 index_name "postgresql-logs"
 <buffer>
 @type file
 path /opt/bitnami/fluentd/logs/buffers/orthanc-logs.buffer
 flush_thread_count 2
 flush_interval 5s
 </buffer>
 </match>

Then we can create the resource, with helm chart pointing to the config map:

helm install fluentd bitnami/fluentd --set aggregator.enabled=false --set forwarder.configMap=fluentd-cm

We can validate the index creation on Elasticsearch:

kubectl port-forward service/elasticsearch-master 9200
curl -XGET localhost:9200/_cat/indices

From Kibana, we can forward the port as instructed above, and browse to the UI. Once logged on to Kibana, we need to add index pattern first before being able to view the content of index.

How about Fluent Bit

Fluentd has an even more lightweight brother Fluent Bit, also a CNCF project, designed by the same team, for environments with tighter resource restrictions. The technical differences are outlined on this page outlines the technical differences. In terms of use case, Fluentd is a log aggregator and Fluent Bit is simply a forwarder. In terms of ecosystem, Fluentd has a stronger ecosystem whereas Fluent Bit is more seen in IoT devices. Read this post for more comparison.

Summary

EFK stack (Elasticsearch, Fluentd and Kibana) on Kubernetes is a natural choice for ELK users. Fluentd is a CNCF project created for integration with Kubernetes. It is good alternative to enterprise solution such as Splunk. There are lots of plugins supported and articles on configuration. While developing a solution I had to spend time reading the input plugin documentations.

Log file navigator (lnav)

Wed, 23 Sep 2020 21:03:00 -0400

I’ve used a number of log viewers in command terminal, on MacOS and Linux server. I read system logs, log4j formats, as well as json formats. Unfortunately, I have not found an ideal (free) log viewer, either on UI or in command terminal.

Out of those I tried lnav is one of the better ones. It’s been around for more than 10 years and is configurable for a variety of formats. It is available in EPEL-repository for Linux and home brew for Mac.

For example, if our log (produced by log4j) looks like this:

2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): start ARTIM 5000ms
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): Client closed connecti
on without sending data
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550) enter state: Sta1
2020-10-06 20:59:25,471,INFO,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): close Socket[addr=/10.100.101.10,port=24976,localport=44104]
2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.AssociationReaper - [platform-dicomServer-44104-574917] Stop check for idle Association(552550)

We can introduce custom formatting, for example:

{
 "dapp" : {
 "title" : "dapp log4j format",
 "description" : "dapp log4j format",
 "regex" : {
 "dapp" : {
 "pattern": "^(?<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}),(?<level>\\w+),(?<component>[\\w-.]+) - \\[(?<thread>[^ ]+)\\] (?<body>.*)$"
 }
 },
 "level-field" : "level",
 "level" : {
 "error" : "ERROR",
 "warning" : "WARN",
 "info" : "INFO",
 "debug" : "DEBUG"
 },
 "value" : {
 "level" : { "kind" : "string", "identifier" : true },
 "component" : { "kind" : "string", "identifier" : true },
 "thread" : { "kind" : "string", "identifier" : true },
 "body" : { "kind" : "string" }
 },
 "highlights" : {
 "DIMSE" : {
 "pattern" : "A-(ASSOCIATE-(RQ|AC)|RELEASE-(RQ|RP)|ABORT)|C-(STORE|MOVE|FIND|ECHO)-(RQ|RSP)",
 "color" : "Red"
 }
 },
 "sample" : [
 {
 "line" : "2020-10-06 12:00:28,500,INFO,dicom.dicom-main - [main] Start listening on port 44104"
 }
 ]
 }
}

Save the content above as ~/.lnav/formats/installed/dapp.json, then load the log file with lnav, lnav will display the log by presenting columns in different colours.

One of the default behaviours is highlighting the IPv4 address, as shown above. This behaviour is however, not optional and currently cannot be turned off, which is reported here.