solution architecture on Digi Hunch

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

File storage vs object storage in the cloud

Thu, 23 Sep 2021 22:54:00 -0400

File storage (e.g. NFS) used to be prevalent until object storage comes in for competition.

The competition

Traditionally, enterprise storage product lines are built around three capabilities, as listed in this table below:

Capability	Typical Implementation	Data served
T1 – Block stroage	DAS (e.g. SAS cable) or SAN (Fibre Cable for FCP protocol, or Ethernet for iSCSI protocol)	Mission critical data that are extremely sensitive to latency (e.g. database). Client has block-level access.
T2 – File storage	NAS (connect via CIFS or NFS protocols). Storage arrays are typically a mix of HDD and SSD. Storage servers are usually deployed in the same location over low latency network. DR location is usually in the same region.	Hot data. Multiple client access at file level. The size of each data request varies from small to medium (e.g. text document)
T3 – Object storage	Hardware agnostic, connect via layer-7 protocol (e.g. S3). Storage backend can be either on premise, or in the cloud, over WAN connection.	Warm and code data. Multiple client access at object level. Traditionally for backup but use cases are expanding. The size of each data request varies significantly, from small to very large (e.g. media content).

In the last couple decades, leading players for T2 have been enterprise storage vendors. They each have developed their secret sauces to tackle the challenges. For example, EMC has OneFS, a parallel distributed file system as the foundation of PowerScale (formerly Isilon) product line. NetApp develops ONTAP, featuring proprietary techniques for storage efficiency (deduplication, compaction and compression).

The leading players in T3 are mostly public cloud provider, such as Amazon’s S3. They might work with enterprise storage vendor behind the scene. But the T3 services appear to the end users as provided by the public cloud. Originally, the use case for T3 was archive only for its virtually unlimited capacity. This is not entirely true today. With the drastic improvement in modern network infrastructure, T3 can also brings satisfactory performance to serve hot data.

A competition between T2 and T3 arises. After all, both offer storage service over Ethernet, and both support multiple clients. Today when developers architect the storage layer of their applications, they need to weigh between supporting T2 and T3. Since NFS is the typical protocol for T2 storage (sorry Windows guys) and S3 is typical T3 storage. This competition essentially boils down to NFS versus S3.

For many, the fancy S3 is a no-brainer. While I have suffered from many NFS drawbacks, and there’s even a whole article by Linux folks about why NFS sucks, is it sentenced to death today? Does it beat S3 in some cases? Do so many organizations still stick to NFS just out of inertia?

To answer these questions, I examine four aspects to explore the differences between file storage via NFS protocol, and object storage in S3.

Data request size

Storage client can make request by byte range of a file. Therefore, data request size, instead of file size, is what ultimately matters. I pick a few data request sizes (1K, 4K, 16K, 64K, 246K, 1024K and 4096K) in my experiment, and want to see how much network traffic a write operation produces using NFS and using S3.

To emulate request size, I created files at each size (using dd command), and copy the entire file to each backend. In the mean time, I use tcpdump to write out traffic across the wire into capture files. The size of capture file gives me an idea of how much network traffic went through the network interface, which is closely related to latency.

For NFS, I mounted the target with sync option. This requires NFS client to write out to server synchronously on file copy. I’ve also set the wsize to be 1M. For S3, I simply use the following CLI command to copy file:

aws s3 cp 1kb.img s3://digihunch5ffafe32ab0fd40f

On the network interface, I use tcpdump to filter traffic through specific TCP port (443 for S3, or 2049 for NFS) and record the size of the capture file:

sudo tcpdump -s0 -pi eth0 dst port 443 or src port 443 -w /tmp/4096kb.cap

The key indicator is the payload size (file size) as a percentage of the capture size. I call it payload ratio. The closer it is to 1, the better. I have the following result from my experiment:

Request	Payload	S3 capture size (byte)	NFS capture size (byte)	S3 payload ratio	NFS payload ratio
1K	1024	9352	4332	0.11	0.24
4K	4096	12640	7404	0.32	0.55
16K	16384	25969	20472	0.63	0.80
64K	65536	79183	69746	0.83	0.94
256K	262144	290035	271408	0.90	0.97
1024K	1048576	1085366	1074076	0.97	0.98
4096K	4194304	4381547	4286910	0.96	0.98

This result indicates that NFS has a higher ratio in all groups. However, its advantage diminishes as the data request size grows. What it tells us is that if your applications workload issues most request in small chunks of data, such as 1K, 4K, then NFS will require much less traffic over the network, and thus less latency.

This essentially explains the use case of NFS against S3: workload with small data requests.

Client Support

NFS is natively supported by Linux operating system kernel. NFS client sits below the virtual file system (VFS) layer, which sits below the system call layer. The NFS client translate system calls into RPC (remote procedure calls). Communication between client and server is completed with RPC, on top of TCP.

NFS architecture

Because of the native support, in most cases, developer can treat NFS mounts as if they were local. For performance to be sustainable as file system grows, the directory structure on NFS should follow a certain naming conventions so that files are evenly distributed across directories. The client should also use list operation as sparse as it can because that operation is expensive across the network.

From developer’s perspective, NFS support is brought in by operating system and does not require much effort. On the other hand, S3 client support is not included by default in the operating system. S3 support requires special library, code changes, and integration effort to manage dependency and library version.

NFS has an advantage on client supportability. However, as we move applications to containers, and as container storage options mature, we will need an intermediary layer (storage class, storage provisioner, CSI driver, etc), NFS, or in general file storage, does not have this advantage any more.

Client-side Cache

The NFS support behind VFS layer also means it can leverage the I/O caching mechanism on the client side, that comes with operating system. Client operating system with sufficient memory can take advantage of this mechanism to give it a performance boost. Check out this guide for NFS cache tuning.

In comparison, S3 does not have a cache mechanism by itself. Either the application needs to implement its own cache mechanism, or a cache architecture needs to be introduced, such as CloudFront.

Consistency and concurrency

A common consistency problem is whether client can read the changes immediately after it writes the file. S3 and NFS make a tie in this round.

S3 originally came with eventual consistency model for read after write since 2006. As of Dec 2020 it introduced strong read-after-write consistency. For more information, refer to the guide here.

NFS has a similar consistency guarantee called close-to-open cache coherency. Any changes made by client are flushed to the server on closing the file, and a cache revalidation occurs when you re-open it.

There are more to consider in terms of consistency. For example, multiple clients tries to write the same file/object at the same time.

On the S3 side, there is a locking mechanism called S3 object lock at object level (no byte-range lock). Without an object lock, when two PUT requests are simultaneously made to an object, the request with the latest timestamp wins. Refer to the section Concurrent application on this page.

As far as NFS goes, managing this kind of consistency problem is not in the scope of the standard. Although there are some tinkers. For example, NFS v4 includes a file locking mechanism. Client can choose to lock the entire file, or a byte range within the file. Locking can be mandatory or advisory.

The convergence

NFS and S3 each has their respective advantage. Enterprise NAS customers have been looking for ways to expand into the cloud for lower storage cost. To combine the advantages of the two, solution providers started to converge file storage and object storage. There are two types of solutions that reflects this trend of convergence. In the first trend, enterprise NAS deployed on premise now have the ability to scale out into the cloud. In the second trend, public cloud just brought enterprise NAS into their product offerings.

Scale-out NAS

NAS is traditionally expensive to scale because it requires physical storage media. The idea of scale-out NAS allows NAS to connect to object storage in the public cloud, making it a hybrid architecture. This essentially makes T3 storage as a backend of T2 and it can be implemented with a virtual storage appliance (VSA). The VSA translate file system activities into API calls for object storage operations. One example is AWS storage gateway. EMC has a similar appliance called ECS and this white paper explains how it proxies file system calls and interact with object backends. NetApp, a vested enterprise NAS provider, also has a counterpart called Cloud Volumes ONTAP (CVO). It works well with NetApp on-premise deployment, but the architecture is similar. Here‘s NetApp’s take on how CVO is different than AWS Storage Gateway.

In the scale-out NAS architecture, the public cloud acts merely as extension to on-premise storage solution, to provide capacity. The NAS on premise serves the storage workload primarily.

Cloud hosted NAS

For applications hosted in public cloud, it makes sense for public cloud provider to operate enterprise NAS storage as a service. The underlying storage technology is provided by storage vendor. It is just installed in the data centre managed by the public cloud vendor, instead of customer’s own data centre.

One example is Azure NetApp Files (ANF). ANF is fully managed services, presented to users as storage volumes. The underlying storage technology is NetApp ONTAP. Because it is offered as a fully managed service, the customers are not able to manage the fine details of the storage, as they could with an ONTAP cluster on premise. This takes a lot of flexibility away from the user.

FSx ONTAP is a managed NetApp storage service by AWS, launched in September 2021. The NetApp arrays are installed in AWS data centre, ready for users to provision from AWS console, or using CLI. The Terraform provider support is not available as of yet. Unlike ANF, FSx ONTAP exposes the ONTAP CLI to users, allowing for advanced storage managed by storage gurus. They can use ONTAP CLI commands to configure custom policy for Snapshot, setup SnapMirror replication, and so forth.

Likewise, PowerScale landed on GCP as public cloud partner to launch Dell Cloud PowerScale for Google Cloud in 2020. However, it seems to require a purchase agreement before APIs are enabled.

Conclusion

Object storage has a great momentum and some sees that as a replacement of file storage in the long run. However file storage has its advantages for small data requests, OS-level cache support, and built-in POSIX compatibility. It will continue to be an option for customers with specific workload. Customer stickiness to file storage is so firm, that public cloud providers now install them in their data centres.

From competition to collaboration, it will be interesting to watch what happens next for enterprise storage.

Follow-up Reading

Tom Lyon’s presentation on why NFS must die.

Secure web application deployment

Sun, 16 May 2021 15:19:41 -0400

In Nov 2020, I created OrthWeb project, a deployment of Orthanc’s server. Orthanc is a DICOM viewer and repo shipped in Docker container. In the deployment project, I use Terraform to provision infrastructure, including a managed PostgreSQL instance, an EC2 instance for docker runtime, and the init script to bring up the web service. I whipped up the project for a demo, and skipped some security configurations. For example, the password was stored in clear text in Terraform configuration. The web certificate is stored in the repository. I recently had some time to fix that. My effort leads up to the conclusion that this requires a better platform (i.e. managed Kubernetes cluster). So I wanted to note down how I got there.

Secret store

In AWS, both parameter store and secret manager can act as secret store. Secrets manager comes at higher cost but some additional features, such as built-in password generator, secret rotation, and cross-account access. We use Secret Manager but we generate password within Terraform because we need to specify password during database provisioning. Secret store requires certain special characters to be eliminated. Terraform can specify the special characters allowed. For EC2 instance to pull from secret manager, the following entities are needed:

A secret store
A VPC endpoint to expose secret store to subnet via private route.
The VPC endpoint needs its own security group
The instance profile of the EC2 instance must contain an IAM role to get secret value
The security group of EC2 instance needs to allow traffic to secret store
The script from EC2 instance uses VPC endpoint

This is a common pattern for interaction between computing object and VPC endpoint. The details are in compute.tf, network.tf, secgrp.tf and secret.tf. The secret name needs to be partially randomized to avoid naming conflict with deactivated secrets.

Passing Secret to container

Here is an example CLI command to pull secret:

$ aws secretsmanager get-secret-value --secret-id DatabaseCreds51c1db4172ae9c54 --query SecretString --output text --endpoint-url https://vpce-0897b168cf1c60df2-khx32o7f.secretsmanager.us-east-1.vpce.amazonaws.com | jq -r .password

The connection is made via private network route (whether the instance is in public or private subnet). Traffic is encrypted in TLS. Once in the operating system, the secret is available as standard output and can be stored to file, or saved in environment variable.

My first attempted approach is docker’s secret store and config so that I do not have to store secret in plain text on the file system. I eventually give up this approach due to several hiccups. First, secret and config are part of Docker swarm service. So it requires initializing docker swarm before I could port in the secret, with the following command:

$ docker swarm init
$ echo mdbuser123 | docker secret create db_un -
$ echo m1p@ssw0rd | docker secret create db_pw -
$ echo 10.2.32.41 | docker config create db_ep -

The content of the secret and config are presented as files to the container file system at different locations, as can be verified this way:

$ docker service create --name="redis" --secret=db_un --secret=db_pw --config=db_ep redis:alpine
$ docker container ls
$ docker exec -it c8ed2a278ca8 sh
# cat /db_ep
# cat /run/secrets/db_un
# cat /run/secrets/db_pw

This is a great way to pass secret and config to container applications. However, since the values are stored as content of file, the main application must be able to load file content as its own configuration value. In my specific scenario, the application expects explicit value in its configuration file, or environment variable.

On the other hand, Docker document states that docker secrets do not set environment variables directly. this was a conscous decision, because env var can unintentionally be leaked between containers. In other word I could present secrets as files but the application cannot use it. There is potentially a workaround here which is great function wise but an additional layer of complexity.

Moreover, I later discovered that this isn’t even a viable approach if I use docker compose. This is because I must declare those entries from secret store or config store as external, and external secrets are not even available to containers created by docker-compose.

With reluctance, I store the config and secret keys and values to a file, and use the env_file section in docker compose to import them as environment variables. The application can pick up environment variables as configuration values.

X509 Certificate

We use a self-signed X509 certificate, along with the private key. The creation is straightforward. However, when I tested on Mac, the browser does not load the page for this reason. Since macOS 10.15, the certificate requires several extensions: ExtendedKeyUsage, Subject alternative names and DNS name. The native openssl from the operating system is outdated (v 1.0.2) and I had to install openssl11 package and create it as follows:

openssl11 req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/private.key -out /tmp/certificate.crt -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=digihunch.com/emailAddress=info@3.237.97.93 -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:digihunch.com

The Mac uses libreSSL backed openSSL utility and can achieve the same with slightly different command line argument.

Next Step

The limitation with passing secret concerns me and I’m looking to move to managed Kubernetes platform where secrets can be ported to environment variable of Pods. We can also consider ECS in AWS which allows to inject sensitive data from secret manager to container.

Public Key Infrastructure (PKI) – Introduction

Thu, 08 Apr 2021 22:07:00 -0400

A public-key infrastructure (PKI) is a set of roles, policies, hardware, software and procedures needed to create, manage, distribute, use, store and revoke digital certificates and manage public-key encryption. The algorithms are based on Publick-key cryptography. The format of the digital certificate is defined in X.509 standard.

Certificate Authority – CA digitally signs and publishes the public key to user. Website requesting certificates start with a key pair. It then converts public key into CSR (certificate signing request), including the identity. Once the identity of requestor is validated, CA will sign the public key of requestor, using its own private key. The output of this is the certificate.

Registration Authority – CA may delegate some roles to registration authority (RA). RA is responsible for accepting requests for certificates and authenticating the entity making the request. However, RAs do not have the signing authority of a CA. Note that Microsoft may have referred to a subordinate CA as an RA, which is incorrect according to X.509 PKI standards.

Validation types

Domain Validation: domain ownerships is usually verified via DNS record.
Organization Validation: the organization name and address are verified and put into the certificate.
Extended Validation: verifies existence and location of the legal entity, as well as domain ownership. EV cannot be issued as a wildcard certificate.

Private Certificate Authorities

You can create private CA and use it to sign certificates. Your user need to manually install and trust your private CA so that all certificates issued from the CA will inherit that trust. For revocation, you will also need to maintain an HTTP server for the certificate revocation list, or an OCSP responder.

Certificate Revocation List (CRL)

SSL certificates include information on how to access a certificate revocation list. Client will download and check this list to make sure the certificate has not been revoked. This mechanism has largely been replaced by OCSP responders.

Online Certificate Status Protocol (OCSP)

The OCSP protocol is a replacement for CRLs, with the benefit of being more real-time and requiring less bandwidth. The general operation is similar: clients are to query to OCSP responder to check if a certificate has been revoked.

Commercial vs non-profit CA

	Commercial (e.g. SSLs.com)	Non-profit (e.g. Let’s Encrypt)
Validation	DV, OV and EV	DV only
Wildcard	Supported	Supported (using DNS-01 challenge via ACME v2)
Cost	Not Free	Free
Expiration	1-3 years	90 days

ACME protocol

Traditionally, there are several command-line utilities such as openssl, cfssl, or keytool (Java) to manage certificate related tasks. The process are mostly manual. The Internet Security Research Group (ISRG) developed the ACME (Automated Certificate management Environment) protocol.

In this protocol, there is a certificate management agent (client) on the given web server. The agent generates a key pair and shares it with the CA at the outset of the validation process. Once validation is finished and the agent is verified as the proven owner of the key pair. It can use its key to digitally sign the CSRs it generates and sends to the CA via HTTPS requests. The CA uses the CSR, along with its associated public key, to issue the certificate and send it back to the agent. The agent downloads and installs it, then notifies the designated contact. The agent can be automated to check in with the CA at given intervals to rotate certificates and keys.

Let’s encrypt adopts ACME protocol by using Boulder on the server side, and the most commonly used client is certbot. Smallstep also introduced ACME support in step CA in 2019.

Lets Encrypt

I have used let’s encrypt several times because it is free and easy to manage with certbot, which can be installed using brew on Mac. Here’s how I quickly generate certificate manually:

DOMAIN=orthwebdemo.digihunch.com
echo $DOMAIN
sudo certbot -d $DOMAIN --manual --preferred-challenges dns certonly
# get ready to change txt record
ls /etc/letsencrypt/live/orthwebdemo.digihunch.com/

The --manual switch starts interactive prompts, which includes configuring TXT record and wait for the update.

Open source implementations

Here is a list of open-source implementations of PKI management:

OpenSSL: classic tool for PKI management. The Mac/BSD implementation and GNU implementation are slightly different.
Keytool: Java’s Key and Certificate Management Tool that supports formats used in Java
Cfssl: introduced by CloudFlare to simplify the PKI management process. On Ubuntu, the apt package name is golang-cfssl
Hashicorp Vault: CA, secret management and encryption.
Boulder: implemented in Go based on ACME protocol. Let’s Encrypt uses Boulder on the server side.
EJBCA: a full-featured, enterprise-grade implementation in Java, managed by Swedish company PrimeKey Solutions AB.

Managed CA as service

AWS Certificate Manager: [Update] as of Sep 2022, the managed CA capability was spun off as a new service called AWS Private Certificate Authority, to distinguish from the certificate management capability.
EJBCA Enterprise, as Azure Market place
Google Cloud Certificate Authority Service API

Automatic deployment of Orthanc on AWS

Sun, 08 Nov 2020 00:54:06 -0400

[Update] I changed reverse proxy from Nginx to Envoy. Here‘s the detail.

[Update] Some security improvement was introduced in may 2021. Here‘s detail.

[Update] Here’s the link to the orthweb repository.

In this project we introduce a medical imaging web service based on Orthanc, an open-source project of DICOM server, and a pipeline to deploy such server automatically and consistently. We deploy Orthanc on AWS automatically. This little project involves a number of technical deets in DevOps, to deliver a web application prototype with an automated deployment pipeline.

A brief on imaging

In medical imaging, scanning devices are the data collectors. It consists of various categories of scanners, such as Computed Tomography (CT), and Ultrasound (US). They are collectively referred to as modality, but vary significantly in terms of image generation and hardware manufacturing. The challenges to exchange data between these heterogeneous scanning devices and centralized computers came around as early as the 1980s, which brought about ACR-NEMA standard in 1985, under the initiative between American College Radiology (ACR) and National Electrical Manufacturers Association (NEMA). The standard lately evolved into DICOM (Digital Imaging Communication in Medicine), a comprehensive set of standard in the ISO framework that governs modern imaging data storage and exchange across several disciplines (radiology, cardiology, pathology, etc) that operate around images in medicine.

In addition to defining a file format to store imaging data, DICOM also includes an upper layer protocol that dictates how two compliant devices (referred as application entity, each identified by AE title) can negotiate a common syntax to transfer objects (e.g. an image, a report or a discovery). Upper layer refers to layer 5-7 in OSI model, or application layer in TCP/IP model.

Imaging server

Once scanner acquires images from patient, they stores the exams to imaging server for persistent storage. The functionalities of such server expands overtime since 1990s and hence go by different names in different eras, such as PACS (Picture Archive and Communication Systems), VNA (Vendor Neutral Archive) and EI (enterprise imaging) archive. Regardless of naming, they can be generally seen as a highly specialized variation of enterprise content management system. They are usually hosted with a centralized database to index clinical information at patient, exam and image levels. The other key component is the persistent storage devices, usually in the form of a NAS.

Orthanc is an open-source initiative for such imaging servers. It provides a DICOM endpoint, allowing scanning devices to store medical images. It also provides a web viewer allowing users to see the images stored. It is released for many platforms, including Docker images.

Infrastructure as code

We use Amazon Web Service (AWS) for infrastructure as service, and Terraform as the tool to provision resources off AWS, in a reliable and consistent mechanism, known as Infrastructure-as-Code. Terraform is an alternative to CloudFormation, AWS’s proprietary infrastructure-as-code technology. Terraform is developed by Hashicorp as an open-source project, and therefore is vendor neutral. It supports multiple public cloud vendor through different providers. Each provider accesses the vendor specific SDK. For example, the AWS provider integrates with AWS SDK. As a result, the code used in one vendor cannot just be applied to a different vendor without a major overhaul. Terraform’s current version is 0.13 as of Oct 2020, and has gone through some syntax changes since version 0.11. Terraform also produces files for state management locally in the working directory.

When executing, Terraform combines all files in the working directory to assess variables, and create required resources. It is compatible with the most of AWS resources. For example, you can specify user data with templates when creating EC2 instances. You can also create managed service instance as long as it is supported by the provider.

Architecture

Orthanc web server stores data in sqlite by default, but also has a plugin to support PostgreSQL, an open-source relational database. AWS has managed service (RDS) based on PostgreSQL. In this project, we create an RDS instance that span across two availability zones for minimum high availability. Orthanc also supports storing imaging data including pixels in PostgreSQL, which obviates the need for a dedicated file storage system.

We deploy the application in Docker’s containers for compatibility and portability. The Orthanc server is shipped in Docker images, available in Docker hub registry. The docker environment is configured as part of EC2 instance bootstrapping, including installing packages with YUM, initializing and customizing environment variables. The docker-compose file, and the auxiliary configuration files are provided in the repo. The bootstrapping script installs git and pulls required files from this GitHub repo.

This demo project does not include load balancing, DNS management, or container orchestration.

Security

Orthanc’s web browser natively supports HTTPS. However, the DICOM port does not support TLS natively, as their development has made clear in the FAQ. This leaves a severe security vulnerability because all patient data (protected health information in HIPPA context) would be sent across the Internet in the clear, visible to every network interface along the route. To address this issue we brought in Nginx as a reverse proxy to work at TCP layer to terminate encrypted traffic for Orthanc’s DICOM end point. DICOM upper layer works on top of TCP layer.

In Nginx literature, this use case is referred to as SSL Termination for TCP Upstream Servers. Note that Nginx is providing layer 4 capability in this use case so the certificate and key configuration should not be placed under http section of the configuration file. This layer 4 capability in fact enables security configurations of all protocol that operates in upper layers and can be used in a broad range of situations. It is also noteworthy that Nginx can re-encrypt the traffic on the way out to upstream, for even tighter security control measure as outlined in this use case.

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
 worker_connections 1024;
}

stream {
 upstream dicom_backend {
 server orthanc-backend:4242;
 }

 server {
 listen 11112 ssl;
 proxy_pass dicom_backend;

 ssl_certificate conf.d/site.pem;
 ssl_certificate_key conf.d/site.pem;
 ssl_protocols SSLv3 TLSv1 TLSv1.2 TLSv1.3;
 ssl_ciphers HIGH:!aNULL:!MD5:ECDH+AESGCM;
 ssl_session_cache shared:SSL:20m;
 ssl_session_timeout 4h;
 ssl_handshake_timeout 30s;
 }
}

It is also helpful to use Nginx to terminate HTTPS traffic, using a pair of certificate and key. When testing with self-signed certificate I realized that Chrome browser has specific requirement on self-signed certificate, or it won’t load the page. So the certificate has to be created as instructed here.

For better security, it is advisable that the RDS instance is provisioned in private subnet, with its data encrypted both in-transit and at-rest. Docker service should also manage sensitive information as secrets.

Summary

The deliverable is stored in this Github repo. The docker part of it can be executed on MacBook with PostgreSQL. The entire hardware stack represented by terraform code, can be executed against AWS to create required resources. Checkout README for further instruction. To emulate a modality, one will need a TLS supported DICOM application entity, Horos is a great project on MacOS to serve this purpose, both as DICOM-compliant sender and a viewer. Alternatively, consider some command-line based DICOM toolkit such as dcmtk, or grassroot dicom.

Zookeeper Summary

Wed, 26 Aug 2020 23:10:00 -0400

Distributed systems

Distributed system involves independent computing entities linked together by network. The components communicate and coordinate with each other to achieve a common goal. In early days, designers and developers often had made some assumptions (aka. fallacies) of distributed computing:

The network is reliable
Latency is zero
Bandwidth is infinite
Network is secure
Topology doesn’t change: in reality, components to a network get removed/added over time. the system should tolerate such changes.
There is one administrator: for distributed systems to function, they interact with external system beyond administrative control.
Transport cost is zero: cost is involved everywhere, in the form of CPU cycles spent, to actual dollars paid to service provider.
Network is homogenous

These fallacies make coordinating distributed computing entities a huge challenge and Zookeeper is introduced to address these challenges. Zookeeper implements common tasks for distributed coordination, such as:

Configuration Management (propagate configuration changes to all worker nodes dynamically)
Naming service
Distributed synchronization (locks and barriers)
Cluster membership operations (e.g. detection of node leave/join)

ZooKeeper is a centralized coordination service for the distributed application. ZooKeeper itself is distributed as well. It runs on its own cluster of servers called a ZooKeeper ensemble, separate from application’s cluster. Distributed consensus, group management, presence protocols, and leader election are implemented by the service so that the application developers do not need to reinvent the wheel by implementing them on their own.

Developers will have to use APIs through ZooKeeper’s client library, which has language bindings for almost all popular programming languages. The client library is responsible for the interactions of an application with the ZooKeeper service. For testing with API access one can alternatively use its Java-based command-line shell (zkCli.sh)

$ zkCli.sh -server zknode:2181

How Zookeeper works

Data Model

ZooKeeper allows distributed process to coordinate with each other through a shared hierarchical namespace of data registers (znodes). The hierarchy start with root node which has child znode(s). Each znode can have their children, as well as store its own data (hence the name data register). The data in a znode is stored in byte format for a maximum of 1MB (ZooKeeper by design is just a coordinator service of host application, so its own data set size is fairly small).

Zookeeper data model

Znodes have two types (set at time of creation)

persistent znode: for storing persistent data, such as configuration. The znodes and their data will exist even if the creator client dies.
ephemeral znode: deleted by ZooKeeper service when the creating client’s session ends (due to disconnection or explicit termination). It can also be explicitly deleted by creator client through delete API call. They cannot have children. Their visibility is controlled by ACL policy

ZooKeeper can assign an incremental sequence number as part of znode name during its creation. This makes a sequential node. Both persistent znode and ephemeral znode can be either sequential or not.

In typical client-server architecture, server is passively open and do not initiate communication to client. Client pulls information from server. This is however an anti-pattern for large scale distributed system. ZooKeeper implements a Watch mechanism where clients can get notifications from ZooKeeper service, instead of having to poll for events. Clients can register with the ZooKeeper service (by setting a watch on znode) for any changes associated with a znode. A watch will only trigger notification once, and needs to be re-registered (by client) for trigger the next notification. A watch is triggered upon:

Any changes to the data of a znode;
any changes to the children of a znode;
Creation of deletion of a znode

ZooKeeper guarantees that notifications are delivered in the order of event occurrence. When a client disconnects from ZooKeeper server, it doesn’t receive any watches until the connection is re-established.

API Operations

The ZooKeeper operations are:

Operation	Description
create	Creates a znode in the specified path
delete	Deletes a znodes from the specified path. Not allowed if the znode has children. version number required
exists	Check if a znode at the specified path exists, and get version number; support watch
getChildren	Get a list of children of a znode; support watch
getData	get the data associated with a znode; support watch
setData	writes data into the data field of a znode. Version number required.
getACL	get the ACL of a znode
setACL	set the ACL in a znode
sync	synchronizes a client’s view of a znode

The write operations (setData, create, delete) are atomic, durable and eventually consistent. Every znode has a stat structure including cZxid, mZxid an dpZxid that keeps track of the ID of the transactions that created, last modified this znode, or pertains to adding or removing its children.

Production znode ensemble with more than one node is running in quorum mode. Updates to ZooKeeper tree by clients must be persistently stored in this quorum of nodes for a transaction to be completed successfully. Odd number of node is recommended to avoid split-brain where network partition causes two subsets of servers in the ensemble function independently, and different clients get different results for the same requests, depending upon the server they are connected to.

All ZooKeeper nodes are listed in the configuration for client application to randomly pick from and try to connect and establish a session. The session is associated with every operation the client executes in a ZooKeeper service. The session also has a timeout period specified by the application client during session establishment. If the connection remains idle for more than the timeout period, the server expires the session. Appropriate session timeout should be set based on network condition. Sessions are kept alive by client sending heartbeat to ZooKeeper service. Application developer needs to handle connection-loss scenarios properly.

Leader Election and Atomic Broadcast

ZooKeeper ensemble contains a leader nodes, follower nodes and observer nodes.

The leader node is elected by the cluster. It handles all write requests.
The follower nodes are leader candidates that are not elected. They are backup to the leader nodes. They handle read request, and receive the updates proposed by the leader, and through a majority consensus mechanism, a consistent state is maintained across the ensemble.
The observer nodes are ineligible as leader candidates. They have otherwise the same function as followers.

The service relies on the replication mechanism to ensure that all updates are persistent in all servers that constitute the ensemble. This is the core mechanism in ZooKeeper, implemented as a special atomic messaging protocol called ZooKeeper Atomic Broadcast (ZAB). ZAB (a variant of Paxos algorithm) ensures the election of new leader in the event of old leader crash, and ensures integrity of data. It defines three states (looking, following and leading) of a node, and goes through four phases (election, discovery, sync, broadcast) in its operation.

All read requests (exists, getData, getChildren) are process locally by the ZooKeeper node where the client is connected to. This makes read operation fast. All write requests (create, delete, and setData) are forwarded to the leader in the ensemble, which carries out the client request as a transaction. A transaction is identified by zxid and is idempotent. Transaction also satisfies the property of isolation (no transaction is interfered with by any other transaction). Only after a majority of the followers acknowledge that they have persisted the change does the leader commit the update.

Transaction processing involves two steps in ZooKeeper: leader election and atomic broadcast. This resembles a two-phase commit protocol (which also includes a leader election and an atomic broadcast)

ZooKeeper use local storage to persist transactions. The transactions are logged to transaction logs, in sync’ed write, requiring a dedicated block device separated from boot device of server. The local storage also keep point-in-time copies (snapshots) of the ZooKeeper tree.

ZooKeeper Recipes

The ZooKeeper recipes defines high-level implementation (construct) of some common distributed coordination mechanism:

Barrier: any thread/process must stop at this point and cannot proceed until all other threads/processes reach this barrier.
Queue: allow FIFO in distributed system
Lock: Fully distributed locks that are globally synchronous, meaning at any snapshot in time no two clients think they hold the same lock.
Leader Election: designate a single process as the organizer of some task distributed among several nodes.
Group membership: node may join or leave a group, which needs to be made available to clients. An alternative to ZooKeeper to manage group membership is gossip protocol.
Service discovery: help client to determine IP and port for a service that are hosted by multiple servers.
Two-phase commit: a mechanism for atomic commitment in two steps: first a commit request phase involving a voting by participants; and second, either a commit action, or an abort action, based on the voting result.

Zookeeper Administration

The official documentation includes all we need to know about administration. In addition, we need to configure log4j for proper logging. As best practices, we also should turn off swapping on ZooKeeper. We should clean up the data directory periodically if auto purge is not enabled. For optimal performance, ZooKeeper transaction log should be configured in a dedicated device.

For monitoring, ZooKeeper responds to a small sets of four-letter commands issued through telnet or nc to server’s client port. This allows the admin to check health of server or diagnose any problems. This requires the following property in zoo keeper config:

4lw.commands.whitelist=stat, ruok, conf, isro, wchc

The value can be set to asterick to allow all four-letter keyword. Once enabled, we can check server status

$ echo ruok | nc localhost 2181
imok

More four-letter commands are listed here. Apart from the four-letter commands, ZooKeeper can also be managed through Java Management Extensions (JMX).

Conclusion

Apache ZooKeeper is a coordination service for distributed application. It has become the solution for high availability for many other projects. Some of Apache’s well known open-source distributed services include:

Apache Hadoop (an umbrella of projects including many components for BigData processing such as Hadoop Common, Hadoop Distributed File System (HDFS), Hadoop YARN (yet another resource negotiator) and Hadoop MapReduce)
Apache HBase: non-relational database on top of HDFS
Apache Hive: data warehouse with SQL-like interface
Apache Kafka: stream processing
Apache Nifi: automated data flow processing.

Some of them, such as Nifi, has an embedded implementation of ZooKeeper ensemble if there isn’t a separate ensemble. There is some limitation with embedded Zookeeper ensemble. First, we cannot start ZooKeeper without starting Nifi service on the same server. Second, we need to orchestrate the configuration so that the ZooKeeper ensemble does not grow too large. We need to keep in mind that the ZooKeeper ensemble is a separate cluster of its own, and the it is not recommended to have more than 7 nodes on ZooKeeper.

Cloud storage overview

Wed, 12 Aug 2020 22:19:00 -0400

In a narrow sense, cloud storage refers to object storage. In a broader sense, it refers to any storage service (block, file or object level) provided by cloud vendors, in a cloud business model. The underlying technology of storage, is the same be it in the cloud or on-premise.

	Block storage	File storage	Object
Interaction with OS	OS has direct byte-level access to disk blocks.	OS manages storage by file, or byte range of file. Files are organized in POSIX hierarchy.	OS reads and writes the entire object, or a byte range, via rest API calls.
Metadata	N/A	Stored in file system, for directory or file	customizable metadata
Common protocol	N/A	NFS	S3
Implementation	SAN (bock device is typically dedicated to a single VM) or DAS	NAS, file storage is usually shared amongst multiple VMs. Locking mechanism is usually in place to keep access in order.	S3
Workload	database storage, scratch data, etc	persistent data, content management, etc	archive data, media streaming, data analytics, static asset serving, etc

Below is a list of common storage services provided by public cloud vendors to day.

	Block Storage	File Storage	Object Storage	Other managed storage service
AWS	Elastic Block Store (EBS)	Elastic File System (EFS) FSx for Windows FSx for Lustre	Simple Storage Service (S3)	Storage Gateway Snow Family DataSync
Azure	Azure Managed Disks	Azure Files	Azure Blobs	Azure Table Azure Queues
GCP	Persistent Disk local SSD	Filestore	Cloud Storage	Cloud Storage for Firebase Data Transfer
Digital Ocean	Volumes Block storage local SSD	N/A	Space object storage (S3 compatible)	Content Delivery Network

Storage Products from common public cloud vendor

Since AWS is the first vendor that provides a full suite of storage service, this post will focus on the storage product lines, as a refresher of AWS cloud storage options: Simple Storage Service, Elastic File Storage and Elastic Block Storage). There will be some overlap with the AWS storage service whitepaper.

Before getting further to details, here’s a reminder of two types of policies in AWS:

	IAM policy	Resource-based policy
Principal	Must be attached to individual user, group, or role to take effect	Needs to be explicitly specified, can be ARN under other AWS account
Element	Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition	Principal/NotPrincipal Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition
Example	Managed policy, custom policy	File system policy, S3 bucket policy, access point policy, etc

Two types of policies

Although the resource is usually assumed in a resource-based policy, the policy usually target a sub-section of a resource (e.g. object with certain prefix), so resource section is still required in resource-based policy. In storage services, we may use S3 bucket policy, access point policy, or file system policy for EFS.

Below we go over the three families of storage service in AWS.

EBS (Elastic Block Storage)

EBS is a distributed system. Each volume is a logical volume, made up of multiple physical devices. EBS data is persistent, and access is dedicated to a single EC2 instance at a time. If EC2 instance failed, the attached EBS volume can be detached, and then re-attached to other instance, in the same Availability Zone. There are two types of EBS:

EC2 Instance store: ephemeral, block-level storage for EC2 instance, no replication by default, no snapshot support. Used as buffers, caches, scratch data, temporary content.
EBS volume (persistent) : used for database, dev/test, enterprise application, etc. There are two sub-categories:
- SSD-backed volumes:
  - Optimized for transnational workloads that requires very low latency
  - Dominant performance attribute is IOPS
  - For frequent, read/write with small size and random I/O
  - Typical use case include relational database (PostgresQL, MySQL) and NoSQL (Cassandra, Mongo)
  - gp2 (general purpose) and io1 (provisioned IOPS)
- HDD-backed volumes:
  - Optimized for large streaming workloads demanding throughput
  - Dominant performance attribute is throughput
  - For workloads with lots of sequential I/O
  - Typical use case icnlude big data, analytics (Kafka, Splunk, Hadoop, data warehousing), file/media server
  - st1 (throughput optimized0 and sc1 (cold HDD)

The four types of EBS are compared here:

Note that the volume can be modified (change type, increase size) after creation. However, you cannot decrease size. If you increase the size, the file system must be extended after the increase.

Another way to deliver better performance is to use EBS-optimized instances. These instances have dedicated network bandwidth for its I/O traffic to and from EBS. Without EBS-optimized instance, the traffic between EBS volume and EC2 instance uses shared network link with EC2, which is subject to latency during heavy traffic. This distinction is similiar to the difference between iSCSI SAN and FC SAN. Also, you may increase read-ahead buffer in OS for better EBS performance.

On EBS, users can create snapshot, a point-in-time incremental backup. When snapshot is restored to a volume, data is loaded lazily in the background, so that volume is available immediately. This also means that initial read of data that is not yet loaded will be subject to latency, known as first read penalty. To achieve target performance, user may run an initialization on the volume, by reading all blocks with data upfront.

In a newly created snapshot, only the data blocks modified since the previous snapshot are stored as is. The rest are pointers to unchanged data blocks in the original snapshot. When a previous snapshot is deleted, AWS ensures changes are reconciled into the newer snapshot so there is no loss of data. Creation of snapshots on many volumes can be automated with Data Lifecycle Manager (DLM).

As far as encryption goes, the best practice is to create your own master key. KMS uses envelop encryption, where the data key encrypts the data, and the master key encrypts the data key. The encryption key is stored in EC2 instance memory only and never written to disk, for security and performance considerations.

EFS (Elastic File Storage)

EFS is a managed implementation of file storage that supports NFS 4.0 and 4.1, with strong data consistency and file locking. An EFS includes a single mount target in (one subnet of) each availability zone. EC2 instance, or on-premise client via Direct Connect, can mount EFS volumes using amazon-efs-utils yum package. EC2 instance can also be configured to automatic mount EFS volume in launch wizard. EFS also has a lifecycle management policy, and a storage class for infrequent access.

Performance wise, EFS has two performance modes and two throughput modes. The two performance modes are:

General Purpose: for latency-sensitive applications and general-purpose workloads. limit of 7k ops/sec, best choice for most workloads
Max I/O: for large-scale and data-heavy applications, with virtually unlimited ability to scale out throughput/IOPS, but with slightly higher latencies. consider this for large scale-out workloads

The two throughput modes are:

Bursting throughput: recommended for the majority of workload. Since file system workload is typically spiky, aws use credit system to determine when the file system throughput can burst. credit accumates idle time, and consumed in retrieval
Provisioned throughput: recommended for higher throughput to storage ratio workload, can increase the provisioned throughput afterwards. but it incurs separate throughput charge

Other ways to achieve higher performance, include parallelization of file operation (e.g. multiple threads, more instances); and increase I/O size for better throughput.

In terms of security, EFS encryption at rest must be selected at the time of file system creation. There is an TLS mount option to encrypt traffic in transit. EFS involves its own resource-based policy called file system policy to manage file-level POSIX permissions. IAM policy is used to manage NFS administration access and client access. EFS access points is also a means to enforce the use of a specific operating system user, and group to access EFS.

S3 (Simple Storage Service)

S3 is one of the earliest and maturest AWS services for object storage. It is very cheap and easy to use, and supports user-defined metadata on objects as well as many peripheral features. There is no limit to the number of objects in a bucket. As the object in bucket increases, S3 scales to request rate by automatically creating more partitions to meet the target number of request per partition. There used to be a performance trick, that requires client to make object key naming pattern distribute across multiple prefixes. It is not required any more as of July 2018.

Versioning can be enabled at bucket level, and suspended afterwards. New version of object is created on every upload, without performance penalty.

S3 integrate well with other event-driven AWS services, such as SNS, SQS, Lambda, etc. Event can fire on request such as PUT, POST, COPY.

Object tags (not to be confused with object metadata) can help categorize storage. It also facilitates access control (i.e. by being referenced in bucket policy or IAM policy), lifecycle policy, analysis and CloudWatch configurations.

S3 select is a way to retrieve only a subset of data from an object based on a SQL expression, to reduce amount of data and help with performance. The input can be json or CSV and output will be in CSV.

S3 Inventory is a tool to audit object replication status and encryption status. It generates CSV report with all objects in the given bucket name, including: key name, version id, islatest, size, last modified date, etag, storage class, multipart upload flag, delete marker, replication status, encryption status. For storage-class analysis, S3 inventory is much faster than list-object API call which parses through all objects.

S3 also has access point, similar to EFS, with unique hostnames that customers create to enforce distinct permissions and network controls for any request made through the access point.

S3 transfer acceleration take advantage of edge locations (at additional charge) to speed up transfer of large object over long distance, by providing a separate end point. It is also helpful for faster uploads over long distances. Apart from transfer acceleration, for faster uploads for large object, user may also consider multi-part upload API when the object reaches 100MB. Orphaned uploaded parts can be cleaned up in lifecycle configuration. For better download performance, take advantage of CloudFront and byte range request.

Kafka high-level Overview

Tue, 21 Jul 2020 23:19:00 -0400

Zookeeper

General definition of distributed system: a software system that is composed of independent computing entities linked together by a computer network whose components communicate and coordinate with each other to achieve a common computational goal. Implementing coordination among components of a distributed system is hard. For example, designated master node becomes single point of failure; cluster needs to detect availability of new nodes as it joins cluster.

Zookeeper is designed to simplify cluster coordination. Zookeeper implements key aspects in cluster coordination, such as distributed consensus, group management, presence protocols and leader election. In order to coordinate a cluster, zookeeper itself also runs in its own cluster, called ensemble. Zookeeper exposes a simple but powerful interface of primitives. Applications can be designed on these primitives implemented through ZooKeeper APIs to solve the problems of distributed synchronization, cluster configuration management, group membership, etc.

Zookeeper Ensemble

Clients can connect to a Zookeeper service by connecting to any member of the ensemble. The members of the ensemble are aware of each other’s state. As long as a majority of the nodes are available, the service will be available. Zookeeper cli (zkCli.sh) can be used to connect to Zookeeper server. they can be downloaded from here.

Zookeeper is integrated with many other services apart from Kafka, such as Nifi and Hadoop.

Kafka

Kafka is a messaging system that is horizontally scalable, fault tolerant. It can also serve as queue storage system and stream processing system. It is distributed and use Zookeeper for cluster coordination. Each node is called a broker.

Topics in Kafka (think of table in database) is a category or feed name to which messages (records) are published. Topic is broken up into ordered commit logs called partitions. Each partition has an ID. Each message in a partition is assigned an offset. Topics that are created in Kafka are distributed across brokers based on the partition, replication, and other factors. Each partition is replicated across several brokers depending on replication factor. For each partition, Kafka elect one replica as the leader of partition.

Writes to a partition is generally sequential. Reading messages can either be from the beginning, or rewind or skip to any port in partition given an offset value. Data in a topic is retained for a configurable period of time.

A message is a unit of data in Kafka, in the format of key-value pair. A key is used to control the message that is to be written to partitions. Messages with the same keys are always written to the same partition (hash map)

A producer publishes new message to a topic. Producers do not care which partition the message is written to and will balance messages over every partition of a topic evenly. Directing messages to a partition is done using the message key and a partitioner, this will generate a hash of the key and map it to a partition.

A consumer is subscribed to one or more topics and read messages sequentially. The consumer keeps track of messages it has consumed by keeping track on the offset of the message. The offset is a bit of metadata (an integer value that continually increases) that kafka adds to each message. Each partition has a unique offset which is stored with the offset of the last consumed message. A consumer can stop and start without losing its current state.

A Kafka broker is designed to operate as part of a cluster. One broker in the cluster also function as the cluster’s controller, which is responsible for administrative operations such as: assigning partitions to brokers; monitoring for broker failures in cluster. A particular partition is owned by a broker and that broker is called the leader of the partition.

All consumers and producers operating on that partition must connect to the leader.

Kafka cluster may replicate across cluster using MirrorMaker.

Reference: Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale

EMC Isilon storage product

Wed, 08 Jul 2020 20:04:00 -0400

EMC has several product lines for different use cases in enterprise data storage. Like may other IT solutions, the website is clouded with marketing terms and slogans, and is purposefully not technical. This makes it difficult for technical staff to grasp the advantage of its product in a glimpse. I personally have to know their product (mostly with Isilon and ECS) well in order to make integration decisions. So I’m putting together this note (updated as of July 2020), with lots of details from their technical white paper.

Overview of EMC storage

At the highest level, the EMC enterprise data storage product lines are categorized into two groups: primary storage (along the lines of block-level storage) and unstructured storage (mostly file and object storage). The primary storage includes the following product:

PowerMax for OLTP database (Oracle, MicrosoftSQL and SAP)
PowerFlex: for Software defined storage, Oracle RAC, Elastic Stack, Kubernetes, Splunk
XtremIO for VMware, VDI, SAP
PowerStore for Database, VMware
PowerVault for Entry-level SAN and DAS environment

This post only expands on the unstructured storage product line, which mainly consists of PowerScale and ECS. ECS (elastic cloud storage) is EMC’s object storage. PowerScale (aka Isilon) is scale-out NAS platform for high-volume storage (up to 50 PB in a single file system), backup and archiving of unstructured data. For the rest of this post, I will still refer to PowerScale as Isilon.

Dell’s official support website is the most resourceful place to get information. For example, when I want to read about Isilon. I start with Dell support, then click on knowledgebase at the top, then go to “servers, storage and networking“, then “storage technical documents and videos“. There I can select a productline such as Isilon.

Overview of Isilon Family

Isilon is a clustered storage system consisting of three or more nodes. A node is a server with OneFS as its operating system. Based on FreeBSD, OneFS is EMC’s proprietary operating system to unify a cluster of nodes into a single shared resource. So OneFS is for Isilon only. It is the basis of Isilon. Isilon has three series of products:

F series: F200, F600, F800 and F810.
H series: typical models are H400, H500 and H600, which seeks to balance performance and capacity
A series: typical models are A200 and A2000 for active and deep archive storage

In June 2020, Dell decoupled OneFS software (with 9.0 released) from server hardware (referred to as PowerScale). Going forward EMC will refer to Isilon as PowerScale for OneFS version newer than 9.0 in spec sheets and white papers.

F200 is the cost-effective choice with SSD for remote office, small hospital, retail outlets, IOT or factory floor. F600 uses NVMe drives instead, and has more ECC memory and faster ethernet backend network. and is higher than F200 in its use case. Both F200 and F600 provide inline data compression and deduplication capabilities. F800 and F810 both use SSD and they are similar. F800 comes with InfiniBand backend network and F810 provides inline data compression and deduplication capabilities. H series tries to strike a balance between performance and capacity so they are pretty much everything in betwee. On the other end, A200 and A2000 are almost the same except for capacity difference.

Isilon’s advantage

Isilon has lots of intelligence built into its solution compared to a traditional NAS. Here are some aspects from its product white paper:

Aspects of Design	Isilon OneFS Scale-Out NAS	Traditional NAS
Network	Separation of front-end and back-end network to isolate node-to-node communication to a private low-latency network. Front-end traffic load balanced with SmartConnect	Single network for both external and internal traffic
File system structure and NameSpace	The storage is completely virtualized to users as a truly single file system with one namespace. There is no partitioning or volumes. The single file tree can grow organically without requiring planning or oversight about how the tree grows. SmartPool handles tiering of files to appropriate disk, without disrupting the single file tree.	An appearance of single namespace is typically achieved through namespace aggregation, where files are still managed in separate volumes, and a simple “veneer” layer glues individual directories to a “top-level” tree via symbolic links. LUNs and volumes, as well as volume limits are still present. Files have to be manually moved from volume-to-volume to load-balance.
Data Layout	OneFS controls the placement of file directly, down to the sector-level on any drive anywhere in the cluster. The addressing scheme for data and metadata is indexed at physical level by a tuple of {node, drive, offset}	Data are sent through RAID and volume management layers, introducing inefficiencies in data layout and providing non-optimized block access.
Redundancy Control	OneFS can flexibly control the type of striping as well as the redundancy level of the storage system at the system, directory and even file-levels.	The entire RAID volume is dedicated to a particular performance type and protection setting.

Isilon terms

The Isilon technology re-implemented the read and write path during file storage and introduced several terms along with its technology.

SmartPools – Job that runs and moves data between the tiers of nodes within the same cluster. Also executes the CloudPools functionality if licensed and configured. FilePolicy is changelist-based SmartPools file pool policy job. SmartPoolsTree enforces SmartPools file policies on a subtree.

Storage Pools – Storage pools provide the ability to define subsets of hardware within a single cluster, allowing file layout to be aligned with specific sets of nodes through the configuration of storage pool policies. The notion of Storage pools is an abstraction that encompasses disk pools, node pools, and tiers.

Disk Pools – Disk pools are the smallest unit within the storage pools hierarchy. OneFS provisioning works on the premise of dividing similar nodes’ drives into sets, or disk pools, with each pool representing a separate failure domain. Disk pools are laid out across all five sleds in each node.

Node Pools – groups of disk pools, spread across similar storage nodes (or equivalent classes). Multiple groups of different node types can work together in a single, heterogeneous cluster. For example, one node pool of all-flash F-Series anodes, one node pool of H-series, and one node pool of A-series. Each node pool only contains disk pools from the same type of storage nodes.

Tiers – groups of nodepools combined into a logical superset to optimize data storage, according to OneFS platform type. this allows customers who consistently purchase highest capacity nodes available to consolidate a variety of node styles within a single tier, and manage them as one logical group. SmartPools users typically deploy 2 to 4 tiers. different node pools under a tier needs to be compatible.

Global Namespace Acceleration (GNA)’s principal goal is to help accelerate metadata read operations by keeping a copy of a cluster’s metadata on high performance, low latency SSD media.

SmartConnect is a load balancer that works at the front-end Ethernet layer to evenly distribute client connections across the cluster. SmartConnect supports dynamic NFS failover and failback to ensure that when a node failure occurs, or preventative maintenance is performed, all in-flight reads and writes are handed off to another node in the cluster to finish its operation without any user or application interruption.

Auto Balance reallocates and rebalances data and make storage space more usable and efficient.

SmartQuotas is directory-level quota management. Note: there is no partitioning, and no need for volume creation in OneFS.

SmartRead creates a data “pipeline” from L2 cache, prefetching into a local “L1” cache, on the captain node, in order to greatly improve sequential-read performance. For high-sequential cases, SmartRead can very aggressively prefetch ahead. SmartRead can control how aggresive the pre-fetching is, and how long data stays in the cache, and optimizes where data is cached.

In-line Data Reduction – the write path involves zero block removal, in-line deduplication, and in-line compression. This is supported in some models only.

Smart Dedupe – post-process, asynchronous deduplication. Smart Dedupe scans the on-disk data for identical blcoks and then eliminate the duplicates. After duplicate blocks are discovered, SmartDedupe movees a single copy of those blocks to a special set of files known as shadow stored. With post-process deduplication, new data is first stored on the storage device and then a subsequent process analyzes the data looking for commonality. This means that initial file write or modify performance is not impacted, since no additional computation is required in the write path, as opposed to in-line deduplication. This is supported on some models only.

OneFS SSD strategy – How OneFS leverage the SSD for performance. It has these options:

L3 cache (implemented at nodepool level)
metadata read
metadata read/write
Global Namespace Acceleration (GNA)
Data on SSD

L3 cache consumes all the SSD in node pool. L3 cannot coexist with other SSD strategies, with the exception of GNA just because L3 cache node pool SSD cannot participate in GNA.

Isilon’s High Availability

The OneFS is distributed across all nodes in the cluster and is accessible by clients connecting to any node in the cluster. Metadata and locking tasks are managed by all nodes collectively and equally in a peer-to-peer architecture. This symmetry is key to the simplicity and resiliency of the architecture. There is no single metadata server, lock manager or gateway node.

The entire cluster forms a single file system with a single namespace that runs across every node equally. No one node controls or “masters” the cluster; all nodes are true peers.

During failover, clients are evenly redistributed across all remaining nodes in the cluster, ensuring minimal performance impact. If a node is brought down for any reason, including a failure, the virtual IP addresses on that node is seamlessly migrated to another node in the cluster. When the offline node is brought back online, SmartConnect automatically rebalances the NFS and SMB3 clients across the entire cluster to ensure maximum storage and performance utilization. This functionality allows for per-node rolling upgrades affording full-availability throughout the duration of the maintenance window.

There are two logical roles in processing an I/O request from client:

The initiator: the node that the client connects to with front-end protocol. The initiator acts as the ‘captain’ for the entire I/O operation.
The participant: Every node in the cluster is a participant for a particular I/O operation.

File Write in Isilon

OneFS employs a patented transaction system during write to eliminate single point of failure. In a write operation, the initiator “captains” or orchestrates the layout of data and metadata, the creation of erasure codes, and the normal operations of lock management and permission control.

When a client connects to a node to write a file, it is connecting to the Initiator. OneFS breaks the file down into atomic units. An atomic unit is a smaller logical chunk of data, also called stripe, or protection groups in the context of data protection. The size of each file chunk is referred to as the stripe unit size. After this division, OneFS then write the stripe individually to the Participant (with disks). This design ensures that data is protected at the specified level as soon as it is being written. Redundancy is built into protection groups, such that if every protection group of a file is safe, then the entire file is safe. In terms of protection mechanism, OneFS can use either Reed-Solomon erasure coding system, or simply mirroring for data protection. Erasure coding is the predominant mechanism with very high performance without sacrificing on-disk efficiency.

The initiator node uses a modified two-phase commit transaction to safely distribute writes to multiple NVRAMs across the cluster. As client initiates write to OneFS cluster, instead of immediately writing to disk, OneFS temporarily writes the data to an NVRAM-based journal cache on the initiator node and acknowledge the write the client. As outlined above, these writes are also mirrored to participant nodes’ NVRANM journals to satisfy the file’s protection requirement. Later, at a more convenient time, OneFS then flush these cached writes to disks asynchronously.

Since NVRADM journals all the transactions that are occurring across every node in the storage cluster. If a node fails mid-transaction, and then re-joins the cluster, the uncommitted cached writes are fully protected, and the only required actions for the node, are to replay its journal from NVRAM, and occasionally for AutoBalance to rebalance files that were involved in the transaction. Writes are never blocked due to a failure. There is no ‘fsck’ or ‘disk-check’ process.

OneFS file system block size is 8KB. A file smaller than 8KB will use a full 8KB block. For larger files, OneFS can maximize sequential performance by taking advantage of a stripe unit consisting of 16 contiguous blocks, for a total of 128KB per stripe unit.

Cache in Isilon

OneFS aggregates the cache present on each node in a cluster into one globally accessible pool of memory by using a messaging system similar to NUMA (non-uniform memory access). This allows all the nodes’ memory cache to be available to each and every node in the cluster. Remote memory is access over internal network with much lower latency than accessing hard disk drives. The internal network as distributed system bus, is a redundant, under-subscribed flat Ethernet up to 40Gb. The oneFS caching subsystem is coherent across the cluster, due to the use of MESI protocol to maintain cache coherency. If the same content exists in the private caches of multiple nodes, this cached data is consistent across all instances.
OneFS uses up to three levels of read cache, plus an NVRAM-backed write cache, or coalescer.

OneFS Caching Hierarchy

L1 cache – prefetches data from remote nodes. Data is prefetched per file, and this is optimized in order to reduce the latency associated with the nodes’ back-end network. The L1 cache refers to memory on the same node as the initiator. It is only accessible to the local node, and typically the cache is not the master copy of the data.

L1 is also known as remote cache because it contains data retrieved from other nodes in the cluster. It is coherent across the cluster but is used only by the node on which it resides and is not accessible by other nodes. Data in L1 cache on storage nodes is aggressively discarded after it is used. L1 cache uses file-based addressing, in which data is accessed via an offset into a file object.

OneFS also uses a dedicated inode cache in which recently requested inodes are kept. The inode cache frequently has a large impact on performance, because clients often cache data, and many network I/O activities are primarily requests for file attributes and metadata, which can be quickly returned from the cached inode.

L2 cache (backend cache) refers to local memory on the node on which a particular block of data is stored. L2 cache is globally accessible from any node in the cluster and is used to reduce the latency of a read operation by not requiring a seek directly from the disk drives.

L2 cache is also known as local cache because it contains data retrieved from disk drives located on that node and then made available for requests from remote nodes. Data in L2 cache is evicted according to a Least Recently Used (LRU) algorithm.

L3 cache, or Smart Flash, is configurable on nodes that contain solid state drives. Smart Flash (L3) is an eviction cache that is populated by L2 cache blocks as they are aged out from memory.

During I/O request, clients talk to L1 cache and write coalescer; L1 cache talks to L2 cache on all cluster nodes. L2 cache buffers to and from disks. L3 cache is optionally enabled per node pool, as an extension from L2. L3 and L2 communicate in backend network.

Name	Medium	Description
L1 Cache (aka front-end cache or remote cache)	RAM (volatile)	holds clean, cluster coherent copies of file system data and metadata blocks requested by clients via front-end network
L2 Cache (aka back-end cache or local cache)	RAM (volatile)	contains clean copies of file system data and metadata on a local node
SmartCache (Write Coalescer)	Battery-backed NVRAM (Persistent)	a persistent journal cache that buffers any pending writes to front-end files that have not been committed to disk
SmartFlash or L3 Cache	SSD (persistent)	contains file data and metadata blocks evicted from L2 cache, effectively increasing L2 cache capacity

File Read in Isilon

The high-level steps for fulfilling a read request with cache interaction involves:

Step 1 – on local node, determine whether part of the requested data is in the local L1 cache:

if so, return to client
if not, request data from remote nodes

Step 2 – on remote nodes, determine whether requested data is in the local L2 or L3 cache:

if so, return to the requesting node
if not, read from disk and return to requesting node

During a read operation, the “captain” node gathers all of the data from the various nodes in the cluster and presents it in a cohesive way to the requestor.

The cluster provides a high ratio of cache to disk (multiple GB per node) that is dynamically allocated for read and write operations as needed. This RAM-based cache is unified and coherent across all nodes in the cluster, allowing a client read request on one node to benefit from I/O already transacted on another node. As the cluster grows larger, the cache benefit increases. For this reason, the amount of I/O to disk on a cluster is generally substantially lower than it is on traditional platforms.

For files marked with an access pattern of concurrent or streaming, OneFS can take advantage of pre-fetching of data based on heuristics used by the SmartRead component

Conclusion

This post provided a high level introduction to EMC storage product line and expanded into some technical details in the read write operation in OneFS/Isilon. Some of the features can be seen in OneFS simulator which is a free tool from EMC.

Performance Analysis

Fri, 19 Jun 2020 16:47:01 -0400

Overview

In 2015, Brendan Gregg posted two great articles on Netflix blog: Linux Performance Analysis in 60 seconds, and Linux Perfomrance Tools. They have great value when I was in a urgency to spot performance issues. The articles cover the essential tools for performance troubleshooting, including:

Check out load averages: w or uptime
Print kernel ring buffer: dmesg -T
Virtual memory status: vmstat 1
Multiple processor staticstics: mpstat -P ALL 1
Task status: pidstat 1
CPU and I/O status: iostat -xz 1
Free memory check: free -m
Network Activity record: sar -n DEV 1
TCP activity record: sar -n TCP,ETCP 1
Display processes: top

We will dive into each of them in the next section.

Basic Troubleshooting

The command w is equivalent of uptime (which shows uptime since boot) and who (which shows logged-in users). It also displays load average for the last 1 minute, 5 minutes and 15 minutes. The number of load average reflects the overall system load (CPU + disks), and it is further discussed in this post with a simple take away:

If the averages are 0.0, then your system is idle.
If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
If they are higher than your CPU count, then you might have a performance problem (it depends).

When Linux load averages increase, you know you have higher demand for resources (CPUs, disks, and some locks), but you aren’t sure which. You will need to switch to other metrics. Brendan recommend don’t spend more than 5 seconds on these numbers.

[dhunch@c7v-ghintapp01 ~]$ w
 12:14:10 up 46 days, 16:41, 3 users, load average: 2.69, 2.44, 2.29
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dhunch pts/0 w6v-ghas01 24Jun20 3days 0.36s 0.30s ssh c7v-bastion
dhunch pts/1 202.95.88.111 12:02 2.00s 0.00s 0.00s w
dhunch pts/4 w6v-ghas01 17Jun20 15days 0.15s 0.07s view readme.txt

Before moving to more insightful metrics, it is also worth a quick look into the kernel ring buffer with dmesg command (dmesg -T | less +G). This will allow us to capture obvious issues such as oom-killer or TCP request dropping.

The vmstat tool reports the statistics of virtual memory. Servers have a fixed amount of physical memory, but they can run a set of applications that use a much larger amount of virtual memory. Application tend to reserve more memory than they need, and they usually operate on only a subset of their memory. In both cases, the operating system can keep the unused parts of memory on disk, and page it into physical memory only if it is needed. For the most part, this kind of memory management works well. But it doesn’t always with Java applications due to Java heap. Once a system start swapping – moving pages of data from main memory to disk, and vice versa, the performance tend to be bad. Systems must be configured so that swapping never occurs.

[dhunch@c7v-ghintapp01 ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 3 0 239360 385928 0 36734692 0 0 1 200 0 0 6 1 93 0 0
 3 0 239360 387732 0 36734704 0 0 0 43 8017 8524 14 0 85 0 0
 3 0 239360 387608 0 36734904 0 0 0 57 6768 7680 14 1 86 0 0
 2 0 239360 389008 0 36734904 0 0 0 44 6366 7300 14 0 86 0 0
 3 0 239360 421728 0 36700144 0 0 0 0 8141 7957 13 0 86 0 0
 2 0 239360 421984 0 36702048 0 0 0 467 8994 8362 14 0 85 0 0

The tool prints key server statistics each line, with the first line showing the average since boot. Here lists the explanation of some columns:

r: number of processes running on CPU and waiting for a turn. This provides a better signal than load averages for determining CPU saturation, as it does not include I/O. To interpret: an “r” value greater than the CPU count is saturation.
swpd: the amount of virtual memory used. This number should align with the used column for Swap row from free command.
buff, cache: buffer and cache. They should align with the buff/cache column form Mem row from free command.
free: free memory in kilobytes. This number should align with the free column for Mem row from free command.
si, so: swap-ins and swap-outs. As mentioned, if these are non-zero, you’re out of memory.
bi, bo: blocks received from and sent to a blcok device (# of block per second)
in, cs: number of interrupt, and context switches per second.
us, sy, id, wa, st: user, system, idle, wait I/O and stolen times. These are breakdowns of CPU time, on average across all CPUs. They should add up to 100% (or close). stolen time is amount of CPU time needed by a guest virtual machine that is not provided by the host. IO wait time is the CPU time waiting for I/O activity. Idle time could be several things: the process may be waiting for something (e.g. a response from database); the process may be blocked by a thread lock; or the process simply has nothing to do. user and system times are CPU times spent on user tasks and kernel tasks, respectively.

Out of these columns, watch for columns r, free, buff, cache, us, sy, id and wa at minimum. The combination of us and sy confirms if CPUs are busy. A constant degree of wa points to a disk bottleneck with too much time spent on pending disk I/O. The sy (kernel time) is necessary for I/O processing but sy stays high (e.g. constantly over 20%), it becomes interesting. Perhaps the kernel is processing I/O inefficiently.

For further per-CPU stats, use mpstat command (-P ALL), to prind CPU time breakdowns per CPU and check for imbalance. A single host CPU can be evidence of a single-threaded application. Here is an example output from a system of 16 CPU cores.

[dhunch@c7v-ghintapp01 ~]$ mpstat -P ALL 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

13:47:23 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
13:47:24 all 13.77 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 86.05
13:47:24 0 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 1 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 3 98.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
13:47:24 4 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 5 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 6 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 7 2.94 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 96.08
13:47:24 8 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 9 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 10 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
13:47:24 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 12 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 13 1.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 14 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

For a per process summary of CPU consumption, use pidstat command. It can be thought of a periodical snapshot of top command, allowing you to watch for patterns. The %CPU column is the total across all CPUs so 5 CPUs have a maximum value of 500.

If vmstate indicates some I/O issue, iostat tool can help us understand block devices, on both the workload applied and the resulting performance. Key columns are:

r/s, w/s, rkB/s, wkB/s: delivered reads, writes, read Kbytes, and write Kbytes per second to the device. Use these for workload characterization. A performance problem may simply be due to an excessive load applied.
await: the average wait time for I/O in milliseconds. This is the time that the application suffers, as it includes both time queued and time being serviced. Larger than expected average times can be an indicator of device saturation, or malfunction.
avgqu-sz: the average number of requests issued to device. Values greater than 1 can be evidence of saturation (although devices can typically operate on requests in parallel, especially virtual devices which front multiple back-end disks.)
%util: device utilization. This is really a busy percent, showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance (which should be seen in await), although it depends on the device. Values close to 100% usually indicate saturation.

I/O problem may either be inefficiencies in application that issues I/O request, or slowing disk unable to keep up with I/O requests. We review two examples here to illustrate each situation. The first output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 23.45 0.00 37.89 0.10 0.00 38.56

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 11.60 0.60 24.20 0.02 0.14 13.35 0.15 6.06 5.33 6.08 0.42 1.04

In the first example, the disk stat loosk up at first glance. The w_await (time to service I/O write) is fairly low at 6.08ms. However, the system is spending 37.89% of its time in the kernel. If all that system time is from the application, it suggest something inefficient is happening. The fact that the system is doing 24.2 writes per second is another clue: that is alot when writing only 0.14 MB per second (MBps). I/O has become a bottleneck, and the next step would be to look into how the application is performing its writes.

The second example output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 35.05 0.00 7.85 47.89 0.00 9.20

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 0.20 1.00 163.40 0.00 81.09 1010.19 142.74 866.47 97.60 871.17 6.08 100.00

In this example, it tells us that processes are spending 47.89% of their time in iowait, and the data to complete the I/O (w_await) is 871ms, the queue size is large, and the disk is writing at 81MB per second. This all points to disk I/O as a problem and that the amount of I/O in the application (or elsewhere in the system) must be reduced.

Bear in mind that poor performing disk I/O isn’t necessarily an application issue. Many techniques are typically used to perform I/O asynchronously, so that the application doesn’t block and suffer the latency directly (e.g. read-ahead for reads, and buffering for writes, also refer to “tail latency“).

Note that the acceptable utilization metric depends on the configuration of block device. If the storage is a logical disk device fronting many back-end disks (e.g. RAID 0), then 100% utilization may just mean that some I/O is being processed 100% of the time, however, the back-end disks may be far from being saturated, and may even be able to handle more work.

The free command gives the breakdown of memory usage. The right two columns are:

buffers: for the buffer cache, used for block device I/O.
cached: for the page cache, used by file systems.

We just want to check that these aren’t near-zero in size, which can lead to higher disk I/O (confirm using iostat), and worse performance. Linux uses free memory for the caches, but can reclaim it quickly if applications need it. So in a way the cached memory should be included in the free memory column. In this case, it’s included in the available column. This website has further details.

To check interface stat, nicstat is a great tool but it isn’t available by default in Linux. Instead we can run sar (-n DEV) to retrieve stats.

[dhunch@c7v-ghintapp01 ~]$ sar -n DEV 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:51:25 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:26 eth0 3089.00 934.00 3815.33 834.61 0.00 0.00 0.00
16:51:26 lo 464.00 464.00 2289.07 2289.07 0.00 0.00 0.00

16:51:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:27 eth0 956.00 586.00 826.66 211.34 0.00 0.00 0.00
16:51:27 lo 213.00 213.00 196.00 196.00 0.00 0.00 0.00

16:51:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:28 eth0 349.00 181.00 52.32 147.19 0.00 0.00 0.00
16:51:28 lo 244.00 244.00 81.13 81.13 0.00 0.00 0.00

Here rxkB/s/s and txkB/s represents receive and transmission throughput, as a measure of workload. If they reach the limit then the interface is the bottleneck.

On top of interface is the TCP layer, which can be monitored with sar again (-n ECP, ETCP). The key metrics are:

active/s: number of locally-initiated (e.g. via connect()) TCP connections per second
passive/s: number of remotely-initiated (e.g. via accept()) TCP connections per second
retrans/s: number of TCP retransmits per second

The active and passive counts are often useful as a rough measure of server load. It might help to think of active as outbound, and passive as inbound, but this isn’t strictly true (e.g. consider a localhost to localhost connection). Retransmits are a sign of network or server issue; it may be an unreliable network (e.g. public Internet), or it may be due to a server being overloaded and dropping packets.

[dhunch@c7v-ghintapp01 ~]$ sar -n TCP,ETCP 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:52:29 active/s passive/s iseg/s oseg/s
16:52:30 0.00 1.00 28.00 35.00

16:52:29 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:30 0.00 0.00 0.00 0.00 0.00

16:52:30 active/s passive/s iseg/s oseg/s
16:52:31 8.00 8.00 200.00 317.00

16:52:30 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:31 0.00 1.00 1.00 0.00 3.00

Last but not least is our favourite command top, which includes many of the metrics covered in previous tools. The downside to top is it is harder to see patterns over time, which may be more clear in tools like vmstat and pidstat, both of which produce rolling output.

Several tools introduced here involves sar, which is a great monitoring tool on its own that we need to be familiar with.

System Activity Report (SAR)

Further to the basic tools, the sar command is very helpful as it stores historical stat every 10 minutes. Sar keeps 18 types of reports, all stored in /var/log/sa/. When viewing the report file, you may pipe the result to less command so it only prints header once. For example, if you would like to print CPU report for the 2nd of the month:

# sar -u -f /var/log/sa/sar02 | less

If you check NFS client statistics for the 31st

 # sar -n NFS -f /var/log/sa/sar31

If you need to check network server statistics for today

# sar -n NFS -f /var/log/sa/sar31

Below are all types of reports:

-u CPU utilization
-w task creation and system switching activity
-W swapping statistics
-B report paging
-b report I/O and transfer rate statistics
-R report memory statistics
-r memory utilization
-S swap space utilization
-H huge pages utilization statistics
-v inode
-q queue length and load average
-y TTY device activity
-d activity for each block device
-n network statistics, DEV (per interface)
-n network statistics, EDEV (error per interface)
-n network statistics, NFS (NFS client)
-n network statistics, NFSD (NFS server)
-n network statistics, SOCK (socket usage)

Berkeley Packet Filter (BPF) Compiler Collection (bcc) tools

For advanced, low-level performance troubleshooting, the BCC tools provide a suite of tools. Here we only cover the installation of it.

In CentOS, install bcc-tools package with yum. When you try to run a command, such as cachestat, if you come across this error:

-bash: cachestat: command not found

Then you will need to add its path to default:

export PATH=$PATH:/usr/share/bcc/tools

Now if you run into this error:

chdir(/lib/modules/3.10.0-1062.12.1.el7.x86_64/build): No such file or directory
Traceback (most recent call last):

The file listed is a symbolic link, and if it is missing, you just need to install kernel-headers that matches the kernel version:

yum install kernel-headers

Then you may use tools in /usr/share/bcc/tools. For example, cachestat help you display page cache hit ratio; gethostlatency shows DNS resolution latency; tcpconnect prints out active tcp connections (made via connect system call):

[root@dhunch ~]# /usr/share/bcc/tools/tcpconnect -t -P 8080 | gawk '{ print strftime("%F %T  "), $0 }'
2020-06-13 00:16:57   TIME(s)  PID    COMM         IP SADDR            DADDR            DPORT
2020-06-13 02:16:57   0.000    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.064    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.438    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080

The command above outputs a time and pid stamped log line every time a TCP connection is made to port 8080; tcpaccept traces passive tcp connections (via accept system call). These tools are not as intrusive as tcpdump.

It is beyond the purpose of this article to get into details of each tool in the BFP suite. The tools are covered in detail in books “BPF Performance Tools” and “Linux Observability with BPF”.

Introduction to Authentication Frameworks (PAM and SSPI)

Tue, 24 Mar 2020 20:19:00 -0400

This article gives a very brief high-level introduction to PAM (Pluggable Authentication Module) and SSPI (Security Support Provider Interface) as authentication frameworks in Linux and Windows respectively.

PAM

The Pluggable Authentication Module (PAM) architecture provides a powerful abstraction for user IAM using pluggable authentication model Unix platforms. It defines a generic API for authentication and hides the underlying mechanisms. Thanks to PAM, administrators can plug different authentication modules and protocols into Linux. This makes different authentication methods and protocols available to applications running on Linux. Here is some of authentication methods and protocols that PAM supports:

Unix file-based authentication (using /etc/passwd or /etc/shadow)
LDAP-based authentication
Kerbero-based authentication
NTLM-based authentication

PAM obviates the need for a separate authentication schemes. It exports methods of the various libraries under its auspices to calling applications. Here is a diagram for PAM on Redhat.

PAM can also enbable single-sign-on(SSO) on the UNIX platform. If the password used for different services are identical, PAM can be used to share the password transparently between the application’s possibly different authentication mechanisms. PAM is configured in /etc/pam.d/ directory.

The Windows equivalent of PAM is the Security Support Provider Interface (SSPI) and its Security Support Provider (SSP) Modules. For example /etc/pam./login manages login module, /tec/pam.d/imap manages imap module.

SSPI

Windows Authentication Architecture involves Local Security Authority (LSA, to authenticate users to local computer only and is managed in local security policy) and Security Support Provider Interface (SSPI).

SSPI is the API that obtains integrated security service for authentication, message integrity, message privacy, and security quality-of-service for any distributed application protocol. SSPI is the implementation of the Generic Security Service API (GSS-API) in Windows operating system. Applications and infrastructure services authenticate users by using the SSPI to abstract calls for authentication. This way, developers do not need to understand the complexities of specific authentication protocols or build authentication protocols into their applications. Here is the architecture diagram:

The SSPI in Windows provides a mechansim that carries authentication token over the existing communication channel between the client computer and the server. When two computers or devices need to be authenticated so that they can communicate securely, the requests for authentication are routed to the SSPI, which completes the authentication process, regardless of the network protocol currently in use.

Here is some exampls of SSPs that are supported by SSPI:

Kerberos SSP (default for Active Directory)
NTLM SSP
Digest SSP
Negotiate SSP (based on SPNEGO, RFC4178)
Credential SSP
Negotiate Extension SSP

Some SSPs such as Kerberos SSP and NTLM SSP use a single protocol. Some (e.g. Negotiate SSP and Credential SSP) combine several protocols to allow application to select what security mechanism they wish to use and negotiate with authentication services.

SASL Authentication Mechanisms

Thu, 19 Mar 2020 22:53:00 -0400

Introduction

Authentication is used in many protocols (such as LDAP binding) and it usually involves sending password. Given the nature of authentication protocol, its traffic encryption is usually mandatory. Simple Authentication and Security Layer (SASL) is introduced to ensure the security during authentication. It is not a single protocol, but rather a framework for authentication and data security involving many protocols. The intent is to decouple authentication mechanisms from application protocols, thus allowing any authentication mechanism (under SASL) to be used in any application protocol (that supports SASL). Application protocols that support SASL typically can also be built on Transport Layer Security (TLS), whose latest versions (1.2 and 1.3) are considered more secure.

None (ANONYMOUS)

The server basically does not authenticate the client. The client connects to the server anonymously. Under SASL framework, this may also be referred to as ANONYMOUS mechanism.

Simple (PLAIN)

In simple authentication method the password is sent to server in the clear. This is subject to eavesdropping and is not secure. It is still surprisingly widespread in legacy configurations probably due to the simplicity of configuration. This option should not be available in cloud environment. Under SASL framework, this may also be referred to as PLAIN mechanism.

CRAM-MD5 and DIGEST-MD5

CRAM-MD5: described in RFC 2195, using HMAC-MD5 algorithm. In this challenge-response scheme based mechanism, the client’s password is protected during authentication, but the application session (e.g. LDAP) traffic is not encrypted. It includes random data from the server and is slightly better than Simple authentication. However, this authentication method is not recommended either.

DIGEST-MD5: described in RFC 2831. This is very similar to CRAM-MD5 but is is somewhat stronger because it includes random data from both the client and server. In addition, it also provides a provision to ensure connection integrity and confidentiality (a data security layer).

GSSAPI

Generic Security Service Application Program Interface (GSS-API) is an API specification for programs to access security services. GSS-API by itself does not provide any security. Instead, security-service vendors provide GSSAPI implementations – usually in the form of libraries installed with their security software. These libraries present a GSSAPI-compatible interface to application developers who can write their application to use only the vendor-independent GSSAPI.

Under SASL, the dominant GSSAPI mechanism implementation in use is Kerberos version 5. GSSAPI allows Kerberos implementations to be API compatible. In many contexts, GSSAPI simply implies Kerberos.

NTLM

NT LAN Manager (NTLM) is a challenge-response based Microsoft security protocols. It is implemented in a Security Support Provider (SSP), which combines the older LAN Manager authentication protocol, NTLMv1, NTLMv2 and NTLM2 Session protocols in a single package. Group policy manages whether these protocols are used or can be used. NTLM passwords are considered weak because they can be brute-forced very easily with modern hardware. It might still be enabled in server configuration as a backup mechanism to Kerberos.

TLS (EXTERNAL)

CRAM-MD5, DIGEST-MD5, GSSAPI, and NTLM are more commonly referred to as SASL mechanisms (in loose terms). These mechanisms allow for a secure password exchange without requiring TLS by trying to address the authentication traffic encryption problem at application layer. Using TLS this can also be address at transport layer. TLS can be used in combination with any of the mechanisms above but usually TLS/Simple mechanism is sufficient. In many occasions the mechanisms under SASL can be replaced by simple authentication encrypted with TLS. Under the SASL framework, this may also be referred to as EXTERNAL mechanism so TLS (in strict terms) is also considered a SASL mechanism.

Summary

This article outlined several authentication protocols under SASL. Since SASL is the framework that intends to govern all authentication protocols, the use case of these protocol can be widespread. For example, in LDAP you can find all of them. Here is a comparison across them:

	Password on wire	Session
Simple	Clear	No Encryption
SASL/CRAM-MD5	Encrypted	No Encryption
SASL/DIGEST-MD5	Encrypted	No Encryption
SASL/GSSAPI	Kerberos	Encryption
TLS:SIMPLE	Encrypted	Encrypted

Common SASL implementation includes Cyrus SASL and GNU SASL. There are also some API implementations that supports some of SASL mechanisms, such as SSPI.

OAuth 2.0 and OIDC 1 of 2

Sat, 14 Mar 2020 21:10:00 -0400

OAuth 2.0 and OpenID Connect (OIDC 1.0) are different but highly related protocols and they are often confused. When we talk about IAM (identity and access management), we should first distinguish between Authentication (AuthN) and Authorization (AuthZ):

AutheNtication (AuthN, aka Identity Management) is about validating user’s identity by verifying that the user trying to connect is actually who it claims itself to be;
AuthoriZation (AuthZ, aka Access Management) refers to granting or denying access to specific resources based on the requesting user’s identity. It is usually performed after a user is identified through authentication. The most common approach is Role-Based Access Control (RBAC).

In a nutshell, OAuth 2.0 deals with authorization. OIDC is a layer later developed on top of OAuth 2.0, to deal with authentication. This post is greatly influenced by a presentation (1 hour) delivered by Nate Barbettini from Okta, with the slides available here. There is also an abridged and illustrated video (16 min) by Okta available here.

OAuth 2.0

OAuth was originally developed by Twitter and Google in 2006 as an open standard for API authorization. OAuth 2.0 is published in 2012. It allows user to delegate authorization. The original scenario is a user signed up to a new application and allows it to automatically import her Gmail contact. The technical problem to solve is: how can a user (Resource Owner) let an app (Client) to access his contact list stored in Google server (Resource Server)? The proposal is that it redirects user to Google Account page (Authorization Server) for user to log in. Then Google Account issues token to the application (Client) with user’s approval. Note that the user did NOT log in to the application itself with her Google account. From the application’s standpoint, the user had been authenticated already, and was simply importing contact after logging in. The roles involved in OAuth 2.0 are:

Resource: the contact list of the user
Resource owner: the user
Client: the application
Resource Server: contact.google.com
Authorization Server: accounts.google.com

The diagram below illustrates the interactions:

OAuth 2.0 Authorization Code Flow

This page has further details for each step. Note that at step 3 to 5 may seem unnecessary because Auth Server could have send Access Token Grant to Application via User-Agent at step 3, which could have eliminate the need for step 4 and 5. In fact this design is to avoid sending critical information (Access Token Grant) to User-Agent (browser) which is considered in secure. In other words, it avoids front channel (User-agent to auth server) and prefers back channel (Client to Auth server) for security. This is the difference between Authorization code flow and the implicit flow.

With OAuth 2.0, there are a number of flows:

Authorization code (front channel and back channel)
Implicit (front channel only, token returned to user agent directly)
Resource owner password credentials (back channel only)
Client credentials (back channel only)

In Authorization Code Flow, the Application (client) needs a one-time registration with the Auth Server and is given a client ID and client secret, which are sent to Auth Server at step 4 along with Access Token Request, to prove the identity of the client application.

Note that OAuth 2.0 is an inherently insecure protocol since it does not support signature, encryption, channel binding or client verification. The protocol relies entirely on the underlying transport layer security (TLS) to provide confidentiality and integrity.

Also note that throughout the process (Authorization Code Flow as an example), the Client application eventually is granted access to user’s data. However, it does not know anything about the user itself. Neither the authorization code grant, nor the access token grant is obligated to present information about the user itself. Therefore, OAuth 2.0 is designed strictly for permission purpose without the intent to address identity issue. In the flow, the auth server does the authentication (for the purpose of granting access to resources, but none of the authentication. A user logs in to client application as Bob, when he requests to imports contact, he is redirected to account.google.com and there he could put in the credential of Alice and therefore load Alice’s Google contacts into Bob’s App account!

Pseudo-authentication with OAuth 2.0

In many real life OAuth 2.0 implementations, at step 3, the Auth server chooses to include a field about the user’s identity. This makes user’s identity visible to the client, and the client is therefore able to confirm user’s identity in its own code. This also allows client application to use OAuth 2.0 as an authentication method, which is referred to as pseudo-authentication. The access token acts as a kind of “valet key” that the application can include with its request to the auth server, as a proof that it has user’s permission to access the resources (or APIs).

Because the identity provider (auth server) typically (but not always) authenticates the user as part of the process of granting an OAuth access token, it’s tempting to view a successful OAuth access token request as an authentication method itself. However, because OAuth was not designed with this use case in mind, making this assumption can lead to major security flaws.

Nate’s presentation outlined some scenarios where OAuth 2.0 is applied up to 2012, and which ones are misuses:

Simple login – pseudo authentication with OAuth 2.0
Single sign-on across sites – pseudo authentication with OAuth 2.0
Mobile app login – pseudo authentication with OAuth 2.0
Delegated authorization – the only intended use case for OAuth 2.0

To address the authentication issue properly, and in a standard approach, we need OpenID Connect.

OpenID Connect (OIDC)

OpenID Connect is an open standard for authentication, promoted by the non-profit OpenID Foundation. It allows user to be authenticated using a third-party service called identity providers. User may choose to use their preferred OpenID Connect providers to log in to websites that accept the OpenID Connect authentication scheme. For example, a user uses her Facebook to login to an online application.

OpenID Connect is an extension to OAuth 2.0 with a just few additions:

In addition to access token, an ID token is returned by the authorization server;
Userinfo end point is provided in case Id token is not sufficient and more user information is needed;
“openid” is passed as a parameter in the Scope during the initial call to the authorization server;

Therefore OpenID Connect is considered an identity layer on top of OAuth 2.0. Many application supports OpenID Connect such as Apache Nifi. OIDC is comparable with SAML in the sense that both provide SSO feature (federated identity). Here is a comparison table:

	OpenID Connect	SAML
Main Purpose	SSO for consumer/mobile applications	SSO for enterprise applications
Load	Relatively light weight	Heavy weight due to the size of XML messages
Use case	Satisfies both authentication and authorization use cases, often combined with OAuth 2.0	Generally not used for API security
Transport	HTTP GET and HTTP POST	HTTP Redirect (GET) binding, SAML SOAP binding, HTTP POST binding, et

Here are more details about their differences. In general SAML is more common in the enterprise world for SSO and it has been around for a while. When developing new applications for enterprise it is advised to consider OIDC first.

OIDC also has authorization code flow, with the additional fields on top of its counterpart in OAuth 2.0. The authorization server returns both access and ID tokens, wrapped in a data structure named JWT (JSON Web Token). The JWT includes a signature field, allowing the client application to verify it with authorization server’s public key. Nate’s presentation proposes the following flows for each application type:

Web application with server backend: authorization code flow
Native mobile app: authorization code flow with PKCE
Java Script app (SPA) with API backend: implicit flow
Microservices and APIs: client credential flow

In addition, this page from Okta developer has a good summary of how to select flow type (grant) based on each use case.

Summary

OpenID Connect is an authentication protocol for the purpose of validating user’s identity. OAuth 2.0 is an authorization protocol. You should use OAuth 2.0 for granting access to your API, or access to user data in other systems. If you need to log user in, or make your accounts available in other systems, you need OIDC.

The next post about OAuth and OIDC was posted in 2023.

Security Assertion Markup Language (SAML)

Sun, 08 Mar 2020 22:23:00 -0400

SAML is an XML-based standard for exchanging authentication and authorization data between IdP (identity provider) and service provider. We can compare SAML with LDAP (as authentication protocol) as both are to provide single-sign-on (SSO) feature.

LDAP is considered traditional configuration in on-premise operation for organizations. The configuration can be complex and administrators needs to complete significant work upfront. AD is notoriously hard to integrate into the cloud. On the other hand, LDAP gives the organization greater level of control over authentication and authorization due to its tighter integration with domain controller. It is prevalent in on-premise enterprise infrastructure and integrate well with OpenVPN, Jenkins, Docker, Kubernetes, etc.

On the other hand, SAML was created in early 2000s with the exclusive purpose of federating identities to web applications. The protocol was introduced assuming an IdP already exists in an organization. The SAML protocol doesn’t intend to replace the IdP, but rather use it to assert the validity of a user’s identity. This timed assertion (declaration that user’s identity is valid for a period of time) will be delivered to a service provider via secure XML exchange.

The benefit of SAML is that an on-premise identity typically stored in Active Directory (AD), could be extended to authenticate its users against web applications. ISVs (independent software vendor) can build web applications that integrates with on-premise AD server to achieve SSO feature. The AD server in this case provides IDaaS (Identity as a service) using its FS (Federation Service) module. Examples of web applications that support SAML integration include Confluence, Zendesk, Slack, Bombgar, etc. In the configuration, you typically need to specify who is the IdP (e.g. Microsoft AD, Okta, etc) and it’s SSO URL.

At a high level, an SSO process using SAML takes places in the following steps:

User tires to reach web application (service provider);
Web application redirects user browser to SSO URL;
User provide credential in the SSO URL;
IdP authenticates the user;
IdP produces SAML response to browser;
Browser passes the SAML response to service provider’s dedicated endpoint
Service provider permits user access to web application

Here is an example of SAML integration guide from an G Suite. The guide outlines how it works and the assertion requirements.

The Wikipedia page for SAML 2.0 provide an example of assertion message.

Lightweight Directory Access Protocol (LDAP)

Mon, 02 Mar 2020 21:11:00 -0400

Introduction

Originally LDAP only refers to the connectivity protocol to the directory server. This term is being used loosely today and it also refers to the actual directory service that supports and complies with LDAP. LDAP v3 is the current version developed in RFC 2251.

A directory is information about some set of entities such as people, organization, or stones. An example of directory would be /etc/passwd file in Linux. A directory server is simply an application with the main purpose of maintaining directories. Typically, the read traffic is high whereas write traffic is low. LDAP is a general-purpose directory server. It can store information about people, or cars, or rocks. You just need to define what a person’s entry looks like as well as what a rock’s entry looks like. The general architecture of LDAP provides the capability nedded for managing large amount of diverse directory entries.

An LDAP entry consists of DN (distinguished name) and attributes. An attribute may have one or more attribute names and they are defined in attribute definitions. Attribute names are not case-sensitive. An attribute may have one or more values if multiple values are allowed for that attribute. Attribute values may be case-sensitive depending on the definition.

A special attributed named objectclass attribute provides information about what type of record it is, and what attributes canbe given to the record. For example, the organization name (o) is required for any entry with an organization object class. While a record may have multiple object classes, one of these object classes must be the structural object class for the record. A structural object class determines what type of object the record is.

In addition to regular attributes, the directory server may also attach special operational attributes to an entry. Operational attributes are used by the directory server itself to store information about entries. Such attributes are not designed for use by end users, and are usually not returned during LDAP searches.

An LDAP schema defines types of records in a directory and how those records might relate to each other.

Information in an LDAP directory is organized into one or more hierarchies where, at the top of the hierarchy, there is a base entry, and other entries are organized in tree-like structures beneath the base entry. Each node on the hierarchy is an entry, with a DN and more than one attributes. This hierarchically organized collection of entries is called a directory information tree (DIT). In DIT, LDAP directories stores data in hierarchical relationships. The root entry sits at the top and subordinate entry is beneath that, which in turn may have its own subordinate entries. Each of these records has its own DN, and its own attributes. The DN of each entry is composed of two parts: the relative DN (RDN) and the full DN of the superior entry.

LDAP is nothing other than a special sort of database that organizes data into tree structures, like a file system hierarchy. This view is more easily seen by comparing an LDAP directory to a relational database system (RDB), where SQL is the protocol and RDBMS is the service. LDAP refers to both the protocol and the service.

OpenLDAP

A common LDAP implementation is openldap. OpenLDAP suite can be broken up into four components:

Servers: slapd (stand-alone LDAP Daemon) provides LDAP services.
Clients: ldapsearch is used to manipulate LDAP data
Utilities: support LDAP servers
Libraries: provide programming interfaces to LDAP

Installing OpenLDAP requires libldap-2.3-0, slapd, ldap-utiles packages. It is configured in /etc/ldap/. An HDB (hierarchical database) needs to be specified in the configuration.

To test as a client, the first thing that must happen is the client must authenticate to the server (via simple bind or SASL Bind). LDAP server verifies the identity, permission as well as password provided by the client.

LDAPTLS_REQCERT=never ldapsearch -x -o ldif-wrap=256 -H ldaps://ldap.digihunch:636/ -b "OU=Admin,OU=Service Department,DC=digihunch,DC=com" -D "gh\ldap-bind-user" -w 'S@f35+P@55w0rd' "(objectclass=user)" -s sub -d 9

The command above first sets client environment variable LDAPTLS_REQCERT to never, in case the client is being asked to provide certificate. Then the ldapsearch command performs the bind.

To search the directory, the client needs to provide the followings:

Base DN: where in the directory to start from
Scope: how deep in the tree to look
Attributes: what information to be retrieved per result
Filter: what to look for

Below is an example of ldapsearch (-b for Base DN, -s for Scope, -S for attributes, stdin for filter):

ldapsearch -x -o ldif-wrap=256 -H ldaps://ldap.digihunch:636/ -b "OU=Admin,OU=Service Department,DC=digihunch,DC=com" -D "gh\ldap-bind-user" -w 'S@f35+P@55w0rd' "(memberof=CN=Security-Admin,OU=Admin,OU=Service Department,DC=digihunch,DC=com)" -s sub -S name

Users with appropriate permissions may also other directory operations using ldapadd, ldapmodify, ldapdelete, ldapcompare, ldapmodrdn, ldappasswd, ldapwhoami, etc

Apart from those in Openldap toolkits, there are many other tools such as Apache Directory Studio that allows you to perform similar functionality with a user interface.

LDAP security

Historically LDAP servers listens to port 389 through which traffic is sent in clear text. This is a bad security practice known as “insecure bind”. To secure LDAP traffic, two prevalent approaches are Secure LDAP and StartTLS.

Secure LDAP was the original attempt to secure LDAP traffic as an addition to LDAP v2. It is also known as LDAPS, LDAP over TLS/SSL or LDAP channel binding (“channel binding” just refers to the establishment of encrypted channel following TLS handshake. It provides a facility to tie an authentication exchange to security services provided at a lower layer. Defined in RFC 5056). Secure LDAP operates on port 636 on the server side and TLS handshake must be established for traffic encryption. Client application usually need to import the certificate of LDAP server. As part of TLS 1.2 protocol, the server may also request client certificate during ServerHello message. The presence of CertificateRequest means the server either demands client certificate, or tries to get client certificate (i.e. TLSVerifyClient is set to demand or try, which is only visible on the server). If client cert is only attempted, the LDAP client may choose to ignore it. If client cert is demanded, then a two-way TLS authentication is required and thus the client must proof its identity to the server. This Secure LDAP configuration requires the server to listen to both 389 and 636 ports on the same server to support both secure and legacy applications, which is unnecessary. Secure LDAP therefore is not the preferred approach.

The standardized way of implementing SSL/TLS in LDAP v.3 is to use the StartTLS method. This method should be implemented whenever possible. If an AD server supports StartTLS, the client can start with a STARTTLS command to the server so that the server begins the TLS encryption process. In the binding phase, TLS handshake follows a LDAP_START_TLS_OID command through port 389.

Here’s the summary of the three LDAP configuration mode:

	Legacy	SecureLDAP (aka LDAPS, LDAP over TLS/SSL)	StartTLS
Listening port	389	636	389
Traffic Encrypted	No	Yes	Yes
Standard	Yes but this should always be avoided since it is insecure	Introduced in the time of LDAP v2, but the option is deprecated (although still supported) by RedHat	Introduced in LDAP v3. This may be left as the only valid option.

Note that one of the recent changes that drives may customer away from the legacy mode is the requirement for LDAP channel binding on Windows servers, with a target date of March 2020. Our current strategy at CS is to direct customer towards Secure LDAP as we do not support StartTLS yet and we know we do support LDAPS. Although Secure LDAP itself is somewhat legacy this would not hold long. According to this Wikipedia page:

The use of LDAP over SSL was common in LDAP Version 2 (LDAPv2) but it was never standardized in any formal specification. This usage has been deprecated along with LDAPv2, which was officially retired in 2003.

The trade off between StartTLS and TLS/SSL exists not only in LDAP protocol, but also in many other protocols such as SMTP (port 2525, 25, 587). StartTLS is also called Opportunistic TLS. The standard is in the relevant RFC documents.

Introduction to Active Directory (AD)

Fri, 28 Feb 2020 21:36:00 -0400

Workgroup, homegroup and Windows Domain

A workgroup is a group of computers on the same local network. A Windows computer not joined to a domain is part of a workgroup. In a workgroup, no computer has control over any other computer and it does not require a password. Any computer can join or leave a workgroup any time. Workgroup was previously for home file and printer sharing and Microsoft later introduced homegroup for more security. Compared to workgroup, all computers in a homegroup needs to be on the same home network (instead of local network). Homegroup is password protected. New computer needs to join homegroup by providing the password.

Windows domains (or domains for short) provide network administrators with a way to manage a large number of PCs and control them from one place and remotely. One or more servers — known as domain controllers — have control over the domain and the computers on it. Computers on a domain has to be on the same local network, either physically or over VPN. Centralized control is essential for corporate operation.

Administrators can join a Windows PC with professional or enterprise license to a domain. Once joined, the computer does not use its own local user accounts. When a user logs into a computer on that domain, the computer authenticates the user account name and password with the domain controller. Also, the computer cannot just leave the domain without administrator access.

Network administrators can change group policy settings on the domain controller. Each computer on the domain will get these settings from the domain controller and they’ll override any local settings users specify on their PCs. All the settings are controlled from a single place. This prevents from users from changing many system settings on a computer joined to a domain. The domain controller is in charge of what a user can do.

Apart from centralized administration, the benefit the users is that they can log in with the same username and password on any computer joined to the domain, if permission allows.

Domain Controller

In order to achieve centralized administration and log-in from any computer in the corporate world, a centralized service called domain controller is introduced. At a high level, a domain controller maintains a list for each of the followings:

Users and their passwords
Computers and their credentials

This is because in a domain, not only the users, but also the computers (workstations or servers) need to be authenticated. For example, when a Windows server boots up, it needs to log on to the domain with its own credential. This way we can control whether the server is allowed to query the domain for information about users. If it is allowed to query the domain, then we can determine whether the user is allowed to log on that server, and eventually, authenticate the user.

For a domain controller, it responsibility to credentials for users and computers, and respond to log in requests (authentication service) is a critical commitment in the enterprise environment. Domain controller is therefore commonly built with high availability and fault tolerance.

Administrators needs to add each new user to the user directory in domain controller. They also needs to register each new computer with the domain controller by joining them to the domain.

Joining a Domain

As explained earlier, joining a Window domain means register a computer in the domain so it has the permission to query the domain to validate users identity and permissions. Both Windows server and Linux server can join a domain. Windows servers usually provide a path through UI to join a domain and password is required.

To join a Linux (e.g. Redhat) server to a domain, we can use a tool called adcli. Here is a good example of using this command to join a domain.

Authentication

A Windows or Linux server in the domain needs to go to the domain controller to authenticate itself and the users. Authentication involves several protocols, including kerberos, NTLM, TLS/SSL and Digest, as part of an extensible architecture. In addition, some protocols are combined into authentication packages such as Negotiate and the Credential Security Support Provider.

The MIT Kerberos Documentation provides some tools (e.g. kinit, klist) to configure and troubleshoot Kerberos protocol.

Active Directory

Since Windows 2000, Active Directory is a complete redesign and re-branding of the entire Windows Domain system. The term Active Directory now refer to either the entire domain system, or the actual database that comprises the Windows Domain information or both.

All of the information that makes up an Active Directory is stored in an X.500 compatible database, typically replicated between domain controllers to ensure high availability and fault tolerance. X.500 is a set of network directory standards. A Windows Domain is a kind of network directory, hence the name Active Directory for its replacement.

Active Directory introduced one important new type of object and concept, Forests. An Active Directory Forest is kind of a list of lists, meaning, it is a collection of Domains that are all related to each other for both security and management purposes.

Here is more details about Active Directory.

Lightweight Directory Access Protocol

As mentioned above, X.500 is a series of computer networking standards covering electronic directory services. ISO incorporated it into OSI suite of protocols. The protocols defined by X.500 include DAP (Directory Access Protocol), DSP (Directory System Protocol), DISP (Directory Information Shadowing Protocol) and DOP (Directory Operational Bindings Management Protocol). DAP is a heavyweight protocol that operates over a full OSI protocol stack and requires a significant amount of computing resources. LDAP (Lightweight Directory Access Protocol), as its alternative, is designed to operate over TCP/IP and provides most of the functionality of DAP at a much lower cost.

Technically speaking, LDAP is a directory access protocol to an X.500 directory service. In early days, the typical architecture involves a proxy. Client connects to the proxy in LDAP and the proxy connects to X.500 server in DAP. Nowadays, it is common that LDAP is directly implemented in X.500 servers.

Because LDAP is based on a simpler subset of the standards contained within the X.500 standard, LDAP was sometimes called X.500-lite. While DAP and the other X.500 protocols can now use the TCP/IP networking stack, LDAP remains a popular directory access protocol.

High Availability and Load Balancer

Wed, 22 Jan 2020 20:49:00 -0400

Overview

Fault tolerance and high availability are two architectural characteristics that people often confuse with each other. High availability focuses on minimizing downtime. It guarantees uptime, but not performance in the event of component failures. Fault tolerance, on the other hand, focuses on stable capacity even in the event of component failures. Fault tolerance has higher bar, and therefore is more expensive.

Suppose an application requires four servers to meet performance goal. Placing two servers in each of the two AZs will meet HA criteria but not FT requirement. In the event of an AZ failure, application can operate at degraded performance yet still be highly available. However, FT requires stable capacity and to meet FT requirement, we’d have to place four servers in each AZ.

High availability can be achieved either by clustering, or load balancing. A cluster involves several nodes, all able to perform the same function, but may take different roles at different times (e.g. primary, standby) in order for the cluster to perform its function as a single system. In Linux, clustering is implemented by pacemaker or corosync. With a high load system, it is common to set up load balancing system to achieve high availability (and fault tolerance).

Load balancing

The idea of load balancing is simple: load goes high and we want to scale horizontally instead of simply upgrading server hardware. At a high level, there has been three approaches to load balancing:

DNS rotating: (aka. DNS round robin) DNS record resolves to multiple IPs, very simple and cheap to implement. Since DNS is cached, the load distribution will come imbalanced and it’s hard to re-balance, making this a very limited approach;
Hardware Load Balancer: using dedicated hardware device to configure load balancing. This option is expensive and only enterprises can afford it (here‘s some pricing information). A classic load balancer operates at layer 3 and 4, which is also known as POLB (plain old load balancer). It is the core functionality of hardware load balancer. The hardware load balancer on the market today usually come with a variety of add-on features, such as advanced load balancing (L4, L7 path-based, script driven), compression, caching, SSL offloading, and even DDoS mitigation, etc. The whole suite of features makes it an Application Delivery Controller (ADC). Therefore many refer to hardware load balancer as hardware-based ADC to highlight the features in addition to POLB. Hardware-based ADCs ship with manufactures hardware, with specialized processors, advanced network hardware, and often ASIC (application specific integrated circuit). At a higher expense, they have better reliability and capacity. Some major market players are:
- F5 – Big IP, F5 also has a good article about history of load balancer.
- Cisco – Citrix Ahttps://www.citrix.com/products/citrix-adc/DC (formerly NetScaler ADC)
- A10 Networks – Thunder (general) and Lightning (cloud)
Software Load Balancer: using software to achieve load balancing. These solutions are affordable, and usually open-source. They can be loaded on commodity hardware (including NIC). Some (e.g. Nginx) refers to themselves as software-based ADC. Major players are:
- HA Proxy
- Nginx
- Linux Virtual Server (LVS, L4 only)

The hardware ADCs are usually supported commercially and there are plenty of resources from their white papers. There is an ongoing debate about whether one is better than the other. However, there is no doubt that a software-based load balancer is more approachable as open-source tools. The line between software and hardware load balancers becomes blurred today as hardware vendors try to adapt their software appliance to commodity hardware. Check out this article. The rest of this post, will focus on software-based load balancer.

Software-based load balancer

We explained that ADC (application delivery controller) is an expanded set of features from load balancer, and will only cover the load balancer part of the feature set in this article.

HAProxy supports both layer 4 and layer 7 load balancing. It supports load balancing based on cookie and session, as well as health check. Since it is layer 4 load balancing, it supports any TCP protocol such as read traffic for MySQL.

Nginx is a high-performance, event-driven, cross-platform layer 7 load balancing application. It works as a reverse proxy where it receives request for the Internet and forwards it to (upstream) internal servers. It consumes less memory than many of its alternatives for layer 7 load balancing. There are many strategies for load balancing such as round robin, by weight, by hash of requesting IP, by upstream response time, or by URL hash. It supports 20-30 k concurrent connections, and support compression and health check. It is known to be very stable and common for small and medium volume. Nginx has a commercial counterpart Nginx Plus with advanced features.

Nginx and HA proxy are commonly used in front end load balancing. For backend traffic such as database (e.g. separating read write traffic), LVS can be used.

Linux Virtual Server

LVS (Linux Virtual Server) is part of standard Linux kernel. It performs layer 4 load balancing based on TCP or UDP and therefore consumes less memory and CPU. Compared to layer 7 load balancing, the performance is generally higher, and the configuration is less complex (with simpler routing rules). LVS is usually configured in a common cluster architecture involving these components:

Load balancer: the front-end machine of the whole cluster systems, and balances requests from clients among a set of servers, so that the clients consider that all the services is from a single IP address.
Server cluster: set of servers running actual business workload
Shared storage: a shared storage space for the servers, such as NFS

Load balancer is the single entry-point of server cluster systems, it can run IPVS that implements IP load balancing techniques inside the Linux kernel, or KTCPVS that implements application-level load balancing inside the Linux kernel. When IPVS is used, all the servers are required to provide the same services and contents, the load balancer forward a new client request to a server according to the specified scheduling algorithms and the load of each server. No matter which server is selected, the client should get the same result. When KTCPVS is used, servers can have different contents, the load balancer can forward a request to a different server according to the content of request. Since KTCPVS is implemented inside the Linux kernel, the overhead of relaying data is minimal, so that it can still have high throughput.

IPVS is also called layer-4 switching, it directs TCP/UDP requests to the real servers behind load balancer. It works in three modes:

Network Address Translation (NAT)
Direct Routing (DR)
Tunnel mode (TUN)

These are three packet-forwarding methods in IPVS. The IPVS is implemented as a module over the netfilter framework, similar to iptables, which is also built on top of netfilter, based on chain and rules.

Summary

We had an overview of high availability, and then expanded on load balancing, an important mechanism to implement high availability. We touched on both hardware-based and software-based load balancing technologies, and dived a little more into Linux Virtual Server. It is worth-noting that LVS is also the foundation of kube-proxy, the load balancing mechanism used in Kubernetes.

Nginx as a reverse proxy for Nifi web UI and Kibana

Thu, 16 Jan 2020 22:22:51 -0400

Nginx can act as a application neutral proxy. One example is to front Nifi. The nifi default configuration provides an HTTP access point, specified in the following entries in nifi.properties:

nifi.web.http.host=192.168.133.5
nifi.web.http.port=8080

Nifi can provide secure port by commenting out the lines above and provide the followings:

nifi.web.https.host=192.168.133.5
nifi.web.https.port=8083

However, it does require configuring JKS keystore for Java, as well as authentication. Customers with existing AD servers are likely to require authentication via LDAP. While Nifi does support LDAP integration according to its administration guide. The configuration is quite involving. You need to configure the identity provider, as well as authorizes. I have personally spent a couple days on this without much progress. The information in the logging isn’t to the point. Restarting nifi also is a long process, making it painful to troubleshoot.

I then moved to Nginx (open-source) as an alternative and it is quite enlightening. I already knew that the SSL termination in nginx is super easy to configure. This time I learned that the opensource community even has a support for LDAP integration. Here is a diagram of how it works:

This approach is outlined in a blog post on Nginx website. The ldap-auth daemon is implemented in Python can can be wrapped up as a systemd service. Once a client sends a request in https, security layer is terminated in nginx, and an authentication request in http is sent to the ldap-auth daemon, which proxies converts the request into LDAP searches and proxies it over to customer’s Active Directory server, for authentication. Once authenticated, the http request can make to one of the backend container or server which hosts Nifi. Below is an example of how this can be configure on RedHat.

Install python3 and python-ldap

RedHat may have both python2 and python3 pre-installed, python2 being the default. We do not want to change the default because other applications such as yum still depends on python2 as of early 2020.

yum -y install python3
yum -y install gcc python3-devel openldap-devel
pip3 install python-ldap

Once python3 is installed, pip3 will be available and we use that to install python-ldap. This is a module in Python3 that will be used by the script that act as ldap daemon.

Configure ldap-auth daemon as systemd service

In the github project for ldap-auth, download nginx-ldap-auth-daemon.py to local location such as /usr/bin, then we create nginx-ldap-auth.service in /etc/systemd/system/ with the following content.

[Unit]
Description=LDAP authentication helper for Nginx
After=network.target network-online.target
 
[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/var/run
ExecStart=/usr/bin/python3 /usr/bin/nginx-ldap-auth-daemon.py
KillMode=process
KillSignal=SIGINT
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

Then, run the following command to load, start and check nginx-ldap-auth service.

systemctl reload-daemon
systemctl start nginx-ldap-auth
systemctl status nginx-ldap-auth

This service will be up and listening to port 8888 for http traffic.

Configure Nginx

Then configure nginx with the following entries in its default.conf file, typically located in /etc/nginx/conf.d.

upstream nifibackend {
 # default: round robin
 server container1.nifi.digihunch.com:8080;
 server container2.nifi.digihunch.com:8080;
}
proxy_cache_path cache/ keys_zone=auth_cache:10m;
# nifi proxy
server {
 listen 8083ssl;
 include /etc/nginx/ssl/default.conf;
 
 location / {
 auth_request /auth-proxy;
 
 proxy_pass http://nifibackend;
 proxy_set_header Host $host:$server_port;
 proxy_set_header X-ProxyScheme https;
 proxy_set_header X-ProxyHost $1;
 proxy_set_header X-ProxyPort 8083;
 proxy_set_header X-ProxyContextPath /;
 }
 
 location /auth-proxy {
 internal;
 proxy_pass http://127.0.0.1:8888;
 
 proxy_pass_request_body off;
 proxy_set_header Content-Length "";
 proxy_cache auth_cache;
 proxy_cache_valid 200 10m;
 
 proxy_cache_key "$http_authorization$cookie_nginxauth";
 proxy_set_header X-Ldap-URL "ldaps://ldap.digihunch.com:636";
 proxy_set_header X-Ldap-BaseDN "OU=Corporate User Accounts,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindDN "CN=Digi Hunch Service Account,OU=Digi,OU=ServiceAccounts,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindPass "myownpasswordtricks";
 proxy_set_header X-CookieName "nginxauth";
 proxy_set_header Cookie nginxauth=$cookie_nginxauth;
 proxy_set_header X-Ldap_Starttls "true";
 proxy_set_header X-Ldap-Template "(&(sAMAccountName=%(username)s)(objectClass=organizationalPerson)(memberOf=CN=GH_SYSADMIN,OU=GHCO,OU=Groups,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org))";
 }
}

We need the full distinguished name of bind user to get this to work. Once configured properly, and user attempts to connect through a browser, Nginx will pop up a prompt for username and password. The username will be plugged into the X-Ldap-Template for further queries. The same HTTP header also allows you to filter by membership that the user is associated with.

Storage Nitty-Gritty 5 of 5 – Replication

Tue, 19 Nov 2019 00:10:23 -0400

Replication Terms

PIT (point in time) replica – snapshot of the source at some specific timestamp;
Continuous Replica – always in-sync with the production data;
Recoverability – enables restoration of data from the replica to the source if data loss or corruption occurs;
Restartability – enables restarting business operations using the replicas;

Local Replication

Use Case:

Alternative source for backup
Fast recovery
Decision-support activities such as data warehousing
Testing platform
Data migration

Consistency in file system replication
File systems buffer the data in the host memory to improve the application response time. The buffered data is periodically written to the disk. In UNIX operating systems, sync daemon is the process that flushes the buffers to the disk at set intervals. In some cases, the replica is created between the set intervals, which might result in the creation of an inconsistent replica. Therefore, host memory buffers must be flushed to ensure data consistency on the replica, prior to its creation.

Flushing the file system buffer

In the illustration above, If the host memory buffers are not flushed, the data on the replica will not contain the information that was buffered in the host. If the file system is unmounted before creating the replica, the buffers will be automatically flushed and the data will be consistent on the replica.

Consistency in database replication
When a database is replicated while it is online, changes made to the database at this time must be applied to the replica to make it consistent. A consistent replica of an online database is created by using the dependent write I/O principle or by holding I/Os momentarily to the source before creating the replica.

A dependent write I/O principle is inherent in many applications and database management systems (DBMS) to ensure consistency. According to this principle, a write I/O is not issued by an application until a prior related write I/O has completed. For example, a data write is dependent on the successful completion of the prior log write.

For a transaction to be deemed complete, databases require a series of writes to have occurred in a particular order. These writes will be recorded on the various devices or file systems.

Another way to ensure consistency is to make sure that the write I/O to all
source devices is held for the duration of creating the replica. This creates a
consistent image on the replica. However, databases and applications might time out if the I/O is held for too long.

Local Replication Technologies

Host-based Local Replication

LVM-based replication: logical volume manager (LVM) is responsible for creating and controlling the host-level logical volumes. Each logical block in a logical volume is mapped to two physical blocks on two different physical volumes. LVM-based replication is part of operating system and comes without additional license cost. However, every write generated by application translates into two writes on the disk, and thus, an additional burden is placed on the host CPU. This can degrade application performance. Presenting an LVM-based logical replica to another host is usually not possible because the replica will still be part of the volume group, which is accessed by one host at any given time. You can’t track changes on LVMs either so it does not support incremental resynchronization.

File system snapshot: a pointer-based replica that requires a fraction of the space used by the production FS. This snapshot can be implemented by either FS or by LVM. It uses the Copy on First Write (CoFW) principle to create snapshot. When a snapshot is created, a bitmap and blockmap are created in the metadata of the Snap FS. The bitmap is used to keep track of blocks that are changed on the production FS after the snap creation. The blockmap is used to indicate the exact address from which the data is to be read when the data is accessed from the Snap FS. Immediately after the creation of the FS snapshot, all reads from the snapshot are actually served by reading the production FS. In a CoFW mechanism, if a write I/O is issued to the production FS for the fi rst time after the creation of a snapshot, the I/O is held and the original data of production FS corresponding to that location is moved to the Snap FS. Then, the write is allowed to the production FS. The bitmap and blockmap are updated accordingly. Subsequent writes to the same location do not initiate the CoFW activity. To read from the Snap FS, the bitmap is consulted. If the bit is 0, then the read is directed to the production FS. If the bit is 1, then the block address is obtained from the blockmap, and the data is read from that address on the Snap FS. Read requests from the production FS work as normal.

File system snapshot

Storage Array-based local replication
the array-operating environment performs the local replication process. The host resources, such as the CPU and memory, are not used in the replication process. Consequently, the host is not burdened by the replication operations. The replica can be accessed by an alternative host for other business operations.

Full-Volume Mirroring – the target is attached to the source and established as a mirror of the source. After all the data is copied and both the source and the target contain identical data, the target can be considered as a mirror of the source. After the synchronization is complete, the target can be detached from the source and made available for other business operations. The target becomes a point-in-time (PIT) copy of the source. After detachment, changes made to both the source and replica can be tracked at some predefined granularity. This enables incremental resynchronization (source to target) or incremental restore (target to source). The granularity of the data change can range from 512 byte blocks to 64 KB blocks or higher.

Full volume mirroring

Pointer-based, Full-Volume Replication – the target is immediately accessible by the BC host after the replication session is activated. Therefore, data synchronization and detachment of the target is not required to access it.

Pointer-based, Virtual Replication – at the time of the replication session activation, the target contains pointers to the location of the data on the source. The target does not contain data at any time. Therefore, the target is known as a virtual replica. the target is immediately accessible after the replication session activation. A protection bitmap is created for all data blocks on the source device. Granularity of data blocks can range from 512 byte blocks to 64 KB blocks or greater.

Network-based local replication: the replication occurs at the network layer between host and storage arrays. By offloading replication from servers and arrays, network-based replication can work across a large number of server platforms and storage arrays, making it ideal for highly heterogeneous environments.

Continuous Data Protection: CDP provides the ability to restore data to any previous PIT. In CDP, data changes are continuously captured and stored in a separate location from the primary storage. With CDP, recovery from data corruption poses no problem because it allows going back to a PIT image prior to the data corruption incident. CDP uses a journal volume to store all data changes on the primary storage. The journal volume contains all the data that has changed from the time the replication session started. The amount of space that is configured for the journal determines how far back the recovery points can go. CDP appliance is an intelligent hardware platform that runs the CDP software and manages local and remote data replications. Write splitters intercept writes to the production volume from the host and split each write into two copies. Write splitting can be performed at the host, fabric, or storage array.

CDP Local Replication Operation: before the start of replication, the replica is synchronized with the source and then the replication process starts. After the replication starts, all the writes to the source are split into two copies. One of the copies is sent to the CDP appliance and the other to the production volume. When the CDP appliance receives a copy of a write, it is written to the journal volume along with its timestamp. As a next step, data from the journal volume is sent to the replica at predefi ned intervals.

Continuous Data Protection

Tracking Changes to Source and Replica
Changes can occur on the replica device if it is used for other business operations. To enable incremental resynchronization or restore operations, changes to both the source and replica devices after the PIT should be tracked.
This is typically done using bitmaps, where each bit represents a block of data. For example, if the block size is 32 KB, then a 1-GB device would require 32,768 bits (1 GB divided by 32 KB). The size of the bitmap would be 4 KB. If the data in any 32 KB block is changed, the corresponding bit in the bitmap is flagged. If the block size is reduced for tracking purposes, then the bitmap size increases correspondingly.
The bits in the source and target bitmaps are all set to 0 (zero) when the replica is created. Any changes to the source or replica are then fl agged by setting the appropriate bits to 1 in the bitmap. When resynchronization or restore is required, a logical OR operation between the source bitmap and the target bitmap is performed. The bitmap resulting from this operation references all blocks that have been modifi ed in either the source or replica.
This enables an optimized resynchronization or a restore operation because it eliminates the need to copy all the blocks between the source and the replica. The direction of data movement depends on whether a resynchronization or a restore operation is performed.
If resynchronization is required, changes to the replica are overwritten with the corresponding blocks from the source. If a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.
If a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.

Tracking Changes

Comparison of local replication technologies

Local Replication in a Virtualized Environment
Typically, local replication of VMs is performed by the hypervisor at the compute level. However, it can also be performed at the storage level using array-based local replication, similar to the physical environment. In the array-based method, the LUN on which the VMs reside is replicated to another LUN in the same array. VM Snapshot captures the state and data of a running virtual machine at a specifi c point in time. The VM state includes VM files, such as BIOS, network confi guration, and its power state (powered-on, powered-off, or suspended). The VM data includes all the files that make up the VM, including virtual disks and memory. A VM Snapshot uses a separate delta file to record all the changes to the virtual disk since the snapshot session is activated. Snapshots are useful when a VM needs to be reverted to the previous state in the event of logical corruptions. Reverting a VM to a previous state causes all settings confi gured in the guest OS to be reverted to that PIT when that snapshot was created. There are some challenges associated with the VM Snapshot technology. It does not support data replication if a virtual machine accesses the data by using raw disks. Also, using the hypervisor to perform snapshots increases the load on the compute and impacts the compute performance.

Remote Replication

Synchronous remote replication – writes must be committed to the source and remote replica (or target), prior to acknowledging “write complete” to the host. Additional writes on the source cannot occur until each preceding write has been completed and acknowledged. This ensures that data is identical on the source and replica at all times. Further, writes are transmitted to the remote site exactly in the order in which they are received at the source. Therefore, write ordering is maintained. If a source-site failure occurs, synchronous remote replication provides zero or near-zero RPO. However, application response time is increased with synchronous remote replication because writes must be committed on both the source and target before sending the “write complete” acknowledgment to the host. The degree of impact on response time depends primarily on the distance between sites, bandwidth, and quality of service (QOS) of the network connectivity infrastructure.

Synchronous replication

In asynchronous remote replication, a write is committed to the source and immediately acknowledged to the host. In this mode, data is buffered at the source and transmitted to the remote site later. Asynchronous replication eliminates the impact to the application’s response time because the writes are acknowledged immediately to the source host. This enables deployment of asynchronous replication over distances ranging from several hundred to several thousand kilometers between the primary and remote sites.

Asynchronous replication

Below are the bandwith requirement for both:

Bandwidth requirement for synchronous replication

Bandwidth requirement for asynchonous replication

Asynchronous replication implementation can also take advantage of locality of reference (repeated writes to the same location). If the same location is written multiple times in the buffer prior to transmission to the remote site, only the final version of the data is transmitted. This feature conserves link bandwidth.

Remote Replication Technologies

Host-Based Remote Replication

LVM-based remote replication: performed and managed at the volume group level. Writes to the source volumes are transmitted to the remote host by the LVM. The LVM on the remote host receives the writes and commits them to the remote volume group.
LVM-based remote replication supports both synchronous and asynchronous modes of replication. LVM-based remote replication is independent of the storage arrays and therefore supports replication between heterogeneous storage arrays.
The replication process adds overhead on the host CPUs. CPU resources on the source host are shared between replication tasks and applications. Because the remote host is also involved in the replication process, it must be continuously up and available.

LVM based remote replication

Host-Based Log Shipping
Database replication via log shipping is a host-based replication technology supported by most databases. Transactions to the source database are captured in logs, which are periodically transmitted by the source host to the remote host. The remote host receives the logs and applies them to the remote database.
RPO at the remote site is fi nite and depends on the size of the log and the frequency of log switching. Available network bandwidth, latency, rate of updates to the source database, and the frequency of log switching should be considered when determining the optimal size of the log file. Host-based log shipping requires low network bandwidth because it transmits only the log fi les at regular intervals.

Host based log shipping

Storage Array-Based Remote Replication

Synchronous replication mode
To optimize the replication process and to minimize the impact on application response time, the write is placed on cache of the two arrays. The intelligent storage arrays destage these writes to the appropriate disks later.
If the network links fail, replication is suspended; however, production work can continue uninterrupted on the source storage array. The array operating environment keeps track of the writes that are not transmitted to the remote storage array. When the network links are restored, the accumulated data is transmitted to the remote storage array. During the time of network link outage, if there is a failure at the source site, some data will be lost, and the RPO at the target will not be zero.

Array-based remote synchronous replication

Asynchronous replication mode Fig 12-8
Data is buffered at the source and transmitted to the remote site later. The source and the target devices do not contain identical data at all times. The data on the target device is behind that of the source, so the RPO in this case is not zero. Asynchronous replication writes are placed in cache on the two arrays and are later destaged to the appropriate disks. Some implementations of asynchronous remote replication maintain write ordering. A timestamp and sequence number are attached to each write when it is received by the source. Writes are then transmitted to the remote array, where they are committed to the remote replica in the exact order in which they were buffered at the source. This implicitly guarantees consistency of data on the remote replicas.

Array-based asynchronous replication

Disk-buffered replication mode: a combination of local and remote technologies. A consistent PIT local replica of the source device is fi rst created. This is then replicated to a remote replica on the target array.
At the beginning of the cycle, the network links between the two arrays are suspended, and there is no transmission of data. While production application runs on the source device, a consistent PIT local replica of the source device is created. The network links are enabled, and data on the local replica in the source array transmits to its remote replica in the target array.

Disk buffered remote replication

Network-based Remote Replication

CDP remote replication
Fig 12-10

Three site replication

Cascade/Multihop: data fl ows from the source to the intermediate storage array, known as a bunker, in the fi rst hop, and then from a bunker to a storage array at a remote site in the second hop. Replication between the source and the remote sites can be performed in two ways: synchronous + asynchronous or synchronous + disk buffered. Replication between the source and bunker occurs synchronously, but replication between the bunker and the remote site can be achieved either as disk-buffered mode or asynchronous mode.

Three-site remote replication cascade/multihop

Triangle/Multitarget: data at the source storage array is concurrently replicated to two different arrays at two different sites. The source-to-bunker site (target 1) replication is synchronous with a near-zero RPO. The source-to-remote site (target 2) replication is asynchronous with an RPO in the order of minutes. The distance between the source and the remote sites could be thousands of miles. The key benefit of three-site triangle/multitarget replication is the ability to failover to either of the two remote sites in the case of source-site failure, with disaster recovery (asynchronous) protection between the bunker and remote sites.

Three-site replication triangle/multitarget

Data migration solutions
Data mobility refers to moving data between heterogeneous storage arrays for cost, performance, or any other reason. It helps implement a tiered storage strategy.
Data migration refers to moving data from one storage array to other heterogeneous storage arrays for technology refresh, consolidation, or any other reason. The array performing the replication operations is called the control array.
Data migration solutions perform push and pull operations for data movement.
These terms are defined from the perspective of the control array. In the push operation, data is moved from the control array to the remote array.
The control device, therefore, acts like the source, while the remote device is the target.
In the pull operation, data is moved from the remote array to the control array.
The remote device is the source, and the control device is the target.

The push and pull operations can be either hot or cold. These terms apply to the control devices only. In a cold operation the control device is inaccessible to the host during replication. Cold operations guarantee data consistency because both the control and the remote devices are offl ine. In a hot operation the control device is online for host operations. During hot push and pull operations, changes can be made to the control device because the control array can keep track of all changes and thus ensure data integrity.

Remote replication and migration in a virtualized environment

In hypervisor-to-hypervisor VM migration, the entire active state of a VM is moved from one hypervisor to another. This method involves copying the contents of virtual machine memory from the source hypervisor to the target and then transferring the control of the VM’s disk fi les to the target hypervisor. Because the virtual disks of the VMs are not migrated, this technique requires both source and target hypervisor access to the same storage.

Hypervisor-to-hypervisor VM migration

In array-to-array VM migration, virtual disks are moved from the source
array to the remote array. This approach enables the administrator to move VMs across dissimilar storage arrays. Array-to-array migration starts by copying the metadata about the VM from the source array to the target. The metadata essentially consists of configuration, swap, and log files. After the metadata is copied, the VM disk file is replicated to the new location.

Array-to-array VM migration

AWS Certified DevOps Engineer Exam Tips

Sat, 17 Aug 2019 23:07:59 -0400

The last 30 days have been exhausting for me. I studied hard on the new AWS Certified DevOps Engineer exam and thank goodness I passed (750 out of 1000 is required). This was the hardest professional certification I ever worked on. The exam was re-launched recently in March 2019 so there is still a shortage of information around the community. I was hoping to share my experience to help demystify this new exam.

I had a somewhat solid background to begin with, having taken the AWS Certified Solution Architect Professional exam (before the 2019 update), and worked quite a bit on CloudFormation, automation and Git. However, I still did not anticipate the exam to be this difficult until I was halfway through and already had the exam and materials paid for.

Without clear guideline on study material other than the white papers, I first skimmed through the ACloudGuru course, which helped me form a high level sense of exam coverage. Nonetheless it does not cover any topic in-depth and therefore by no means makes an essential part of my study. I checked out LinuxAcademy course and they are much more in-depth for the major topics indeed. I like the course material in Lucid chart. However, the LinuxAcademy course along does not cover everything you need to know.

What I found extremely helpful is the free training videos from the official training website. I strongly recommend the 7-hour course Exam Readiness: AWS Certified DevOps Engineer – Professional. The instructor did a great job outlining the services and knowledge areas in the assessment . The course also comes with quality sample questions with answers and explanations on what the thinkings are behind the correct answers or why some choices are obviously wrong. I went through these questions twice and feel much better at not only understanding the question, but also understanding the intent of the question.

Apart from the Exam Readiness course, other free introductory courses from the official training website are helpful as well especially for those services that you only need to know the basics. Most of those courses are 5 ~ 10 minutes long, with brief but sufficient introduction and a demo session. The other extremely helpful resource is the official practice questions. The practice exam is harder than the actual exam but they closely resemble the actual question style in the exam. Unfortunately, no answer is provided but they made me spend time finding answers across the documentations and blogs. It is worth-noting that the AWS blogs provides plenty of use cases that are covered in the questions.

When it comes to the real exam, it covers a lot more topics than its predecessor. Many questions are long and confusing. And I wish I could run a diff command to highlight the differences between choices. During exam preparation you really need to train yourself on reading efficiently. I found myself sometimes eyeball through all four choices at the same time, which get my mind scattered.

During my study, I divide all services into three categories based on the level of familiarity, and here is my list:

Category 1. Know these services very well, in and out:

AWS ElasticBeanstalk, OpsWorks Stacks, OpsWorks Chef Automate, CloudFormation, CloudWatch, CodeBuild, CodeCommit, CodeDeploy, CodePipeline, CodeStar, Lambda, API Gateway, Config, Trusted Advisor, CloudTrail, Systems Manager, Autoscaling Group in EC2, DynamoDB

Category 2. Know these services well, but not necessarily down to every single detail:

Amazon Kinesis Firehose, Kinesis Analytics, Kinesis Streams, Step Functions, Elastic Load Balancer, Secrets Manager, Serverless Application Model (SAM), Route53, RDS, Certificate Manager, ElasticSearch, ECS, ECR

Category 3. Know about these services at a high level, but do not skip any:

AWS Organization, X-Ray, GuardDuty, Macie, Inspector, Service Catalog, KMS, Batch, Athena, Single-Sign-On, Data LifeCycle Manager, CloudSearch, Health Dashboard, Glue, QuickSight, LightSail

Although I did not mention much about the white papers, I want to highlight their importance again. I would not attempt the exam without reading and understanding the required white papers.

If you aspire to take the AWS certified DevOps Engineer exam I hope this helps you a little bit. Good luck.

Application I/O Characteristics

Sat, 20 Apr 2019 22:09:00 -0400

There are many unknown factors and randomness when a solution architect determines storage requirement for an application. However, this process should still be as scientific as it can be and here are some of the important considerations regards application I/O characteristics.

Random and Sequential

	Random I/O	Sequential I/O
Description	Successive read/write operations from noncontiguous addresses – accesses that are spread across the addressable capacity of the LUN.	Successive read/write operations from contiguous addresses: one logical block address after another. In sequential I/O access, disk seek time is reduced because the read/write head moves little to access the next block.
Example	Messaging OLTP application	Data Backup

Reads and Writes

Another aspect of the I/O workload is the ratio of read I/Os to write I/Os generated by application. The sum of the read and write rate is the I/O rate (number of I/O operations per second). The application’s I/O rate is one of the important factors that determine the minimum number of disks required for application. In storage systems, cache plays an important role to improve the system performance. The table below summarizes how read I/O and write I/O interact with cache.

I/O Type	READ	WRITE
Random	Hard to effectively cache because of difficulty in predicting prefetch; Requires multiple fast disks for good performance	Caching is effective, resulting in a response time better than disk response time.
Sequential	Caching is extremely effective due to predictability of prefetch; Reads are done at cache speeds;	Caching is effective; cache is flushed quickly because entire disk stripe can be written.

Here are some typical read versus write ratio for common business applications:

OLTP: 67% reads and 33% writes
DSS (decision support, aka data warehouse or business intelligence): 80%~90% reads to data tables including frequent table scans (sequential reads)
Backup: As long as the file system is not fragmented, file-based backups are sequential

I/O Request Size

The size of I/O generated by an application may vary depending upon the type of the application. Some of the overhead to execute an I/O is fi xed. If data exists in large chunks, it is more effi cient to transmit larger blocks because a host can move data faster by using larger I/Os than smaller I/Os. The response time of each large transaction is longer than the response time for a single small transaction, but the combined service time of many smaller transactions is greater than a single transaction that contains the same amount of data.

Application	Seek Type	I/O Request Size	Proportion of I/O as writes
Microsoft Exchange	Random	32KB	Moderate to high
SAP/Oracle Applications	Random	~8KB	Depends on application
RDBMS: Data entry/OLTP	Random	Database or file system page size	Moderate to high
RDBMS: Online transaction logs	Sequential	512 byte+	High, except for archiving process
RDBMS: Temp Space	Random	Database or file system page size	Very high
Web file server	75% random, 25% sequential	4KB, 8KB, 64KB	95% read, 5% write
Web server log	100% sequential	8KB	100% write
Media Streaming	100% sequential	64KB	98% write; 2% read
OS paging	100% sequential	64KB	98% write; 2% read

Lightsail – create a WordPress site in one hour

Wed, 10 Apr 2019 01:30:44 -0400

My challenge with my just revived wordpress.com site is the plug-ins. Even paid premium users cannot install plug-ins for diagram, etc. I don’t want to upgrade to business plan so I decided to build my own.

14 years ago in university, I prototyped an intranet site using the Windows, Apache, MySQL, and PHP. It took a few weeks. Nowadays, this is referred to as WAMP stack with WordPress. AWS has a post on the best practices for hosting WordPress. However, I just need a single all-in-one server and there are some instruction on that. With Amazon Lightsail it was made a lot easier. The annual cost to host a WordPress site would be $55 in US, given that virtually nobody visits it except myself. The cost consists of:

Domain registration for $13 a year
Lightsail blueprint for $3.5 a month

There are several benefit of using this Lightsail blueprint, such as:

You can actually SSH into the EC2 instance. This is important to myself in the habit of probing into how things work;
Resources are all packaged into a fixed price plan. Remember the pesky accidental AWS charges?
You have the whole LAMP stack pre-installed, with the bitnami image for WordPress;
Start small but with scalability.

To start with Lightsail is extremely intuitive. Just click and launch:

There is a good instruction here on YouTube. Once launched successfully, there are some post configurations:

Request a static IP and assign it to the EC2 instance;
Register a domain (within Lightsail, with Route53 or any other domain registrar) and create an A record referencing the IP address;
Create a certificate and set up anto-renewal. Here is an instruction but the steps differ if DNS is managed outside of Lightsail;
Redirect http to https. Refer to the instruction from bitnami;
Export from wordpress.com and import the xml into this little site.

Notice that all services relevant to hosting are packaged into a single service called bitnami. In order to restart service, instead of restarting php, httpd and mysql individually, you can run the following:

# service bitnami stop
# service bitnami start

In fact, in /etc/rc5.d or /etc/init.d, I do not find the individual services for httpd or mysql. There is actually an instance of MySQL database in the Instance:

# mysql -u root -p
mysql > show databases;

There you go! Now the site is live. This little instruction will come in handy for rebuilding when this site is blown away 🙂

Migrate to new LightSail instance

I have been using the same LightSail instance for 3.5 years and it’s been stable. By Dec 2022, the version of PHP (7.2.17) has been outdated and I decided to launch a new LightSail instance based on new version of image, which comes with newer version of PHP (8.1.13). Here is a note of my steps:

Create a new LightSail instance with the same SSH key, assign a new static IP to it. As soon as the new instance is created, we can access it by IP on HTTP, using the initial credential provided.
We need the latest version of All-in-One WP Migration plugin to perform the migration. Ensure its latest version is installed on both old and new instance.
Export site to file using All-in-One WP Migration plugin. It should produce a single file with wpress extension. For this site, the site is 450MB by Dec 2022
For the new site, try to upload the file and notice that the max size allowed is 80MB. I made the following changes:
- Change PHP attributes. Edit /opt/bitnami/php/etc/php.ini at the following attributes:
  - post_max_size = 512M
  - upload_max_filesize = 512M
- Change WordPress attributes. Create /opt/bitnami/wordpress/.htaccess with the following attributes:
  - php_value upload_max_filesize 512M
  - php_value post_max_size 512M
  - php_value memory_limit 512M
  - php_value max_execution_time 0
  - php_value max_input_time 300
- Restart services related to wordpress
Now upload the .wpress file and it should take less than 5 minutes to upgrade. After the process, the new site is imported. Delete useless plugins.
Port the certificate files, including keys, certificate and certificate chains. In my case, I had to edit the following files:
- /opt/bitnami/apache/conf/vhosts/wordpress-https-vhost.conf
- /opt/bitnami/apache/conf/vhosts/wordpress-vhost.conf
- /opt/bitnami/apache/conf/bitnami/bitnami.conf
- /opt/bitnami/apache/conf/bitnami/bitnami-ssl.conf
Configure HTTP->HTTPS redirect on the new server. Revisit the files above.
Change DNS record to point to the new server.
Browse the site, check page and posts and pay attention to: images, tables, code blocks.
One common issue is images do not display. Check the image URL. The URL might have been replaced by bad values during import. Use a plugin (e.g. Go Live Update Urls) to correct the URLs in tables.
Use a plugin to find out broken links such as image, etc and fix the broken links. This should be an ongoing maintenance activity but is particularly worth doing on newly migrated site.

Other little things

There are many little things to fix. For example, I need to give transparent images white background. This is because when the site’s background is black and the transparent area in the diagram will be black too, making diagrams (authored in white background) illegible. To do this, I have to add the following section to the additional CSS setting of the active theme:

WordPress is dynamic site and if you don’t edit the site often, and don’t want to start a static site from scratch (e.g. using Hugo or Jekyll frameworks), you can use WP2Static plugin. Make sure to configure S3 bucket and CloudFront accordingly.

Now, enjoy blogging.