Load Balancing on Digi Hunch

GraphQL and gRPC

Fri, 07 Oct 2022 14:49:00 -0400

Big Picture

For inter-process communication at a high level, the two styles are asynchronous and synchronous styles:

Asynchronous event-driven style: involving an event broker as a middle man.
Synchronous request-response style: including several families of technologies:
- RPC (Remote Procedure Call):
  - CORBA (Common Object Request Broker Architecture)
  - Java RMI (Remote Method Invocation)
- SOAP (Simple Object Access Protocol)
- REST (Representational State Transfer)
- gRPC
- GraphQL
- Apache Thrift

RPCs, built on top of TCP/UDP, are usually complex to implement. SOAP improved it and can operate on HTTP. Many large companies today still used SOAP for message exchange. However, it has a limitation with complex format and specifications for XML messaging, giving rise to REST. REST is not a standard, but rather a loosely defined architectural style. Its payload can be in any format (XML, JSON, etc) specified in the header. As long as the API conforms to certain guidelines (criteria outlined in this article), we can consider the API RESTful. REST has been steadily replacing SOAP in the past few years. In this post, I start with REST, then dive into gRPC and GraphQL.

REST

The de facto method of building microservices using REST architectural style is use HTTP protocol with JSON format payload. JSON format is human readable, but not optimized for machine-to-machine communication. So there is some room for compression. To emulate a web request in REST, one can use curl, a common utility to emulate any HTTP client.

REST has its shortcomings. For example, the interface between REST client and server is not strongly typed. You can choose to use OpenAPI/Swagger specification to define types but it is still not tightly integrated. There is no enforcement on the format of the payload either. RESTful services are quite bulky, inefficient, and error-prone. gRPC and GraphQL emerged to address different challenges with REST. GraphQL operates on HTTP and we can view it as a layer on top of REST in a broad sense. gRPC on the other hand, operates on HTTP /2, and thereby inherits many advantages from it.

HTTP /2

HTTP/2 is the second major version of HTTP. It overcomes some issues with HTTP/1.1 on security, speed, etc. This post is a good rundown of the difference between HTTP /2 and HTTP /1.1, which account for many of the advantages of gRPC. A thorough discussion on the differences between the two HTTP versions is beyond what this post can cover. One of the important difference with HTTP/2, is that all communication between a client and server is performed over a single TCP connection that can carry any number of bidirectional flows of bytes. This makes gRPC a high-performance RPC framework. In HTTP/2, the key concepts to understand are:

Stream: a bidirectional flow of bytes within an established connection. A stream may carry one or more messages;
Frame: the smallest unit of communication in HTTP/2. Each frame contains a frame header, which at a minimum identifies the stream to which the frame belongs.
Message: a complete sequence of frames that map to a logical HTTP message that consists of one or more frames.

The request message is always triggered by the client. During the interaction, the client and server break down the message into frames, interleave them, and then reassemble them on the other side. In this way HTTP /2 multiplex the messages, and enables the following communication patterns:

Simple RPC: a single request and a single response in the communication;
Server Streaming RPC: a single request and message followed by multiple response messages;
Client streaming RPC: client sends multiple messages and the server replies with one response message;
Bi-directional RPC: client setups connection by sending header frames. Once connection is established, both client and server send messages simultaneously without waiting for the other to finish;

The streaming communication patterns fundamentally improves performance, enabling the duplex streaming capability for gRPC.

gRPC

At the protocol level, gRPC has the following advantages over REST:

well-defined service interface and schema
strongly typed data
duplex streaming (thanks to HTTP2)
built-in commodity features (e.g. authentication, encryption, resiliency, service discovery, etc)

I previously touched on gRPC protocol in the context of Envoy and etcd. Envoy makes use of gRPC for its control plane, where it fetches configuration from management server(s) and in filters, such as for rate limiting or authorization checks. Etcd store implements gRPC protocol for client utility to communicate with.

Since the typical use case of gRPC is internal communication, I have not been able to find a playground online. To get a taste of gRPC client-server interaction, just play with etcd on MacOS. gRPC is language neutral. To start developing, refer to the tutorial in different languages (e.g. Golang, Python). When developing a gRPC application, the first thing to do is define a service interface in IDL (interface definition language). gRPC uses protocol buffers as the IDL to define the service interface. Protocol buffers are a language-agnostic, platform-neutral, extensible mechanism to serializing structured data. Using that service interface definition, we can generate the server-side code known as a server skeleton. Also you can generate the client-side code, known as a client stub. The methods that you specify in the service interface definition can be remotely invoked by the client side as easily as making a local function invocation.

gRPC also has some disadvantages. Currently, the ecosystem is still small. When we have a service interface, we have to maintain the interface across versions.

GraphQL

In most of the use cases of gRPC and GraphQL, GraphQL works for external-facing services/APIs while internal services backing the APIs are implemented using gRPC. Honeypot created good documentary for GraphQL available here.

We use GraphQL for external facing services because it gives API client the ability to query. An SQL query allows client to filter requested data based on conditions, and define the interested columns in the data return. This capability is missing in the REST style guideline. One may choose to implement their RESTful service to support their client’s query requirement, GraphQL standardizes this capability with typed data, thereby prevents unnecessary network round trips, over- and under-fetching of data.

The official site is a good reference for learning. One needs to know concepts around queries and mutations, schema and types (scalars, variable, fragment, interfaces, unions) to understand how query works.

This page lists a number of publicly available services in GraphQL. For example, country information service is available here. On the web page you can put in a query like this:

query myCountry {
 countries(filter:{code:{in:"CA"}}) {
 code,
 name,
 capital
 }
}

The query (with the name of myCountry) above asks to return entries with countries as type. Then it filters the result by the condition that the country code must include “CA”. The return should include code, name and capital columns. The web page looks like this:

GraphQL is on HTTP, so I can emulate the call with curl and get the same result:

curl --request POST https://countries.trevorblades.com/ \
--header 'Content-Type: application/json' \
--data-raw '{
 "query" : "query myCountry { countries (filter: { code: { in: \"CA\" } }) {code name capital} }"
}'

The return prints the same result:

The document has a page on library options for different languages, such as Graphene and Ariadne for Python. You can use Ariadne and Flask to build GraphQL API as this post suggests.

The example on this page has a better example that highlights how GraphQL allows client to ask multiple questions at the same time, and how the response includes different pieces of answers. GraphQL backend often needs to connect to different downstream system. This requires a proxy service to connect to disparate data sources. Examples of such services with such capabilities include Apollo, AWS AppSync, or ApiGee.

Summary

For API protocols, we often compare among REST, gRPC and GraphQL. REST is loosely defined and widely adopted for the past few years. The use case of REST diverged into two areas: external facing API and internal service-to-service communication. gRPC and GraphQL are relatively new and they each are good for one of the use cases.

Microservice pattern based on gRPC and GraphQL (source: “gRPC up & running” by Indrasiri & Kuruppu)

The gRPC protocol on HTTP2 is often used in internal communication between microservices. GraphQL offers query capability and is therefore often used to face external client.

Service Proxy – from Nginx to Envoy

Wed, 09 Mar 2022 21:36:00 -0400

Update (Nov 20, 2022): 1. Envoy’s configuration schema can be hard to get used to. It is lacking examples because the documentation is mostly generated. Use its examples directory to find real-life configuration examples. 2. the Envoy implementation in the example project has been reverted in favour of Nginx.

Envoy proxy is the underlying technology for Istio, as well as a number of other service mesh products, such as AppMesh (AWS), Consul (Hashicorp) and OpenServiceMesh (Azure). Most of the capabilities of Isito is ultimately provided by Envoy proxy. Envoy has a page outlining its differences with similar technologies. I decided to take a look into Envoy by replacing Nginx with it.

Rate limiting and Circuit Breaker

In most SDLC, it is application developers that create backend APIs or server applications. Most developers specializes in application features, and cannot fathom all the nuances with the TCP/IP network stack. Nginx allows them to push network concerns (non-business features) to a dedicated proxy to handle the dynamics in network connection. Nginx can be configured as both a reverse proxy (handling incoming connection on behalf of the process) and a forward proxy (handling outgoing connection on behalf of the process). This is the prototype of sidecar pattern, an important idea behind service mesh. For example, when a sudden increase in connection to the server-side application, the server process could be either unresponsive (refer to “the queuing knee“, and Little’s Law), or OOM killed. When such interruptions are not automatically recovered, a downtime is caused. This traditionally requires some congestion control strategy for TCP/IP queue but two features provided by a network proxy can help circumvent this situation: rate limiting, and circuit breaking. Rate limiting keeps more requests above threshold from entering the queue. Circuit breaker releases downstream pressure by cutting out existing in-queue request. Nginx added both over the years but Circuit breaker still remains a premium feature exclusive to Nginx Plus. Envoy on the other hand has them free when it was launched.

Envoy also supports other advanced traffic management such as traffic shaping, and mirroring. It is on top of those features that Istio introduces its own abstraction such as virtual service, destination rules to its users. In that sense, we can think of Istio as a configurator (control plane) for Envoy proxy (data plane), similar to Ansible to Nginx proxy instances.

Dynamic Configuration via API

I used Nginx previously with traditional environment and loved its flexibility. As the system grows, I started to feel the pain of management overhead. With one of the production system, there were 25 + instances of Nginx each running on a VM and I managed configuration files with Ansible. Ansible pushes out configuration files and triggers a reload from each Nginx instance. In the cloud-native era where Pods are ephemeral, this kind of overhead would snowball to an unmanageable level. Envoy was designed for cloud-native applications, with all these kinds of problems in mind. Envoy has dynamic configuration. The majority of the configurations can be pulled from xDS API, or file system. Updating configuration drains connections gracefully without runtime having to reload the file. The idea of centrally managing Nginx instances with Ansible, also evolved into the concept of control plane.

TLS origination

In the Orthweb project, I used Nginx to proxy TLS and HTTP traffic, and performed TLS termination on both ports. This is know as TLS offloading. The traffic between the proxy and the upstream service takes place in the clear, even though they do not travel across different network interfaces in most cases. For a true end-to-end encryption, it is helpful to also encrypt the traffic between proxy and upstream server. This requires the capability of securing TCP traffic to upstream server. With Nginx, the ability to secure HTTP traffic to upstream server is offered in open-source. The ability to secure TCP traffic is available in Nginx Plus, or with self-compiled binary. In Envoy, both are available using the UpstreamTlsContext configuration.

More pros

In another post, I also discussed Nginx as a LDAP proxy to front services such as Kibana and Nifi. It requires a proxy service (ldap-auth in this case), to defer auth to third party. Envoy has this capability using a filter with extension for external authorization. Istio also exposes this capability, an enabler for the configuration proposed in my previous post. Envoy also uses WebAssembly for its extensibility.

Another useful feature is protocol detection. It can use filters to detect protocol (TLS or regular TCP) and route traffic to predefined destination.

Performance wise, this benchmark from 2018 ran a comparison among the popular options where Envoy leads by a margin.

Observability (logging, metrics and tracing) are well supported in Envoy. User can configure format of logs that takes effect immediately. There are many metrics that works with Prometheus and they are expandable using filters. On the tracing side, Envoy supports integration with jaeger, zipkin and datadog.

Basics of Envoy

The configuration of Envoy is more involving. There is an Envoy course by Tetrate, as well as two blog entries for envoy 101: Envoy as gateway proxy and File-based dynamic configuration. Another good way to get started is the Sandboxes projects, which covers a number of different areas of configuration. The admin port (by default at port 9901. Istio’s default is 15000) provides helpful information. If we need to turn on debug on some features, we can do so with curl:

curl -X POST http://localhost:9901/logging?client=debug

Stats are exposed at the same port:

curl -X GET http://localhost:9901/stats

When packets are received at a listener, the is first processed by listener filters. Then, depending on filter match, one or more network filter chains will further process the packet, including further actions, as illustrated below:

Envoy supports dynamic configuration, which uses a set of discovery services (xDS) APIs. Some of the important xDS APIs include:

LDS (Listener Discovery Service) – allows you to add listeners dynamically while Envoy is running
RDS (Route Discovery Service) – allows you to dynamically update routes for HTTP connection managers
CDS (Cluster Discovery Service) – allows you to update cluster definitions dynamically
EDS (Endpoint Discovery Service) – allows you to add or remove endpoints dynamically
Secret DS

The relation can be illustrated in this diagram below:

This post from Tetrate has more examples.

Nginx to Envoy

For the advantages of Envoy, I decided to migrate from Nginx to Envoy on my Orthweb project. Using Envoy as service proxy is not where Envoy is mostly used (as sidecar), but it is how Envoy was originally used at Lyft to replace ELB in 2015.

The original Nginx configuration was referenced in this old blog post. The Envoy setup also covers both TCP (DICOM) and HTTP (HTTPS) traffic. For HTTP traffic, it also encrypts the traffic to upstream. Below is what it looks like:

admin:
 address:
 socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
 listeners:
 - name: https_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 443
 filter_chains:
 - filters:
 - name: envoy.filters.network.http_connection_manager
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
 codec_type: AUTO
 stat_prefix: ingress_http
 route_config:
 name: local_route
 virtual_hosts:
 - name: app
 domains:
 - "*"
 routes:
 - match:
 prefix: "/"
 route:
 cluster: service-https
 http_filters:
 - name: envoy.filters.http.router
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 - name: dicomtls_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 11112
 filter_chains:
 - filters:
 - name: envoy.filters.network.tcp_proxy
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
 stat_prefix: downstream_cx_total
 cluster: service-dicomtls
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 validation_context:
 allow_expired_certificate: true
 trusted_ca: {"filename": "/etc/ssl/certs/site.pem"}
 require_client_certificate: false
 clusters:
 - name: service-https
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-https
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 8042
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
 - name: service-dicomtls
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-dicomtls
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 4242
layered_runtime:
 layers:
 - name: static_layer_0
 static_layer:
 envoy:
 resource_limits:
 listener:
 https_listener:
 connection_limit: 1000
 overload:
 global_downstream_max_connections: 5000

From Nginx to Envoy, to achieve nearly the same functionalities, it takes 100 lines of configuration instead of less than 30. The configuration also appears more abstract, which is one of the cons of Envoy to consider before the migration.

Update from Jan 2025 – To align with what most uses in Orthanc community, the project switched back to using Nginx. The envoy configuration is kept here.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

From Microservice to Service Mesh

Thu, 25 Nov 2021 00:31:19 -0400

We all know what microservice is now but how does service mesh assist with microservice development.

Microservice

Microservice as an architecture was firstly conceptualized in this article by Martin Fowler in 2014. It covers the pros (strong module boundaries, independent deployment, technology diversity) and cons (dealing with distributed system, eventual consistency, operational complexity). The reality is, many teams develops their product with the microservice architectural pattern.

The implementation of microservice architecture involves a lot of programming patterns, and tools. The creation of these patterns and tools are usually done in a separate dedicated project so developers can focus on business logics. When building software, developers only need to interact with libraries and frameworks. Libraries (e.g. log4j) provides dependencies, and developers needs to write code to call the libraries. On the other hand, frameworks (e.g. Spring, Flask) not only provides tools, but also implements a pattern. It addresses a set of common problems such as authentication, expose http service, logger and database connectivity. Once set up, the framework will call the code that developers write (unlike in libraries).

Spring – an example of Microservice framework

When it comes to microservice, a well-known appliction framwork is the Spring framework. It solves problems such as:

Application context and dependency injection (for Inversion of Control, or IOC)
Database access and transaction management
Expose rest APIs (using spring MVC)

There’s an entire ecosystem of projects under Spring framework. This framework is a huge system requiring a lot of configuration efforts. This is where Spring Boot helps.

Spring Boot makes it easy to create stand-alone, production-grade Spring based applications that you can just run. It features the “convention over configuration” paradigm to save programmers from boiler plate configuration. SpringBoot gives you a standalone application ready to run without complicated deployment steps.

Managing configuration in property files does not scale in the time of microservice. Spring Cloud provides configuration as a service (in line with everything else microservice framework). It doesn’t necessarily have to be hosted in the cloud. It is comparable to Apachee Zookeeper, Etcd (distributed key value store), Hashicorp Consul and Netflix OSS (Eureka, Ribbon, Hystrix). You can pull from Git repo. The mission of Spring Cloud is to eliminate boilerplate associated with distributed systems problems for Spring Boot applications.

Many developers use Spring Boot along with Spring Cloud to build microservices. This page contains a diagram for such architecture.

In this architecture, Spring cloud helps with service discovery, traffic routing, circuit-breaking, distributed tracing and monitoring. It can also act as API gateway (in place of Nginx).

API Gateway and API Management

Microservices relies on API. Let’s distinguish API gateway, and API management (this long post has some good information).

API Gateway is a microserivce pattern. The idea is a single point of entry for all clients. The API gateway either proxy an incoming request to the appropriate service, or it may fan out a request to multiple services. The other important aspect is the API gatway can expose a different API for reach client.

API Gateway Pattern

A variation of this pattern is the Backends for frontends pattern, where it defines a separate API gateway for each kind of client. The API Gateway may authenticate user and pass an Access Token containing information about the user to the services. It may use a circuit breaker to invoke services. To summarize, the key functions of an API Gateway in this pattern is:

Unified entry point for multiple API implementations
Protocol transformation
Request morphing
Client specific logics

This page from Azure has a good comparison between API Gateway pattern vs direct connection between client and microservice. Note that API Gateway can also refers to API Gateway product, which implements the functions above. For example

Spring Cloud Gateway
Solo.io Gloo
Netflix Zuul

API management acts as a proxy for an existing API implementations. Typical functions include:

AuthN and AuthZ
Service Discovery Ingreation
Load balancing (e.g. L7 path based routing)
Logging, tracing (track user), correlation
Response Caching
Retry policies, circuit breaker, QoS
Enforce policy
Track usage and monetization
metrics (duration)
rate limiting and throttling
Request morphing (header, query string and claims transformation)
IP whitelisting

API management are usually implemented as tightly controlled shared infrastructure owned by either a “platform team”, “integration team”, or other API infrastructure teams. Examples of API management product (including SaaS) are:

Google Cloud Apigee
Mulesoft
Kong

In real world, people often use API management produce and API gateway product interchangeably.

Service Mesh

Many dub Service Mesh the next generation of Microservice. So what is the relationship between microservice and service mesh. The Microservice architectural pattern creates the need for API gateway pattern. To address this pattern, the API Gateway products first emerged. Service mesh emerged later. Service mesh and API gateway have a common set of features.

In this presentation (a tale of two frameworks) from 2018 (early days of service mesh), two teams discussed microservice (spring cloud) and service mesh (istio) approaches. There is a slide about when to use which. Many teams since have moved to Service Mesh for feature richness. This is a case study from 2021.

There are a number of service mesh technologies, such as Consul, Isito and Linkerd. Here is a comparison chart. Even though all those projects are open-source, there is some competition already. Linkerd is the first to bring up the concept of service mesh in this blog. It also purportedly has better performance than Istio. However, it does not use Envoy proxy. Istio is good at marketing. It has higher adoption rate and is feature rich. However, Google did not donate Istio project to CNCF as many expected. Instead, it created its own governing body, the Open Usage Commons. The Istio is not an open-governance project, which potentially diverge from CNCF in the future [1]. Hashicorp Consul initially was built for service discovery and distributed key/value store. It supports Kubernetes and VM. However, it still lacks observability features.

Apart from the three major technologies, other players tries to push for standardization of service mesh. The most influential initiative is the SMI (service mesh interface), pushed by Microsoft. The idea is a separation of standard and implementation, so late players will have a chance. OpenServiceMesh is Microsoft’s reference implementation of SMI. The SMI is something to watch for but it remains pretty week thus far. Google’s platform has Anthos Service Mesh which is a commercial distribution based on Istio. AWS has its own AppMesh technology, also using Envoy proxy.

[1] Update from Apr 25, 2022 – Istio applied to become CNCF project.

Istio Lab – Ingress and Egress

Tue, 16 Nov 2021 21:28:00 -0400

Istio is a popular open-source service mesh implementation using Envoy proxy. One of the benefit of using Istio is the ingress and egress it brings to native Kubernetes platform. This article is a hands-on guide to test Istio ingress and egress gateways on Minikube. It was tested on my MacBook. All the information in this post can be found in several different pages from Istio and Minikube documentation. The purpose of this article is a single cheat sheet for myself to quickly set up Istio ingress and egress testing.

Install Istio on Minikube

We start with configuring Minikube on MacBook (or change the command accordingly on other platform), and install metal LB load balancer. The steps with Istio also works with a regular Kubernetes platform.

minikube start --memory=12288 --cpus=6 --kubernetes-version=v1.20.2 --nodes 3 --container-runtime=containerd --driver=hyperkit
minikube addons enable metallb

Then we need to give load balancer an IP address range before setting up the load balancer. Minikube adds an interface named bridge100 to MacOS host environment. The IP address of the host can be found via the following command:

minikube ssh "ping host.minikube.internal -c 1"

For example, my host IP address is 192.168.64.1, so I dedicate the IP range 192.168.64.64 to 192.168.64.80 to the MetalLB load balancer. Then I need to provide this range when configuring the load balancer:

minikube addons configure metallb
kubectl -n metallb-system get po

After the configuration, make sure the MetalLB pod is up. Then we can install istio to the Kubernetes cluster. The steps are completed using istioctl following the instruction, I created a quick and dirty script to put the steps together:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
ln -sfn $(ls -d -- istio-*) istio
if istioctl x precheck; then
 echo ready to install istio and lable namespace for istio-injection
 istioctl install --set profile=demo -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

The script above download the istio zip file, unzips it, installs Istio (in demo mode and set outboundTrafficPolicy to REGISTRY_ONLY), labels default namespace for proxy injection. To use istioctl without spelling out full path, we can run the following command again separately outside of script:

export PATH=$(realpath istio/bin):$PATH

To validate, check istioctl version command.

Install Sample application

For simplicity, the rest of this article uses httpbin to test Istio’s ingress and egress. For completeness, this section gives the steps to install bookinfo application as it is used in other Istio testing as well. It also help understand how to ingress to an application with Istio using an IP provided by the load balancer. Skip this section if just need to test with the simple httpbin application. The Bookinfo application comes within the istio 1.12 directory. In the following steps from the istio directory, we install some observability add-ons as well as the bookinfo application. Then we test access to the website.

kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/kiali.yaml
kubectl apply -f samples/addons/jaeger.yaml
kubectl apply -f samples/addons/grafana.yaml

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS productpage:9080/productpage | grep -o "<title>.*</title>"

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
curl -s "http://${GATEWAY_URL}/productpage" | grep -o "<title>.*</title>"

On my laptop, the value of $GATEWAY_URL is 192.168.64.64:80. Therefore I can browse to site http://192.168.64.64:80/productpage with local browser. We can also check kiali dashboard:

istioctl dashboard kiali

Then we are good to examine ingress and egress.

Prep Work

The following custom resources are related to traffic management in istio:

Virtual Service (vs)
Destination Rule (dr)
Service Entry (se)
Gateway (gw, Istio’s ingress or egress)

We will expand on the role of each in a separate article. To execute command from within a Pod, we also use sleep application which is essentially a Pod with a sleeper thread for the purpose allowing user to run troubleshooting command. The sleep application is provided in istio directory:

kubectl apply -f samples/sleep/sleep.yaml
kubectl get po -l app=sleep

Find out the Pod name to execute command. Then we use httpbin website to emulate an external service, and use the sleep Pod (suppose the name is sleep-557747455f-q99cz) to access the service:

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/headers

As we previously set outboundTrafficPolicy.mode=REGISTRY_ONLY, this step is expected to output nothing. If this is not the case, it means the cluster may have ALLOW_ALL as outbount traffic policy, and you will need to follow this guide to set it to REGISTRY_ONLY. With REGISTRY_ONLY, in order to allow access, we need to register a service entry, by applying the document below:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: httpbin
spec:
 hosts:
 - httpbin.org
 ports:
 - number: 80
 name: http
 protocol: HTTP
 resolution: DNS

Now we should be able to access httpbin service as it is registered:

kubectl get se httpbin -o yaml

Using the exec command above again, we should see output from the curl command.

Ingress Testing

We deploy an httpbin service in order to test ingress. The YAML declaration is prepared by Istio installation package:

kubectl apply -f samples/httpbin/httpbin.yaml

Then we deploy the ingress:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: httpbin-gateway
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - "httpbin.digihunch.com"

The selector above indicates that the gateway deployed to a Pod labeled with istio: ingressgateway. Then we create a corresponding virtual service for this gateway:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: httpbin
spec:
 hosts:
 - "httpbin.digihunch.com"
 gateways:
 - httpbin-gateway
 http:
 - match:
 - uri:
 prefix: /status
 - uri:
 prefix: /delay
 route:
 - destination:
 port:
 number: 8000
 host: httpbin

Note that the hosts setup must match the configuration in gateway. The virtual service defines two services at URIs /status and /delay to expose, and we can confirm the creation of virtual service:

kubectl get vs httpbin -o yaml

We can then test with curl command from MacOS host to verify that the ingress is working. We need to use the IP address of the service (192.168.64.64 with port 80 as previously indicated):

curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/status/200
curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/delay/2

We use -HHost to emulate this request going to httpbin.digihunch.com (otherwise we’d have to fake DNS entry in /etc/hosts). Both command should give their return without error. The second command will give return after a delay of 2 seconds.

This validates the success of ingress setup.

Egress Testing

To visit an external service, there are four options:

Just access external service from Pods, when you have global.outboundTrafficPolicy.mode=ALLOW_ANY
Use ServiceEntry (as illustrated above in Prep Work)
Bypass envoy proxy (not recommended, because there is no role of service mesh)
Configure an Egress Gateway

In the Prep Work, we’ve seen how to use Service Entry. However, there is no control of how Pods access the external service. We are going to configure an Egress Gateway to manage the outgoing traffic flow. This allows you to manage route, and apply observability features to the outgoing traffic, something the Service Entry alone will not provide. The egress gateway have two typical use cases as explained on Istio website.

First, we want to confirm that the egress pod exists, with a quick command:

kubectl get po -n istio-system -l app=istio-egressgateway && kubectl get se

It should exist as we installed istio as demo version. We then also need a service entry, as we have already configured in Prep Work. At this point, the egress gateway resource has not been created.

Now, in a separate terminal, we monitor the log of egress pod

kubectl logs -f istio-egressgateway-7d5d69dcfd-kvwxv -n istio-system

With the -f (follow) session on, if we visit external service as we did during Prep Work, there is no new entry pop up at the tail of the log. This is because the external access is still not managed by egress. Now we need to add egress, with the following resources applied:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: istio-egressgateway
spec:
 selector:
 istio: egressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - httpbin.org

Then we setup the virtual service with related rules:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: vs-for-egressgateway
spec:
 hosts:
 - "httpbin.org"
 gateways:
 - istio-egressgateway
 - mesh
 http:
 - match:
 - gateways:
 - mesh
 port: 80
 route:
 - destination:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subset: httpbin
 port:
 number: 80
 weight: 100
 - match:
 - gateways:
 - istio-egressgateway
 port: 80
 route:
 - destination:
 host: httpbin.org
 port:
 number: 80
 weight: 100

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: dr-for-egressgateway
spec:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subsets:
 - name: httpbin

With all applied, we can repeat the access command using curl, and watch for the log tail.

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/ip

The log should output a line saying curl command access, which indicates that this outbound access is managed by istio egress gateway. The log line looks like below and in the example the IP address 10.244.2.4 is the Pod IP address.

[2021-11-22T01:22:47.547Z] "GET /ip HTTP/2" 200 - via_upstream - "-" 0 45 818 816 "10.244.2.9" "curl/7.80.0-DEV" "9ec0a193-77f0-90a5-b132-21c3259307da" "httpbin.org" "3.216.167.140:80" outbound|80||httpbin.org 10.244.2.4:42372 10.244.2.4:8080 10.244.2.9:52710 - -

Looking at the virtual service declaration, you might noticed multiple routes in the virtual service. The first route (mesh) tells sidecars to route request to egress gateway. The second route (istio-egressgateway) tells egress gateway to route traffic to external service as destination.

In real life, there’s more to consider. For example, we usually want the host to allow on egress to be a wild card host such as *.api.google.com. While the hosts field in virtual service can be wild card, the route destination in virtual service has to be a list of fully qualified domains. The other situation is the destination website may issue an HTTP redirect to a different host, which also needs to be registered as a destination. These details can add up to administrative effort.

Egress example

An other real life scenario is the use of authorization policy. Authorization policy rules use SPIFEE identity of a Pod. If a Pod initiates egress connection to destination host in TLS, then the Pod’s SPIFFEE identity and its identity for external TLS connection are in conflict.

With Azure AKS

The steps above also works on other cloud platform such as AKS. This is an AKS cluster that I tested the steps. The Terraform code provisions a private cluster, as well as a Bastion Host. From the Bastion host, we can perform the same steps as above.

When Istio gets installed, it will created a Service object called istio-ingressgateway in the istio-system namespace. The service is of load balancer type and by default will create a public load balancer in Azure AKS.

The test steps will also work for private load balancers. To tell Azure to create private load balancer on a specific subnet, the Service object needs to include annotation. We have to customize Istio installation in order to do that, using the following overlay file (azoverlay.yaml):

apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
metadata:
 namespace: istio-operator
 name: custom-istiocontrolplane
spec:
 profile: demo
 values:
 gateways:
 istio-ingressgateway:
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "suitable-porpoise-lb-subnet"

This is inspired by this post. Then the command to install Istio becomes:

istioctl install -f azoverlay.yaml -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

This concludes the hand-on guide for Istio ingress and egress labs. I will need to write another post explaining the related concepts.

Service and Ingress -Traffic Management in Kubernetes

Sun, 04 Jul 2021 01:30:00 -0400

Update 2022-08 – Read my latest article on ingress traffic management.

In this post we discuss the traffic management in Kubernetes, specifically on Service and Ingress objects. Let’s start with a traditional architecture:

In this traditional architecture, we run application as processes on the operating system on each virtual machine. The application process is bound to a certain ports on the operating system, and is wrapped into services (e.g. systemd). On the same virtual machine, there is also a reverse proxy service (e.g. Nginx). There are several main functional areas as listed below, and how they are fulfilled in traditional architecture:

Requirement	Detail	Typically fulfilled by
L4 Load balancing	TCP/UDP traffic routing, operating at L3 and L4	Network Load Balancer
TLS termination	Terminate TLS traffic, operating at L4	TLS termination is available in many products such as Load Balancer (L4/L7), Nginx, or the application itself.
Path-based routing	Route request based on URI path, operating at L7	Nginx, modern L7 Load Balancer.
Authentication	Integrate with external identity store, operating at L7	Nginx, modern L7 Load Balancer.

These requirements are the problems that Kubernetes needs to solve in its own architecture. They are solved by different abstraction objects in Kubernetes. Before getting to traffic management, we first need to expose an application.

Service

During traditional application deployment, we often need to organize a group of homogenous application instances as a single target for batch operation. The Pod object is an abstraction of a single application instance. The Deployment object is an abstraction of a group of homogenous Pods. The purpose of Deployment object is for Pod orchestration only. It is not designed to expose the application. To define how we want to expose an application, we use Service object.

The service object does not carry exactly the same functionalities as an operating system service. It connects to the frontend (client), as well as to the backend (server). There are two ways to connect to a backend:

To connect to Pods as backend, use selector and label; the target port is Pod’s port. This is the most common use case.
To connect to a custom backend (e.g. external database, services in different namespaces, during workload migration), define an Endpoints object (including address and port), and target the port;

On the frontend, there are several ways to expose service to client, as defined in ServiceType property. Each represents a level of exposure:

ClusterIP (default): the service gets an internal IP address in the cluster. This is the lowest level of exposure. The service is only reachable from within the cluster. This is a good choice when the service is for internal assumption, such as database.
NodePort: the service is exposed at a static port on each node. The port must be in a range pre-specified during cluster provisioning (default 30000-32767). Each node proxies traffic to that port to the service. Without a load balancer, each node is a point of entry on its own.
LoadBalancer: this option works with external load balancer in cloud deployments. The actual creation of the load balancer happens asynchronously, and information about the provisioned balancer is published in the Service’s .status.loadBalancer field. Some cloud providers allow you to specify the loadBalancerIP. The benefit Load Balancer over NodePort, is it provides a single point of entry (for each service).
ExternalName: rare use case with custom endpoint object.

Headless service

With service type ClusterIP, if you explicitly specify "None" for the cluster IP (.spec.clusterIP), the service is considered a headless service. With a headless service, a cluster IP is not allocated, kube-proxy does not handle these services, and there is no load balancing or proxying done by the platform for them. Each connection to the service is forwarded to one randomly selected backing pod. Hence the document points out that you can use a headless Service to interface with other service discovery mechanisms, without being tied to Kubernetes implementation. The behaviour differs slightly based on whether selectors are present, but both resembles DNS routing with multiple A record.

Virtual IP

Kubernetes manages service traffic with virtual IP. When clients connect to virtual IP (VIP), the traffic is automatically transported to an appropriate endpoint. Virtual IP is implemented with kube-proxy. Kube-proxy can work in three modes: userspace, iptables and IPVS. I discussed these terms in this post last year. The takeaway is that IPVS is the recommended mode.

Ingress

Ingress in Kubernetes cannot match up with a counterpart in traditional architecture. It is mainly for path-based request routing. Also, do not confuse Ingress object with Ingress rule as a policy type in Network Policy object. Ingress is a high level abstraction and should be considered over Service object when the followings are involved in the routing.

Content-based or path-based L7 routing
Multiple protocols (e.g. gRPC, WebSockets)
Authentication

Ingress usually work with service object (ClusterIP), as illustrated in Kubernetes documentation:

Also note that if you have a service other than HTTP or HTTPS, that you need to expose to the Internet, it is recommended to use a service object of NodePort or LoadBalancer type.

We call Ingress a high-level abstraction. Ingress object (aka ingress resource) itself does not expose application. It simply defines a set of routing rules. The implementation is provided by another object (Ingress Controller), who enforces the routing rules by monitoring and manage traffic using its own Service and Pods. You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect. There are a number of Ingress Controllers to choose from.

Ingress Resource

In an Ingress resource, annotations are used to configure some options, depending on the corresponding Ingress Controller. What annotation can be used depends on the the specific Ingress Controller. The backend can be either a service, or a resource. A common usage for a Resource backend is to ingress data to an object storage backend with static assets. You can define DefaultBackend for an Ingress.

Each Ingress should specify a class, a reference to an IngressClass resource that contains additional configuration including the name of the controller that should implement the class. Before the IngressClass resource and ingressClassname field were added in Kubernetes 1.8, Ingress classes were specified with a kubernetes.io/ingress.class annotation on the Ingress. This annotation was never formally defined, but was widely supported by Ingress controllers. For example, here is the annotations supported by Nginx Controllers.

Below is the yaml output of the ingress from Kubernetes documentation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: minimal-ingress
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 rules:
 - http:
 paths:
 - path: /testpath
 pathType: Prefix
 backend:
 service:
 name: test
 port:
 number: 80

Ingress Controller

Ingress Controller exists in the form of Pods, usually as daemonSet, sometimes as a deployment. The Pods listens for requests to create or modify Ingress within the cluster, and converts the rules in the manifest into configuration directives for a load balancing components. Below is all the components related to Ingress Controller:

> kubectl -n ingress-nginx get all
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-s7486 0/1 Completed 0 11d
pod/ingress-nginx-admission-patch-sjt2q 0/1 Completed 2 11d
pod/ingress-nginx-controller-5b74bc9868-6vmjc 1/1 Running 18 11d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.106.25.194 localhost 80:31774/TCP,443:31576/TCP 11d
service/ingress-nginx-controller-admission ClusterIP 10.102.38.191 <none> 443/TCP 11d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 11d

NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-5b74bc9868 1 1 1 11d

NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 9s 11d
job.batch/ingress-nginx-admission-patch 1/1 25s 11d

Ingress Controller can be implemented by load balancer resource from cloud platform, or Nginx. When you have one ingress resource and one controller, the matching is assumed. When you have multiple controllers, you need to use the mechanism from the ingress controller to ensure correct matching.

Nginx is a popular controller and there are a couple of implementations as illustrated here. Let’s take a look at Nginx Controller as an example. The troubleshooting guide states that, For each Ingress/VirtualServer resource, the Ingress Controller generates a corresponding NGINX configuration file in the /etc/nginx/conf.d folder. Additionally, the Ingress Controller generates the main configuration file /etc/nginx/nginx.conf, which includes all the configurations files from /etc/nginx/conf.d. In the Rancher ingress example above, we can check the nginx configuration with the commands below:

kubectl exec ingress-nginx-controller-5b74bc9868-6vmjc -n ingress-nginx -- cat /etc/nginx/nginx.conf | less

It is important to understand the difference between a load-balancer type service and an ingress. The documentation for ingress states that: An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer. This is because ingress operates at layer 7, so routes connections based on http host header or url path. Load balanced services operate at layer 4 so can load balance arbitrary tcp/udp/sctp services. Ingress should be backed by L7 load balancer, whereas load-balancer service should be backed by L4 load balancer.

Nginx Ingress Controller

There are several flavours of Nginx ingress controllers that cause much confusion. It is clarified on a blog post on Nginx website. To recap:

Community version: Found in the kubernetes/ingress-nginx repo, the community Ingress controller is based on Nginx Open Source, with docs on Kuberentes.io. It is maintained by the Kubernetes community with assistance from the F5 Nginx team.
Nginx version: Found in the nginxinc/kubernetes-ingress repo, the NGINX Ingress Controller is developed and maintained directly by F5 NGINX team, with docs on docs.nginx.com. It is available in two editions:
- NGINX Open Source-based
- NGINX Plus-based

There are also a number of other Ingress controller based on NGINX, such as Kong, but their names are easily distinguished. If you’re not sure which version you’re using, check the container image, then compare the image name with the repos listed above.

Load Balancer

Kubernetes by itself does not have an object for Load Balancer. The function of traditional Load Balancer is implemented through Service and Ingress objects in Kubernetes, both of which can be satisfied by a load balancer object from the cloud platform (service-managed load balancer and ingress-managed load balancer). Alternatively, you may stand up a standalone load balancer independent of the Kubernetes cluster, which is not recommended.

If your architecture is complex and you have a lot of services (e.g. using microservice), then the overhead of managing everything with Service and Ingress in Kubernetes can be significant. In that case, consider delegating these tasks to a service mesh.

Troubleshooting

There isn’t a single recipe for troubleshooting service and ingress on Kubernetes. There are some good general guide lines here and here, in addition to the guides (here and here) from official documentation. To run networking command from within the Pod network, you can launch a Pod using nicolaka netshoot image.

Bottom line

We compared service and ingress in Kubernetes. In real life, we use both, and oftentimes along with CRDs of service mesh.

Docker storage

Tue, 03 Nov 2020 20:22:00 -0400

Microservices are all about stateless and ephemeral workloads, and containers are great microservices. This may suggest that that Docker is all about ephemeral storage. In fact, Docker supports both non-persistent and persistent storage, such as database, kafka, etc.

Non-persistent storage is automatically created, alongside the container and is tied to the lifecycle of the container. On Linux system, it is /var/lib/docker/ as part of container. This is referred to as local storage.

Docker has a concept of volume, which is essentially a file or a directory. Volumes are for persistent data. they are de-coupled from containers and are not tied to the lifecycle of any container. Volume allows process in docker container to bypass the default uionFS, and stores file or directory on host machine. It also allows different containers to share data. You may mount a volume to a container. even if container is deleted, volume persists.

By default, Docker creates new volumes with the built-in local driver. Local volumes are only available to containers on the node they’re created on. There are also third-party drivers as plugins that provides advanced options to integrate external storage system with Docker. (NAS, SAN, etc)

There are more than 25 volume plugins that you can specify with -d switch, to cover all three categories of storage

Block storage tends to be high performance and good for small-block random access workloads.
File storage is high performance, shared amongs multiple containers with NFS or SMB protocols.
Object storage is good for long term storage of large data blobs that do not change frequently. It is often content addressable and relatively low performance.

Note that if you share volume with multiple containers, the application needs to worry about data collision.

You may use docker volume create command to create volume. Note that there is no quota management within docker so the partition needs to be managed at operating system level.

Implementation of Volume

Remember that Docker image is built on multi-layer file system. When we run a container, Docker places a read-write layer on top of the image, such that the active files in running container are all placed in this read-write layer. When container is deleted, so are the files. The file system in Docker is a pseudo file system implemented in unionFS. Volumes bypasses the uionFS and directly accesses the host file system.

When we create a Docker volume, Docker places the volume data to /var/lib/docker/volumes and under each directory named after volume, creates a directory _data, which is attached to the corresponding container.

You can even mount an NFS volume to container. Reference here.

We mentioned UnionFS a couple times so far. UnionFS is a light-weight, layered file system. It can mount the contents of multiple directories to the same directory, to form a single file system. User can use unionFS like a directory. It is the foundation of Docker image and container and enables saving of spaces.

There are three common types of union FS: AUFS, DeviceMapper, and OverlayFS.

AUFS file system

AUFS is the earliest driver that Docker uses for file system, most common in Ubuntu and Debian. To check if the system support AUFS, check out the documentation here.

AUFS is recommended in Ubuntu or Debian. For CentOS and Redhat, it needs to be installed and make sure the command above returns aufs. To configure AUFS, create file /etc/docker/daemon.json and add:

{
 "storage-driver":"aufs"
}

Then restart docker service. Run “docker info” and examine the Storage Driver section, as documented here.

AUFS layers multiple directories on a single Linux host and presents them as a single directory. These directories are called branches in AUFS terminology, and layers in Docker terminology. The unification process is referred to as a union mount.

Layers of a Ubuntu container

This section describes how the layers work and this section describes how it reads and writes files (Copy-on-Write (CoW) strategy to maximize storage efficiency and minimize overhead). CoW characterized AUFS.

AUFS has not been adopted in the Linux kernel mainline for lack of maintainability. So for CentOS, the recommended file system driver is devicemapper.

Devicemapper file system

Devicemapper is a technical framework to map physical block device to virtual block device, introduced since kernel 2.6.9. So it’s essentially different from AUFS. The Logical Volume Manager (LVM) in Linux is also implemented based on devicemapper.

The three critical components in devicemapper are:

mapped device: a virtual device that devicemapper provides to client
target device: the underlying physical device or a section of it.
map table: keeps track of the offset, range, etc between mapped and target devices.

Devicemapper uses target driver to block, filter, and forward I/O requests (e.g. Raid, encryption, think provisioning, etc). In thin provisioning, storage driver only assigns spaces that are needed. Docker uses snapshot technology in thin provisioning. This part of the documentation provides further details as to how device mapper works.

Ubuntu and busybox image layers

Devicemapper has to modes:

loop-lvm: in dev and test environment
direct-lvm: recommended in production

Here is the performance best practice. To configure devicemapper, create /etc/docker/daemon.json file and add:

{
 "storage-driver":"devicemapper"
 "storage-opts":[
 "dm.directlvm_device=/dev/xdf",
 "dm.thinp_percent=95",
 "dm.thinp_metapercent=1",
 "dm.thinp_autoextend_threshold=80",
 "dm.thinp_autoextend_percent=20",
 "dm.directlvm_device_force=false"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure direct-lvm mode is on.

Since devicemapper uses block device to store files, it is faster than directly operate on file system. It is adopted as default driver as unionFS for a long time, ensuring stable performance under Red Hat and CentOS.

OverlayFS file system

Earlier versions of OverlayFS (known as overlay driver) is not stable. Later version is known as overlay2, which is very stable and recommended in overlay2. It requires:

Docker version higher than 17.06.02;
Kernel version higher than 3.10.0-514 for CentOS and RHEL; or higher than 4.0 for other distributions of Linux;
Using with xfs file system with d_type turned on

In production environment, it is recommended to moutn /var/lib/docker to separate disk or partition, to prevent the directory getting full from impacting the host OS. The option pquota is recommended for mounting options in /etc/fstab.

To configure storage driver, create file /etc/docker/daemon.json, with the following content:

{
 "storage-driver":"overlay2",
 "storage-opts":[
 "overlay2.size=20G",
 "overlay2.override_kernel_check=true"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure storage driver is overlay2 and d_type is true.

The way overlay2 works is similar to AUFS, involving union mount process, with lowerdir, upperdir and merged. More details are here, including how overlay2 works with file read and file write (e.g. CopyOnWrite).

Today, overlay2 driver is officially recommended by Docker for its stability and performance, it should be used if all the conditions are met.

Docker Compose, Docker Stack and Docker Swarm

Sun, 24 May 2020 21:58:03 -0400

This posting covers some basic docker orchestration tools.

Docker Compose

Docker Compose’s predecessor is a tool called Fig developed by Orchard, which was acquired by Docker in 2014, with Fig renamed to Docker Compose. Docker Compose is the official container management tool. It is essentially a python script that parses yaml file, to make Docker API calls to manage containers dynamically. It is installed along with Docker on MacOS and Windows. On Linux, you will have to download package with curl command and install manually. Docker Compose has three versions so far and we should create new template with v3.

The Docker compose yaml template consists of three parts:

services: similar to docker run
- build: specify Dockerfile to build image
- cap_add, cap_drop: specify kernel capabilities (e.g. NET_ADMIN, SYS_ADMIN)
- command: override default startup command by container
- container_name
- depends_on
- devices: map host device to container
- dns
- dns_search:
- entryppoint: override entry point from image
- env_file: specify file that stores environment variable
- environment: specify environment variable
- image: specify the location of image
- pid: share the PID namespace with host
- ports: expose network ports. HOST:CONTAINER
- networks
- volumes: mount host volume to container
networks: similar to docker network create
volumes: similar to docker volume create

Here is a typical structure of docker compose yaml template (wordpress):

version: "3.8"
services:
 mysql:
 image:mysql:5.7
 volumes:
 - mysql_data:/var/lib/mysql
 restart: always
 environment:
 MYSQL_ROOT_PASSWORD:root
 MYSQL_DATABASE:mywordpress
 MYSQL_USER:digihunch
 MYSQL_PASSWORD:hunchdigi
 wordpress:
 depends_on:
 - mysql
 image: wordpress:php7.4
 ports:
 - "8080:80"
 restart:always
 environment:
 WORDPRESS_DB_HOST:mysql:3306
 WORDPRESS_DB_USER:digihunch
 WORDPRESS_DB_PASSWORD: hunchdigi
 WORDPRESS_DB_NAME: digihunch 
networks:
 frontend:
 backend:
volumes
 mysql-data: {}

In summary, Docker Compose is an orchestration tool for single host, typically seen in development and test environment with dependencies between services.

Docker Stack

A stack is a set of related services and infrastructure that gets deployed and managed as a unit. A docker stack file has the same format as Docker Compose file, with the only requirement that the version: key specify a value of 3.0. The other difference between Docker Stacks and Docker Compose, is that stacks do not support builds. All images have to be built prior to deploying the stack.

From the stack file, Docker first executes the network section and create networks that do not exist. Then it goes through other elements. A service is a JSON collection(dictionary) that contains a bunch of keys. The image key is the only mandatory key in the service objects, which will be pulled from Docker Hub by default. Ports key maps the port of Swarm to the port of each service replica. By default, all ports are mapped using ingress mode. This means they’ll be mapped and accesible from every node in the Swarm -even nodes not running a replica. The alternative is host mode, where ports are only mapped on Swarm nodes running replicas for the service.

The environment key lets you inject environment variables into services replica.

The secrets key defines two secrets – revprox_cert and revprox_key. These must be defined in the top-level secrets key, and must exist on the system. Secrets get mounted into service replicas as a regular file. The secrets defined in this service will be mounted in each service replica as /run/secrets/revprox_cert and /run/secrets/revprox_key, unless otherwise specified.

The volumes key is used to mount pre-created volumes and host directories into a service replica.

The networks key ensures that all replicas for the service will be attached to the front-tier network. The network specified here must be defined in the networks top-level key, and if it doesn’t already exist, Docker will create it as an overlay.

The service also defines a placement constraint under the deploy key. This ensures that replicas for this service will always run on Swarm worker nodes. Placement constraints are a form of topology-aware scheduling, and can be a great way of influencing scheduling decisions.

When Docker stops a container, it issues a SIGTERM to the process with PID 1 inside the container. The container (its PID 1 process) then has a 10-second grace period to perform any clean-up operations. If it doesn’t handle the signal, it will be forcibly terminated after 10 seconds with a SIGKILL. The stop_grace_period property overrides this 10 second grace period.”
Although you may scale a docker service as part of a stack with scale command, it is not recommended. Instead, stack file should be used as the ultimate source of truth (declarative method vs imperative method). All changes to the stack should be made to the stack file, and the updated stack file used to redeploy the app.

Docker Swarm

For multi-host cluster, Docker Swarm facilitates the deployment of micro-services. Docker Swarm is:

a cluster of Docker hosts: enterprise-grade, secure communication, PKI with automation, dynamic addition of nodes
an orchestration engine, with deployment automation, deploying native swarm apps (using Docker API) and Kubernetes apps.

A Docker nodes can be physical servers, VMs, cloud instances, etc. Nodes are configured as managers or workers. Managers look after the control plane of the cluster, and dispatches tasks to workers. Managers forms a distributed management cluster on its own, and they use Raft protocol to ensure consistency. Workers accept tasks from managers and execute them. Swarm mandatorily uses TLS to encrypt communications, authenticate nodes, and authorize roles, with Automatic key rotation.

The atomic unit of scheduling on a swarm is the service. When a container is wrapped in a service, we call it a task or a replica, and the service construct adding things like scaling, rolling updates and simple rollbacks.

To initialize a swarm, we need to have the following ports open. Then we can initialize the first manager node, join additional manager nodes, and then join workers.

2377/tcp: for secure client-to-swarm communication
7946/tcp & udp: for control plane gossip
4789/udp: for VXLAN-based overlay networks

A Docker node can exist either in single-engine mode as stand alone, or in swarm mode as part of a swarm. Service only exist in swarm mode. Running docker swarm init on a Docker host in single-engine mode will switch that node into swarm mode, create a new swarm, and make the node the first manager of the swarm. Then additional nodes can be joined as managers or workers.

Swarm managers have native support for high availability, through an active-passive, multi-manager HA. Only one manager is considered active (the leader), which is the only one that will ever issue live commands against the swarm. If a passive manager receives commands for the swarm, it proxies them across to the leader.

Managers are either leaders or followers. This is Raft terminalogy because swarm uses an impelementation of the Raft consensus althorithm to power manager HA. As to HA, the following two best practices apply:

deploy an odd number of managers
don’t deploy too many managers (3 or 5 recommended, never more than 7)

Having an odd number of managers reduces the chances of split-brain conditions. Having less than 7 managers ensures that achieving consensus is quick.

With a service, we can specify name, port mappings, network to attach to, and images, as well as desired state for an application service. It is recommended in production environment to use docker-compose template to specify service. Services have replication mode, and the default is replicated. This will deploy a desired number of replicas and distribute them as evenly as possible across the cluster. The other mode is global, which runs a single replica on every node in the swarm.

Running “docker service scale” command can scale the number of service replicas from 5 to 10, which in the background updates the service’s desired state to the newly specified number of replicas. Behind the scenes, Swarm also runs a scheduling algorithm that defaults to balancing replicas as evenly as possible across the node in the swarm. Docker makes it super easy to push updates to deployed applications. With rolling update, you may specify number of replicas to update at a time, and cool-off period per update.

Here is a great article on the difference between Docker Swarm and Kubernetes.

Nginx as a reverse proxy for Nifi web UI and Kibana

Thu, 16 Jan 2020 22:22:51 -0400

Nginx can act as a application neutral proxy. One example is to front Nifi. The nifi default configuration provides an HTTP access point, specified in the following entries in nifi.properties:

nifi.web.http.host=192.168.133.5
nifi.web.http.port=8080

Nifi can provide secure port by commenting out the lines above and provide the followings:

nifi.web.https.host=192.168.133.5
nifi.web.https.port=8083

However, it does require configuring JKS keystore for Java, as well as authentication. Customers with existing AD servers are likely to require authentication via LDAP. While Nifi does support LDAP integration according to its administration guide. The configuration is quite involving. You need to configure the identity provider, as well as authorizes. I have personally spent a couple days on this without much progress. The information in the logging isn’t to the point. Restarting nifi also is a long process, making it painful to troubleshoot.

I then moved to Nginx (open-source) as an alternative and it is quite enlightening. I already knew that the SSL termination in nginx is super easy to configure. This time I learned that the opensource community even has a support for LDAP integration. Here is a diagram of how it works:

This approach is outlined in a blog post on Nginx website. The ldap-auth daemon is implemented in Python can can be wrapped up as a systemd service. Once a client sends a request in https, security layer is terminated in nginx, and an authentication request in http is sent to the ldap-auth daemon, which proxies converts the request into LDAP searches and proxies it over to customer’s Active Directory server, for authentication. Once authenticated, the http request can make to one of the backend container or server which hosts Nifi. Below is an example of how this can be configure on RedHat.

Install python3 and python-ldap

RedHat may have both python2 and python3 pre-installed, python2 being the default. We do not want to change the default because other applications such as yum still depends on python2 as of early 2020.

yum -y install python3
yum -y install gcc python3-devel openldap-devel
pip3 install python-ldap

Once python3 is installed, pip3 will be available and we use that to install python-ldap. This is a module in Python3 that will be used by the script that act as ldap daemon.

Configure ldap-auth daemon as systemd service

In the github project for ldap-auth, download nginx-ldap-auth-daemon.py to local location such as /usr/bin, then we create nginx-ldap-auth.service in /etc/systemd/system/ with the following content.

[Unit]
Description=LDAP authentication helper for Nginx
After=network.target network-online.target
 
[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/var/run
ExecStart=/usr/bin/python3 /usr/bin/nginx-ldap-auth-daemon.py
KillMode=process
KillSignal=SIGINT
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

Then, run the following command to load, start and check nginx-ldap-auth service.

systemctl reload-daemon
systemctl start nginx-ldap-auth
systemctl status nginx-ldap-auth

This service will be up and listening to port 8888 for http traffic.

Configure Nginx

Then configure nginx with the following entries in its default.conf file, typically located in /etc/nginx/conf.d.

upstream nifibackend {
 # default: round robin
 server container1.nifi.digihunch.com:8080;
 server container2.nifi.digihunch.com:8080;
}
proxy_cache_path cache/ keys_zone=auth_cache:10m;
# nifi proxy
server {
 listen 8083ssl;
 include /etc/nginx/ssl/default.conf;
 
 location / {
 auth_request /auth-proxy;
 
 proxy_pass http://nifibackend;
 proxy_set_header Host $host:$server_port;
 proxy_set_header X-ProxyScheme https;
 proxy_set_header X-ProxyHost $1;
 proxy_set_header X-ProxyPort 8083;
 proxy_set_header X-ProxyContextPath /;
 }
 
 location /auth-proxy {
 internal;
 proxy_pass http://127.0.0.1:8888;
 
 proxy_pass_request_body off;
 proxy_set_header Content-Length "";
 proxy_cache auth_cache;
 proxy_cache_valid 200 10m;
 
 proxy_cache_key "$http_authorization$cookie_nginxauth";
 proxy_set_header X-Ldap-URL "ldaps://ldap.digihunch.com:636";
 proxy_set_header X-Ldap-BaseDN "OU=Corporate User Accounts,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindDN "CN=Digi Hunch Service Account,OU=Digi,OU=ServiceAccounts,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindPass "myownpasswordtricks";
 proxy_set_header X-CookieName "nginxauth";
 proxy_set_header Cookie nginxauth=$cookie_nginxauth;
 proxy_set_header X-Ldap_Starttls "true";
 proxy_set_header X-Ldap-Template "(&(sAMAccountName=%(username)s)(objectClass=organizationalPerson)(memberOf=CN=GH_SYSADMIN,OU=GHCO,OU=Groups,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org))";
 }
}

We need the full distinguished name of bind user to get this to work. Once configured properly, and user attempts to connect through a browser, Nginx will pop up a prompt for username and password. The username will be plugged into the X-Ldap-Template for further queries. The same HTTP header also allows you to filter by membership that the user is associated with.