Distributed Systems on Digi Hunch

Etcd – the key-value store for Kubernetes

Tue, 14 Jun 2022 00:10:00 -0400

Etcd in Kubernetes

In Kubernetes architecture, etcd is the data store. It stores the desired state of Kubernetes object. API server is the only client that connects to etcd (via gRPC protocol). Cluster builder specifies the endpoint of etcd as a parameter to the kube-api-server process. Other Kubernetes components, whether in the control plane or from the nodes, connect to API server. API server translates their request into etcd query, and then translates etcd query result into what its clients ask for. For this reason, communication with etcd accounts for a lot of network traffic in a Kubernetes cluster.

The etcd store is a CNCF project for “a distributed, reliable key-value store for critical data in a distributed system”, developed by CoreOS team. So it is essentially a distributed key-value store for any distributed application. If an application runs on Kubernetes, it can leverage etcd store, by keeping their configurations in ConfigMap and Secret objects. One key feature is to watch for specific keys or directories for changes, and react to the changes. Voila! This is the underlying mechanism for controller!

A Kubernetes cluster may have stacked etcd deployment or connect to an external etcd store.

stacked etcd architecture

external etcd architecture

In managed Kubernetes services such as EKS in AWS and AKS in Azure, users usually do not directly access etcd store. However, it is still a very important component to understand. Its use case includes:

Configuration sharing
Service discovery
Consistency
Watching mechanism
Expiry and extension of key

The consistency use case is based on Raft protocol for distributed consensus.

Raft protocol

I am not an expert in distributed consensus protocols and nor do I intent to cover it in depth. At a high level, I have heard of three of them so far:

Etcd uses Raft protocol
Zookeeper uses ZAB protocol
Cassandra uses paxos protocol

Here is a good intro to the three protocols. Instead of getting into the fine details, I would like to discuss why we need such a consensus protocol (or consensus mechanism) in distributed systems, which are also decentralized systems.

Centralized, Decentralized, Distributed systems

The reason a distributed system needs consensus protocol, is that a distributed system lacks a single source of truth as centralized systems do. Different parts of the distributed system may receive different signals but they must come to agreement of a single plan to act. Lamport studies this with an analogy of Byzantine Generals problem, and first proposed Paxos protocol. Paxos has been an important foundation to modern distributed systems. In Paxos, consensus is achieved in two phases, which creates the problem of livelocks. Raft is an alternative to Paxos, and is widely adopted today. Here is a link to an animated illustration for Raft protocol. The Raft protocol is also used in Redis. It has three roles: Leader, Candidate, and follower. ZAB protocol is similar to Raft, where it needs to select a leader.

Etcd Lab

In troubleshooting, if we suspect that the response from API server is inconsistent with etcd store, we want to directly connect to it.

Managed Kubernetes services do not expose their etcd store. We can use KinD or Minikube. There are two types of jump box to access etcd store: using etcd Pod, or SSH to a Node. To connect to etcd, we also need the X509 key, certificate and CA’s certificate, in addition to the endpoint, usually an IP with port 2389. When I connect to Pod shell, I find the command shell not easy to use. They might miss basic command such as ls, or do not support auto completion.

Take KinD for example, we first create a secret, then we can connect to the node with docker CLI command:

kubectl create ns myns
kubectl -n myns create secret generic mysecret --from-literal key1=value1
kubectl -n myns get secret mysecret -o jsonpath='{.data.key1}' | base64 -d
docker exec -it control /bin/bash

From the node,

apt update && apt install etcd-client
etcdctl version
nc -vz localhost 2379
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

export ETCDCTL_API=3
export ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379'
etcdctl member list write out=table

Now we can see the secret object directly with etcd store:

etcdctl get /registry/secrets/myns/mysecret

With get query, when using –prefix, we can use –keys-only switch to list keys without values:

etcdctl get --prefix /registry/api --keys-only
etcdctl get --prefix /registry/namespace -wjson

We can write key-value with put command:

etcdctl put myloc 0
etcdctl get myloc -wjson

In Kubernetes, all the key names start with / which makes the key looks like a POSIX path. Every Kubernetes object is stored in etcd with a unique key following a self-explanatory naming pattern. To display the path, we can also use debug log that records the call to API server:

kubectl get ns myns -v9

Look for curl command such as:

I0523 22:51:43.517728 32347 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.23.6 (darwin/amd64) kubernetes/ad33385" 'https://127.0.0.1:64081/api/v1/namespaces/myns'

From there we can see the etcd query as the URI is namespaces/myns, which we use in etcdctl query path:

etcdctl get /registry/namespaces/myns

Every type of Kubernetes object has a storage.go file in their implementation that defines how api server should write object. Here is an example for Pod object.

Etcd also supports watch command to watch for changes. For example:

etcdctl watch --prefix /registry/namespace # watch output k create ns newns

Now we create a namespace with kubectl:

kubectl create ns myns

The output from etcdctl will reflect the change. The communication between etcdctl and etcd is gRPC protocol. The output is based on stream, as we can see from the watch result.

Etcd Maintenance

Like any distributed store, etcd needs maintenance and operation work. For example, we can check endpoint status with endpoint command:

etcdctl endpoint status

We can also backup and restore etcd store with etcdctl command:

etcdctl snapshot save /tmp/backup.db

This was an question in CKA exam. In real life, when the workload scales up, the etcd store may come across many pitfalls, such as degraded performance, unresponsiveness, some etcd member going down, network partition on etcd store causing split brain. It is important to ensure efficient communication between API server and etcd store. The etcdctl provides defrag and compact commands for common maintenance activities.

Istio Operation Gotchas

Sat, 19 Mar 2022 11:09:00 -0400

In this post I discuss a few aspects when putting istio in operation.

Installation

Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to Istio’s perceived complexity.

Currently, the recommended installation methods are istioctl or Helm. Using Istio Operator manifest (with out istioctl) is discouraged. As to the Helm chart installer, it was once deprecated (around early 2020), but was later re-introduced in the fall of 2021. This back-and-forth had caused some aversion. As of Nov 2021, their re-introduced Helm charts version dropped alpha tag. The Helm repo consists of separate Helm charts, for Istio CRD (base), control plane (istiod), each gateway and CNI respectively. A typical deployment therefore requires multiple Helm Releases (example here). Istio document still considers Helm support as alpha, so I assume the most reliable method to install Istio is istioctl.

The istioctl utility can be used with many options to customize Istio install. For example, we can supply a YAML declaration input (to -f switch) to customize installation behaviours. The YAML file declares a CRD. Two different CRDs have been used: IstioOperator and IstioControlPlane. According to this blog and this post, since Istio 1.5 in early 2020, we’re supposed to IstioOperator CRD exclusively. The IstioControlPlane CRD is left only for legacy support. As stated in the current documentation:

The istioctl command supports the full IstioOperator API via command-line options for individual settings or for passing a yaml file containing an IstioOperator custom resource (CR).

With IstioOperator CRD, we still have a number of options to tweak the install behaviours. Here is a summary of potential options:

use IstioOperator API via IstioOperator CRD (without using “values” or overlay fields)
specify an attribute value in argument, including a pre-built profile e.g. –set meshConfig.accessLogFile=/dev/stdout, –set profile=demo
use K8sObjectOverlay by using “k8s/overlays/patches” field in IstioOperatorCRD
use Helm API by using “values” field in IstioOperatorCRD

The key-value specified in –set switch overrides the same key-value supplied in the IstioOperator CRD. So option 2 overrides option 1. The value for profile can also be empty if you’d rather start from scratch. However too many –set switches makes the command wordy so we should build our own IstioOperator CRD

For option 4, the document hyper-links Helm API to a section from version istio 1.4, and I appears to exist only for legacy (pre-2020 Helm support) compatibility. Option 3 (K8sObjectOverlay) would be helpful when a field cannot be conveniently customized with option 1 and we have to patch the object like in Kustomization.

So the most practical approach is IstioOperator CRD for per-component customization, potentially with K8sObjectOverlay. No matter which option, istioctl compiles the installation manifest before applying it against Kubernetes API. This manifest can be previewed using “istioctl manifest” command, so that you can take a look before installation.

Thank you Isito for so much confusion just to land on a working installation method. Below is the content of az-istio-operator.yaml file that I use for my installation:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istio-install-customization
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.1
 revision: 1-13-1
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 outboundTrafficPolicy:
 mode: "REGISTRY_ONLY"
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 ingressGateways:
 - name: istio-ingressgateway
 #namespace: istio-ingress
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 11
 minReplicas: 1
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-lb-subnet"
 egressGateways:
 - name: istio-egressgateway
 enabled: true

I can preview the install, run the install and validate installation status:

$ istioctl manifest generate -f az-istio-operator.yaml
$ istioctl install -f az-istio-operator.yaml -y --verify
$ istioctl verify-install -f az-istio-operator.yaml
$ kubectl -n istio-system get IstioOperator installed-state-istio-install-customization-1-13-1 -o yaml | less

Note that in the IstioOperator declaration I marked the revision. This is helpful when I run multiple versions of control plane (e.g. during upgrade). I can check revisions of istiod with:

$ istioctl x revision list

We can delete installed istio components

$ istioctl x uninstall -f az-istio-operator.yaml
$ istioctl x uninstall --revision 1-11-5

In practice, it is helpful to use separate operators each for a different component (pilot, ingressGateways, egressGateways). This makes maintenance and upgrade easier.

Debugging

The documentation has a page for common problems that one needs to be familiar with. It covers not only problems, but also steps to troubleshoot each kind of problem (e.g. authorization policy).

Although istioctl is pretty confusing as an installation tool, it is a good utility for many troubleshooting activities. We should probably add its path to PATH environment variable and add the export command (e.g. export PATH=”$HOME/istio/bin:$PATH“) to .zshrc or .bashrc. Istioctl has a few useful subcommands, some of which are only available as experimental and therefore needs to be following an x. Some common commands are given below:

To analyze Istio problems:

$ istioctl analyze -n istio-system

To look at proxy configuration of an Envoy instance at different levels, use proxy-config sub-command or pc for shorthand:

$ istioctl proxy-config <clusters|listeners|routes|endpoints|bootstrap|log|secret|stats>
$ istioctl pc cluster deploy/istio-ingressgateway -n istio-system # see what Envoy cluster an ingress gateway knows about
$ istioctl proxy-config log deploy/httpbin --level "rbac:debug"
$ istioctl pc log <pod_name> -n <namespace> --level connection:debug
$ istioctl pc secret -n istio-system deploy/istio-ingressgateway # check certificates loaded to a gateway
$ istioctl proxy-config listeners deploy/istio-ingressgateway -n istio-system # query envoy listener configuration
$ istioctl pc routes deploy/istio-ingressgateway -n istio-system --name http.8080 # query envoy route configuration

The last two commands set logging level to the specified workload. If we want to set logging level at mesh level, we can use these commands:

$ istioctl admin log --level authorization:debug
$ istioctl admin log

To look at synchronization status of each envoy in the mesh, use proxy-status sub-command, or ps for shorthand:

$ istioctl ps # ensure data plane is in sync

If an item for a workload shows STALE instead of SYNCED, it means that the configuration has not been pushed from control plane to that instance of Envoy proxy. Check if the Istio configuration change is valid. If the system is newly installed and there is no ingress or egress gateway resources declared, the RDS column for ingress or egress may show “NOT SENT”. The far right column displays the version of istiod connected.

To describe applied istio config:

$ istioctl describe <pod|service>
$ istioctl describe po workload1 -n my-workload # detect misconfigurations on workload

To view dashboard:

$ istioctl dashboard <envoy|grafana|prometheus>

To check authorization policy on a Pod,

$ istioctl x authz check mypod -n workload

To validate istio configuration in a file:

$ istioctl validate -f resource_authorization_policy.yaml

Sometimes we need to turn on access logging just on the envoy proxy on the Gateway Pod. In that case, we will need to apply Istio’s Envoy filter object. This filter is applied to Pods labelled as gateway. It patches the existing filter chain with the additional defined in the manifest:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
 name: hcm-gw-access-log
 namespace: istio-system
spec:
 workloadSelector:
 labels:
 istio: ingressgateway
 configPatches:
 - applyTo: NETWORK_FILTER
 match:
 context: GATEWAY
 listener:
 filterChain:
 sni: demo.digihunch.com
 filter:
 name: "envoy.filters.network.http_connection_manager"
 patch:
 operation: MERGE
 value:
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
 access_log:
 - name: envoy.access_loggers.file
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
 path: /dev/stdout
 format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The object above enables access logging on the Gateway Pods only, without impacting other Envoy proxies.

Multi-tenancy

In large enterprises, Istio is usually installed by a platform/operation team. The entire platform is shared by multiple application teams. Istio should not be seen as a responsibility of a single party. It is necessary to break down the Istio CRDs and define the responsibility of each CRD. For example:

Gateways are placed in the istio-system namespace (or a dedicated istio-ingress namespace in some case). Platform team can decide whether a Gateway is considered a shared infrastructure, or each tenant (application team) uses their own Gateway. In the servers/hosts field of Gateway declaration, add namespace before host to suggest that the routing behaviour of that host must be defined in a certain namespace.
Virtual Services can be managed in two models as well. They can be managed individually and are all placed in each tenant namespace. Alternatively in a shared responsibility model, Virtual Service can be created in istio-system namespace, and the processing of each match can be delegated to a Virtual Service in each tenant namespace, using the Delegate feature of Virtual Service. Another field that can help with multi-tenancy is the “gateways” field, you can specify a value of mesh to indicate the virtual service is available to the entire mesh.
Destination Rules are usually placed in each Tenant’s workspace
Peer Authentication also depends on the configuration. If a mesh level configuration is enforced, it is easier for the Platform Team to manage it and this should be the setup for new clusters. If for historical reasons Peer Authentication is enforced per tenant namespace, it can be delegated to each application team. It can make communication troubleshooting more complex.
Control Plane and observability workloads such as Kiali are usually the responsibility of Platform team.
Request Authentication and Authorization Policy should be the responsibility of individual application team.

In a multi-tenancy management model, Istio related resource should be subject to admission control policy, as well as scrutiny by the security and platform teams. Google Anthos has a good page on the constraint templates for Istio resources.

As to restricting traffic between namespaces, apart from the measures from this previous post, we can also use Sidecar CRD to restrict outbound traffic. As its documentation states:

Sidecar describes the configuration of the sidecar proxy that mediates inbound and outbound communication to the workload instance it is attached to. By default, Istio will program all sidecar proxies in the mesh with the necessary configuration required to reach every workload instance in the mesh, as well as accept traffic on all the ports associated with the workload. The Sidecar configuration provides a way to fine tune the set of ports, protocols that the proxy will accept when forwarding traffic to and from the workload. In addition, it is possible to restrict the set of services that the proxy can reach when forwarding outbound traffic from workload instances.

The documentation page also includes two examples. The first is a sidecar at the mesh level that restricts outbound traffic to the same namespace that the sidecar is in, and the istio-system namespace:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: istio-config
spec:
 egress:
 - hosts:
 - "./*"
 - "istio-system/*"

The second example overrides the mesh level default above, and allows egress traffic to three specified namespaces:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: prod-us1
spec:
 egress:
 - hosts:
 - "prod-us1/*"
 - "prod-apis/*"
 - "istio-system/*"

As to inbound control, we can expose a Virtual Service to other namespaces by using the exportTo field to specify which other namespaces the Virtual Service should be exported to. If no namespaces are specified then the virtual service is exported to all namespaces by default.

Sometimes we want a virtual services to expose to both outside of the mesh via Ingress, and within the mesh, and we hope to use the same hostname. For this requirement, we can use the ServiceEntry CRD. ServiceEntry enables adding additional entries into Istio’s internal service registry.

From Microservice to Service Mesh

Thu, 25 Nov 2021 00:31:19 -0400

We all know what microservice is now but how does service mesh assist with microservice development.

Microservice

Microservice as an architecture was firstly conceptualized in this article by Martin Fowler in 2014. It covers the pros (strong module boundaries, independent deployment, technology diversity) and cons (dealing with distributed system, eventual consistency, operational complexity). The reality is, many teams develops their product with the microservice architectural pattern.

The implementation of microservice architecture involves a lot of programming patterns, and tools. The creation of these patterns and tools are usually done in a separate dedicated project so developers can focus on business logics. When building software, developers only need to interact with libraries and frameworks. Libraries (e.g. log4j) provides dependencies, and developers needs to write code to call the libraries. On the other hand, frameworks (e.g. Spring, Flask) not only provides tools, but also implements a pattern. It addresses a set of common problems such as authentication, expose http service, logger and database connectivity. Once set up, the framework will call the code that developers write (unlike in libraries).

Spring – an example of Microservice framework

When it comes to microservice, a well-known appliction framwork is the Spring framework. It solves problems such as:

Application context and dependency injection (for Inversion of Control, or IOC)
Database access and transaction management
Expose rest APIs (using spring MVC)

There’s an entire ecosystem of projects under Spring framework. This framework is a huge system requiring a lot of configuration efforts. This is where Spring Boot helps.

Spring Boot makes it easy to create stand-alone, production-grade Spring based applications that you can just run. It features the “convention over configuration” paradigm to save programmers from boiler plate configuration. SpringBoot gives you a standalone application ready to run without complicated deployment steps.

Managing configuration in property files does not scale in the time of microservice. Spring Cloud provides configuration as a service (in line with everything else microservice framework). It doesn’t necessarily have to be hosted in the cloud. It is comparable to Apachee Zookeeper, Etcd (distributed key value store), Hashicorp Consul and Netflix OSS (Eureka, Ribbon, Hystrix). You can pull from Git repo. The mission of Spring Cloud is to eliminate boilerplate associated with distributed systems problems for Spring Boot applications.

Many developers use Spring Boot along with Spring Cloud to build microservices. This page contains a diagram for such architecture.

In this architecture, Spring cloud helps with service discovery, traffic routing, circuit-breaking, distributed tracing and monitoring. It can also act as API gateway (in place of Nginx).

API Gateway and API Management

Microservices relies on API. Let’s distinguish API gateway, and API management (this long post has some good information).

API Gateway is a microserivce pattern. The idea is a single point of entry for all clients. The API gateway either proxy an incoming request to the appropriate service, or it may fan out a request to multiple services. The other important aspect is the API gatway can expose a different API for reach client.

API Gateway Pattern

A variation of this pattern is the Backends for frontends pattern, where it defines a separate API gateway for each kind of client. The API Gateway may authenticate user and pass an Access Token containing information about the user to the services. It may use a circuit breaker to invoke services. To summarize, the key functions of an API Gateway in this pattern is:

Unified entry point for multiple API implementations
Protocol transformation
Request morphing
Client specific logics

This page from Azure has a good comparison between API Gateway pattern vs direct connection between client and microservice. Note that API Gateway can also refers to API Gateway product, which implements the functions above. For example

Spring Cloud Gateway
Solo.io Gloo
Netflix Zuul

API management acts as a proxy for an existing API implementations. Typical functions include:

AuthN and AuthZ
Service Discovery Ingreation
Load balancing (e.g. L7 path based routing)
Logging, tracing (track user), correlation
Response Caching
Retry policies, circuit breaker, QoS
Enforce policy
Track usage and monetization
metrics (duration)
rate limiting and throttling
Request morphing (header, query string and claims transformation)
IP whitelisting

API management are usually implemented as tightly controlled shared infrastructure owned by either a “platform team”, “integration team”, or other API infrastructure teams. Examples of API management product (including SaaS) are:

Google Cloud Apigee
Mulesoft
Kong

In real world, people often use API management produce and API gateway product interchangeably.

Service Mesh

Many dub Service Mesh the next generation of Microservice. So what is the relationship between microservice and service mesh. The Microservice architectural pattern creates the need for API gateway pattern. To address this pattern, the API Gateway products first emerged. Service mesh emerged later. Service mesh and API gateway have a common set of features.

In this presentation (a tale of two frameworks) from 2018 (early days of service mesh), two teams discussed microservice (spring cloud) and service mesh (istio) approaches. There is a slide about when to use which. Many teams since have moved to Service Mesh for feature richness. This is a case study from 2021.

There are a number of service mesh technologies, such as Consul, Isito and Linkerd. Here is a comparison chart. Even though all those projects are open-source, there is some competition already. Linkerd is the first to bring up the concept of service mesh in this blog. It also purportedly has better performance than Istio. However, it does not use Envoy proxy. Istio is good at marketing. It has higher adoption rate and is feature rich. However, Google did not donate Istio project to CNCF as many expected. Instead, it created its own governing body, the Open Usage Commons. The Istio is not an open-governance project, which potentially diverge from CNCF in the future [1]. Hashicorp Consul initially was built for service discovery and distributed key/value store. It supports Kubernetes and VM. However, it still lacks observability features.

Apart from the three major technologies, other players tries to push for standardization of service mesh. The most influential initiative is the SMI (service mesh interface), pushed by Microsoft. The idea is a separation of standard and implementation, so late players will have a chance. OpenServiceMesh is Microsoft’s reference implementation of SMI. The SMI is something to watch for but it remains pretty week thus far. Google’s platform has Anthos Service Mesh which is a commercial distribution based on Istio. AWS has its own AppMesh technology, also using Envoy proxy.

[1] Update from Apr 25, 2022 – Istio applied to become CNCF project.

Intro to PLG stack -Prometheus, Loki and Grafana

Sun, 03 Oct 2021 12:59:00 -0400

Last month we discussed log shipping with EFK. This week I spent sometime checking out its alternative Loki. Having been exposed to the ELK stack extensively, I am also interested in exploring the counterparts in this new stack, such as Prometheus, Loki and Grafana. So I need to address the issues of shipping both metrics and logs.

Let’s start by clarifying the terms:

Grafana is a visualizer. It supports many backends such as Prometheus, Loki, Elasticsearch, CloudWatch and Azure Monitor. It is the flagship product of Grafana Labs.

Prometheus is a time-series database and alerting platform. To push metrics to Premethus, you can either integrate your application with client library (in their term, instrumenting), or configure an existing exporters for a third party application such as PostgreSQL. Prometheus collects and stores its metrics as time series data ( i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels) and it comes with basic visualization capability. Premetheus is a CNCF project since 2016 and is maintained by Grafana Labs.

Loki is a log aggregation system, also developed by Grafana Labs. Loki does not index the contents of the logs. Instead it groups entries into streams, and indexes a set of labels for each log stream. You can use grafana or logcli to consume the logs. Loki supports clients such as Fluentd, Fluentbit, Logstash and Promtail.

Promtail is a log collection agent built for Loki.

The roles of the components such as Prometheus, Loki, Grafana and Promtail are similar to the ELK stack. Grafana resembles Kibana. Promtail resembles Filebeat. Premethus exporters resemble Metricbeat. Both Premetheus and Loki resemble Elasticsearch in some aspects. Premetheus keeps metrics and Loki persists log streams.

These tools are heavily used in Kubernetes. For a simple start, I’d like to just configure two minimally working pipelines on my MacBook without any containerization. My example setup is to achieve the followings:

Fake up some log lines and ship them to Loki
Ship OS metrics to Prometheus
Display the metrics and logs with Grafana

Here’s the diagram of what we want to implement:

This is simple enough to build a quick and dirty setup

Configure Node Exporter

We can use home brew to install node_exporter on MacOS. For Linux, use the equivalent package management tool:

brew install node_exporter
brew services start node_exporter
curl http://localhost:9100/metrics | grep "node_"

The node_exporter collect metrics and make it available for scrape using port 9100 as shown above.

Configure Prometheus

Install Prometheus with homebrew and start the service:

brew install prometheus
brew services start prometheus

If curl to port 9090 with GET returns “Found”, then Prometheus is successfully installed. We also want to configure it so it scrapes node exporter for metrics. Edit the configuration file, in my case, /usr/local/etc/prometheus.yml, by adding the followings:

 - job_name: node
 static_configs:
 - targets: ['localhost:9100']

This tells Prometheus to scrape metrics at specified interval. Restart Prometheus and browse to http://localhost:9090/ for Prometheus UI. Click on Status -> Targets and you can see the node as an export. Click on Graph and Execute a query for example “node_memory_free_bytes” and click on Graph. You should see a plot of the metric value.

We can later configure to display the chart in Grafana.

Generate log lines

We use a tool called flog to generate fake logs:

brew tap mingrammer/flog
brew install flog
flog -f rfc3164 -l -d 300ms -t log -o /tmp/test.log -w

The command above produces log in RFC3164 format to /tmp/test.log, one line every 300ms. The log does not rotate.

Configure Loki

Install Loki with homebrew and start the service. We want to configure Loki before Promtail so it is ready to receive logs. You might as well install LogCLI to interact with Loki.

brew install loki
brew install logcli
brew services start loki

If nc to port 3100 returns success, then Loki is successfully installed.

Configure Promtail

Install promtail using homebrew:

brew install promtail

Note that Promtail is not installed as a homebrew service. Although it can be manually configured as a service, I’d rather stay focused and use command line just for the time being. A copy of configuration file is located in /usr/local/etc/promtail-local-config.yaml but it needs to be modified first with a job to tell it where to scrap the log lines. The configuration looks like this:

server:
 http_listen_port: 9080
 grpc_listen_port: 0

positions:
 filename: /tmp/positions.yaml

clients:
 - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: app
 static_configs:
 - targets:
 - localhost
 labels:
 job: applogs
 __path__: /tmp/test.log

Then, use the command as in the documentation to start promtail:

promtail -config.file /usr/local/etc/promtail-local-config.yaml

Port 9080 will be open when Promtail is running. Promtail will push logs to Loki. You can use LogCLI to interact with Loki and see the latest log lines pushed to Loki.

logcli labels job
logcli query '{job="applogs"}'

However, unlike Prometheus, Loki itself does not have any visualization capabilities. We will need Grafana to display the log nicely on the web.

Configure Grafana

We use Grafana to visualize both the logs and metrics. To install Grafana on Mac:

brew install grafana
brew services start grafana

If curl to port 3000 with GET returns “Found”, then Grafana is successfully installed. Browse to localhost:3000, with default credential admin and admin. From the UI, add two data sources. For the first data source, specify Prometheus as type and localhost:9000 as destination. For the second, specify Loki as the destination http://localhost:3100

The steps to see logs in Loki is pretty much the same as on this page of its documentation.

Exploring logs

Exploring metrics data is similar. Click on Explore on the side bar, then select Prometheus from the dropdown as data source. Then execute a query such as “node_memory_free_bytes”.

Explore Metrics

With this quick and dirty configuration, we established a good understanding of what Prometheus, Loki and Grafana do. Next, we will move all these configurations to K8s cluster and understand some specific points of configurations.

Spark, Cassandra and Python

Tue, 15 Sep 2020 16:24:09 -0400

In this post we touch briefly on Apache Spark as a cluster computing framework that supports a number of drivers to pipe data in, and that its stunning performance thanks much to resilient distributed dataset (RDD) as its architectural foundation. In this hands-on guide, we expand on how to configure Spark, and use Python to connect to Cassandra data source.

Spark supports Sala, Java and Python shells. I’m not familiar with Scala but I have had Python background and know it’s importance in big data processing. One key data structure with big data processing in Python is Pandas data frame. Spark has the ability to map its own data frame to Pandas data frame.

Spark also needs a third party connector to connect to Cassandra. This connector is provided by Datastax in this open-source project called spark-cassandra-connector. The Github page includes a README with compatibility matrix, which is very important to understand before any configuration works. However, the Github is only the source code repository for anyone to build the project themselves. An alternative source of the dependency is this page from Maven repository. When running Spark we can simply reference that page URL as dependency.

Suppose we install spark onto CentOS, we download and unzip this package to somewhere such as user directory (~). Assuming we already have Open JDK 1.8 installed, when we run spark binary, it places cache and jar files in ~/.ivy2, potentially we need to manually move the following dependencies to ~/.ivy2/jars:

org.codehaus.groovy_groovy-json-2.5.7.jar
com.github.jnr_jffi-1.2.19.jar
org.codehaus.groovy_groovy-2.5.7.jar

These jar files are available for download from Maven’s repository as well if you wish provide them as package dependencies. We have two flavours of interactive shells to connect to Spark: the Scala shell (spark-shell) and python shell (PySpark)

Scala Shell

We can enter the default scala shell by

$ ./bin/spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 --conf spark.cassandra.connection.host=10.10.10.151 --verbose

During the start, note a stdout line that says:

Spark context Web UI available at http://spark-host:4040

Then we can open that tcp port on iptables and view that job in browser. From within scala shell we can test connectivity to Cassandra with the following commands:

>>> val new_exam = spark.read.format("org.apache.spark.sql.cassandra").options(Map("table" -> "new_exam","keyspace" -> "examarchive")).load()

Python Shell

Python Shell (aka PySpark) brings Python shell which is known to many engineers from system admin or development background. By default, python 2 will be used. To specify python version, set some environment variables before we start pyspark with cassandra connector package specified:

$ export PYSPARK_PYTHON=python3
$ export PYSPARK_DRIVER_PYTHON=python3
$ export SPARK_HOME=/home/dhunch/spark-2.4.6-bin-hadoop2.7
$ export PATH=$SPARK_HOME/bin:$PATH
$ ./bin/pyspark --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 --conf spark.cassandra.connection.host=10.10.10.151

Once you’re in the interactive shell, you can start with loading required python libraries, and test your connectivity:

>>> from pyspark import SparkContext, SparkConf
>>> from pyspark.sql import SQLContext

>>> load_options = { "table": "new_exam", "keyspace": "examarchive"}
>>> df=spark.read.format("org.apache.spark.sql.cassandra").options(**load_options).load()
>>> df.show()
>>> df.write.csv('/tmp/mycsv.csv')

>>> #df.registerTempTable("ne")
>>> df.createTempView("ne")

>>> tw1=sqlContext.sql("select count(*) from ne")
>>> tw1.show()

>>> qrdf2=sqlContext.sql("select study_key, image_count from ne where current_exam_version=exam_version")
>>> qrdf2.write.csv('/tmp/tw2')

Note that the load method returns type pyspark.sql.dataframe.DataFrame, which is already a distributed data structure. So there is no need to parallelize it with parallelize() method. As of Spark 2.0, we are supposed to use createTempView() method instead of the old registerTempTables() method. Read this for further information.

Python Application

With interactive shell you run one or several commands at a time. We can build a python script and submit the whole script as an application. This is an example command:

./bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 sample.py

Note that the sample.py script name must be provided after –packages switch. Otherwise, you will get an error saying missing dependency (Failed to find data source: org.apache.spark.sql.cassandra). In the script, we can manipulate the data from Cassandra with greater flexibility. For example, we can map one field to several fields. For example, if one of the fields stores an XML document, the script can drill down the XML tree structure parse out values at different levels of child nodes, into separate data base columns. Here is an example of python script where we register a custom UDF declared in python and apply it to some existing columns to build new columns:

#! /usr/bin/python3
# To submit this script as an application to spark:
# ./bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 examstat.py
# Note that the script name must be placed after --packages 

import sys,datetime,re
import xml.etree.ElementTree as ET
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, SparkSession
from pyspark.sql.functions import udf 
from pyspark.sql.types import StringType,StructType,StructField

cluster_seeds=['dest_cass_host']

def pTrimExamCode(raw_code):
 return 'NULL' if raw_code is None or raw_code=='None' else str(raw_code).replace(',','').rstrip('\r\n')

def is_valid_date(date_str):
 isValidDate=bool(re.match("^(19|20)\d\d(0[1-9]|1[012])(0[1-9]|[12][0-9]|3[01])$",date_str))
 if isValidDate:
 try:
 datetime.datetime(int(date_str[:4]),int(date_str[4:6]),int(date_str[6:8]))
 except ValueError:
 isValidDate=False
 return isValidDate

def pPullTags(study_key,raw_xml_field):
 ns={"vc":"http://medical.nema.org/mint"}

 StudyDateTag='None'
 StudyDescriptionTag='None'
 try:
 if raw_xml_field is not None:
 summary_tree=ET.fromstring(str(raw_xml_field)) # str function outputs 'None' or null object
 xml_find_res=summary_tree.find("vc:attributes/vc:attr[@tag='00080020']",ns)
 if xml_find_res is not None: StudyDateTag=str(xml_find_res.attrib.get('val'))
 
xml_find_res=summary_tree.find("vc:attributes/vc:attr[@tag='00081030']",ns)
 if xml_find_res is not None: StudyDescriptionTag=str(xml_find_res.attrib.get('val'))
 except:
 print("----------------------->>>>>>>>> examstat: error parsing metadta for study_key "+study_key)
 
 return (StudyDateTag,StudyDescriptionTag)

# custom StructType for the output tuple
XMLExtractType=StructType([
 StructField("StudyDate",StringType(),False),
 StructField("StudyDescription",StringType(),False)])

if __name__ == "__main__":
 sparkSession=SparkSession.builder \
 .appName('examstat') \
 .config('spark.cassandra.connection.host',','.join(cluster_seeds)) \
 .master('local[*]') \
 .getOrCreate()
 load_options = {"table": "new_exam", "keyspace": "examarchive"}
 sqlContext=SQLContext(sparkSession)

 # pyspark.sql.dataframe.DataFrame is already a distributed data structure. No need to parallelize it.
 df0=sqlContext.read.format('org.apache.spark.sql.cassandra').options(**load_options).load()
 df0.createTempView("new_exam")

 # pyspark.sql.functions.udf(python function,output type)
 sparkSession.udf.register("uTrimExamCode",udf(pTrimExamCode,StringType()))
 sparkSession.udf.register("uPullTags",udf(pPullTags,XMLExtractType))

 # use custom UDFs uTrimExamCode and uPullTags to calculate new columns and remove dups and deleted studies
 df1=sqlContext.sql("select study_key as StudyKey,uTrimExamCode(exam_id) as ExamCode,image_count as ImgCnt,Total_pixel_data_size as PixelSize, uPullTags(study_key,metadata_summary) as XMLExtract, metadata_summary from new_exam where exam_version=current_exam_version and is_deleted=False")
 df1.createTempView("uniq_study")
 
 # map the four fields in XMLExtract to separate columns. we take this as separate step as we don't want uPullTags to execute multiple times in previous step 
 df2=sqlContext.sql("select StudyKey,ExamCode,ImgCnt,PixelSize,XMLExtract.StudyDate as StudyDate,XMLExtract.StudyDescription as StudyDescription from uniq_study")
 df2.createTempView("uniq_study_stat")

 # Run analytical query
 df3=sqlContext.sql("SELECT ExamCode, round(avg(PixelSize)/1024/1024) as avg_size_mb, round(sum(PixelSize)/1024/1024/1024,2) as total_size_gb,count(StudyKey) as study_count FROM uniq_study_stat GROUP BY ExamCode order by study_count desc")
 #data frames are lazily loaded and processing not started until the following call
 df3.write.csv('/tmp/examstat_'+datetime.datetime.now().strftime("%m%d%H%M%S"))

It is important to understand the concept of lazy evaluation in Spark RDD here. The execution of function to RDD does not start until an action is triggered (eg. show method, or write method). Spark maintains the record of which operation is being called through DAG (directed acyclic graph). Such record is referred to as a transformation. We need to understand whether each RDD method is a transformation, or an action so we know whether it will be lazily evaluated (here’s more information).

This is a major difference between Apache Spark and Hadoop MapReduce. With MapReduce, developer spend a lot of time in minimizing the number of MapReduce passes. It happens by clubbing the operations together.

Zookeeper Summary

Wed, 26 Aug 2020 23:10:00 -0400

Distributed systems

Distributed system involves independent computing entities linked together by network. The components communicate and coordinate with each other to achieve a common goal. In early days, designers and developers often had made some assumptions (aka. fallacies) of distributed computing:

The network is reliable
Latency is zero
Bandwidth is infinite
Network is secure
Topology doesn’t change: in reality, components to a network get removed/added over time. the system should tolerate such changes.
There is one administrator: for distributed systems to function, they interact with external system beyond administrative control.
Transport cost is zero: cost is involved everywhere, in the form of CPU cycles spent, to actual dollars paid to service provider.
Network is homogenous

These fallacies make coordinating distributed computing entities a huge challenge and Zookeeper is introduced to address these challenges. Zookeeper implements common tasks for distributed coordination, such as:

Configuration Management (propagate configuration changes to all worker nodes dynamically)
Naming service
Distributed synchronization (locks and barriers)
Cluster membership operations (e.g. detection of node leave/join)

ZooKeeper is a centralized coordination service for the distributed application. ZooKeeper itself is distributed as well. It runs on its own cluster of servers called a ZooKeeper ensemble, separate from application’s cluster. Distributed consensus, group management, presence protocols, and leader election are implemented by the service so that the application developers do not need to reinvent the wheel by implementing them on their own.

Developers will have to use APIs through ZooKeeper’s client library, which has language bindings for almost all popular programming languages. The client library is responsible for the interactions of an application with the ZooKeeper service. For testing with API access one can alternatively use its Java-based command-line shell (zkCli.sh)

$ zkCli.sh -server zknode:2181

How Zookeeper works

Data Model

ZooKeeper allows distributed process to coordinate with each other through a shared hierarchical namespace of data registers (znodes). The hierarchy start with root node which has child znode(s). Each znode can have their children, as well as store its own data (hence the name data register). The data in a znode is stored in byte format for a maximum of 1MB (ZooKeeper by design is just a coordinator service of host application, so its own data set size is fairly small).

Zookeeper data model

Znodes have two types (set at time of creation)

persistent znode: for storing persistent data, such as configuration. The znodes and their data will exist even if the creator client dies.
ephemeral znode: deleted by ZooKeeper service when the creating client’s session ends (due to disconnection or explicit termination). It can also be explicitly deleted by creator client through delete API call. They cannot have children. Their visibility is controlled by ACL policy

ZooKeeper can assign an incremental sequence number as part of znode name during its creation. This makes a sequential node. Both persistent znode and ephemeral znode can be either sequential or not.

In typical client-server architecture, server is passively open and do not initiate communication to client. Client pulls information from server. This is however an anti-pattern for large scale distributed system. ZooKeeper implements a Watch mechanism where clients can get notifications from ZooKeeper service, instead of having to poll for events. Clients can register with the ZooKeeper service (by setting a watch on znode) for any changes associated with a znode. A watch will only trigger notification once, and needs to be re-registered (by client) for trigger the next notification. A watch is triggered upon:

Any changes to the data of a znode;
any changes to the children of a znode;
Creation of deletion of a znode

ZooKeeper guarantees that notifications are delivered in the order of event occurrence. When a client disconnects from ZooKeeper server, it doesn’t receive any watches until the connection is re-established.

API Operations

The ZooKeeper operations are:

Operation	Description
create	Creates a znode in the specified path
delete	Deletes a znodes from the specified path. Not allowed if the znode has children. version number required
exists	Check if a znode at the specified path exists, and get version number; support watch
getChildren	Get a list of children of a znode; support watch
getData	get the data associated with a znode; support watch
setData	writes data into the data field of a znode. Version number required.
getACL	get the ACL of a znode
setACL	set the ACL in a znode
sync	synchronizes a client’s view of a znode

The write operations (setData, create, delete) are atomic, durable and eventually consistent. Every znode has a stat structure including cZxid, mZxid an dpZxid that keeps track of the ID of the transactions that created, last modified this znode, or pertains to adding or removing its children.

Production znode ensemble with more than one node is running in quorum mode. Updates to ZooKeeper tree by clients must be persistently stored in this quorum of nodes for a transaction to be completed successfully. Odd number of node is recommended to avoid split-brain where network partition causes two subsets of servers in the ensemble function independently, and different clients get different results for the same requests, depending upon the server they are connected to.

All ZooKeeper nodes are listed in the configuration for client application to randomly pick from and try to connect and establish a session. The session is associated with every operation the client executes in a ZooKeeper service. The session also has a timeout period specified by the application client during session establishment. If the connection remains idle for more than the timeout period, the server expires the session. Appropriate session timeout should be set based on network condition. Sessions are kept alive by client sending heartbeat to ZooKeeper service. Application developer needs to handle connection-loss scenarios properly.

Leader Election and Atomic Broadcast

ZooKeeper ensemble contains a leader nodes, follower nodes and observer nodes.

The leader node is elected by the cluster. It handles all write requests.
The follower nodes are leader candidates that are not elected. They are backup to the leader nodes. They handle read request, and receive the updates proposed by the leader, and through a majority consensus mechanism, a consistent state is maintained across the ensemble.
The observer nodes are ineligible as leader candidates. They have otherwise the same function as followers.

The service relies on the replication mechanism to ensure that all updates are persistent in all servers that constitute the ensemble. This is the core mechanism in ZooKeeper, implemented as a special atomic messaging protocol called ZooKeeper Atomic Broadcast (ZAB). ZAB (a variant of Paxos algorithm) ensures the election of new leader in the event of old leader crash, and ensures integrity of data. It defines three states (looking, following and leading) of a node, and goes through four phases (election, discovery, sync, broadcast) in its operation.

All read requests (exists, getData, getChildren) are process locally by the ZooKeeper node where the client is connected to. This makes read operation fast. All write requests (create, delete, and setData) are forwarded to the leader in the ensemble, which carries out the client request as a transaction. A transaction is identified by zxid and is idempotent. Transaction also satisfies the property of isolation (no transaction is interfered with by any other transaction). Only after a majority of the followers acknowledge that they have persisted the change does the leader commit the update.

Transaction processing involves two steps in ZooKeeper: leader election and atomic broadcast. This resembles a two-phase commit protocol (which also includes a leader election and an atomic broadcast)

ZooKeeper use local storage to persist transactions. The transactions are logged to transaction logs, in sync’ed write, requiring a dedicated block device separated from boot device of server. The local storage also keep point-in-time copies (snapshots) of the ZooKeeper tree.

ZooKeeper Recipes

The ZooKeeper recipes defines high-level implementation (construct) of some common distributed coordination mechanism:

Barrier: any thread/process must stop at this point and cannot proceed until all other threads/processes reach this barrier.
Queue: allow FIFO in distributed system
Lock: Fully distributed locks that are globally synchronous, meaning at any snapshot in time no two clients think they hold the same lock.
Leader Election: designate a single process as the organizer of some task distributed among several nodes.
Group membership: node may join or leave a group, which needs to be made available to clients. An alternative to ZooKeeper to manage group membership is gossip protocol.
Service discovery: help client to determine IP and port for a service that are hosted by multiple servers.
Two-phase commit: a mechanism for atomic commitment in two steps: first a commit request phase involving a voting by participants; and second, either a commit action, or an abort action, based on the voting result.

Zookeeper Administration

The official documentation includes all we need to know about administration. In addition, we need to configure log4j for proper logging. As best practices, we also should turn off swapping on ZooKeeper. We should clean up the data directory periodically if auto purge is not enabled. For optimal performance, ZooKeeper transaction log should be configured in a dedicated device.

For monitoring, ZooKeeper responds to a small sets of four-letter commands issued through telnet or nc to server’s client port. This allows the admin to check health of server or diagnose any problems. This requires the following property in zoo keeper config:

4lw.commands.whitelist=stat, ruok, conf, isro, wchc

The value can be set to asterick to allow all four-letter keyword. Once enabled, we can check server status

$ echo ruok | nc localhost 2181
imok

More four-letter commands are listed here. Apart from the four-letter commands, ZooKeeper can also be managed through Java Management Extensions (JMX).

Conclusion

Apache ZooKeeper is a coordination service for distributed application. It has become the solution for high availability for many other projects. Some of Apache’s well known open-source distributed services include:

Apache Hadoop (an umbrella of projects including many components for BigData processing such as Hadoop Common, Hadoop Distributed File System (HDFS), Hadoop YARN (yet another resource negotiator) and Hadoop MapReduce)
Apache HBase: non-relational database on top of HDFS
Apache Hive: data warehouse with SQL-like interface
Apache Kafka: stream processing
Apache Nifi: automated data flow processing.

Some of them, such as Nifi, has an embedded implementation of ZooKeeper ensemble if there isn’t a separate ensemble. There is some limitation with embedded Zookeeper ensemble. First, we cannot start ZooKeeper without starting Nifi service on the same server. Second, we need to orchestrate the configuration so that the ZooKeeper ensemble does not grow too large. We need to keep in mind that the ZooKeeper ensemble is a separate cluster of its own, and the it is not recommended to have more than 7 nodes on ZooKeeper.

Kafka high-level Overview

Tue, 21 Jul 2020 23:19:00 -0400

Zookeeper

General definition of distributed system: a software system that is composed of independent computing entities linked together by a computer network whose components communicate and coordinate with each other to achieve a common computational goal. Implementing coordination among components of a distributed system is hard. For example, designated master node becomes single point of failure; cluster needs to detect availability of new nodes as it joins cluster.

Zookeeper is designed to simplify cluster coordination. Zookeeper implements key aspects in cluster coordination, such as distributed consensus, group management, presence protocols and leader election. In order to coordinate a cluster, zookeeper itself also runs in its own cluster, called ensemble. Zookeeper exposes a simple but powerful interface of primitives. Applications can be designed on these primitives implemented through ZooKeeper APIs to solve the problems of distributed synchronization, cluster configuration management, group membership, etc.

Zookeeper Ensemble

Clients can connect to a Zookeeper service by connecting to any member of the ensemble. The members of the ensemble are aware of each other’s state. As long as a majority of the nodes are available, the service will be available. Zookeeper cli (zkCli.sh) can be used to connect to Zookeeper server. they can be downloaded from here.

Zookeeper is integrated with many other services apart from Kafka, such as Nifi and Hadoop.

Kafka

Kafka is a messaging system that is horizontally scalable, fault tolerant. It can also serve as queue storage system and stream processing system. It is distributed and use Zookeeper for cluster coordination. Each node is called a broker.

Topics in Kafka (think of table in database) is a category or feed name to which messages (records) are published. Topic is broken up into ordered commit logs called partitions. Each partition has an ID. Each message in a partition is assigned an offset. Topics that are created in Kafka are distributed across brokers based on the partition, replication, and other factors. Each partition is replicated across several brokers depending on replication factor. For each partition, Kafka elect one replica as the leader of partition.

Writes to a partition is generally sequential. Reading messages can either be from the beginning, or rewind or skip to any port in partition given an offset value. Data in a topic is retained for a configurable period of time.

A message is a unit of data in Kafka, in the format of key-value pair. A key is used to control the message that is to be written to partitions. Messages with the same keys are always written to the same partition (hash map)

A producer publishes new message to a topic. Producers do not care which partition the message is written to and will balance messages over every partition of a topic evenly. Directing messages to a partition is done using the message key and a partitioner, this will generate a hash of the key and map it to a partition.

A consumer is subscribed to one or more topics and read messages sequentially. The consumer keeps track of messages it has consumed by keeping track on the offset of the message. The offset is a bit of metadata (an integer value that continually increases) that kafka adds to each message. Each partition has a unique offset which is stored with the offset of the last consumed message. A consumer can stop and start without losing its current state.

A Kafka broker is designed to operate as part of a cluster. One broker in the cluster also function as the cluster’s controller, which is responsible for administrative operations such as: assigning partitions to brokers; monitoring for broker failures in cluster. A particular partition is owned by a broker and that broker is called the leader of the partition.

All consumers and producers operating on that partition must connect to the leader.

Kafka cluster may replicate across cluster using MirrorMaker.

Reference: Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale