elasticsearch on Digi Hunch

Log Shipping in Kubernetes with EFK stack

Sat, 04 Sep 2021 21:50:00 -0400

I first worked on log shipping with ELK stack three years ago. In the context of Kubernetes cluster, log shipping has similar challenges. In this post I will discuss the set up of log shipping with Kubernetes cluster using EFK stack

Logging Architecture

As discussed, if the Kubernetes cluster has a runtime in compliant with CRI (e.g. containerd), then the stdout and stderr of the Pod is stored on the node, in the location /var/log/containers/.

When creating log shipping solution, it is important to use a compliant cluster (e.g. minikube) to ensure what you develop will work across environments.

The Kubernetes document has a section on logging architecture which is a good start point. It outlines several different patterns. Logging at the node level is turned on by default and does not require special configuration, as explained in the section above. The EFK pattern is close to the diagram under using a node logging agent for cluster-level logging.

Node logging agent

The diagram above is stolen from Kubernetes documentation. In EFK stack, the agent is a daemonset running fluentd Pod.

EFK stack

While we can use ELK (Elasticsearch, Logstash, Kibana) stack for log shipping, EFK (Elasticsearch, Fluentd, Kibana) is generally recommended in Kubernetes cluster. We compare the two in the following table:

	ELK	EFK
Development language	All in Java. Logstash and Filebeat both require JVM. Managed as an open-source project by Elastic company	Fluentd in Ruby and does not require JVM to run. Fluentd is a CNCF project built to integrate with Kubernetes.
Typical pattern	Filebeat acts as a lightweight collector to monitor the source log. Logstash as aggregator to receive from filebeat, and push to Elasticsearch	The fluentd Pod can be configured to serve as forwarder and aggregator based on configuration. fluentd-forwarder is deployed as daemonset on node, and ship the result to fluentd-aggregator, which may run in a separate cluster. The fluentd-aggregator pushes processed results to Elasticsearch.
Metrics	use metric beat for data collection	scrape metrics from prometheus server
Event Routing	tag-based	if-then statement

ELK and EFK stacks comparison

The main problems in log shipping are:

column mapping : identify column patterns in each log line and map them to appropriate column in Elasticsearch.
multi-line processing: identify when a logging entry spread across multiple lines and process accordingly.

Suppose we want to congregate the logs from stdout and stderr of PostgreSQL pods. The raw output in /var/log/container on the node, looks like this:

2021-08-28T15:27:46.75370563Z stdout F server stopped
2021-08-28T15:27:46.757173069Z stderr F postgresql-repmgr 15:27:46.75 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:46.883126928Z stderr F postgresql-repmgr 15:27:46.88 INFO ==> Registering Primary...
2021-08-28T15:27:47.017164653Z stderr F postgresql-repmgr 15:27:47.01 INFO ==> Loading custom scripts...
2021-08-28T15:27:47.023334611Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
2021-08-28T15:27:47.026169813Z stderr F postgresql-repmgr 15:27:47.02 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.343607487Z stderr F postgresql-repmgr 15:27:47.34 INFO ==> Stopping PostgreSQL...
2021-08-28T15:27:47.448111425Z stdout F waiting for server to shut down.... done
2021-08-28T15:27:47.448172479Z stdout F server stopped
2021-08-28T15:27:47.453722807Z stderr F postgresql-repmgr 15:27:47.45 INFO ==> ** PostgreSQL with Replication Manager setup finished! **
2021-08-28T15:27:47.453829953Z stdout F
2021-08-28T15:27:47.503516746Z stderr F postgresql-repmgr 15:27:47.50 INFO ==> Starting PostgreSQL in background...
2021-08-28T15:27:47.532558987Z stdout F waiting for server to start....2021-08-28 15:27:47.532 GMT [273] LOG: pgaudit extension initialized
2021-08-28T15:27:47.533307459Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-28T15:27:47.533466407Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv6 address "::", port 5432
2021-08-28T15:27:47.537987947Z stdout F 2021-08-28 15:27:47.537 GMT [273] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-28T15:27:47.547956465Z stdout F 2021-08-28 15:27:47.547 GMT [273] LOG: redirecting log output to logging collector process
2021-08-28T15:27:47.548005463Z stdout F 2021-08-28 15:27:47.547 GMT [273] HINT: Future log output will appear in directory "/opt/bitnami/postgresql/logs".
2021-08-28T15:27:47.551741571Z stdout F 2021-08-28 15:27:47.551 GMT [275] LOG: database system was shut down at 2021-08-28 15:27:47 GMT
2021-08-28T15:27:47.558012894Z stdout F 2021-08-28 15:27:47.557 GMT [273] LOG: database system is ready to accept connections
2021-08-28T15:27:47.618577092Z stdout F done
2021-08-28T15:27:47.618708978Z stdout F server started
2021-08-28T15:27:47.630065958Z stderr F postgresql-repmgr 15:27:47.62 INFO ==> ** Starting repmgrd **
2021-08-28T15:27:47.638116348Z stderr F [2021-08-28 15:27:47] [NOTICE] repmgrd (repmgrd 5.2.1) starting up
2021-08-28T15:27:47.65317627Z stderr F INFO: set_repmgrd_pid(): provided pidfile is /opt/bitnami/repmgr/tmp/repmgr.pid
2021-08-28T15:27:47.653232015Z stderr F [2021-08-28 15:27:47] [NOTICE] starting monitoring of node "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:27:47.681683703Z stderr F [2021-08-28 15:27:47] [NOTICE] monitoring cluster primary "orthweb-postgresql-ha-postgresql-0" (ID: 1000)
2021-08-28T15:28:11.742865958Z stderr F [2021-08-28 15:28:11] [NOTICE] new standby "orthweb-postgresql-ha-postgresql-1" (ID: 1001) has connected

From this snippet of log, we can see each line in stdout or stderr is appended with a timestamp. There are multi-line log entries but each is still appended with a timestamp. This is just how kubernetes keeps the log file for Pod stdout and stderr. To handle that, we need to first take out the real log line, and then process multi-line.

We will go over the installation of EFK stack and the mechanism to address the two challenges above.

Install Elasticsearch and Kibana

To install Elasticsearch, we use the helm chart provided by the official repository:

helm repo add elastic https://helm.elastic.co

If we run multiple pods on the same hosts, then we need some customized values in order to get the installation to work. The values.yaml file looks like this:

---

antiAffinity: "soft"

esJavaOpts: "-Xmx128m -Xms128m"

# Allocate smaller chunks of memory per pod.
resources:
 requests:
 cpu: "100m"
 memory: "512M"
 limits:
 cpu: "1000m"
 memory: "512M"

Then we can “preview” what values are used for installation, with helm’s template command:

helm template elasticsearch elastic/elasticsearch -f values.yaml

The effect of the antiAffinity property allows multiple Elasticsearch Pod to be scheduled on the same node. This is not required in production with multiple nodes. To install Elasticsearch and Kibana, run:

helm install elasticsearch elastic/elasticsearch -f values.yaml
helm install kibana elastic/kibana

The Kibana service is exposed on port 5601 of the cluster. To access the port on cluster, we need to run port-forward command as below:

kubectl port-forward deployment/kibana-kibana 5601

Install Fluentd

There are different ways to configure Fluentd. For example, in the forwarder-aggregator pattern, a forwarder Pod is a Daemonset on each Kubernetes node. The forwarder pushes to the aggregator, at port 24224. This page has some configuration details. This pattern is similar to filebeat -> logstash pattern in ELK stack.

In our case, we use a simplified pattern, with a fluentd daemonset acting as collector and then forward to Elasticsearch. Similarly, in ELK stack we can use filebeat to push to Elasticsearch without Logstash. The only reason is the entire architecture isn’t as complicated as requiring an aggregator.

We need to create configmap as below:

kind: ConfigMap
apiVersion: v1
metadata:
 name: fluentd-cm
 namespace: default
 labels:
 app.kubernetes.io/component: forwarder
 app.kubernetes.io/instance: fluentd
 app.kubernetes.io/managed-by: Helm
 app.kubernetes.io/name: fluentd
 helm.sh/chart: fluentd-1.3.0
 annotations:
 meta.helm.sh/release-name: fluentd
 meta.helm.sh/release-namespace: default
data:
 fluentd.conf: |
 # Ignore fluentd own events
 <match fluent.**>
 @type null
 </match>

 # HTTP input for the liveness and readiness probes
 <source>
 @type http
 port 9880
 </source>

 # Throw the healthcheck to the standard output instead of forwarding it
 <match fluentd.healthcheck>
 @type null
 </match>

 # Get the logs from the containers running in the node
 <source>
 @type tail
 read_from_head true
 tag kubernetes.*
 path /var/log/containers/orthweb-postgresql-ha-postgresql-**.log
 pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
 <parse>
 @type regexp
 expression ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[^Z]*Z)\s(?<stream>[^\s]+)\s(?<character>[^\s])\s(?<message>.*)$
 </parse>
 </source>
 # enrich with kubernetes metadata
 <filter kubernetes.**>
 @type kubernetes_metadata
 @id filter_kube_metadata
 kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
 verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
 ca_file "#{ENV['KUBERNETES_CA_FILE']}"
 skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
 skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
 skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
 skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
 </filter>


 <match kubernetes.var.log.containers.orthweb-postgresql-ha-postgresql-**.log>
 @type elasticsearch
 include_tag_key true
 host "elasticsearch-master.default.svc.cluster.local"
 port "9200"
 index_name "postgresql-logs"
 <buffer>
 @type file
 path /opt/bitnami/fluentd/logs/buffers/orthanc-logs.buffer
 flush_thread_count 2
 flush_interval 5s
 </buffer>
 </match>

Then we can create the resource, with helm chart pointing to the config map:

helm install fluentd bitnami/fluentd --set aggregator.enabled=false --set forwarder.configMap=fluentd-cm

We can validate the index creation on Elasticsearch:

kubectl port-forward service/elasticsearch-master 9200
curl -XGET localhost:9200/_cat/indices

From Kibana, we can forward the port as instructed above, and browse to the UI. Once logged on to Kibana, we need to add index pattern first before being able to view the content of index.

How about Fluent Bit

Fluentd has an even more lightweight brother Fluent Bit, also a CNCF project, designed by the same team, for environments with tighter resource restrictions. The technical differences are outlined on this page outlines the technical differences. In terms of use case, Fluentd is a log aggregator and Fluent Bit is simply a forwarder. In terms of ecosystem, Fluentd has a stronger ecosystem whereas Fluent Bit is more seen in IoT devices. Read this post for more comparison.

Summary

EFK stack (Elasticsearch, Fluentd and Kibana) on Kubernetes is a natural choice for ELK users. Fluentd is a CNCF project created for integration with Kubernetes. It is good alternative to enterprise solution such as Splunk. There are lots of plugins supported and articles on configuration. While developing a solution I had to spend time reading the input plugin documentations.

Intro to Big Data Projects

Thu, 10 Sep 2020 21:33:00 -0400

Modern applications produce super large datasets beyond what traditional data-processing application can handle. Big data is a discipline that specialize in processing such data. For example, analysis, information extraction etc. The scale of large dataset grows well beyond the capacity of a single computer, which calls for computing power delivered by multi-node clustered systems. Intensive computing tasks are completed in a distributed system consisting multiple nodes each performing some tasks, known as High-Performance Computing Cluster (HPCC).

Cluster computing inherit the challenges of distributed system. Moreover, two main challenges to solve are: distributed storage, and distributed computation. In Apache Hadoop projects, HDFS and MapReduce address these two challenges respectively. Now the Hadoop ecosystem has evolved to include several core projects:

HDFS

A distributed file system for reliably storing huge amount of unstructured, semi-structured or structured data in the form of files. Parts of a single large file can be stored on different nodes across the cluster. HDFS works in master-slave mode:

NameNode (master): holds file system namespace, controls access, keep track of DataNodes and replication factor
DataNode (slave): stores user data

HDFS is Java-based so is portable across all platforms. User interact with HDFS using a command-line interface called “FS shell”. There is also an interface called FUSE (filesystem in userspace) to mount HDFS to Linux OS. Since HDFS supports commodity hardware it is great for storing data for further processing. However, HDFS is not suitable for storing data related to applications requiring low latency access, nor is it good for simultaneous writes to the same file. Also HDFS is not suitable for large number of small files because the metadata for each file needs to be stored on the NameNode and is held in memory. Here is the architecture guide for HDFS, and this page expands further on the read and write operations in HDFS.

Compared to NAS(e.g. NFS), HDFS is distributed by design. The data blocks are distributed across different nodes. NFS storage may or may not be distributed depending on the implementation. HDFS is designed to work with MapReduce paradigm, where computation is moved to the data. In NAS, data is stored separately from the computations. Lastly, NAS is usually made up of enterprise grade hard drive but HDFS works with commodity hardware.

MapReduce

Hadoop MapRecude is a distributed algorithm framework that allows parallel processing of huge amounts of data. It breaks a large chunk into smaller ones to be processed separately on different data nodes and automatically gather the results across the multiple nodes to return a single result. If the duration of linear data processing can be done during night hours, it makes sense to choose Hadoop MapReduce. MapReduce runs on Hadoop cluster but also supports other database formats like Cassandra and HBase. MapReduce includes:

Job: a unit of work to be performed as requested by the client.
Task: Jobs are divided into sub-jobs known as tasks. The tasks can be run independent of each other on different nodes. There are two types of tasks:
- Map task is performed by map() function to process one or more chunks of data and produce the output results
- Reduce task is performed by reduce() function to consolidate the results produced by each of the map task
JobTracker: like the storage (HDFS), the computation (MapReduce) also works in master-slave fashion. A JobTracker node acts as the master to schedule task on appropriate nodes, coordinate execution of tasks, get the result back after execution of each task, re-execute failed tasks, and monitor overall progress. There is only one JobTracker node per Hadoop Cluster.
TaskTracker: a TaskTracker node acts as teh slave and is responsible for executing a task assigned to it by the JobTracker. There are usually a number of JobTracker nodes in a Hadoop Cluster. They execute the heavy lifting tasks.
Data Locality: if MapReduce cannot place the data and the compute on the same node, data locality put the compute on the node nearest to the respective data node(s) which contains the data to be processed.

The MapReduce programming model includes these steps: input->split->map->combine->shuffle&sort->reduce->output.

MapReduce programming model

YARN

YARN (yet another resource negotiator) is a system to schedule applications and services on an HDFS cluster and manage the cluster resources like memory and CPU. The two components are:

ResourceManager: receives the processing requests, and then passes the parts of requests to corresponding NodeManager accordingly based on the needs. ResourceManager is a central authority.
NodeManager: installed on every DataNode, is responsible for execution of the task on every single DataNode, monitoring the resource usage and reporting to the ResourceManager.

HBase

A key-value pair NoSQL database based on HDFS storage, with column family data representation, and mater-slave replication. HBase is based on Google’s BigTable concept (similar to Cassandra). It runs on a cluster of commodity hardware and scales linearly. Compared with Cassandra, HBase doesn’t have a query language of its own. You will have to work with JRuby-based shell, or Apache Hive. HBase is also a master-slave architecture and it uses Zookeeper as a status manager. In that sense, Cassandra is a “self-sufficient” database technology whereas HBase relies on other components in Hadoop. This article also compares the data model difference between the two.

Hive

Hive is a SQL interface over MapReduce for developers and analysts who prefer SQL interface over native Java MapReduce programming to query and manage large datasets residing in HDFS. With Hive you can map a tabular structure on to data stored in distributed storage. The Hive queries are written in SQL-like language known as HiveQL, executed via MapReduce. When a HiveQL query is issued, it triggers a Map and/or Reduce job(s) to perform the operation defined in the query.

Pig

A scripting interface over MapReduce for developers who prefer scripting interface over the native Java MapReduce programming. It is a runtime environment with a shell (named Grunt Shell) for execution of MapReduce jobs via a high-level scripting language called Pig Latin. Pig is an abstraction (high-level programming language) on top of a Hadoop cluster. The Pig Latin query/command are complied into one or more MapReduce jobs and then executed on Hadoop cluster. The most common commands in Pig are:

DUMP: displays the results to screen
STORE: stores the results to HDFS

Hadoop Ecosystem

There are some other Apache projects, which are sometimes considered as in the Hadoop ecosystem as well:

Oozie: worflow scheduling system to manage Hadoop jobs. In Oozie, a workflow is defined as a collection of control flow nodes and action nodes in a directed acyclic graph. Control flow nodes define the beginning and the end of a workflow, as well as a mechanism to control the workflow execution path. Action nodes are the mechanism by which a workkflow triggers the execution of a computation/processing task, such as MapReduce, Pig, etc.
Sqoop (SQL-to-Hadoop): a command-line interpreter tool for importing data from database (e.g. MySQL, data warehouse, etc) into the Hadoop environment (e.g. HDFS, Hive). It can also export the data back.
Flume: data ingestion for streaming logs into Hadoop environment. Flume is a distributed and reliable service for collecting and aggregating huge amounts of log data.
ZooKeeper: distributed service coordinator, as previously discussed. It is based on a Paxos algorithm variant called ZAB protocol.
Ambari: a framework for provisioning, managing and monitoring Hadoop clusters.

Hortonworks sandbox provide a VM image that have some Hadoop services pre-installed for beginners to get a taste of how it works all together.

Spark

Hadoop is used in the industry owing to a simple programming model (MapReduce) but the speed and waiting time (between queries and running the program). Spark is introduced to speed up the computing process. Spark uses Hadoop for storage (HDFS) and processing. It extends the MapReduce model to efficiently use more types of computations which includes interactive queries and stream processing.

Spark started as a sub-project of Hadoop in 2009 but since 2014 Apache has run it as a top-level project. It is a lightning-fast in-memory cluster computing technology. The features are:

Speed: in-memory computing makes super fast processing;
Built-in APIs supports multiple languages: Scala, Python and Java;
Advanced analytics – apart from map and reduce, Spark also has libraries that supports SQL query, near real-time stream processing, Graph algorithms and machine learning.

Spark can run in standalone mode, on Mesos, or with YARN cluster manager. The document also provides guide on deployment on EC2 and Kubernetes. Spark contains these components:

Spark Core: the underlying general execution engine for spakr platform that all other functionality is built upon. It provides in-memory computing and referencing datasets in external storage systems.
SparkSQL: a components on top of Spark Core that introduces a new data abstraction called SchemaRDD, which supports both structured and semi-structured data.
Spark Streaming: perform streaming analytics on top of Spark Core. It ingests data in mini-batches and performs RDD (Resilient Distributed Datasets) transformation on the fly.
MLib: a distributed machine learning framework
GraphX: a distributed graph-processing framework

The speed of Spark is owing to its fundamental data structure – Resilient Distributed Datasets (RDD), an immutable distributed collection of objects. Each dataset in RDD (object collection) is divided into logical partitions, which can be computed on different nodes of the cluster. The object can be any type of Python, Java or Scala object, including user-defined classes. There are two ways to create RDDS:

Parallelizing an existing collection in your driver program
Referencing a dataset from external storage system (e.g. HDFS, HBase) or data source offering a Hadoop Input Format

You can also create RDD based on other existing RDDs. This page explains further how RDD speeds up computing compared to MapReduce.

AWS Certified DevOps Engineer Exam Tips

Sat, 17 Aug 2019 23:07:59 -0400

The last 30 days have been exhausting for me. I studied hard on the new AWS Certified DevOps Engineer exam and thank goodness I passed (750 out of 1000 is required). This was the hardest professional certification I ever worked on. The exam was re-launched recently in March 2019 so there is still a shortage of information around the community. I was hoping to share my experience to help demystify this new exam.

I had a somewhat solid background to begin with, having taken the AWS Certified Solution Architect Professional exam (before the 2019 update), and worked quite a bit on CloudFormation, automation and Git. However, I still did not anticipate the exam to be this difficult until I was halfway through and already had the exam and materials paid for.

Without clear guideline on study material other than the white papers, I first skimmed through the ACloudGuru course, which helped me form a high level sense of exam coverage. Nonetheless it does not cover any topic in-depth and therefore by no means makes an essential part of my study. I checked out LinuxAcademy course and they are much more in-depth for the major topics indeed. I like the course material in Lucid chart. However, the LinuxAcademy course along does not cover everything you need to know.

What I found extremely helpful is the free training videos from the official training website. I strongly recommend the 7-hour course Exam Readiness: AWS Certified DevOps Engineer – Professional. The instructor did a great job outlining the services and knowledge areas in the assessment . The course also comes with quality sample questions with answers and explanations on what the thinkings are behind the correct answers or why some choices are obviously wrong. I went through these questions twice and feel much better at not only understanding the question, but also understanding the intent of the question.

Apart from the Exam Readiness course, other free introductory courses from the official training website are helpful as well especially for those services that you only need to know the basics. Most of those courses are 5 ~ 10 minutes long, with brief but sufficient introduction and a demo session. The other extremely helpful resource is the official practice questions. The practice exam is harder than the actual exam but they closely resemble the actual question style in the exam. Unfortunately, no answer is provided but they made me spend time finding answers across the documentations and blogs. It is worth-noting that the AWS blogs provides plenty of use cases that are covered in the questions.

When it comes to the real exam, it covers a lot more topics than its predecessor. Many questions are long and confusing. And I wish I could run a diff command to highlight the differences between choices. During exam preparation you really need to train yourself on reading efficiently. I found myself sometimes eyeball through all four choices at the same time, which get my mind scattered.

During my study, I divide all services into three categories based on the level of familiarity, and here is my list:

Category 1. Know these services very well, in and out:

AWS ElasticBeanstalk, OpsWorks Stacks, OpsWorks Chef Automate, CloudFormation, CloudWatch, CodeBuild, CodeCommit, CodeDeploy, CodePipeline, CodeStar, Lambda, API Gateway, Config, Trusted Advisor, CloudTrail, Systems Manager, Autoscaling Group in EC2, DynamoDB

Category 2. Know these services well, but not necessarily down to every single detail:

Amazon Kinesis Firehose, Kinesis Analytics, Kinesis Streams, Step Functions, Elastic Load Balancer, Secrets Manager, Serverless Application Model (SAM), Route53, RDS, Certificate Manager, ElasticSearch, ECS, ECR

Category 3. Know about these services at a high level, but do not skip any:

AWS Organization, X-Ray, GuardDuty, Macie, Inspector, Service Catalog, KMS, Batch, Athena, Single-Sign-On, Data LifeCycle Manager, CloudSearch, Health Dashboard, Glue, QuickSight, LightSail

Although I did not mention much about the white papers, I want to highlight their importance again. I would not attempt the exam without reading and understanding the required white papers.

If you aspire to take the AWS certified DevOps Engineer exam I hope this helps you a little bit. Good luck.

Log shipping through ELK

Wed, 19 Sep 2018 17:26:51 -0400

A common devops task is build logging pipeline with ELK stack (Elasticsearch, Logstash, Kibana). Suppose the application is written in Java and currently use log4j’s RollingFileAppender to generate log files locally. We can use log4j’s socket appender to write to Logstash, which further pushes the log stream to Elasticsearch cluster.

In this model, failure to push a log line results in the following in log4j output:

2018-09-06 11:05:39,778,ERROR,stderr - [AsyncAppender-Dispatcher-Thread-672] log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed)

The logstash log displays the socket exception as well:

[2018-09-06T10:12:21,935][DEBUG][logstash.inputs.log4j ] Accepted connection {:client=>"192.168.111.56:58118", :server=>"0.0.0.0:4560"}
[2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] filter received {"event"=>{"method"=>"?", "thread"=>"676774870@qtp-1804103302-6", "message"=>"Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.", "priority"=>"DEBUG", "type"=>"log4j", "path"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "@timestamp"=>2018-09-06T15:12:21.950Z, "file"=>"?:?", "@version"=>"1", "host"=>"192.168.111.56:58118", "logger_name"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "class"=>"?", "timestamp"=>1536246741950}}
[2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] output received {"event"=>{"method"=>"?", "thread"=>"676774870@qtp-1804103302-6", "message"=>"Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.", "priority"=>"DEBUG", "type"=>"log4j", "path"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "@timestamp"=>2018-09-06T15:12:21.950Z, "file"=>"?:?", "@version"=>"1", "host"=>"192.168.111.56:58118", "logger_name"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "class"=>"?", "timestamp"=>1536246741950}}
[2018-09-06T10:12:22,041][DEBUG][logstash.inputs.log4j ] Closing connection {:client=>"192.168.111.56:58118", :exception=>java.io.InvalidObjectException: Object type java.util.Hashtable is not allowed.}

The troubleshooting isn’t very straightforward. So its alternative is preferred. The alternative is to keep the existing RollingFileAppender as well as the local log files, but use a filebeat agent for each application node. Here is the diagram:

ELK Log Shipping Pipeline

Filebeat is a very light agent to be installed with the application on the same server or container.
Streams of text information congregate to logstash nodes. These log stash service pushes the converged stream to ElasticSearch cluster
ElasticSearch cluster ingest the stream for indexing
Kibana is responsible for viewing.

This is an example architecture with many potential variations. For example, log stash service may be deployed on the same hosts with ElasticSearch cluster. If there aren’t many application nodes, filebeat may directly push its output to ElasticSearch. This architecture provides a lot of flexibility and scalability.

With log4j format, two challenges to address are:

identifying multi-line entry in the log
mapping sections of a log entry into fields in Elastisearch

For example, the following log entry reflects both challenges above:

2018-09-06 11:52:18,022,DEBUG,service.dataportal.web.retrieve - [687285925@qtp-564051174-3350] creating streaming output request 9868594 and requid=fb9a2cad-e24b-4eb8-ad62-b027184e7b9b
2018-09-06 11:52:18,054,ERROR,org.glassfish.jersey.server.ServerRuntime$Responder - [687285925@qtp-564051174-3350] An I/O error has occurred while writing a re
sponse message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.mortbay.jetty.EofException
 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135)
 at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662)
 at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
 at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
 at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
 at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:326)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)
 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
 at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
 at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.mortbay.jetty.EofException
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:789)
 at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:568)
 at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1010)
 at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:330)
 at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:287)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:305)
 at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79)
 at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.spi.ContentEncoder.aroundWriteTo(ContentEncoder.java:137)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
 ... 40 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
 at java.net.SocketOutputStream.socketWrite0(Native Method)
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
 at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)
 at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:122)
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:723)
 ... 58 more

To address multi-line entry we need to tell filebeat how to identify the start of a line through its multiline.pattern configuration option:

- type: log
 enabled: true
 paths:
 - /var/log/dhunch/app.log
 multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
 multiline.negate: true
 multiline.match: after
 fields:
 product:
 name: dhunchapp
 log:
 type: app_log
 content: diagnostic

When filebeat pushes log entries to logstash, there will be an additional “time” column added, which reflects the time the log is ingested. The entire log4j message line is all put in a single field called “message”, this message field contains the timestamp from application, the thread, class, logging level and the actual diagnostic message. These are not mapped to separate sections making it difficult to search in Elasticsearch.

To address section mapping, the configuration is made on the logstash side. We will take advantage of a plugin called grok filter. The function of grok filter here, is to parse the original “message” field, map different sections into separate columns, respectively called logtime, loglevel, logclass, logthread and logmsg. The grok filter expression is as below:

filter {
 grok {
 match => { "message" => "%{TIMESTAMP_ISO8601:logtime},%{LOGLEVEL:loglevel},%{NOTSPACE:logclass} - \[%{DATA:logthread}\] %{GREEDYDATA:logmsg}" }
 }
 date {
 match => [ "logtime" , "yyyy-MM-dd HH:mm:ss,SSS" ]
 timezone => "America/Chicago"
 target => "@timestamp"
 }
 mutate {
 replace => [ "message" , "%{logmsg}" ]
 }
 mutate {
 remove_field => [ "logmsg" ]
 }
}

The date plugin indicates to Elasticsearch to treat this field as datetime with specified time zone. The mutate plugin below essentially renames the “logmsg” column to “message”, which allows Elasticsearch to understand this field as log message.