Elastic/Logstash/Kibana Integration

We can integrate the Poppins application logfile with the Elastic/Logstash/Kibana stack so we can monitor the Poppins backup jobs.

Poppins logfile

The Poppins application logfile looks like this:

"2015-11-12 00:28:21" "ocampo" "SUCCESS" "00:13:21" "/root/poppins/logs/ocampo.2015-11-12_001500.poppins.success.log" "0.1"
"2015-11-12 00:45:00" "martins" "SUCCESS" "00:30:00" "/root/poppins/logs/martins.2015-11-12_001500.poppins.success.log" "0.1"
"2015-11-12 01:14:39" "hong" "ERROR" "00:59:38" "/root/poppins/logs/hong.2015-11-12_001500.poppins.error.log" "0.1"
"2015-11-12 01:22:19" "kingsley" "SUCCESS" "01:07:18" "/root/poppins/logs/kingsley.2015-11-12_001500.poppins.success.log" "0.1"

We will split the Poppins logfile into individual, meaningful field entries which can be indexed by Elasticsearch. Once they have been indexed, we can query them with Kibana.

Let’s dissect the logfile entries. Let’s take the following entry:

"2015-11-12 00:28:21" "ocampo" "SUCCESS" "00:13:21" "/root/poppins/logs/ocampo.2015-11-12_001500.poppins.success.log" "0.1"
Type of field Arbitrary name we choose to map it to Field in the logstash logfile Summary
Date/Time timestamp (“2015-11-12 00:28:21”) Start of backup job.
Hostname servername (“ocampo”) Remote host.
Word result (“SUCCESS”) Result of the backup job: SUCCESS, ERROR or WARNING.
Time value duration (“00:13:21”) Duration of the backup job.
Some text logfile (“/root/poppins/logs/..”) Location of the remote host logfile on disk.
Version number poppinsversion (“0.1”) Poppins version

Import into Logstash

Creating a custom import filter for Logstash can be a tricky operation. Using Grok Constructor, we can easily construct the Logstash input filter:

Logfile:

"2015-11-12 00:28:21" "ocampo" "SUCCESS" "00:13:21" "/root/poppins/logs/ocampo.2015-11-12_001500.poppins.success.log" "0.1"

Logstash Filter:

"%{TIMESTAMP_ISO8601:timestamp}" "%{HOSTNAME:servername}" "%{WORD:result}" "%{TIME:duration}" "%{GREEDYDATA:logfile}" "%{GREEDYDATA:poppinsversion}"

Now that we’ve created our Logstash filter for our Poppins log entries, we need to implement it into our logstash input filter. Because we created this filter ourselves, we will need to define it.

Create a Logstash filter for Poppins as follows:

Filename: “14-poppinslogs.conf“
Contents:
filter {
if [type] == "poppinslogs" {
grok {
patterns_dir => "/<path>/<to>/<patterndir>/patterns"
match => { "message" => "%{POPPINSLOG}" }
}
}
}

In the above example we use our own self defined patterns, but the following definition would have worked as well:

filter {
if [type] == "poppinslogs" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}" "%{HOST:server}" "%{WORD:result}" "%{TIME:startrun}" "%{GREEDYDATA:logfile}" "%{GREEDYDATA:poppinsversion}"
}

The defination %{POPPINSLOG} is defined in the following file:

“/<path>/<to>/<patterndir>/patterns/poppinslogs”

The content looks like this::

POPPINSCONSTRUCT "%{TIMESTAMP_ISO8601:timestamp}" "%{HOST:server}" "%{WORD:result}" "%{TIME:startrun}" "%{GREEDYDATA:logfile}" "%{GREEDYDATA:poppinsversion}"
POPPINSLOG %{POPPINSCONSTRUCT}

Forward

Finally, we need to configure the logstashforwarder to send the poppinslogs to the Logstash server.

Thanks to Xavier Tomaszynski for this article.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s