Wednesday, June 22, 2016

Graylog - Log analytics

GrayLog
We are all familiar with the new hype of the ELK stack (https://www.elastic.co/webinars/introduction-elk-stack).
The idea of sending logs to a central place with the option of textual analysis using elasticsearch is a very big advantage. In this blog I will go over a different stack which is called graylog. Graylog actually started before ELK and in the architecture has the option to switch out elasticsearch if a new analyzer comes along.
The main advantage of graylog over ELK is that graylog has user management of the dashboard, and also housekeeping for elasticsearch so it does not grow too big.
Each tool has it’s pros and cons, so if you need to choose the right one for you, have a look at http://blog.takipi.com/the-7-log-management-tools-you-need-to-know/.
The basic architecture of graylog is as follows:
simple_setup.png
As you can see, graylog advises that all communications go through the graylog server and not directly to the other services (mainly elasticsearch). Though if you prefer you can query the elasticsearch directly (see https://github.com/searchbox-io/Jest for a client api that communicates over rest with es).

Configuration

You can use the instructions from the graylog web site for installations (http://docs.graylog.org/en/2.0/pages/installation.html), i will just add my experience from installing on amazon ec2 platform.
Note: We would like to have all our data on a /data partition.

MongoDB

MongoDB is used for the database for all user management and configuration of the web site. In addition if there are errors sending documents to the elasticsearch (that can happen if the there are errors in the field types), a copy of the document will be stored in the mongodb for analysis.

Setup repo

The repo needs to be updated with the following information:
sudo vi /etc/yum.repos.d/mongodb-org-3.2.repo
Add to file:
[mongodb-org-3.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.2/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc

Install

After updating the mongodb-org-3.2.repo file, run to install:
sudo yum install -y mongodb-org

Configuration

Since we want all the data files to be under /data/mongodb, we need to create the folder, and assign the permissions (user group mongod:mongod):


mkdir /data/mongodb
sudo chown mongod:mongod /data/mongodb/
We then need to configure mongo to use the new location. Update the /etc/mongo.conf file:
dbPath: /data/mongodb

Start Service

sudo service mongod start
Check the log file to see all is ok:
sudo less /var/log/mongodb/mongod.log

ElasticSearch

The next piece of the puzzle is the elasicsearch. Here also you need to add the repo, for more information see: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html

Setup repo


Create repo file
sudo vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Install

After updating the elasticsearch.repo file, run to install:
sudo yum install elasticsearch

Configuration

Since we want all the data files to be under /data/elasticsearch, we need to create the folder, and assign the permissions (user group elasticsearch:elasticsearch):


mkdir /data/elasticsearch
sudo chown elasticsearch:elasticsearch /data/elasticsearch/
We then need to configure elasticsearch to use the new location. Update the elasticsearch.yml file:
sudo vi /etc/elasticsearch/elasticsearch.yml
The following parameters need to be updated:
path.data: /data/elasticsearch
cluster.name: graylog
Network.host: 0.0.0.0

Start Service

sudo service elasticsearch start
Check the log file to see all is ok:
sudo less /var/log/elasticsearch/graylog.log

Java 8

The following instructions are taken from the following site:

Download java
Update the yum packages:
yum update
Download the latest java version:
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.rpm"

Install the rpm:
sudo rpm -ivh jdk-8u91-linux-x64.rpm
Check the version by typing
java -version

GrayLog

Download

Install

sudo yum install graylog-server

Configuration

Please go over document, http://docs.graylog.org/en/2.0/pages/installation/manual_setup.html since parameters might have changed between versions
Edit server.conf
sudo vi /etc/graylog/server/server.conf
is_master = true
password_secret = result of  “pwgen -N 1 -s 96”
root_password_sha2 = echo -n yourpassword | shasum -a 256
elasticsearch_cluster_name = graylog (must be the same that we put in elasticsearch)
Start Service
sudo service graylog-server start
Check the log file to see all is ok:
sudo tail -100f /var/log/graylog-server/server.log

NGINX

Graylog has a web interface and a rest service. Each one uses a different port. For full configuration see: http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html?highlight=nginx

Installation

sudo yum install nginx

Configuration

Put the following configuration in the file (note that the ip of the machine must be the correct one) :
/etc/nginx/conf.d/graylog2.conf
sudo vi /etc/nginx/conf.d/graylog2.conf
server
{
 listen      80 default_server;
 listen      [::]:80 default_server ipv6only=on;
 server_name your_ip;
location /api/
   {
       proxy_set_header    Host $http_host;
       proxy_set_header    X-Forwarded-Host $host;
       proxy_set_header    X-Forwarded-Server $host;
       proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_pass          http://127.0.0.1:12900/;
   }
 location /
   {
       proxy_set_header    Host $http_host;
       proxy_set_header    X-Forwarded-Host $host;
       proxy_set_header    X-Forwarded-Server $host;
       proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header    X-Graylog-Server-URL http://your_ip/api;
       proxy_pass          http://127.0.0.1:9000;
   }
}
From the main configuration (etc/nginx/nginx.conf) remove the server section

Start Service

sudo service nginx start
Check the log file to see all is ok:
sudo less /var/log/nginx/error.log
sudo less /var/log/nginx/access.log


Running and configuration Graylog

To access the graylog server you must open on amazon the http port.

First login

Use the username “admin” with the password you created the hash for before.

Setup Inputs

You need to decide what information you are going to be sending to the server and how you want to send it.
Simple options:
  1. HTTP post messages
You must first configure the input source
From the system -> inputs form
Example of how to send:
curl -XPOST http://graylog.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'
  1. UDP messages (GELP)
You must first configure the input source
  1. From the system -> inputs form
  2. Select “GELF UDP” and choose “launch new input”
  3. Give a name, and choose the node
  4. Click save

Sending Messages from code to graylog

Using GELF from code

To have your log files sent to the graylog server you should add the following appender.
Some of the advantages of GELF is that it is udp, but supports log events larger than a udp package (supports chunking & compression - see http://docs.graylog.org/en/2.0/pages/gelf.html)
<appender name="GRAYLOG2" class="com.github.pukkaone.gelf.logback.GelfAppender">
 <graylogHost>${graylog.host}</graylogHost>
 <originHost>Production_US</originHost>
 <levelIncluded>true</levelIncluded>
 <loggerIncluded>true</loggerIncluded>
 <markerIncluded>false</markerIncluded>
 <mdcIncluded>true</mdcIncluded>
 <threadIncluded>false</threadIncluded>
</appender>

Creating fields in event for later queries

The text of the log event will be sent to elasticseach for analysis. If you need more fields that are not in the text that you want to be part of the event, logback has a feature called Mapped Diagnostic Context (MDC) - see http://logback.qos.ch/manual/mdc.html.
In short what it does, is in the code you call static methods (it is thread safe) and all the parameters before a call to the log. For example:
MDC.put("first", "Dorothy");
MDC.put("last", "Parker");
logger.info("I am not a crook.");
This will add the parameters “first” and “last” as separate parameters that will be sent to the log. To use this you must configure the GelfAppender: <mdcIncluded>true</mdcIncluded>
For regular logs if you want to use the parameters you need to add them to the pattern matching, for examples: <Pattern>%X{first} %X{last} - %m%n</Pattern>.
When using MDC do not forget to remove the parameters after sending to the log, since they will stay on the threadlocal storage, and be used for the next log. For example:
MDC.remove("first");
MDC.remove("last");





Posted by at No comments:
Labels: , , , , ,
Subscribe to: Posts (Atom)