Difference between revisions of "Logstash"

 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
[[Category:Linux]]
 
[[Category:Linux]]
 +
 +
[[File:Icon logstash.png|32px|caption|Logstash]] ''Logstash'' retrieves, extracts and sends data to the ''ElasticSearch'' server.
 +
 +
''Logstash'' needs to be install on all the servers / clients you want to get logs from.
 +
 +
 +
Related topics:
 +
* [[File:Grok icon.png|link=Logstash grok expressions|160px|caption|Logstash grok expressions]] [[Logstash grok expressions]]
 +
* [[File:Configuration examples.png|link=Logstash configuration examples|64px|caption|Logstash configuration examples]] [[logstash configuration examples]]
  
  
 
=Installation=
 
=Installation=
  
You can install logstash either manually or as an APT-GET package. I recommend you to use the ''manual'' installation because the ''automatic'' one will chroot you in /var/log. If your application is using logs that are somewhere else, then you'll be screwed.  
+
You can install logstash either manually or as an APT-GET package.  
 +
 
 +
I recommend you to use the ''manual'' installation because the ''automatic'' one will chroot you in /var/log. If your application is using logs that are somewhere else, then you'll be screwed.  
  
  
Line 18: Line 29:
 
cd /tmp
 
cd /tmp
 
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
 
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar xjvf logstash-1.4.2.tar.gz
+
tar xzvf logstash-1.4.2.tar.gz
 
rm logstash-1.4.2.tar.gz
 
rm logstash-1.4.2.tar.gz
 
mv logstash-1.4.2/ /opt/
 
mv logstash-1.4.2/ /opt/
Line 188: Line 199:
 
* Add a new environment variable in your <code>/etc/profile</code> || <code>/etc/environment</code>
 
* Add a new environment variable in your <code>/etc/profile</code> || <code>/etc/environment</code>
  
<syntaxhighlight lang="bash">
+
Put:
# Where should Logstash keep track of each file?
 
export SINCEDB_DIR="/etc/logstash/db"
 
</syntaxhighlight>
 
 
 
 
 
 
 
=Configuration=
 
 
 
 
 
==GROK==
 
 
 
Grok is used to split a log message into fields.
 
 
 
 
 
 
 
===Grok tools===
 
 
 
You can create your own ''grok'' patterns and test them with the on-line processor over here: http://grokdebug.herokuapp.com/
 
 
 
 
 
 
 
===Apache2 error===
 
 
 
Create configuration file:
 
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
vim /etc/logstash/grok/apache2ErrorLog.grok
+
SINCEDB_DIR=/etc/logstash/db
 
</syntaxhighlight>
 
</syntaxhighlight>
  
  
Put the following content:
+
Apply changes:
 
 
<syntaxhighlight lang="perl">
 
HTTPERRORDATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
 
APACHEERRORLOG \[%{HTTPERRORDATE:timestamp}\] \[%{WORD:severity}\] \[client %{IPORHOST:clientip}\] %{GREEDYDATA:message_remainder}
 
</syntaxhighlight>
 
 
 
 
 
 
 
===IpTables===
 
 
 
Create configuration file:
 
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
vim /etc/logstash/grok/iptables.grok
+
source /etc/environment
 
</syntaxhighlight>
 
</syntaxhighlight>
  
  
Put the following content:
 
  
<syntaxhighlight lang="perl">
 
NETFILTERMAC %{COMMONMAC:dst_mac}:%{COMMONMAC:src_mac}:%{ETHTYPE:ethtype}
 
ETHTYPE (?:(?:[A-Fa-f0-9]{2}):(?:[A-Fa-f0-9]{2}))
 
# IPv6 + v4
 
IPTABLES %{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME} .* IN=(%{WORD:in_device})? OUT=(%{WORD:out_device})? *(MAC=%{NETFILTERMAC})? \
 
SRC=%{IP:src_ip} DST=%{IP:dst_ip} *(LEN=%{INT:pkt_length})? *(TOS=%{BASE16NUM:pkt_tos})? *(PREC=%{BASE16NUM:pkt_prec})? \
 
*(TTL=%{INT:pkt_ttl})? ID=%{INT:pkt_id} .* *(PROTO=%{WORD:protocol}) SPT=%{INT:src_port} DPT=%{INT:dst_port} \
 
*(WINDOW=%{INT:pkt_window})? *(RES=%{BASE16NUM:pkt_res})? .* *(URGP=%{INT:pkt_urgp})?
 
# IPv4 only
 
IPTABLES_V4 %{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME} .* IN=(%{WORD:in_device})? OUT=(%{WORD:out_device})? *(MAC=%{NETFILTERMAC})? \
 
SRC=%{IPV4:src_ip} DST=%{IPV4:dst_ip} *(LEN=%{INT:pkt_length})? *(TOS=%{BASE16NUM:pkt_tos})? *(PREC=%{BASE16NUM:pkt_prec})? \
 
*(TTL=%{INT:pkt_ttl})? ID=%{INT:pkt_id} .* *(PROTO=%{WORD:protocol}) SPT=%{INT:src_port} DPT=%{INT:dst_port} \
 
*(WINDOW=%{INT:pkt_window})? *(RES=%{BASE16NUM:pkt_res})? .* *(URGP=%{INT:pkt_urgp})?
 
</syntaxhighlight>
 
  
 +
=Manual commands=
  
 
+
The following command(s) are just here for my personal reference:
===Fail2ban===
 
 
 
Create configuration file:
 
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
vim /etc/logstash/grok/fail2ban.grok
+
cd /opt/logstash/bin
 +
./logstash -f /etc/logstash/conf.d/ -t --verbose
 
</syntaxhighlight>
 
</syntaxhighlight>
  
  
Put the following content:
 
  
<syntaxhighlight lang="perl">
+
=Grok expressions=
FAIL2BAN %{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria}: %{LOGLEVEL:level} \[%{WORD:service}\] Ban %{IPV4:clientip}
 
</syntaxhighlight>
 
  
 +
See [[Logstash grok expressions]]
  
  
  
==Logstash (Common services)==
+
=Configuration examples=
 
 
More informations about GeoIP: http://logstash.net/docs/latest/filters/geoip
 
 
 
===Apache2===
 
 
 
Requirements:
 
* Make sure your logs are in "/var/log/apache2" or adjust the paths
 
* Make sure your using the COMBINED logs (default in Apache 2.4+)
 
 
 
 
 
Logstash configuration extract:
 
 
 
<syntaxhighlight lang="make">
 
input {
 
file {
 
    path => [ "/var/log/apache2/access.log", "/var/log/apache2/ssl_access.log", "/var/log/apache2/other_vhosts_access.log" ]
 
    type => "apache-access"
 
}
 
file {
 
    path => "/var/log/apache2/error.log"
 
    type => "apache-error"
 
}
 
}
 
 
 
filter {
 
# ------------------------ Parse services logs into fields ---------------------------
 
# APACHE 2
 
if [type] == "apache-access" {
 
# To process log data (message's content) using some regex or precompiled GROK pattern
 
grok {
 
match => [ "message", "%{COMBINEDAPACHELOG}"]
 
}
 
# To extract log's time according to a date pattern
 
date {
 
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
 
}
 
# Extraction browser information, if available.
 
if [agent] != "" {
 
useragent {
 
source => "agent"
 
}
 
}
 
if [clientip] != "" {
 
geoip {
 
source => "clientip"       
 
                                target => "apache_clientip"
 
                                add_tag => [ "geoip" ]
 
                        }
 
                }
 
}
 
 
 
if [type] == "apache-error" {
 
grok {
 
match => [ "message", "%{APACHEERRORLOG}"]
 
# Directory where to find the custom patterns
 
patterns_dir => ["/etc/logstash/grok"]
 
}
 
                if [clientip] != "" {
 
geoip {
 
source => "clientip"       
 
                                target => "apache_clientip"
 
                                add_tag => [ "geoip" ]
 
}
 
}
 
}
 
}
 
 
 
output {
 
  ...
 
}
 
 
 
</syntaxhighlight>
 
 
 
 
 
 
 
===IpTables===
 
 
 
Requirements:
 
* Make sure you are logging dropped packets into a dedicated file. See [[Firewall log dropped]]
 
 
 
Logstash configuration extract:
 
 
 
<syntaxhighlight lang="make">
 
input {
 
file {
 
    path => "/var/log/iptables.log"
 
    type => "iptables"
 
}
 
}
 
 
 
 
 
filter {
 
# IPTABLES
 
if [type] == "iptables" {
 
grok {
 
match => ["message", "%{IPTABLES}"]
 
patterns_dir => ["/etc/logstash/grok"]
 
}
 
                # Default 'geoip' == src_ip. That means it's easy to display the DROPPED INPUT :)
 
                if [src_ip] != "" {
 
geoip {
 
        source => "src_ip"
 
                                add_tag => [ "geoip" ]
 
        target => "src_geoip"
 
}
 
}
 
                if [dst_ip] != "" {
 
geoip {
 
        source => "dst_ip"
 
                                add_tag => [ "geoip" ]
 
        target => "dst_geoip"
 
}
 
}
 
}
 
}
 
 
 
output {
 
  ...
 
}
 
 
 
</syntaxhighlight>
 
 
 
 
 
 
 
===Fail2ban===
 
 
 
Logstash configuration extract:
 
 
 
<syntaxhighlight lang="make">
 
input {
 
file {
 
    path => "/var/log/fail2ban.log"
 
    type => "fail2ban"
 
}
 
}
 
 
 
 
 
filter {
 
# Fail2ban
 
if [type] == "fail2ban" {
 
grok {
 
match => ["message", "%{FAIL2BAN}"]
 
patterns_dir => ["/etc/logstash/grok"]
 
}
 
                if [ban_ip] != "" {
 
geoip {
 
        source => "ban_ip"
 
                                add_tag => [ "geoip" ]
 
        target => "ban_geoip"
 
}
 
}
 
}
 
}
 
 
 
output {
 
  ...
 
}
 
 
 
</syntaxhighlight>
 
 
 
 
 
 
 
===Syslog===
 
 
 
Logstash configuration extract:
 
 
 
<syntaxhighlight lang="make">
 
input {
 
file {
 
    path => [ "/var/log/syslog", "/var/log/auth.log", "/var/log/mail.info" ]
 
    type => "syslog"
 
}
 
}
 
 
 
 
 
filter {
 
# SYSLOG
 
if [type] == "syslog" {
 
grok {
 
match => ["message", "%{SYSLOGBASE}"]
 
}
 
}
 
}
 
 
 
output {
 
  ...
 
}
 
 
 
</syntaxhighlight>
 
 
 
 
 
 
 
===Tomcat===
 
 
 
... TO BE DONE ...
 
 
 
 
 
 
 
 
 
 
 
==Logstash (Application logs)==
 
 
 
To be done: LOG4J logs
 
 
 
 
 
 
 
 
 
 
 
=Manual commands=
 
 
 
The following command(s) are just here for my personal reference:
 
 
 
<syntaxhighlight lang="bash">
 
cd /opt/logstash/bin
 
./logstash -f /etc/logstash/conf.d/ -t --verbose
 
</syntaxhighlight>
 
 
 
  
 +
See [[logstash configuration examples]]
  
  

Latest revision as of 16:45, 5 February 2015


Logstash Logstash retrieves, extracts and sends data to the ElasticSearch server.

Logstash needs to be install on all the servers / clients you want to get logs from.


Related topics:


Installation

You can install logstash either manually or as an APT-GET package.

I recommend you to use the manual installation because the automatic one will chroot you in /var/log. If your application is using logs that are somewhere else, then you'll be screwed.


Manual installation (recommended)

Be careful: Logstash version must match the ElasticSearch version for better performances.


cd /tmp
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar xzvf logstash-1.4.2.tar.gz
rm logstash-1.4.2.tar.gz
mv logstash-1.4.2/ /opt/
cd /opt
ln -s /opt/logstash-1.4.2 /opt/logstash


  • Create configuration directories
mkdir -p /etc/logstash/conf.d
mkdir /etc/logstash/grok
mkdir /etc/logstash/db
chmod -R 777 /etc/logstash


  • touch log file
touch /var/log/logstash.log
chmod -R 777 /var/log/logstash.log


  • Create an init.d script
cd /etc/init.d
vim logstash.sh


Parse the following content:

#!/bin/sh
### BEGIN INIT INFO
# Provides: logstash
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start daemon at boot time
# Description: Enable service provided by daemon.
### END INIT INFO

. /lib/lsb/init-functions

if [ $(id -u) -ne 0 ]; then
	echo -e " " 
	echo -e "!!!!!!!!!!!!!!!!!!!!" 
	echo -e "!! Security alert !!" 
	echo -e "!!!!!!!!!!!!!!!!!!!!" 
	echo -e "You need to be root or have root privileges to run this script!\n\n"
	echo -e " " 
	exit 1
fi

# Where should Logstash keep track of each file?
export SINCEDB_DIR="/etc/logstash/db"

# Logstash params
name="logstash"
logstash_bin="/opt/logstash/bin/logstash"
logstash_conf="/etc/logstash/conf.d/"
logstash_log="/var/log/logstash.log"
pid_file="/var/run/$name.pid"

start () {
	commandOpts="agent -f $logstash_conf --log ${logstash_log} --verbose"
	log_daemon_msg "Starting $name" "$name"
	if start-stop-daemon --start --quiet --oknodo --pidfile "$pid_file" -b -m --exec $logstash_bin -- $commandOpts; then
		log_end_msg 0
	else
		log_end_msg 1
	fi
}
testConfig () {
	echo "#############################"
	echo " Logstash configuration test"
	echo "#############################"
	command="${logstash_bin} -f $logstash_conf --verbose -t"
	$command
}
stop () {
	log_daemon_msg "Stopping $name" "$name"
	start-stop-daemon --stop --quiet --oknodo --pidfile "$pid_file"
}
status () {
	status_of_proc -p $pid_file "" "$name"
}

case $1 in
	start)
		if status; then exit 0; fi
		start
		;;
	stop)
		stop
		;;
	reload)
		stop
		start
		;;
	restart)
		stop
		start
		;;
	status)
		status && exit 0 || exit $?
		;;
	testConfig)
		testConfig
		;;
	*)
		echo "Usage: $0 {start|stop|restart|reload|status|testConfig}"
		exit 1
		;;
esac
exit 0


  • Create symlinks
ln -s /etc/init.d/logstash.sh /usr/bin/logstash


  • Register application as a service (optional)
cd /etc/init.d
update-rc.d logstash.sh defaults


Automatic installation

Source: http://logstash.net/docs/latest/repositories

  • Add Logstash repository: see Sources#ELK
  • Install application
apt-get install logstash logstash-contrib

>> Binaries in /opt/logstash

>> Configuration in /etc/logstash/conf.d/

>> Logs in /var/log/logstash/


  • Create a folder for logstash to keep track of each file
mkdir -p /etc/logstash/db
chmod -R 777 /etc/logstash/


  • Add a new environment variable in your /etc/profile || /etc/environment

Put:

SINCEDB_DIR=/etc/logstash/db


Apply changes:

source /etc/environment



Manual commands

The following command(s) are just here for my personal reference:

cd /opt/logstash/bin
./logstash -f /etc/logstash/conf.d/ -t --verbose


Grok expressions

See Logstash grok expressions


Configuration examples

See logstash configuration examples


References