Difference between revisions of "ElasticSearch"

 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
[[Category:Linux]]
 
[[Category:Linux]]
 +
 +
[[File:Icon elastic search.png|32px|caption|ElasticSearch]] ''ElasticSearch'' is the central point of the ELK architecture. This is where data will be aggregated and persisted.
  
  
Line 6: Line 8:
 
To install and use ELK you need:  
 
To install and use ELK you need:  
  
* '''JAVA 1.7.55+'''  
+
* '''JAVA 1.7.55+''' (Java 8 is recommended)
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
Line 17: Line 19:
 
* '''Firewall rule'''
 
* '''Firewall rule'''
  
Open the ports 9200 + 9300, allow multicast too. See [[Firewall INPUT filters#ElasticSearch|FW input]] && [[Firewall OUTPUT filters#IT_ports|FW output]]
+
Open the TCP ports 9200 + 9300, allow multicast too.  
 +
 
 +
See [[Firewall INPUT filters#ElasticSearch|FW input]] && [[Firewall OUTPUT filters#IT_ports|FW output]]
  
  
Line 23: Line 27:
  
 
See [[Template:Menu content web#Web_server|Apache2 setup]]
 
See [[Template:Menu content web#Web_server|Apache2 setup]]
 +
 +
 +
* Python
 +
 +
<syntaxhighlight lang="bash">
 +
apt-get install python3
 +
apt-get install python-pip
 +
</syntaxhighlight>
  
  
Line 69: Line 81:
  
 
<syntaxhighlight lang="yaml">
 
<syntaxhighlight lang="yaml">
cluster.name: VEHCO        # line 33      DAXIONGMAO
+
cluster.name: VEHCO        # line 33       
node.name: "VEHCO_MASTER"  # line 40     dev.daxiongmao.eu
+
node.name: "VEHCO_MASTER"  # line 40
  
 
                             ### [...] At the end
 
                             ### [...] At the end
 
http.cors.enabled: true
 
http.cors.enabled: true
http.cors.allow-origin: https://dev.daxiongmao.eu     # http://192.168.1.203
+
http.cors.allow-origin: https://smartcards.vehco.com     # http://192.168.1.203
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 82: Line 94:
 
==Extensions (plugins)==
 
==Extensions (plugins)==
  
You need to install some extensions (plugins) to get the full power of ElasticSearch. The following plugin list is the one recommended by ElasticSearch team.  
+
You need to install some extensions (plugins) to get the full power of ElasticSearch.  
 +
 
 +
The following plugins are the ones recommended by the ElasticSearch team.  
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
Line 107: Line 121:
  
 
Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.
 
Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.
 +
 +
 +
==ElasticSearch tooling==
 +
 +
===Curator===
 +
 +
Curator allows you to remove the old indices.
 +
 +
 +
'''Installation'''
 +
 +
<syntaxhighlight lang="bash">
 +
pip install elasticsearch-curator
 +
</syntaxhighlight>
 +
 +
 +
 +
'''Usage'''
 +
 +
<syntaxhighlight lang="bash">
 +
# Display VEHCO- indices
 +
curator show --show-indices --prefix vehco-
 +
# Remove indices that are more than 10 days old
 +
</syntaxhighlight>
 +
 +
 +
'''References'''
 +
 +
* Curator: https://github.com/elasticsearch/curator/
 +
  
  
Line 118: Line 162:
 
/etc/init.d/elasticsearch start
 
/etc/init.d/elasticsearch start
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
 +
=Delete indices=
 +
 +
Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html
 +
 +
 +
<syntaxhighlight lang="bash">
 +
curl -XDELETE 'http://localhost:9200/smartcard-monitoring-2014*/'
 +
</syntaxhighlight>
 +
 +
replace ''smartcard-monitoring-2014*'' by your own expression.
 +
  
  

Latest revision as of 15:54, 5 February 2015


ElasticSearch ElasticSearch is the central point of the ELK architecture. This is where data will be aggregated and persisted.


Requirements

To install and use ELK you need:

  • JAVA 1.7.55+ (Java 8 is recommended)
java -version

Java version must be > 1.7.0_55


  • Firewall rule

Open the TCP ports 9200 + 9300, allow multicast too.

See FW input && FW output


  • Apache2 server

See Apache2 setup


  • Python
apt-get install python3
apt-get install python-pip


ElasticSearch

Installation

Source: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html


  • Add ELK repository: see Sources#ELK
  • Install application
apt-get install elasticsearch

>> Binaries in /usr/share/elasticsearch

>> Configuration in /etc/elasticsearch

>> Logs in /var/log/elasticsearch


  • Register application as a service
cd /etc/init.d
update-rc.d elasticsearch defaults 95 10


Configuration

Edit the configuration file:

vim /etc/elasticsearch/elasticsearch.yml


Set your CLUSTER and NODE name + allow Kibana access.

cluster.name: VEHCO         # line 33      
node.name: "VEHCO_MASTER"   # line 40

                            ### [...] At the end
http.cors.enabled: true
http.cors.allow-origin: https://smartcards.vehco.com     # http://192.168.1.203

!! You need to adjust your IP || hostname according to your needs.


Extensions (plugins)

You need to install some extensions (plugins) to get the full power of ElasticSearch.

The following plugins are the ones recommended by the ElasticSearch team.

cd /usr/share/elasticsearch/bin
./plugin -install karmi/elasticsearch-paramedic
./plugin -install mobz/elasticsearch-head
./plugin -install royrusso/elasticsearch-HQ


More information about each plugin:


You can access the plugins using the /_plugin/ URL:


You can search for more plugins on Google or the official ElasticSearch web-site.

Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.


ElasticSearch tooling

Curator

Curator allows you to remove the old indices.


Installation

pip install elasticsearch-curator


Usage

# Display VEHCO- indices
curator show --show-indices --prefix vehco-
# Remove indices that are more than 10 days old


References



Start ElasticSearch

service elasticsearch start 

## OR ##
/etc/init.d/elasticsearch start


Delete indices

Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html


curl -XDELETE 'http://localhost:9200/smartcard-monitoring-2014*/'

replace smartcard-monitoring-2014* by your own expression.



References