logstash

Setting up Logstash, for Apache access logs

Configuration to get Apache Access logs

In this case, we will run LogStash on each server where an Apache web server is running. In our Apache setup, we've enabled the Apache Combined Access Log for each of our Apache Virtual servers.
In this case, we have one virtual machine running Apache 2.4.18 with a different access log per Virtual Host. So an extract of the configuration looks like:

<VirtualHost site1>
    DocumentRoot /var/www/site1
    <...>
    CustomLog /var/log/apache2/site1.log combined
</VirtualHost>

Setting up Logstash, for Syslog

Configuration to get Syslog messages

This is the configuraton of this Logstash instance. We will use the syslog input model to listen for syslog messages from all our hosts.
We will start the Logstash on server "logstash-runner", then we will configure Rsyslog.

input {
  syslog {
    port => "10514"
    add_field => {
      "source" => "syslog"
    }
  }
}
filter {
  if [source] == "syslog" {
    grok {

Setting up Logstash, for Tweets

Configuration to get Tweets

input {
  twitter {
    consumer_key => "<your twitter API consumer key>"
    consumer_secret => "<your twitter API consumer secret>"
    oauth_token => "<your twitter API application token>"
    oauth_token_secret => "<your twitter API application token secret>"
    keywords => [ "thevoicebe" ]
    add_field => { "source" => "tweets" }
    full_tweet => false
  }
}
filter {
  if [type] == "tweets" {
    mutate {

Setting up Logstash - the basics

Setting up Logstash, the basics


Installing Logstash on the server (done with Ubuntu 16.04 LTS).
    server1(admin) ~$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
    server1(admin) ~$ sudo apt-get install apt-transport-https
    server1(admin) ~$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
    server1(admin) ~$    sudo apt-get update && sudo apt-get install logstash