Logstash

Collect, Enrich & Transport Data.

Logstash is a flexible, open source, data collection, enrichment, and transport pipeline designed to efficiently process a growing list of log, event, and unstructured data sources for distribution into a variety of outputs, including Elasticsearch.

logstash

  • Java 7 or higher is required for Logstash 2.x
houbinbindeMacBook-Pro:bin houbinbin$ java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

Install in Mac

Title: Install logstash Download->Config: logstash.conf Config->Run: bin/logstash -f conf/logstash.conf
  • Download

Download and unzip

houbinbindeMacBook-Pro:Downloads houbinbin$ tar -zxf logstash-all-plugins-2.4.0.tar.gz

move it into tools package

houbinbindeMacBook-Pro:Downloads houbinbin$ ls | grep -i logstash
logstash-2.4.0
logstash-all-plugins-2.4.0.tar.gz
houbinbindeMacBook-Pro:Downloads houbinbin$ mv logstash-2.4.0 ~/it/tools/logstash
  • Config

Prepare config file logstash.conf

houbinbindeMacBook-Pro:Downloads houbinbin$ cd ~/it/tools/logstash/
houbinbindeMacBook-Pro:logstash houbinbin$ ls
CHANGELOG.md		Gemfile			LICENSE			bin			vendor
CONTRIBUTORS		Gemfile.jruby-1.9.lock	NOTICE.TXT		lib
houbinbindeMacBook-Pro:logstash houbinbin$ mkdir conf
houbinbindeMacBook-Pro:logstash houbinbin$ cd conf/
houbinbindeMacBook-Pro:config houbinbin$ ls
houbinbindeMacBook-Pro:config houbinbin$ vi logstash.conf
houbinbindeMacBook-Pro:config houbinbin$ ls
logstash.conf

Edit it’s content like this for test:

input {
      stdin{}
}
# filter is optional
#filter {
#}
output {
    stdout{
        codec => rubydebug
    }
}
  • Run
houbinbindeMacBook-Pro:logstash houbinbin$ bin/logstash -f conf/logstash.conf
Settings: Default pipeline workers: 8
Pipeline main started

Enter content testing in terminal to test:

testing
{
       "message" => "testing",
      "@version" => "1",
    "@timestamp" => "2016-10-16T03:45:01.064Z",
          "host" => "houbinbindeMacBook-Pro.local"
}

Usage

  • Show plugin list
$   bin/plugin list
The use of bin/plugin is deprecated and will be removed in a feature release. Please use bin/logstash-plugin

houbinbindeMacBook-Pro:logstash houbinbin$ bin/logstash-plugin
Usage:
    bin/logstash-plugin [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
    SUBCOMMAND                    subcommand
    [ARG] ...                     subcommand arguments

Subcommands:
    install                       Install a plugin
    uninstall                     Uninstall a plugin
    update                        Update a plugin
    pack                          Package currently installed plugins
    unpack                        Unpack packaged plugins
    list                          List all installed plugins
    generate                      Create the foundation for a new plugin.

So, we can use bin/logstash-plugin list to show plugin list.

Default, it has plugin named logstash-input-log4j, it’s for log4j, not log4j2

1、It’s a pity that this plugin only support logstatsh version (1.5+, 2.1]

2、Before you install this plugin, you should start logstash first

start logstash

houbinbindeMacBook-Pro:logstash houbinbin$ bin/logstash -f conf/logstash.conf
Settings: Default pipeline workers: 8
Pipeline main started

install

houbinbindeMacBook-Pro:logstash houbinbin$ bin/logstash-plugin install logstash-input-log4j2
LogStash::GemfileError: duplicate gem logstash-filter-date
         add_gem at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/gemfile.rb:102
             gem at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/gemfile.rb:200
          (eval) at (eval):109
   instance_eval at org/jruby/RubyBasicObject.java:1598
           parse at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/gemfile.rb:188
            load at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/gemfile.rb:19
         gemfile at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/command.rb:4
  verify_remote! at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/install.rb:50
         execute at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/install.rb:28
             run at /Users/houbinbin/it/tools/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
         execute at /Users/houbinbin/it/tools/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
             run at /Users/houbinbin/it/tools/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
             run at /Users/houbinbin/it/tools/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
          (root) at /Users/houbinbin/it/tools/logstash/lib/pluginmanager/main.rb:43

En, that’s a question… It seems we should go another way ==!

Build Log Sys

ELK Blog zh_CN

Log4j2+ELK zh_CN

We want to build log system use log4j2 with ELK(ElasticSearch+Logstash+Kibana)

Log4j2

  • log4j2.xml

The whole project demo is here

The info about SocketAppender

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <Socket name="Logstash" host="127.0.0.1" port="7000" protocol="TCP">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Socket>
    </Appenders>

    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="Logstash"/>
        </Root>
    </Loggers>
</Configuration>
  • LogService.java

and the simple log as

public class LogService {
  static final Logger logger = LogManager.getLogger(LogService.class);

  public static void main(String[] args) {
    logger.info("log test...");
  }
}

LogStatsh

  • add micro_wiki.conf
input {
  tcp {
    host => "0.0.0.0"
    port => "7000"
    mode => "server"
    type => "microwiki"
    add_field => {
      "name" => "Ryo"
    }
  }
}
filter {
}
output {
  stdout {
    codec => rubydebug
  }
}
  • run logstash
houbinbindeMacBook-Pro:logstash-2.4.0 houbinbin$ bin/logstash -f conf/micro_wiki.conf
Settings: Default pipeline workers: 8
Pipeline main started
  • run java
{
       "message" => "16:44:10.428 [main] INFO  com.ryo.service.LogService - log test...",
      "@version" => "1",
    "@timestamp" => "2016-10-16T08:44:10.430Z",
          "host" => "127.0.0.1",
          "port" => 53150,
          "type" => "microwiki",
          "name" => "Ryo"
}

ElasticSearch

  • Edit the micro_wiki.conf

In order to let the logger of Logstash trans into ElasticSearch, we edit the micro_wiki.conf of Logstash, like this:

input {
  tcp {
    host => "0.0.0.0"
    port => "7000"
    mode => "server"
    type => "microwiki"
    add_field => {
      "name" => "Ryo"
    }
  }
}
filter {
}
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    action => "index"
    codec => rubydebug
    index => "microwiki-%{+YYYY.MM.dd}"
    template_name => "microwiki"
  }
}

After edit this, we should restart logstash, you may meet error like

Could not start TCP server: Address in use {:host=>"0.0.0.0", :port=>7000, :level=>:error}
Pipeline aborted due to error {:exception=>"Errno::EADDRINUSE", :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'",
"org/jruby/RubyIO.java:871:in `new'", "/Users/houbinbin/it/tools/logstash/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:244:in
`new_server_socket'", "/Users/houbinbin/it/tools/logstash/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:79:in `register'",
"/Users/houbinbin/it/tools/logstash/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:330:in `start_inputs'",
"org/jruby/RubyArray.java:1613:in `each'", "/Users/houbinbin/it/tools/logstash/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:329:in
`start_inputs'", "/Users/houbinbin/it/tools/logstash/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:180:in `start_workers'",
"/Users/houbinbin/it/tools/logstash/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in `run'",
"/Users/houbinbin/it/tools/logstash/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"],
:level=>:error}

use command lsof -n -P| grep 7000

idea      4138 houbinbin  txt      REG                1,4   1257000  100928 /Library/Fonts/Copperplate.ttc
Google    4142 houbinbin  txt      REG                1,4   1257000  100928 /Library/Fonts/Copperplate.ttc
java      6699 houbinbin    9u    IPv6 0xf1b9bfa3b241e019       0t0     TCP *:7000 (LISTEN)

use command sudo kill -9 id to kill it.

Last, start logstash bin/logstash -f conf/micro_wiki.conf

  • Edit the elasticsearch.yml
cluster.name: MicroWiki-Cluster
node.name: microwiki-node1
network.host: 127.0.0.1
http.port: 9200
  • Run elasticsearch

Use bin/elasticsearch -d Starts Elasticsearch in the background

  • Run java and use elasticsearch to search
LOGGER.info("log test with 2016-10-16 17:06:02...");

Enter http://localhost:9200/microwiki-2016.10.16/_search in browser, and get:

{
    "took":21,
    "timed_out":false,
    "_shards":{
        "total":5,
        "successful":5,
        "failed":0
    },
    "hits":{
        "total":1,
        "max_score":1,
        "hits":[
            {
                "_index":"microwiki-2016.10.16",
                "_type":"microwiki",
                "_id":"AVfMvhC_IjTkofXOa5qh",
                "_score":1,
                "_source":{
                    "message":"17:06:30.417 [main] INFO com.ryo.service.LogService - log test with 2016-10-16 17:06:02...",
                    "@version":"1",
                    "@timestamp":"2016-10-16T09:06:30.421Z",
                    "host":"127.0.0.1",
                    "port":53511,
                    "type":"microwiki",
                    "name":"Ryo"
                }
            }
        ]
    }
}

Kibana

  • Edit kibana.yml

Connect kibana with elasticsearch.

# Kibana is served by a back end server. This controls which port to use.
# server.port: 5601
server.port: 5601

# The host to bind the server to.
# server.host: "0.0.0.0"
server.host: 127.0.0.1

# If you are running kibana behind a proxy, and want to mount it at a path,
# specify that path here. The basePath can't end in a slash.
# server.basePath: ""

# The maximum payload size in bytes on incoming server requests.
# server.maxPayloadBytes: 1048576

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://localhost:9200"
  • Run
  houbinbindeMacBook-Pro:kibana houbinbin$ bin/kibana

  log   [17:15:05.664] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
  log   [17:15:05.688] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [17:15:05.704] [info][status][plugin:kbn_vislib_vis_types@1.0.0] Status changed from uninitialized to green - Ready
  log   [17:15:05.721] [info][status][plugin:markdown_vis@1.0.0] Status changed from uninitialized to green - Ready
  log   [17:15:05.724] [info][status][plugin:metric_vis@1.0.0] Status changed from uninitialized to green - Ready
  log   [17:15:05.726] [info][status][plugin:spyModes@1.0.0] Status changed from uninitialized to green - Ready
  log   [17:15:05.739] [info][status][plugin:statusPage@1.0.0] Status changed from uninitialized to green - Ready
  log   [17:15:05.745] [info][status][plugin:table_vis@1.0.0] Status changed from uninitialized to green - Ready
  log   [17:15:05.749] [info][listening] Server running at http://127.0.0.1:5601
  log   [17:15:10.753] [info][status][plugin:elasticsearch@1.0.0] Status changed from yellow to yellow - No existing Kibana index found
  log   [17:15:13.332] [info][status][plugin:elasticsearch@1.0.0] Status changed from yellow to green - Kibana index ready
  • Visit

Enter localhost:5601, and add index like this:

add index

默认为 logstash-*, 可以修改为 microwiki-* 即可。

可关注是否有log4j2的插件,不用如此麻烦。

More

logstash learn zh_CN

logstash api zh_CN