搭建ELK日志系统

ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成。

ElasticSearch是一个基于Apache Lucene(TM)的开源搜索引擎。Elasticsearch也使用Java开发并使用Lucene作为其核心来实现所有索引和搜索的功能，但是它的目的是通过简单的RESTful API来隐藏Lucene的复杂性，从而让全文搜索变得简单。

运行Elasticsearch

./bin/elasticsearc

如果想在后台以守护进程模式运行，添加-d参数。

修改配置config/elasticsearch.yml

cluster.name: my-application
node.name: node1
path.data: /path/to/data
path.logs: /path/to/logs
http.port: 9200

打开另一个终端进行测试

curl http://localhost:9200/
{
  "name" : "OYcr1Wi",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "H4eKj3cNToySgNscOVeVWA",
  "version" : {
    "number" : "5.0.1",
    "build_hash" : "080bb47",
    "build_date" : "2016-11-11T22:08:49.812Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.1"
  },
  "tagline" : "You Know, for Search"
}

这说明你的ELasticsearch集群已经启动并且正常运行

logstash

Logstash的功能如下：

其实它就是一个 收集器 而已，我们需要为它指定Input和Output（当然Input和Output可以为多个）。由于我们需要把Java代码中Log4j的日志输出到ElasticSearch中，因此这里的Input就是Log4j，而Output就是ElasticSearch。

unzip logstash-5.0.1.zip
sudo mv logstash-5.0.1 /usr/local/logstash
cd /usr/local/logstash/
vim config/logstash.yml

使用命令行命令调试：

./bin/logstash -e 'input { stdin { } } output { stdout {} }'
#The stdin plugin is now waiting for input:
hello
2016-11-22T14:14:57.851Z learnLinux hello
./bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'
goodnight
{
    "@timestamp" => 2016-11-22T14:35:40.557Z,
      "@version" => "1",
          "host" => "learnLinux",
       "message" => "goodnight",
          "tags" => []
}

通过修改output { stdout {}}的参数，我们改变了Logstash输出格式．类似我们可以修改input,filter,output配置，生成我们想要的格式化日志方便查询．

输出到elasticsearch,新建config/simple.conf文件

input { stdin { } }
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

./bin/logstash  -f config/simple.conf
#在终端输入一条访问日志
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
{
        "request" => "/xampp/status.php",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
        "message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
           "tags" => [],
       "referrer" => "\"http://cadenza/xampp/navi.php\"",
     "@timestamp" => 2013-12-11T08:01:45.000Z,
       "response" => "200",
          "bytes" => "3891",
       "clientip" => "127.0.0.1",
       "@version" => "1",
           "host" => "learnLinux",
    "httpversion" => "1.1",
      "timestamp" => "11/Dec/2013:00:01:45 -0800"
}

Kibana

Kibana 是一款基于 Apache 开源协议，使用JavaScript语言编写，为 Elasticsearch 提供分析和可视化的 Web 平台。它可以在 Elasticsearch 的索引中查找，交互数据，并生成各种维度的表图。

sudo mv kibana-5.0.1-linux-x86_64 /usr/local/kibana
cd /usr/local/kibana/
vim config/kibana.yml
server.port: 5601
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"
#启动kibana
./bin/kibana

查看kibana显示的数据

注意右上角是查询的时间范围，如果没有查找到数据，那么你就可能需要调整这个时间范围了，这里我选择Today：

Elasticsearch权威指南中文版