ELK

Nginx+Logstash+Elasticsearch+Kibana搭建网站日志分析系统笔记

Posted by jintang on 2016-11-12

原文地址：https://www.iamle.com/archives/1926.html

前言

流程,nignx格式化日志成json,通过logstash直接采集到elasticsearch,然后通过kibana gui界面展示分析

要点nignx日志成json格式,避免nignx默认日志是空格,需要正则匹配,导致logstash占过多cpu
elasticsearch机配置防火墙,只让指定的logstash机访问
kibana只监听本地127.0.0.1使用nignx方向代理,nginx中配置Http Basic Auth账号密码登陆

比较粗略的笔记,备忘
安装java
yum install java-1.8.0-openjdk*

nginx配置

为了让nignx机跑logstash采集日志负载最低，建议直接生成json的方式，直接就可以用logstash读取写入到Elasticsearch

http{} 中定义 格式化日志成json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
log_format logstash_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"http_x_forwarded_for":"$http_x_forwarded_for",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"request":"$request",'
'"url":"$uri",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';

server内输出日志 access_log可以配置多个同时输出，可以保留你以前的

access_log /data/wwwlogs/www.iamle.log iamle.com;
access_log /data/wwwlogs/www.iamle.com.logstash_json.log logstash_json;

nginx机安装Logstash1.5.x

1
2
3
4
5
6
7
8
9
10
11
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/logstash.repo <<EOF
[logstash-1.5]
name=logstash repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
EOF
yum clean all
yum install logstash

在目录 /etc/logstash/conf.d/
建立配置文件 nginx_json.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
input {
    file {
        path => "/data/wwwlogs/www.iamle.com.logstash_json.log"
        codec => json
    }
}
filter {
    mutate {
        split => [ "upstreamtime", "," ]
    }
    mutate {
        convert => [ "upstreamtime", "float" ]
    }
}

output {
    elasticsearch {
        host => "elk.server.iamle.com"
        protocol => "http"
        index => "logstash-%{type}-%{+YYYY.MM.dd}"
        index_type => "%{type}"
        workers => 5
        template_overwrite => true
    }
}

service logstash start

日志存储机安装Elasticsearch1.7.x提供数据底层支持

1
2
3
4
5
6
7
8
9
10
11
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-1.7]
name=Elasticsearch repository for 1.7.x packages
baseurl=http://packages.elastic.co/elasticsearch/1.7/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF
yum clean all
yum install elasticsearch

配置文件

配置数据保存位置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
vim /etc/elasticsearch/elasticsearch.yml
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
path.data: /data
目录会自动生成,只需要指定一个空目录就可以了

service elasticsearch start

centos7
systemctl start elasticsearch
systemctl status elasticsearch
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled)
Active: active (running) since Fri 2015-09-04 15:37:08 CST; 1s ago
Docs: http://www.elastic.co
Main PID: 19376 (java)
CGroup: /system.slice/elasticsearch.service
└─19376 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -X...
Sep 04 15:37:08 elk systemd[1]: Starting Elasticsearch...
Sep 04 15:37:08 elk systemd[1]: Started Elasticsearch.
检查是否已经成功开启
ss -ltnp |grep 9200

centos7配置firewalld固定ip可访问elasticsearch
systemctl start firewalld.service
systemctl status firewalld.service

只允许nignx机访问elasticsearch机9200 9300端口

1
2
3
4
5
6
7
8
9
10
11
firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \
source address="10.8.8.2" \
port protocol="tcp" port="9200" accept"

firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \
source address="10.8.8.2" \
port protocol="tcp" port="9300" accept"
firewall-cmd --reload

iptables -L -n |grep 9200
ACCEPT tcp -- 10.8.8.2 0.0.0.0/0 tcp dpt:9200 ctstate NEW

安装Kibana4展示Elasticsearch中的数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
tar zxvf kibana-4.1.1-linux-x64.tar.gz
cd kibana-4.1.1-linux-x64
修改配置文件
vim /usr/local/kibana-4.1.1-linux-x64/config/kibana.yml
# Kibana is served by a back end server. This controls which port to use.
port: 5601

# The host to bind the server to.
#监听本地地址 用nignx反向代理
host: "127.0.0.1"

nohup ./bin/kibana &

检查是否已经成功开启
ss -ltnp |grep 5601

使用nignx反向代理kibana
nginx配置Http Basic Auth账号密码登陆
http://trac.edgewall.org/export/10770/trunk/contrib/htpasswd.py （nginx wiki里推荐的）
运行示例
chmod 777 htpasswd.py
./htpasswd.py -c -b htpasswd username password

#-c为生成文件 htpasswd为文件名

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
server
{
    listen 80;
    #listen [::]:80;
    server_name elk.server.iamle.com;

    location / {
        auth_basic "Password please";
        auth_basic_user_file /usr/local/nginx/conf/htpasswd;
        proxy_pass http://127.0.0.1:5601/;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

老版本
https://download.elastic.co/kibana/kibana/kibana-3.1.3.tar.gz
https://www.elastic.co/downloads/past-releases/kibana-3-1-3

参考
http://kibana.logstash.es/
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html