ELKF日志系统收集nginx日志（非SSL非集群）-工具盒子

当前系统版本RockyLinux8.9，配置4核8G，nginx安装参考wlnmp一键安装包，自行关闭或配置selinux、firewalld。

日志收集流程：Filebeat将日志数据发送到Logstash，Logstash进行过滤、转换，然后将数据发送到Elasticsearch进行存储。最后，Kibana通过可视化界面允许用户对存储在Elasticsearch中的数据进行查询和分析。

1、安装jdk11

我这里使用是Oracle的jdk-11.0.21_linux-x64_bin.rpm

2、安装elasticsearch

我这里使用的是直接从官方下载的rpm包（elasticsearch-8.11.4-x86_64.rpm），上传到服务器进行安装
yum install elasticsearch-8.11.4-x86_64.rpm -y

|---|------------------------------------------------| | 1 | yum install elasticsearch-8.11.4-x86_64.rpm -y |

安装完成后，看到大致如下内容：

The generated password for the elastic built-in superuser is : TaO5MlQt2SAZJzWvv16F（elasticsearch密码）
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic（重置elastic内置超级用户的密码）
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana（为Kibana生成token）
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node（为Elasticsearch节点生成token）

3、创建elasticsearch存储及日志目录
mkdir -p /data/elasticsearch/{logs,data}

|---|------------------------------------------| | 1 | mkdir -p /data/elasticsearch/{logs,data} |

4、修改目录权限
chown -R elasticsearch.elasticsearch /data/elasticsearch

|---|----------------------------------------------------------| | 1 | chown -R elasticsearch.elasticsearch /data/elasticsearch |

5、配置elasticsearch

修改elasticsearch.yml配置
vim /etc/elasticsearch/elasticsearch.yml

|---|------------------------------------------| | 1 | vim /etc/elasticsearch/elasticsearch.yml |

将
path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch #bootstrap.memory_lock: true #network.host: 192.168.0.1 #http.port: 9200

|-----------|----------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 | path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch #bootstrap.memory_lock: true #network.host: 192.168.0.1 #http.port: 9200 |

修改为
path.data: /data/elasticsearch/data path.logs: /data/elasticsearch/log bootstrap.memory_lock: true network.host: 本机IP http.port: 9200

|-----------|---------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 | path.data: /data/elasticsearch/data path.logs: /data/elasticsearch/log bootstrap.memory_lock: true network.host: 本机IP http.port: 9200 |

修改jvm.options配置

vim /etc/elasticsearch/jvm.options

将

-Xms4g ## -Xmx4g

|-----|---------------------| | 1 2 | ## -Xms4g ## -Xmx4g |

修改为
-Xms4g -Xmx4g

|-----|---------------| | 1 2 | -Xms4g -Xmx4g |

6、配置内存锁定

因为开启了bootstrap.memory_lock: true选项，所以需要以下配置。
vim /etc/sysctl.conf

|---|----------------------| | 1 | vim /etc/sysctl.conf |

vm.swappiness=0 vm.max_map_count=262144

|-----|-----------------------------------------| | 1 2 | vm.swappiness=0 vm.max_map_count=262144 |

保存退出
sysctl -p

|---|-----------| | 1 | sysctl -p |

7、编辑启动服务配置
systemctl edit elasticsearch

|---|------------------------------| | 1 | systemctl edit elasticsearch |

[Service] LimitMEMLOCK=infinity LimitNOFILE=65535 LimitNPROC=4096

|---------|---------------------------------------------------------------------| | 1 2 3 4 | [Service] LimitMEMLOCK=infinity LimitNOFILE=65535 LimitNPROC=4096 |

按F2，然后按y，再按Enter键保存

或按Ctrl + O，然后按Enter键保存。按Ctrl + X退出。

8、启动elasticsearch
systemctl daemon-reload systemctl enable elasticsearch.service systemctl start elasticsearch.service

|-------|------------------------------------------------------------------------------------------------------| | 1 2 3 | systemctl daemon-reload systemctl enable elasticsearch.service systemctl start elasticsearch.service |

9、访问验证

验证方式一：当前主机访问
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

|---|--------------------------------------------------------------------------------------| | 1 | curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200 |

正常的话会看到如下内容
{ "name" : "whsir", "cluster_name" : "elasticsearch", "cluster_uuid" : "T1l5HuLuT8qbFyhkXk_9sw", "version" : { "number" : "8.11.4", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "da06c53fd49b7e676ccf8a32d6655c5155c16d81", "build_date" : "2024-01-28T10:05:08.438562403Z", "build_snapshot" : false, "lucene_version" : "9.8.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }

|-------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | { "name" : "whsir", "cluster_name" : "elasticsearch", "cluster_uuid" : "T1l5HuLuT8qbFyhkXk_9sw", "version" : { "number" : "8.11.4", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "da06c53fd49b7e676ccf8a32d6655c5155c16d81", "build_date" : "2024-01-28T10:05:08.438562403Z", "build_snapshot" : false, "lucene_version" : "9.8.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } |

验证方式二：浏览器访问（IP替换为你实际的地址）

https://IP:9200/

用户名：elastic
密码（就是前面安装好生成的）：TaO5MlQt2SAZJzWvv16F

10、安装kibana

同样该包组是从官方提前下载好的，上传至服务器上直接安装
yum install kibana-8.11.4-x86_64.rpm -y

|---|-----------------------------------------| | 1 | yum install kibana-8.11.4-x86_64.rpm -y |

11、编辑kibana配置文件
vim /etc/kibana/kibana.yml

|---|----------------------------| | 1 | vim /etc/kibana/kibana.yml |

将
#server.port: 5601 #server.host: "localhost" #i18n.locale: "en"

|-------|-----------------------------------------------------------------| | 1 2 3 | #server.port: 5601 #server.host: "localhost" #i18n.locale: "en" |

修改为
server.port: 5601 server.host: "IP" i18n.locale: "zh-CN"

|-------|----------------------------------------------------------| | 1 2 3 | server.port: 5601 server.host: "IP" i18n.locale: "zh-CN" |

12、启动kibana
systemctl daemon-reload systemctl enable kibana.service systemctl start kibana.service

|-------|----------------------------------------------------------------------------------------| | 1 2 3 | systemctl daemon-reload systemctl enable kibana.service systemctl start kibana.service |

13、生成kibana token

稍等一会，浏览器访问Kibana

http://IP:5601/

生成token

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

生成验证码

/usr/share/kibana/bin/kibana-verification-code

输入用户名密码登录，这里的用户名密码就是elastic的用户名密码

用户名：elastic
密码（就是前面安装好生成的）：TaO5MlQt2SAZJzWvv16F

至此kibana安装完成

14、安装logstash
yum install logstash-8.11.4-x86_64.rpm -y

|---|-------------------------------------------| | 1 | yum install logstash-8.11.4-x86_64.rpm -y |

15、配置logstash
cp /etc/logstash/logstash-sample.conf /etc/logstash/conf.d/filebeat.conf

|---|--------------------------------------------------------------------------| | 1 | cp /etc/logstash/logstash-sample.conf /etc/logstash/conf.d/filebeat.conf |

vim /etc/logstash/conf.d/filebeat.conf

|---|----------------------------------------| | 1 | vim /etc/logstash/conf.d/filebeat.conf |

注意将下方示例配置中的hosts地址修改为实际的地址，以及es的密码，truststore_password字段密码为前面生成CA时你自行设置的密码。
input { beats { port => 5044 } } output { elasticsearch { hosts => ["https://ES的IP:9200"] index => "nginx_log" user => "elastic" password => "TaO5MlQt2SAZJzWvv16F" ssl_certificate_verification => false } }

|-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | input { beats { port => 5044 } } output { elasticsearch { hosts => ["https://ES的IP:9200"] index => "nginx_log" user => "elastic" password => "TaO5MlQt2SAZJzWvv16F" ssl_certificate_verification => false } } |

16、启动服务
systemctl daemon-reload systemctl enable logstash.service systemctl start logstash.service

|-------|--------------------------------------------------------------------------------------------| | 1 2 3 | systemctl daemon-reload systemctl enable logstash.service systemctl start logstash.service |

至此logstash安装完成

17、安装filebeat
yum install filebeat-8.11.4-x86_64.rpm -y

|---|-------------------------------------------| | 1 | yum install filebeat-8.11.4-x86_64.rpm -y |

18、配置filebeat
vim /etc/filebeat/filebeat.yml

|---|--------------------------------| | 1 | vim /etc/filebeat/filebeat.yml |

enabled: false改为enabled: true

/var/log/*.log改为- /data/logs/nginx/*.log，这个是我nginx日志的路径，可结合实际情况进行设置

139和141行注释

152和154行注释放开

154行配置具体IP hosts: ["IP:5044"]

19、启动服务
systemctl daemon-reload systemctl enable filebeat.service systemctl start filebeat.service

|-------|--------------------------------------------------------------------------------------------| | 1 2 3 | systemctl daemon-reload systemctl enable filebeat.service systemctl start filebeat.service |

至此filebeat配置完成

20、验证

在kibana中找到开发工具