使用Telegraf+GrayLog实现Linux业务系统服务异常时自动推送钉钉告警

实现过程参考Telegraf官方文档

https://docs.influxdata.com/telegraf/v1.24/get_started/
https://github.com/influxdata/telegraf/blob/release-1.24/plugins/inputs/exec/README.md
https://sbcode.net/grafana/telegraf-inputs-exec-monitor-ssh-sessions/

一、GrayLog上配置Telegraf的GELF UDP方式接入Input和Stream

步骤较简单，下面只展示配置时的一些截图

(图片点击放大查看)

记得GrayLog上开放input设置的端口

firewall-cmd --permanent --zone=public --add-port=12201/udp
firewall-cmd --reload

(图片点击放大查看)

二、业务服务器上安装Telegraf并配置telegraf.conf

1、业务服务器上创建一个XX服务检测脚本

(图片点击放大查看)

vim /opt/service_check.sh 
#!/bin/sh
status=$(/usr/bin/systemctl status sshd | grep Active | awk -F "since" '{print $1}')
echo $status

chmod 777 /opt/service_check.sh

2、telegraf.conf配置文件生成并修改

rpm -ivh telegraf-1.24.3-1.x86_64.rpm telegraf --sample-config --input-filter exec --output-filter graylog > telegraf.conf

vim telegraf.conf

(图片点击放大查看)

最终的telegraf.conf 配置文件如下

#cat telegraf.conf \| grep -v \^# \| grep -v \^$ \| grep -v \^.\*## \| grep -v \^.\*#
[global_tags]
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = ""
  omit_hostname = false
[[outputs.graylog]]
  servers = ["udp://192.168.31.170:12201"]
[[inputs.exec]]
  commands = [ "sh /opt/service_check.sh" ]
  timeout = "10s"
  name_override = "sshd_service_status_check"
  data_format = "value"
  data_type = "string"
  interval = "45s"

cd /etc/telegraf/
mv telegraf.conf telegraf.conf_default
cp /root/telegraf.conf ./
chmod 644 telegraf.conf
systemctl start telegraf

启动报错，原因为telegraf.conf的权根

(图片点击放大查看)

三、GrayLog上查看telegraf日志并配置告警

(图片点击放大查看)

其中 PrometheusAlert告警模板

(图片点击放大查看)

## \[Graylog告警信息\](.check_result.Event.Source)
### \<font color=#FF0000\>告警描述：{{.event_definition_description}}\</font\>
{{ range $k,$v:=.backlog }}
##### \<font color="#FF0000"\>告警时间\</font\>：{{GetCSTtime $v.timestamp}} \</br\>
##### \<font color="#FF0000"\>告警服务器名称\</font\>：{{$v.source}} \</br\>
##### \<font color="#FF0000"\>告警服务器IP地址\</font\>：{{$v.fields.gl2_remote_ip}} \</br\>
##### \<font color="#FF0000"\>服务目前状态\</font\>：{{$v.fields.value}} \</br\>
{{end}}

GrayLog告警配置过程截图