Prometheus+Alertmanager实现企业微信告警

Prometheus+Alertmanager实现企业微信告警(python脚本调用机器人url实现)

prometheus部署详见 https://blog.csdn.net/dragonQuncle/article/details/133983718?spm=1001.2014.3001.5502

软件包-版本：
Alertmanager-0.23.0：https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-amd64.tar.gz

部署Alertmanager及配置企业微信告警

安装Alertmanager

step 1.下载

wget https://github.com/prometheus/alertmanager/releases/download/v0.25.1/alertmanager-0.25.1.linux-amd64.tar.gz

step 2.安装

tar -xzf  alertmanager-0.25.1.linux-amd64.tar.gz     -C  /opt/
cd  /opt
mv   alertmanager-0.25.1.linux-amd64/   alertmanager
chown -R  prometheus:prometheus /opt/alertmanager/

step 3.创建服务脚本systemd管理

vim /usr/lib/systemd/system/alertmanager.service

[Unit]
Description=alertmanager
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/alertmanager/alertmanager \
--config.file=/opt/alertmanager/alertmanager.yml \
--log.level=debug

ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

step 4.启动

systemctl daemon-reload
systemctl enable alertmanager
systemctl start  alertmanager
systemctl status alertmanager

企业微信告警

step 1.创建机器人
在企业微信创建群聊，添加机器人会有webhook地址。

step 2.修改prometheus配置文件
在prometheus.yml文件里开启alertmanager服务端口和自定义规则

vim   /opt/prometheus/prometheus.yml

step 3.创建目录编写规则yml文件

mkdir /opt/prometheus/rules/
cd  /opt/prometheus/rules/

vim  /opt/prometheus/rules/node.yml

groups:
# 报警组组名称
- name: alters
  #报警组规则
  rules:
   #告警名称，需唯一
  - alert: cpu使用率大于75%
    #promQL表达式
    expr: sum(avg without (cpu)(irate(node_cpu_seconds_total{mode!='idle'}[5m]))) by (instance) > 0.75
    #满足此表达式持续时间超过for规定的时间才会触发此报警
    for: 1m
    labels:
      #严重级别
      severity: warning
    annotations:
     #发出的告警标题
      summary: "实例 {{ $labels.instance }} CPU 使用率过高"
      #发出的告警内容
      description: "实例{{ $labels.instance }} CPU 使用率超过 75% (当前值为: {{ $value }})"
  - alert: 内存使用率大于90%
    expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)/node_memory_MemTotal_bytes > 0.90
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "实例 {{ $labels.instance }} 内存使用率过高"
      description: "实例 {{ $labels.instance }} 内存使用率 90% (当前值为: {{ $value }})"
 2.vim node_alters.yml
 groups:
- name: Alerthost
  rules:
  - alert: 服务器宕机
    expr: avg by (instance) (up{job="host"}) == 0
    for: 15s       #控制在触发告警之前，测试表达式的值必须为true的时长
    labels:
      severity: '突发事件'
    annotations:
      description: "实例 {{ $labels.instance }} 服务器已宕机，请进行检查."
      summary: "{{ $labels.instance }} 机器已经宕机超过15秒"
  - alert: 磁盘使用率大于80%
    expr: 100 - (node_filesystem_free_bytes{mountpoint="/",fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"} * 100) > 80
    for: 2m
    labels:
      severity: warning
    annotations:
      description: "{{ $labels.instance  }} ： {{ $labels.job  }} ：{{ $labels.mountpoint  }} 这个分区使用大于百分之80% (当前值：{{ $value }})"
      summary: "Instance {{ $labels.instance  }} ：{{ $labels.mountpoint }} 分区使用率过高"

step 4.访问普罗米页面，看是否添加成功

step 5.在普罗米服务端添加python自动化脚本(需要安装python3环境、flask、request模块 )

vim /usr/bin/app.py
import json
import os
from flask import request, Flask   
from dingtalkchatbot.chatbot import DingtalkChatbot     

def Open(s):
    Path = './temp.json'
    if os.path.exists(Path):
        os.remove(Path)     
    f = open(Path, 'w')
    print(s, file = f)
    f.close()
 

def GetData():
    PostData = request.get_data()  
    Data = json.loads(PostData)  
    JsonData = json.dumps(Data, ensure_ascii=False, indent=4)  
    return Data

app = Flask(__name__)
app.config['JSON_AS_ASCII'] = False
 
 
@app.route('/webhook/test/', methods=['POST'])
def IssueCreate():
    # Open(GetData())
    status = GetData()['status']
    alertname = GetData()['alerts'][0]['labels']['alertname']
    serverity = GetData()['alerts'][0]['labels']['severity']
    instance = GetData()['alerts'][0]['labels']['instance']
    start_time = GetData()['alerts'][0]['startsAt']
    message = str(
                     '##' + 'Prometheus告警：%s' + '##' + '\n'
                     '状态：%s' %status + '\n'
                     '告警信息：%s' %alertname + '\n'
                     '告警级别：%s' %serverity + '\n'
                     'IP地址：%s' %instance + '\n'
                     '开始时间：%s' %start_time + '\n'
    )
    print(message)
    send_message(message)
    return "OK", 200
def send_message(message):
    webhook = '机器人地址'
    xiaoding = DingtalkChatbot(webhook)
    xiaoding.send_text(msg=message, is_at_all=True)     
 
if __name__ == '__main__':
    app.run(debug = False, host = '0.0.0.0', port = 8888)

step 6.配置脚本自启动文件

vim /usr/lib/systemd/system/app.service
[Unit]
Description=QR Code Reader Service
After=multi-user.target
Conflicts=getty@tty1.service

[Service]
Type=simple
ExecStart=/usr/bin/python3 /usr/bin/app.py
StandardInput=tty-force

[Install]
WantedBy=multi-user.target

step 7.启动

systemctl daemon-reload
systemctl enable  app.service
systemctl start   app.service

step 8.在alertmanager的yml文件修改

vim /opt/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m   #处理超时时间
route:
  group_by: ['alertname']
  group_wait: 10s    #等待时间
  group_interval: 10s   #相同的Gourp之间发送告警通知的时间间隔
  repeat_interval: 5m   #重复报警的间隔时长
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://本机ip:8888/webhook/test/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

检测配置文件是否正确：

step 9.重启服务alertmanager，看企业微信是否可以收到告警信息。

systemctl restart alertmanager.service

撒花！！！