plg（Loki+Promtail+Grafana）监控nginx日志、messages日志监控平台

Loki 是一个水平可扩展，高可用性，多租户的日志聚合系统 。它的设计非常经济高效且易于操作，因为它不会为日志内容编制索引，而是为每个日志流编制一组标签。

Loki 组成：

    loki : 主服务器，负责存储日志和处理查询

    promtail : 代理，负责收集日志并将其发送给 loki

    Grafana : Go语言开发的开源数据可视化工具，可以做数据监控和数据统计，带有告警功能

登录官网：loki官网Like Prometheus, but for logs. Contribute to grafana/loki development by creating an account on GitHub.https://github.com/grafana/loki/releases/

一、loki安装配置

1.1 环境准备

----root用户操作
###创建用户
useradd loki
passwd loki###创建安装目录
mkdir /opt/loki###授权
chown -R loki:loki /opt/loki
chmod -R 755 /opt/loki----loki用户操作
###下载二进制包
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.4/loki-linux-amd64.zip"###解压二进制包
$ unzip "loki-linux-amd64.zip"### make sure it is executable
$ chmod a+x "loki-linux-amd64"

1.2 编写配置文件

$ vim loki-local-config.yml

更多配置请参考：https://grafana.com/docs/loki/latest/configuration/

auth_enabled: falseserver:http_listen_port: 3100grpc_listen_port: 9096ingester:lifecycler:address: 127.0.0.1ring:kvstore:store: inmemoryreplication_factor: 1final_sleep: 0schunk_idle_period: 10mchunk_retain_period: 30s
schema_config:configs:- from: 2020-05-15store: boltdbobject_store: filesystemschema: v11index:prefix: index_period: 168h
storage_config:boltdb:directory: /opt/loki/indexfilesystem:directory: /opt/loki/chunks  # 块存储路径limits_config:enforce_metric_name: falsereject_old_samples: true          # 是否拒绝老样本reject_old_samples_max_age: 168h  # 168小时之前的样本将会被删除ingestion_rate_mb: 200ingestion_burst_size_mb: 300per_stream_rate_limit: 1000MBmax_entries_limit_per_query: 10000
chunk_store_config:max_look_back_period: 168h        # 为避免查询超过保留期的数据，必须小于或等于下方的时间值
table_manager:retention_deletes_enabled: true   # 保留删除开启retention_period: 168h            # 超过168h的块数据将被删除ruler:storage:type: locallocal:directory: /opt/loki/rulesrule_path: /opt/loki/rules-tempalertmanager_url: http://192.168.0.1:9093    # alertmanager地址ring:kvstore:store: inmemoryenable_api: trueenable_alertmanager_v2: true

1.3 编写启动脚本

$ vim restart-loki.sh

#!/bin/bash
echo "stop loki"
ps -ef | grep loki-linux-amd64 | grep -v grep | awk '{print $2}'| xargs kill -9 echo "Begin start loki"
sleep 1
str=$"\n"
nohup ./loki-linux-amd64 --config.file=loki-local-config.yml &
sstr=$(echo -e $str)
echo $sstr

1.4 增加执行权限

$chmod +x restart-loki.sh

1.5 loki的启动

cd /opt/loki
./restart-loki.sh

1.6 loki日志查看

[root@localhost loki]# tail -10f nohup.out 
level=info ts=2023-03-26T08:40:48.324631438Z caller=checkpoint.go:614 msg="starting checkpoint"
level=info ts=2023-03-26T08:40:48.324805699Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=wal/checkpoint.000022
level=info ts=2023-03-26T08:45:18.327347566Z caller=checkpoint.go:501 msg="atomic checkpoint finished" old=wal/checkpoint.000022.tmp new=wal/checkpoint.000022
level=info ts=2023-03-26T08:45:18.32797424Z caller=checkpoint.go:572 msg="checkpoint done" time=4m30.002891425s
level=info ts=2023-03-26T08:45:48.325049331Z caller=checkpoint.go:614 msg="starting checkpoint"
level=info ts=2023-03-26T08:45:48.32524561Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=wal/checkpoint.000023
level=info ts=2023-03-26T08:50:18.327325742Z caller=checkpoint.go:501 msg="atomic checkpoint finished" old=wal/checkpoint.000023.tmp new=wal/checkpoint.000023
level=info ts=2023-03-26T08:50:18.327880793Z caller=checkpoint.go:572 msg="checkpoint done" time=4m30.002463306s
level=info ts=2023-03-26T08:50:48.325234397Z caller=checkpoint.go:614 msg="starting checkpoint"
level=info ts=2023-03-26T08:50:48.325402194Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=wal/checkpoint.000024

二、日志代理Promtail安装和配置

负责收集日志并将其发送给 loki，在需要采集日志的主机上都安装Promtail

2.1 Promtail环境准备

mkdir /opt/promtail
chown -R loki:loki /opt/promtail
chmod -R loki:loki /opt/promtail###下载
curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.4/promtail-linux-amd64.zip"###解压
unzip loki-linux-amd64.zip # 解压

2.2 编写Promtail配置文件

vim promtail-local-config.yml

server:http_listen_port: 9080grpc_listen_port: 0positions:filename: /opt/promtail/positions.yamlclients:- url: http://10.X.X.125:3100/loki/api/v1/push # 填写好Loki地址scrape_configs:
- job_name: messagelogstatic_configs:- targets:- localhostlabels:#job: messageloghost: namenode01__path__: /var/log/messages- job_name: agentlogstatic_configs:- targets:- localhostlabels:#job: agentloghost: namenode01__path__: /ssd/ssd0/agentlog/agent.log- job_name: datanodelogstatic_configs:- targets:- localhostlabels:
#      #job: datanodeloghost: namenode01__path__: /ssd/ssd0/datanodelog/*datanode*.log

2.3 编写promtail启动脚本

vi restart-promtail.sh

#!/bin/bash
echo "Begin stop promtail"
ps -ef | grep promtail-linux-amd64 | grep -v grep | awk '{print $2}' | xargs kill -9echo "Begin start promtail...."
nohup ./promtail-linux-amd64 --config.file=promtail-local-config.yml > ./promtail-9080.log 2>&1 &

2.4 增加执行权限

$chmod +x restart-promtail.sh

2.5 启动promtail服务

[root@localhost promtail]# cd /opt/promtail[root@localhost promtail]# ./restart-promtail.sh 
Begin stop promtail
Begin start promtail....
[root@localhost promtail]# 

三、 Grafana可视化安装和配置

    Grafana : Go语言开发的开源数据可视化工具，可以做数据监控和数据统计，带有告警功能

3.1 Grafana安装配置

Download Grafana | Grafana LabsOverview of how to download and install different versions of Grafana on different operating systems.https://grafana.com/grafana/download

###CentOS系统通过rpm方式安装wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.4.3-1.x86_64.rpmsudo yum install grafana-enterprise-9.4.3-1.x86_64.rpm -y[root@localhost grafana]# rpm -qa|grep grafana
grafana-enterprise-9.4.3-1.x86_64
[root@localhost grafana]# 

3.2 启动grafana

grafana会占用服务器3000端口

##启动
systemctl start grafana-server.service##停止
systemctl stop grafana-server.service##重启
systemctl restart grafana-server.service##设置开机自启动
systemctl enable grafana-server.service##状态查看
systemctl status grafana-server.service

3.3 grafana导入nginx dashboard

Grafana Loki Dashboard for NGINX Service Mesh | Grafana LabsLoki v2+ showcase using JSON NGINX access logs.https://grafana.com/grafana/dashboards/12559-grafana-loki-dashboard-for-nginx-service-mesh/

3.4 grafana配置告警邮箱

编辑Grafana配置文件，设置发件人
docker安装的Grafana：vim /home/monitor/grafana/config/grafana.ini
yum安装：vim /etc/grafana/grafana.ini

在[smtp] 标签下 修改配置

# 配置邮件服务器
[smtp]
enabled = true# 发件服务器
host = smtp.mail.139.com:465# smtp 密码
password = PAASmail2019# smtp账号
user = paas-support@139.com# 发信邮箱
from_address = paas-support@139.com# 发信人
from_name = Grafana

保存好配置文件之后，重启Grafana

##重启
systemctl restart grafana-server.service

3.5 测试grafana告警发送

点击页面左边的铃铛，并选择 Contact points

然后选择 Add contact point 创建一个 contact point,填写 Name 和收件人邮箱地址

然后点击Test ，提示成功后，刚填写的邮箱应该就可以收到如下内容的测试邮件 

查看邮箱，成功收到邮件

 

参考：简易日志系统LPG生产环境实践指南 | 坑我已经帮你们踩好了

参考：PLG日志平台搭建: Promtail + Loki + Grafana 全步骤_promtail loki_dxccccccccccc的博客-CSDN博客

参考：Promtail + Loki + Grafana 构建日志监控告警系统（一）_promtail+loki_5D金豆豆的博客-CSDN博客

参考:

PLG日志平台搭建: Promtail + Loki + Grafana 全步骤_promtail loki_dxccccccccccc的博客-CSDN博客

参考：

搭建Loki、Promtail、Grafana轻量级日志系统（centos7）_loki promtail_成钰的博客-CSDN博客

参考：

轻量日志收集系统loki_loki日志收集_wyl9527的博客-CSDN博客

参考：

轻量级日志可视化平台Grafana Loki接入nginx访问日志-阿里云开发者社区

参考：

PLG 云原生日志系统在压测中的落地实践 · TesterHome

参考：

快速部署Grafana日志监控+Nginx封禁IP-阿里云开发者社区

参考：

使用使用grafana中文插件-阿里云开发者社区