nginx+rsyslog+kafka+clickhouse+grafana 实现nginx 网关监控

2023-12-30 05:52
文章标签 实现 nginx clickhouse 网关 监控 kafka grafana rsyslog

本文主要是介绍nginx+rsyslog+kafka+clickhouse+grafana 实现nginx 网关监控，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

需求

  • 我想做一个类似腾讯云网关日志最终以仪表方式呈现，比如说qps、p99、p95的请求响应时间等等

流程图

数据流转就像标题
nginx ----> rsyslog ----> kafka —> clickhouse —> grafana

部署

kafka

kafka 相关部署这里不做赘述，只要创建一个topic 就可以
这里kafka地址是 192.168.1.180,topic是``

rsyslog 设置

rsyslog 具体是啥东西这个我这里也不做介绍，本人也是一个后端开发不是做运维的，只知道这个东西性能不错，算是logstash 平替把

# 安装rsyslog-kafka 插件
yum install -y rsyslog-kafka
# 创建一个配置vim /etc/rsyslog.d/rsyslog_nginx_kafka_cluster.conf

conf 内容

module(load="imudp")
input(type="imudp" port="514")# nginx access log ==> rsyslog server(local) ==> kafka
module(load="omkafka")template(name="nginxLog" type="string" string="%msg%")if $inputname == "imudp" then {# 这里的名字和下面nginx 配置tag 相同if ($programname == "nginx_access_log") thenaction(type="omkafka"template="nginxLog"# kafka 地址broker=["192.168.1.180:9092"]# topic 名字topic="rsyslog_nginx"partitions.auto="on"confParam=["socket.keepalive.enable=true"])
}:rawmsg, contains, "nginx_access_log" ~

最后重启rsyslog 即可

nginx

http 节点中设置如下

log_format jsonlog '{''"host": "$host",''"server_addr": "$server_addr",''"remote_addr":"$remote_addr",''"time_format":"$time_ms",''"time_sec":$times,''"timestamp":$timestamp,''"method":"$request_method",''"request_url":"$request_uri",''"status":$status,''"upstream_name":"$upstream_name",''"http_user_agent":"$http_user_agent",''"upstream_addr":"$upstream_addr",''"trace_id":"$http_traceid",''"upstream_status":"$upstream_status",''"upstream_response_time":"$upstream_response_time",''"request_time":$request_time,''"nginx_host":"$nginx_host"''}';# 配置rsyslog，tag 和上文rsyslog 中相同！！！access_log syslog:server=192.168.1.179,facility=local7,tag=nginx_access_log,severity=info jsonlog;#access_log  /var/log/nginx/access.log  jsonlog;

配置实际转发upstream

upstream testcontrol{server 192.168.10.123:8081;
}server {listen       18081;server_name  _;# 设置up nameset $upstream_name 'testcontrol';include /etc/nginx/conf/time.conf;location / {proxy_set_header traceid $http_traceid;proxy_set_header n-t $msec;#proxy_set_header X-Real-IP $remote_addr;proxy_pass http://testcontrol;}location ~ /status {stub_status   on;}
}

/etc/nginx/conf/time.conf; 配置

set $nginx_host '192.168.8.64';
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})") {set $year $1;set $month $2;set $day $3;set $hour $4;set $minutes $5;set $seconds $6;set $time_sec "$1-$2-$3 $4:$5:$6";}# 获取毫秒时间戳，并拼接到$time_zh之后if ($msec ~ "(\d+)\.(\d+)") {# 时间戳-msset $timestamp $1$2;# 时间戳-sset $times $1;# 格式化之后的时间msset $time_ms $time_sec.$2;# traceId，方便查日志set $http_traceid $nginx_host#$timestamp#$request_id;
}

clickhouse

先创建一个 nginx_gw_log 表


create table app_logs.nginx_gw_log
(timestamp              DateTime64(3, 'Asia/Shanghai'),server_addr            String,remote_addr            String,time_sec               DateTime('Asia/Shanghai'),method                 String,request_url            String,status                 Int16,upstream_name          String,http_user_agent        String,upstream_addr          String,trace_id               String,upstream_status        String,upstream_response_time String,request_time           Int32,nginx_host              String,host                   String
) engine = MergeTree PARTITION BY toYYYYMMDD(timestamp)ORDER BY timestampTTL toDateTime(timestamp) + toIntervalDay(1);

上面设置了过期1天，可以自由修改
添加kafka 引擎

create table app_logs.nginx_gw_log_kafka
(timestamp              DateTime64(3, 'Asia/Shanghai'),server_addr            String,remote_addr            String,time_sec               DateTime('Asia/Shanghai'),method                 String,request_url            String,status                 Int32,upstream_name          String,http_user_agent        String,upstream_addr          String,trace_id               String,upstream_status        String,upstream_response_time String,request_time           Decimal32(3),nginx_host              String,host                   String
) engine = Kafka()SETTINGS kafka_broker_list = '192.168.1.180:9092',kafka_topic_list = 'rsyslog_nginx',kafka_group_name = 'nginx_ck',kafka_format = 'JSONEachRow',kafka_skip_broken_messages = 100000,kafka_num_consumers = 1;

最后设置mv

CREATE MATERIALIZED VIEW app_logs.nginx_gw_log_mv TO app_logs.nginx_gw_log AS
SELECT timestamp,server_addr,remote_addr,time_sec,method,request_url,status,upstream_name,http_user_agent,upstream_addr,trace_id,upstream_status,upstream_response_time,toInt32(multiply(request_time, 1000)) as request_time,host
FROM app_logs.nginx_gw_log_kafka;

这里做了一个类型转换，因为nginx，request-time 单位是s，我想最终呈现在grafana 中是ms，所以这里做了转换，当然grafana中也可以做

上面都配置完了之后可以先验证下，保证数据最终到ck，如果有问题，需要再每个节点调试，比如先调试nginx->rsyslog ，可以先不配置kafka 输出，配置为console或者文件输出都可以，具体这里就不写了

grafana

{"annotations": {"list": [{"builtIn": 1,"datasource": {"type": "grafana","uid": "-- Grafana --"},"enable": true,"hide": true,"iconColor": "rgba(0, 211, 255, 1)","name": "Annotations & Alerts","target": {"limit": 100,"matchAny": false,"tags": [],"type": "dashboard"},"type": "dashboard"}]},"editable": true,"fiscalYearStartMonth": 0,"graphTooltip": 0,"id": 17,"links": [],"liveNow": false,"panels": [{"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"},"custom": {"axisCenteredZero": false,"axisColorMode": "text","axisLabel": "","axisPlacement": "auto","barAlignment": 0,"drawStyle": "line","fillOpacity": 0,"gradientMode": "none","hideFrom": {"legend": false,"tooltip": false,"viz": false},"lineInterpolation": "linear","lineWidth": 1,"pointSize": 5,"scaleDistribution": {"type": "linear"},"showPoints": "auto","spanNulls": false,"stacking": {"group": "A","mode": "none"},"thresholdsStyle": {"mode": "off"}},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]}},"overrides": []},"gridPos": {"h": 9,"w": 12,"x": 0,"y": 0},"id": 2,"options": {"legend": {"calcs": [],"displayMode": "list","placement": "bottom","showLegend": true},"tooltip": {"mode": "single","sort": "none"}},"pluginVersion": "9.1.1","targets": [{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT time_sec as a, count() as num FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime )\r\ngroup by a order by a desc ;\r\n","refId": "A","selectedFormat": 4}],"title": "qps","type": "timeseries"},{"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"},"custom": {"axisCenteredZero": false,"axisColorMode": "text","axisLabel": "","axisPlacement": "auto","barAlignment": 0,"drawStyle": "line","fillOpacity": 0,"gradientMode": "none","hideFrom": {"legend": false,"tooltip": false,"viz": false},"lineInterpolation": "linear","lineWidth": 1,"pointSize": 5,"scaleDistribution": {"type": "linear"},"showPoints": "auto","spanNulls": false,"stacking": {"group": "A","mode": "none"},"thresholdsStyle": {"mode": "off"}},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]}},"overrides": []},"gridPos": {"h": 9,"w": 12,"x": 12,"y": 0},"id": 3,"options": {"legend": {"calcs": [],"displayMode": "list","placement": "bottom","showLegend": true},"tooltip": {"mode": "single","sort": "none"}},"pluginVersion": "9.1.1","targets": [{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT time_sec as a, toInt32(quantile(0.95)(request_time))  as num FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime )\r\ngroup by a order by a desc ;","refId": "p95","selectedFormat": 4},{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"hide": false,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT time_sec as a, toInt32(quantile(0.99)(request_time))  as num FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime )\r\ngroup by a order by a desc ;","refId": "p99","selectedFormat": 4}],"title": "响应时间统计","type": "timeseries"},{"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"custom": {"align": "auto","displayMode": "auto","inspect": false},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]}},"overrides": []},"gridPos": {"h": 9,"w": 12,"x": 0,"y": 9},"id": 4,"options": {"footer": {"fields": "","reducer": ["sum"],"show": false},"showHeader": true},"pluginVersion": "9.1.1","targets": [{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT request_time as k,request_url,trace_id  FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime )\r\n order by k desc limit 10;\r\n","refId": "A","selectedFormat": 4}],"title": "最耗时","type": "table"},{"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"custom": {"align": "auto","displayMode": "auto","inspect": false},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]}},"overrides": []},"gridPos": {"h": 9,"w": 12,"x": 12,"y": 9},"id": 5,"options": {"footer": {"fields": "","reducer": ["sum"],"show": false},"showHeader": true},"pluginVersion": "9.1.1","targets": [{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT request_url, count() as num FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime )\r\ngroup by request_url order by request_url desc ;","refId": "A","selectedFormat": 4}],"title": "接口请求数量","type": "table"},{"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"fieldConfig": {"defaults": {"custom": {"align": "auto","displayMode": "auto","inspect": false},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]}},"overrides": []},"gridPos": {"h": 9,"w": 12,"x": 0,"y": 18},"id": 6,"options": {"footer": {"fields": "","reducer": ["sum"],"show": false},"showHeader": true},"pluginVersion": "9.1.1","targets": [{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT DISTINCT host  FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime );\r\n","refId": "A","selectedFormat": 4}],"title": "upstream","type": "table"},{"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"fieldConfig": {"defaults": {"custom": {"align": "auto","displayMode": "auto","inspect": false},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]}},"overrides": []},"gridPos": {"h": 9,"w": 12,"x": 12,"y": 18},"id": 8,"options": {"footer": {"fields": "","reducer": ["sum"],"show": false},"showHeader": true},"pluginVersion": "9.1.1","targets": [{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT  upstream_status,count() as num  FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime ) group by upstream_status;\r\n","refId": "A","selectedFormat": 4}],"title": "upstream","type": "table"},{"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"fieldConfig": {"defaults": {"mappings": [],"thresholds": {"mode": "percentage","steps": [{"color": "green","value": null},{"color": "orange","value": 70},{"color": "red","value": 85}]}},"overrides": []},"gridPos": {"h": 9,"w": 12,"x": 12,"y": 27},"id": 7,"options": {"orientation": "auto","reduceOptions": {"calcs": ["lastNotNull"],"fields": "","values": false},"showThresholdLabels": false,"showThresholdMarkers": true},"pluginVersion": "9.1.1","targets": [{"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"format": 1,"meta": {"builderOptions": {"database": "app_logs","fields": [],"filters": [{"condition": "AND","filterType": "custom","key": "timestamp","operator": "WITH IN DASHBOARD TIME RANGE","restrictToFields": [{"label": "timestamp","name": "timestamp","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"},{"label": "startTime","name": "startTime","picklistValues": [],"type": "DateTime64(3, 'Asia/Shanghai')"}],"type": "datetime"}],"groupBy": [],"limit": 100,"metrics": [{"aggregation": "count","field": ""}],"mode": "list","orderBy": [],"table": "tk_wx_control_req_kafka","timeField": "timestamp","timeFieldType": "DateTime64(3, 'Asia/Shanghai')"}},"queryType": "sql","rawSql": "SELECT count()  FROM \"app_logs\".${table_name} where ( timestamp  >= $__fromTime AND timestamp <= $__toTime );\r\n","refId": "A","selectedFormat": 4}],"title": "请求总数","type": "gauge"}],"schemaVersion": 37,"style": "dark","tags": [],"templating": {"list": [{"current": {"selected": false,"text": "nginx_gw_log","value": "nginx_gw_log"},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"definition": "show  tables from app_logs where name = 'nginx_gw_log' ","hide": 0,"includeAll": false,"label": "table_name","multi": false,"name": "table_name","options": [],"query": "show  tables from app_logs where name = 'nginx_gw_log' ","refresh": 1,"regex": "","skipUrlSync": false,"sort": 0,"type": "query"},{"current": {"selected": false,"text": "ClickHouse_1.178","value": "ClickHouse_1.178"},"hide": 0,"includeAll": false,"label": "datasource","multi": false,"name": "datasource","options": [],"query": "grafana-clickhouse-datasource","queryValue": "","refresh": 1,"regex": "","skipUrlSync": false,"type": "datasource"},{"current": {"isNone": true,"selected": false,"text": "None","value": ""},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"definition": "select upstream_name from app_logs.nginx_gw_log","hide": 0,"includeAll": false,"label": "upstream_name","multi": false,"name": "upstream_name","options": [],"query": "select upstream_name from app_logs.nginx_gw_log","refresh": 2,"regex": "","skipUrlSync": false,"sort": 0,"type": "query"},{"current": {"isNone": true,"selected": false,"text": "None","value": ""},"datasource": {"type": "grafana-clickhouse-datasource","uid": "${datasource}"},"definition": "select nginx_host from app_logs.nginx_gw_log where upstream_name = \"$upstream_name\"","hide": 0,"includeAll": false,"label": "nginx_host","multi": false,"name": "nginx_host","options": [],"query": "select nginx_host from app_logs.nginx_gw_log where upstream_name = \"$upstream_name\"","refresh": 1,"regex": "","skipUrlSync": false,"sort": 0,"type": "query"}]},"time": {"from": "now-5m","to": "now"},"timepicker": {},"timezone": "","title": "nginx_gw","uid": "lZrbSYOIkA","version": 6,"weekStart": ""
}

最终呈现效果

在这里插入图片描述

隐患点

  • clickhouse 本身查询qps 就不高，如果数据量很大可以考虑集群或者其他的存储，doris、es等等
  • 不知道nginx+rsyslog 对性能有多少影响，目前测试单机nginx大几千的qps 都没啥问题

有其他更好方案的小伙伴留言哦

这篇关于nginx+rsyslog+kafka+clickhouse+grafana 实现nginx 网关监控的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！


http://www.chinasem.cn/article/551901。

相关文章

流媒体平台/视频监控/安防视频汇聚EasyCVR播放暂停后视频画面黑屏是什么原因？

流媒体平台/视频监控/安防视频汇聚EasyCVR播放暂停后视频画面黑屏是什么原因？

视频智能分析/视频监控/安防监控综合管理系统EasyCVR视频汇聚融合平台，是TSINGSEE青犀视频垂直深耕音视频流媒体技术、AI智能技术领域的杰出成果。该平台以其强大的视频处理、汇聚与融合能力，在构建全栈视频监控系统中展现出了独特的优势。视频监控管理系统EasyCVR平台内置了强大的视频解码、转码、压缩等技术，能够处理多种视频流格式，并以多种格式（RTMP、RTSP、HTTP-FLV、WebS
阅读更多...
hdu1043（八数码问题，广搜 + hash（实现状态压缩） ）

hdu1043（八数码问题，广搜 + hash（实现状态压缩） ）

利用康拓展开将一个排列映射成一个自然数，然后就变成了普通的广搜题。 #include<iostream>#include<algorithm>#include<string>#include<stack>#include<queue>#include<map>#include<stdio.h>#include<stdlib.h>#include<ctype.h>#inclu
阅读更多...
综合安防管理平台LntonAIServer视频监控汇聚抖动检测算法优势

综合安防管理平台LntonAIServer视频监控汇聚抖动检测算法优势

LntonAIServer视频质量诊断功能中的抖动检测是一个专门针对视频稳定性进行分析的功能。抖动通常是指视频帧之间的不必要运动，这种运动可能是由于摄像机的移动、传输中的错误或编解码问题导致的。抖动检测对于确保视频内容的平滑性和观看体验至关重要。 优势 1. 提高图像质量 - 清晰度提升：减少抖动，提高图像的清晰度和细节表现力，使得监控画面更加真实可信。 - 细节增强：在低光条件下，抖
阅读更多...
【C++】_list常用方法解析及模拟实现

【C++】_list常用方法解析及模拟实现

相信自己的力量，只要对自己始终保持信心，尽自己最大努力去完成任何事，就算事情最终结果是失败了，努力了也不留遗憾。💓💓💓 目录   ✨说在前面 🍋知识点一：什么是list？ •🌰1.list的定义 •🌰2.list的基本特性 •🌰3.常用接口介绍 🍋知识点二：list常用接口 •🌰1.默认成员函数 🔥构造函数（⭐） 🔥析构函数 •🌰2.list对象
阅读更多...
【Prometheus】PromQL向量匹配实现不同标签的向量数据进行运算

【Prometheus】PromQL向量匹配实现不同标签的向量数据进行运算

✨✨ 欢迎大家来到景天科技苑✨✨ 🎈🎈 养成好习惯，先赞后看哦~🎈🎈 🏆 作者简介：景天科技苑 🏆《头衔》：大厂架构师，华为云开发者社区专家博主，阿里云开发者社区专家博主，CSDN全栈领域优质创作者，掘金优秀博主，51CTO博客专家等。 🏆《博客》：Python全栈，前后端开发，小程序开发，人工智能，js逆向，App逆向，网络系统安全，数据分析，Django，fastapi
阅读更多...
让树莓派智能语音助手实现定时提醒功能

让树莓派智能语音助手实现定时提醒功能

最初的时候是想直接在rasa 的chatbot上实现，因为rasa本身是带有remindschedule模块的。不过经过一番折腾后，忽然发现，chatbot上实现的定时，语音助手不一定会有响应。因为，我目前语音助手的代码设置了长时间无应答会结束对话，这样一来，chatbot定时提醒的触发就不会被语音助手获悉。那怎么让语音助手也具有定时提醒功能呢？ 我最后选择的方法是用threading.Time
阅读更多...
Android实现任意版本设置默认的锁屏壁纸和桌面壁纸（两张壁纸可不一致）

Android实现任意版本设置默认的锁屏壁纸和桌面壁纸（两张壁纸可不一致）

客户有些需求需要设置默认壁纸和锁屏壁纸  在默认情况下 这两个壁纸是相同的  如果需要默认的锁屏壁纸和桌面壁纸不一样 需要额外修改 Android13实现 替换默认桌面壁纸： 将图片文件替换frameworks/base/core/res/res/drawable-nodpi/default_wallpaper.*  （注意不能是bmp格式） 替换默认锁屏壁纸： 将图片资源放入vendo
阅读更多...
C#实战|大乐透选号器[6]：实现实时显示已选择的红蓝球数量

C#实战|大乐透选号器[6]：实现实时显示已选择的红蓝球数量

哈喽，你好啊，我是雷工。 关于大乐透选号器在前面已经记录了5篇笔记，这是第6篇； 接下来实现实时显示当前选中红球数量，蓝球数量； 以下为练习笔记。 01 效果演示 当选择和取消选择红球或蓝球时，在对应的位置显示实时已选择的红球、蓝球的数量； 02 标签名称 分别设置Label标签名称为：lblRedCount、lblBlueCount
阅读更多...
搭建Kafka+zookeeper集群调度

搭建Kafka+zookeeper集群调度

前言 硬件环境 172.18.0.5        kafkazk1        Kafka+zookeeper                Kafka Broker集群 172.18.0.6        kafkazk2        Kafka+zookeeper                Kafka Broker集群 172.18.0.7        kafkazk3
阅读更多...
Kubernetes PodSecurityPolicy：PSP能实现的5种主要安全策略

Kubernetes PodSecurityPolicy：PSP能实现的5种主要安全策略

Kubernetes PodSecurityPolicy：PSP能实现的5种主要安全策略 1. 特权模式限制2. 宿主机资源隔离3. 用户和组管理4. 权限提升控制5. SELinux配置 💖The Begin💖点点关注，收藏不迷路💖 Kubernetes的PodSecurityPolicy（PSP）是一个关键的安全特性，它在Pod创建之前实施安全策略，确保P
阅读更多...

Copyright China编程 23002807@qq.com

沪ICP备2023033072号-2