【Graylog】索引模板mapping自定义

本文主要是介绍【Graylog】索引模板mapping自定义，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

文章目录

前言

一、Logstash输入到Elasticsearch

二、进入主题

1.根据拿到的template.json分析

2.新建文件

3.插入

4.验证

5. 遇到的错误

总结

前言

Graylog使用中，总会有千奇百怪的要求，比如有同事就想全盘挪移ELK那套东西到Graylog，目前迁移一个业务日志到平台中，要求使用Logstash输入到Elasticsearch，Graylog管控索引声明周期，负责查询功能。

一、Logstash输入到Elasticsearch

这一步很好实现，但是如果想要在Graylog中可以查看到，需要下一番功夫，我遇到的问题有以下两点。

时间戳问题，@timestamp转换为timestamp，还要对时间进行矫正，要不然会有八个小时的时差
补全字段，如果想要在Graylog可以搜索到，需要补全字段。

二、进入主题

1.根据拿到的template.json分析

通过查看json文件，发现大部分都是keyword类型，所以根据可以不用加入到json中，最特殊的是其中一个使用ik分词器的(安装分词器，分词器的安装比较烂大街，就不在赘述了)所以使用Graylog默认的mapping肯定是行不通的。

查看官方文档，很容易就找到了创建外挂mapping，我的理解是，新建索引集后，会根据默认的mapping+外挂的mapping共同生成一个索引mapping，参考文档：Graylog文档。

2.新建文件

新建一个json文件，新增字段的mapping。

{"template": "cce_*","mappings": {"message": {"properties": {"cluster_name": {"type": "keyword"},"log_level": {"type": "keyword"},"type": {"type": "keyword"},"log_time": {"type": "keyword"},"namespace_name": {"type": "keyword"},"pod_name": {"type": "keyword"},"log_type": {"type": "keyword"},"cluster_uuid": {"type": "keyword"},"container_name": {"type": "keyword"},"log_data": {"search_analyzer": "ik_smart","analyzer": "ik_max_word","type": "text","fields": {"keyword": {"ignore_above": 256,"type": "keyword"}}},"workload_name": {"type": "keyword"},"collect_time": {"type": "date"}}}}
}

3.插入

curl -X PUT -d @'graylog-custom-mapping-7x.json' -H 'Content-Type: application/json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'

返回true即可。

4.验证

轮换索引集，查看mapping，成功。

{ - "cce_2": { - "mappings": { - "message": { - "dynamic_templates": [ - { - "internal_fields": { - "match": "gl2_*","match_mapping_type": "string","mapping": { - "type": "keyword"}}},{ - "store_generic": { - "match_mapping_type": "string","mapping": { - "type": "keyword"}}}],"properties": { - "_time": { - "type": "date"},"cluster_name": { - "type": "keyword"},"cluster_uuid": { - "type": "keyword"},"collect_time": { - "type": "date"},"container_name": { - "type": "keyword"},"full_message": { - "type": "text","analyzer": "ik_max_word"},"gl2_message_id": { - "type": "keyword"},"gl2_processing_timestamp": { - "type": "date","format": "yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp": { - "type": "date","format": "yyyy-MM-dd HH:mm:ss.SSS"},"gl2_remote_ip": { - "type": "keyword"},"gl2_remote_port": { - "type": "long"},"gl2_source_input": { - "type": "keyword"},"gl2_source_node": { - "type": "keyword"},"log_data": { - "type": "text","fields": { - "keyword": { - "type": "keyword","ignore_above": 256}},"analyzer": "ik_max_word","search_analyzer": "ik_smart"},"log_level": { - "type": "keyword"},"log_time": { - "type": "keyword"},"log_type": { - "type": "keyword"},"message": { - "type": "text","analyzer": "ik_max_word"},"namespace_name": { - "type": "keyword"},"pod_name": { - "type": "keyword"},"source": { - "type": "text","analyzer": "analyzer_keyword","fielddata": true},"source_type": { - "type": "keyword"},"streams": { - "type": "keyword"},"time_day": { - "type": "date"},"time_hours": { - "type": "keyword"},"time_min": { - "type": "keyword"},"time_min_zero": { - "type": "keyword"},"time_month": { - "type": "date"},"time_nano": { - "type": "long"},"time_sec": { - "type": "keyword"},"timestamp": { - "type": "date","format": "yyyy-MM-dd HH:mm:ss.SSS"},"type": { - "type": "keyword"},"workload_name": { - "type": "keyword"}}}}}
}