学习Logstash
Directory Layout
Type | Description | Config Setting |
---|---|---|
home | installation path | |
bin | Binary scripts including logstash to start Logstash and logstash-plugin to install plugins | |
settings | configuration files path | path.settings |
conf | Logstash pipeline configuration files | path.config |
logs | Log files | path.logs |
plugins | Local, non Ruby-Gem plugin files | path.plugins |
All these config settings are set in logstash.yml
Define different pipeline configurations in path.config directory. Logstash tries to load all files in that directory.
Grammar
- section, {}, used to define a scope of plugin
- field reference, [], get value of filed
- nested, [][]…
- print value, %{}
- value types: array([]), boolean(true/false), hash({}), number, string
more refer: - Structure of a Config File - Accessing Event Data and Fields in the Configuration - Using Environment Variables
Pipeline Configuration
Pipeline Configuration defines sections of plugins used for event processing pipeline.
logstash 会自动读取path.config目录下所有*.conf文件,然后在内存里拼接成一个完整的大配置文件,再去执行.
logstash列出目录下所有文件时,是字母排序的.而logstash配置段的filter和output都是顺序执行,所以顺序非常重要. 采用多文件管理的用户,推荐采用数字编号方式命名配置文件,同时在配置中,严谨采用if判断限定不同日志的动作。
Work Flow: input | decode | filter | encode | output
[Codec][https://www.elastic.co/guide/en/logstash/current/codec-plugins.html]用来处理decode/encode事件.
对日志统一采用UTC时间存成long类型的数据,是国际安全/运维界的一个通识.
filer和output都是按顺序执行.
Input
Syslog
在使用LogStash::Inputs::Syslog时, 使用TCP协议传输数据.
强烈建议使用LogStash::Inputs::TCP和LogStash::Filters::Grok配合实现同样的syslog功能. 可以很好的优化Syslog的处理性能.
Filter
Grok
grok表达式的完整语法格式
`%{PATTERN_NAME:capture_name:data_type}`
date_type: int/float
可使用Grok Debugger来测试自己的grok表达式