Logstash 转换日志

Logstash 提供了各种插件来转换已解析的日志。这些插件可以在日志中 添加、删除、和 更新字段，以便在输出系统中更好地理解和查询。

我们正在使用 Mutate 插件 在输入日志的每一行中添加一个字段名称 user。

安装 Mutate 过滤器插件

安装mutate过滤器插件；我们可以使用以下命令。

>Logstash-plugin install Logstash-filter-mutate

logstash.conf

在这个配置文件中，Mutate Plugin 被添加到 Aggregate Plugin 之后以添加一个新字段。

input {
   file {
      path => "C:/tpwork/logstash/bin/log/input.log"
   }
} 
filter {
   grok {
      match => [ "message", "%{LOGLEVEL:loglevel}-
         %{NOTSPACE:taskid}-%{NOTSPACE:logger}-
         %{WORD:label}(-%{INT:duration:int})?" ]
   }
   if [logger] == "TRANSACTION_START" {
      aggregate {
         task_id => "%{taskid}"
         code => "map['sql_duration'] = 0"
         map_action => "create"
      }
   }
   if [logger] == "SQL" {
      aggregate {
         task_id => "%{taskid}"
         code => "map['sql_duration'] ||= 0 ; 
            map['sql_duration'] += event.get('duration')"
      }
   }
   if [logger] == "TRANSACTION_END" {
      aggregate {
         task_id => "%{taskid}"
         code => "event.set('sql_duration', map['sql_duration'])"
         end_of_task => true
         timeout => 120
      }
   }
   mutate {
      add_field => {"user" => "lidihuo.com"}
   }
}
output {
   file {
      path => "C:/tpwork/logstash/bin/log/output.log"
   }
}

运行 Logstash

我们可以使用以下命令运行 Logstash。

>logstash –f logstash.conf

input.log

以下代码块显示了输入日志数据。

INFO-48566-TRANSACTION_START-start
INFO-48566-SQL-transaction1-320
INFO-48566-SQL-transaction1-200
INFO-48566-TRANSACTION_END-end

输出.log

可以看到输出事件中有一个名为"user"的新字段。

{
   "path":"C:/tpwork/logstash/bin/log/input.log",
   "@timestamp":"2016-12-25T19:55:37.383Z",
   "@version":"1",
   "host":"wcnlab-PC",
   "message":"NFO-48566-TRANSACTION_START-start\r",
   "user":"lidihuo.com","tags":["_grokparsefailure"]
}
{
   "duration":320,"path":"C:/tpwork/logstash/bin/log/input.log",
   "@timestamp":"2016-12-25T19:55:37.383Z","loglevel":"INFO","logger":"SQL",
   "@version":"1","host":"wcnlab-PC","label":"transaction1",
   "message":" INFO-48566-SQL-transaction1-320\r",
   "user":"lidihuo.com","taskid":"48566","tags":[]
}
{
   "duration":200,"path":"C:/tpwork/logstash/bin/log/input.log",
   "@timestamp":"2016-12-25T19:55:37.399Z","loglevel":"INFO",
   "logger":"SQL","@version":"1","host":"wcnlab-PC","label":"transaction1",
   "message":" INFO-48566-SQL-transaction1-200\r",
   "user":"lidihuo.com","taskid":"48566","tags":[]
}
{
   "sql_duration":520,"path":"C:/tpwork/logstash/bin/log/input.log",
   "@timestamp":"2016-12-25T19:55:37.399Z","loglevel":"INFO",
   "logger":"TRANSACTION_END","@version":"1","host":"wcnlab-PC","label":"end",
   "message":" INFO-48566-TRANSACTION_END-end\r",
   "user":"lidihuo.com","taskid":"48566","tags":[]
}

输出是 Logstash 管道的最后一个阶段，它将过滤器数据从输入日志发送到指定的目的地。 Logstash 提供多个输出插件，将过滤后的日志事件存储到各种不同的存储和搜索引擎。 存储日志Logstash ...