思享工具箱

logstash grok正则调试

摘要：
S+）“”（？S+）”“（？
logstash 正则调试；
nginx 配置；
log_format  main  '$remote_addr [$time_local] "$request" ';


logstash:
 "message" =>"%{IPORHOST:clientip} [%{HTTPDATE:time}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}""
 
 输出:
 
 {
        "message" => " 121.40.205.143 [29/Aug/2016:12:36:32 +0800] "GET /favicon.ico HTTP/1.1" - 404 2319 "-" "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN" 0.000 -",
       "@version" => "1",
     "@timestamp" => "2016-08-29T04:39:16.608Z",
           "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
           "host" => "0.0.0.0",
           "type" => "uat_nginx_access",
       "clientip" => "121.40.205.143",
           "time" => "29/Aug/2016:12:36:32 +0800",
           "verb" => "GET",
        "request" => "/favicon.ico",
    "httpversion" => "1.1"
}

此时grok 能正常匹配:

            "message" => "%{IPORHOST:clientip} [%{HTTPDATE:time}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" 
			%{NUMBER:http_status_code} %{NUMBER:bytes} "(?<http_referer>S+)" "(?<http_user_agent>S+)" "(?<http_x_forwarded_for>S+)""
			
			
			    log_format  main  '$http_host $server_addr $remote_addr [$time_local] "$request" '
                      '$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '
                      '$request_time $upstream_response_time';
					  
继续加；
    log_format  main  '$remote_addr [$time_local] "$request"'
                       '$status $body_bytes_sent';
					   
					  
日志格式:
 121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /resources/plugins/artDialog/ui-dialog.css HTTP/1.1"304 0
 121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /wechat/images/account/icons.7a340e21.png HTTP/1.1"304 0
 121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /wechat/images/nav-icon.44c2022c.png?v=1 HTTP/1.1"304 0
 121.40.205.143 [29/Aug/2016:12:51:19 +0800] "GET /favicon.ico HTTP/1.1"404 2319
 121.40.205.143 [29/Aug/2016:12:51:19 +0800] "GET /favicon.ico HTTP/1.1"404 2319
 121.40.205.143 [29/Aug/2016:12:52:25 +0800] "GET /favicon.ico HTTP/1.1"404 2319
 121.40.205.143 [29/Aug/2016:12:52:25 +0800] "GET /favicon.ico HTTP/1.1"404 2319
 121.40.205.143 [29/Aug/2016:12:53:28 +0800] "GET /favicon.ico HTTP/1.1"404 2319
 
 
filter {
    grok {
        match=> {
       "message" =>"%{IPORHOST:clientip} [%{HTTPDATE:time}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}"%{NUMBER:http_status_code} %{NUMBER:bytes}"
       
        }

		
logstash 输出:
Pipeline main started
{
             "message" => " 121.40.205.143 [29/Aug/2016:12:56:10 +0800] "GET /favicon.ico HTTP/1.1"404 2319",
            "@version" => "1",
          "@timestamp" => "2016-08-29T04:58:54.908Z",
                "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
                "host" => "0.0.0.0",
                "type" => "uat_nginx_access",
            "clientip" => "121.40.205.143",
                "time" => "29/Aug/2016:12:56:10 +0800",
                "verb" => "GET",
             "request" => "/favicon.ico",
         "httpversion" => "1.1",
    "http_status_code" => "404",
               "bytes" => "2319"
}					   



继续；


 121.40.205.143 [29/Aug/2016:13:00:16 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"
 121.40.205.143 [29/Aug/2016:13:00:22 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"
 121.40.205.143 [29/Aug/2016:13:00:30 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"
 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/login.html HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/account.html"
 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/images/login/icon_01.6e839367.png HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/css/wechat.2a00a782.css"
 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/images/login/icon_02.5065faba.png HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/css/wechat.2a00a782.css"
 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /resources/plugins/jquery/jquery.md5.js?v=1 HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"
 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/js/libs/dialog-min.88247f5e.js?v=1 HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"
 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/js/login.a87fbd64.js HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"


{
       "message" => " 121.40.205.143 [29/Aug/2016:13:05:24 +0800] "GET /wechat/account_balance.html HTTP/1.1"200 3059 "https://uatest.winfae.com/wechat/account.html" 
	   "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN"",

121.40.205.143 [29/Aug/2016:13:05:24 +0800] "GET /wechat/account_balance.html HTTP/1.1"200 3059 "https://uatest.winfae.com/wechat/account.html" "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN"
121.40.205.143 [29/Aug/2016:13:05:45 +0800] "GET /wechat/home.html?useragent=android_h5_zjcap&apiver=2 HTTP/1.1"200 11601 "-" "okhttp/2.6.0"



{
             "message" => " 121.40.205.143 [29/Aug/2016:13:13:11 +0800] "GET /wechat/js/regain.431efde9.js HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/regain.html" "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN"",
            "@version" => "1",
          "@timestamp" => "2016-08-29T05:15:55.609Z",
                "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
                "host" => "0.0.0.0",
                "type" => "uat_nginx_access",
            "clientip" => "121.40.205.143",
                "time" => "29/Aug/2016:13:13:11 +0800",
                "verb" => "GET",
             "request" => "/wechat/js/regain.431efde9.js",
         "httpversion" => "1.1",
    "http_status_code" => "304",
               "bytes" => "0",
        "http_referer" => "https://uatest.winfae.com/wechat/regain.html"
		
S+ 和 [^
	
f]+ 语法一样   非空格

		

 
 my $str="  begin 123.456 end  ";  
if ($str =~/(?<request_time>d+.d+)/)  
   {  
    my ($request_time) = ($+{request_time});    
   print $request_time."
";};  
zjtest7-frontend:/root/0825# perl a1.pl   
123.456  

 "http_referer" => "https://uatest.winfae.com/wechat/regain.html"
 
  "(?<http_referer>S+)"
  
  
 "(?<http_user_agent>S+)"	
 "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN"
 
 
              "message" => " 121.40.205.143 [29/Aug/2016:13:54:08 +0800] "GET /resources/plugins/artDialog/ui-dialog.css HTTP/1.1"200 9985 "https://uatest.winfae.com/wechat/home.html?useragent=ios_h5_zjcap&apiver=2&WKWebView=1" "ios_h5_zjcap"",
            "@version" => "1",
          "@timestamp" => "2016-08-29T05:56:53.217Z",
                "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
                "host" => "0.0.0.0",
                "type" => "uat_nginx_access",
            "clientip" => "121.40.205.143",
                "time" => "29/Aug/2016:13:54:08 +0800",
                "verb" => "GET",
             "request" => "/resources/plugins/artDialog/ui-dialog.css",
         "httpversion" => "1.1",
    "http_status_code" => "200",
               "bytes" => "9985",
        "http_referer" => "https://uatest.winfae.com/wechat/home.html?useragent=ios_h5_zjcap&apiver=2&WKWebView=1",
     "http_user_agent" => "ios_h5_zjcap"
}

{
       "message" => " 121.40.205.143 [29/Aug/2016:13:59:35 +0800] "GET /resources/js/toolbar.49fc367e.js?_v=${last.updated}&_=1472450673142 HTTP/1.1"200 1800 "https://uatest.winfae.com/products/productList.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36"",
      "@version" => "1",
    "@timestamp" => "2016-08-29T06:02:18.775Z",
          "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
          "host" => "0.0.0.0",
          "type" => "uat_nginx_access",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

标签：#logstash#正则#nginx | 浏览：187 | 发布日期：

免责声明：文章转载自《logstash grok正则调试》仅用于学习参考。如对内容有疑问，请及时联系本站处理。

上篇oracle sqlplus及常用sql语句L131下篇

宿迁高防，2C2G15M，22元/月；香港BGP，2C5G5M，25元/月 雨云优惠码：MjYwNzM=

相关文章

Logstash: 启动监控及集中管理-总结

Logstash: 启动监控 配置文件：logstash.yml xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username: logstash_system # 系统自带的用户 xpack.monitoring.elasticsearch.password: yX8nV8M...

生产环境中使用Docker Swarm的一些建议

译者按: 实践中会发现，生产环境中使用单个Docker节点是远远不够的，搭建Docker集群势在必行。然而，面对Kubernetes, Mesos以及Swarm等众多容器集群系统，我们该如何选择呢？它们之中，Swarm是Docker原生的，同时也是最简单，最易学，最节省资源的，至少值得我们多了解一下。本文将介绍一些非常实用的建议。 原文: Tips for...

查找nginx安装的路径以及相关安装操作命令

查找nginx安装的路径以及相关安装操作命令 Linux环境下，怎么确定Nginx是以那个config文件启动的？ [root@localhost ~]# ps -ef | grep nginxroot 21196 1 0 23:40 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/ngin...

windows下Nginx实现socket代理功能【转载】

一、下载：http://nginx.org/en/download.html 二、下载后解压：   三、修改配置文件 修改解压文件夹下conf文件夹内的nginx.conf配置文件    ps：下载的配置文件中是不支持socket代理的，我们需要手动写入配置 stream{        upstream socket_server{           ...

Nginx+Tomcat动静分离及Nginx优化

目的：nginx处理用户请求的静态页面，tomcat处理用户请求jsp页面，来实现动态分离，nginx处理静态页面效率远高于tomcat，这样一来就能更好的提高并发，处理性能。 准备软件： 下载jdk1.7：http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880...

前后端分离实践（一）

前言 最近这一段时间由于Nodejs的逐渐成熟和日趋稳定，越来越多的公司中的前端团队开始尝试使用Nodejs来练一下手，尝一尝鲜。 一般的做法都是将原本属于后端的一部分相对于业务不是很重要的功能迁移到Nodejs上面来，也有一些公司将NodeJS作为前后端分离的一个解决方案去施行。而像淘宝网这类的大型网站也很早的完成了前后端的分离，给我们这样的后来者提供了...

最新文章

随机推荐

思享工具箱导航