使用Awk分析Nginx日志大于某个时间内非200状态的数据

Question

使用PHP写了一个同步ES和Redis的服务脚本,每分钟不定时会收到很多http请求,有点担心会失败,打算做一个补跑的脚本,目前是想通过awk分析nginx日志 检索大于某个时间非200状态的数据进行从新发送,但是Awk不怎么会用，只是完成了一部分。。

想要的结果：
通过定时任务15分钟运行一次,然后查询大十五分钟前非200状态的数据,但是现在是用grep指定时间做不到这种效果(现在自己写的可以做到指定分钟内,感觉1分钟运行一次,数据太大会死循环)，研究了半天也没个结果,有没有大佬可否帮忙在个基础上改进一下

Nginx 日志格式

"06/May/2019:03:03:00 +0000" "GET / HTTP/1.1" "41" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "500" "500" "0.021" "0.021" "[20953311,21829304,22006993]"
"06/May/2019:03:07:11 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "500" "500" "0.058" "0.058" "[20953311,21829304,22006993]"
"06/May/2019:03:15:47 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "500" "500" "0.055" "0.055" "[20953311,21829304,22006993]"
"06/May/2019:03:16:44 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "200" "200" "0.051" "0.051" "[20953311,21829304,22006993]"
"06/May/2019:03:40:26 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "200" "200" "0.057" "0.057" "[22769461]"
"06/May/2019:03:41:11 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "500" "500" "0.058" "0.058" "[20953311,21829304,22006993]"
"06/May/2019:03:41:47 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "500" "500" "0.055" "0.055" "[20953311,21829304,22006993]"
"06/May/2019:03:42:44 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "200" "200" "0.051" "0.051" "[20953311,21829304,22006993]"
"06/May/2019:03:43:26 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "200" "200" "0.057" "0.057" "[22769461]"

目前自己研究出来的AWK命令

cat file.log |grep "06/May/2019:03:16" | awk -F "[\"\"]" '{if($18!=200){print $26}}'

Answer 1

试试 GoAccess, 开源的, 现成的

官网
https://goaccess.io/

这篇写得不错
https://www.cnblogs.com/yjf51...

Answer 2

awk -F "/" '{print $3 $0}' a.txt | awk -F" "  '{print $1, $0}' | awk '$1 > "2019:03:15:47" && $16 != "\"200\"" {print $16}'

每次awk后都需要保存整行内容到下一个管道命令，你可以试试看。你的文档最后展示的结果是下面这个样子，你再处理一下就好了。

2019:03:41:11 2019:03:41:11 +0000" "POST "06/May/2019:03:41:11 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "500" "500" "0.058" "0.058" "[20953311,21829304,22006993]"
2019:03:41:47 2019:03:41:47 +0000" "POST "06/May/2019:03:41:47 +0000" "POST / HTTP/1.1" "20" "-" "curl/7.29.0" "10.1.7.31" "-" "127.0.0.1:9000" "500" "500" "0.055" "0.055" "[20953311,21829304,22006993]"