Examples¶
Note
Please feel free to contribute your own examples on github
Simple Text¶
Histogram Values in Text File¶
$ cat input.txt
1
2
1
3
1
0
$ rare histo input.txt
1 3
0 1
2 1
3 1
Matched: 6 / 6 (Groups: 4)
Extract Numbers from Text¶
$ rare filter --match "(\d+)" input.txt
Extract matched value as JSON¶
$ rare f --match "(?P<val>\d+)" -e "{.}" simple.log
{"val": 1}
{"val": 2}
{"val": 1}
Histogram of Numbers in Text¶
$ rare histo --match "(\d+)" -e "{1}" -x input.txt
1 3
0 1
2 1
3 1
Matched: 6 / 6 (Groups: 4)
# Or with Bars/percentages
./rare histo --match "(\d+)" -e "{1}" -x simple.log
1 7 [58.3%] ||||||||||||||||||||||||||||||||||||||||||||||||||
3 2 [16.7%] ||||||||||||||
2 2 [16.7%] ||||||||||||||
0 1 [ 8.3%] |||||||
Matched: 12 / 12 (Groups: 4)
# Logarithmic Scale
./rare histo --match "(\d+)" -e "{1}" --scale log10 -x simple.log
1 7 [58.3%] ||||||||||||||||||||||||||||||||||||||||||
3 2 [16.7%] |||||||||||||||
2 2 [16.7%] |||||||||||||||
0 1 [ 8.3%]
Matched: 12 / 12 (Groups: 4)
Nginx¶
Highlight / Extract HTTP Code and Size¶
# Will colorize HTTP code and size in full log
$ rare filter -m "(\d{3}) (\d+)" access.log
# Will only display http code and size
$ rare filter -m "(\d{3}) (\d+)" -e "{1} {2}" access.log
HTTP Status Histogram¶
Parse error codes and graph in a histogram
$ rare h -m "\" (\d+)" -e "{1}" -z -x testdata/*
404 5,557,374 [66.4%] ||||||||||||||||||||||||||||||||||||||||||||||||||
200 2,564,984 [30.6%] |||||||||||||||||||||||
400 243,282 [ 2.9%] ||
405 5,708 [ 0.1%]
408 1,397 [ 0.0%]
Matched: 8,373,328 / 8,383,717 (Groups: 8)
Extracting Page Sizes¶
Page sizes, ignoring 0-sized pages
$ rare h -m "\" (\d+) (\d+)" -e "{bytesize {bucket {2} 1024}}" -i "{lt {2} 1024}" -z -x testdata/*
234 KB 3,602 [14.6%] ||||||||||||||||||||||||||||||||||||||||||||||||||
149 KB 2,107 [ 8.5%] |||||||||||||||||||||||||||||
193 KB 1,519 [ 6.2%] |||||||||||||||||||||
192 KB 1,470 [ 6.0%] ||||||||||||||||||||
191 KB 1,421 [ 5.8%] |||||||||||||||||||
Matched: 24,693 / 8,383,717 (Groups: 96) (Ignored: 8,348,635)
Extract number of bytes sent by bucket, and format¶
This shows an example of how to bucket the values into size of 1000. In this case, it doesn't make
sense to see the histogram by number of bytes, but we might want to know the ratio of various orders-of-magnitudes.
$ rare histo -m '"(\w{3,4}) ([A-Za-z0-9/.]+).*" (\d{3}) (\d+)' -e "{bucket {4} 10000}" -n 10 -b access.log
0 144239 ||||||||||||||||||||||||||||||||||||||||||||||||||
190000 2599
10000 1290
180000 821
20000 496
30000 445
40000 440
200000 427
140000 323
70000 222
Matched: 161622 / 161622
Groups: 1203
Table of URLs to HTTP Status¶
Know how your URLs are responding by their http statuses
$ rare t -m "\"(\w+) (.+).*\" (\d+) (\d+)" -e "{$ {3} {substr {2} 0 20}}" -z testdata/*
404 200 400
/ HTTP/1.1 0 127,624 5,681
/ HTTP/1.0 0 5,222 0
/test.php HTTP/1.1 3,241 0 0
/1.php HTTP/1.1 2,508 0 0
/qq.php HTTP/1.1 1,908 0 0
/index.php HTTP/1.1 1,776 0 0
/shell.php HTTP/1.1 1,750 0 0
/cmd.php HTTP/1.1 1,588 0 0
/x.php HTTP/1.1 1,573 0 0
/log.php HTTP/1.1 1,261 0 0
/confg.php HTTP/1.1 1,253 0 0
/api.php HTTP/1.1 1,241 0 0
/ss.php HTTP/1.1 1,233 0 0
/mirror/distros/vlit 0 1,122 0
/robots.txt HTTP/1.1 1,056 0 0
/vendor/phpunit/phpu 1,055 0 0
/aaa.php HTTP/1.1 954 0 0
/hell.php HTTP/1.1 948 0 0
/z.php HTTP/1.1 948 0 0
Matched: 465,348 / 470,163 (R: 2396; C: 8)
Heatmap of Requests per Week¶
$ rare heatmap -m '\[(.+?)\].*" (\d+)' -e "{timeattr {time {1}} yearweek}" -e "{2}" access.log
- 0 4 22,602 9 45,204
2019-34..2019-43..2019-52..2020-17..2020-25..2020-33...2020-9
200 11--111111-11111-111111111-1111111-111111-11111-1111-11--1-1-
206 -------------------------------------------------------------
301 -------------------------------------------------------------
304 -------------------------------------------------------------
400 -------------------------------------------------------------
404 33415264914143253212111-1411-13-131-1312-122111--13-1-1-13211
405 -------------------------------------------------------------
408 -------------------------------------------------------------
Matched: 1,035,666 / 1,035,666 (R: 8; C: 61)
Bargraph status codes per year¶
NOTE: For stacking (-s), the results will be color-coded (not shown here)
$ rare bars -z -m "\[(.+?)\].*\" (\d+)" -e "{buckettime {1} year}" -e "{2}" testdata/*
| 200 | 206 | 301 | 304 | 400 | 404 | 405 | 408
2019 ||||||||||||||||||||||||||||||||||||||| 3,741,444
2020 ||||||||||||||||||||||||||||||||||||||||||||||||| 4,631,884
Matched: 8,373,328 / 8,383,717