Skip to main content
Version: v1.3.0

Understand logs processing

info

cscli explain relies on your local setup, parsers, scenario to display its data. It requires a working local crowdsec setup.

cscli explain: understand what is going on#

cscli explain allows you to understand how your logs are processed and in which scenarios they end up.

This can be done with a single line, with a given logfile, or via a full dsn :

cscli explain --file ./myfile.log --type nginx cscli explain --log "Sep 19 18:33:22 scw-d95986 sshd[24347]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4" --type syslogcscli explain --dsn "file://myfile.log" --type nginx

Hint: if your are creating/collecting data on the fly (over a network, for example) and want to avoid temporary files, you can use cscli explain --file /dev/fd/0 or cscli explain -dsn "file://dev/fd/0" to refer to standard input.

The typical output looks like this :

β–Ά sudo cscli explain --file ./x.log --type nginx 
line: xx.xx.xx.xx - - [05/Nov/2017:07:23:41 +0100] "GET /Og9vl1%0d%2019s58%3Atest HTTP/1.1" 404 136 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko" "-"    β”œ s00-raw    |   β”œ 🟒 crowdsecurity/non-syslog (first_parser)    |   β”” πŸ”΄ crowdsecurity/syslog-logs    β”œ s01-parse    |   β”” 🟒 crowdsecurity/nginx-logs (+22 ~2)    β”œ s02-enrich    |   β”œ 🟒 crowdsecurity/dateparse-enrich (+1 ~1)    |   β”œ 🟒 crowdsecurity/geoip-enrich (+12)    |   β”œ 🟒 crowdsecurity/http-logs (+7)    |   β”” 🟒 crowdsecurity/whitelists (unchanged)    β”œ-------- parser success 🟒    β”œ Scenarios        β”œ 🟒 crowdsecurity/http-crawl-non_statics        β”” 🟒 crowdsecurity/http-probing

Lines represent if and how a parser interacted with the line: how many fields were added (+), modified (~), or deleted (-). Other visual clues include: unchanged a parser didn't modify an event (meaning it is not relevant) or [whitelisted] if a whitelist matched the event.

What happens :

  1. Our log line line: xxx.xxx.xxx.xx - - [05/Nov/2017:07:23:41 +0100] "GET ... starts being parsed
  2. It enters the first stage s00-raw
  • It goes through the syslog-logs and non-syslog parsers of crowdsecurity/syslog-logs
    • as syslog-logs is successful and has onsuccess: next_stage, line can move to next stage
  1. It enters the stage s01-parse
  1. It enters the stage s02-enrich
  1. At this point, the log line successfully exited the parsing process and is ready to be processed by scenarios :
  1. Please note that this feature does not track overflows themselves, it only points out if the event was able to enter the scenario

Hopefully, this feature will help users understand behavior when debugging crowdsec or creating parsers and/or scenarios!

Verbose mode#

When troubleshooting parsers, the --verbose/-v option offers extra information. Every changes made to the event will be displayed along below the associated parser.

β–Ά ./cscli -c dev.yaml explain --file /tmp/xx --type nginx --verboseline: 10.42.42.42 - - [23/Oct/2017:10:46:06 +0200] "GET /admin/2019p1mnsa8q1ktU HTTP/1.1" 404 136 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko (Wallarm DirBuster)" "-"    β”œ s00-raw    |   β”œ 🟒 crowdsecurity/non-syslog (first_parser)    |   β”” πŸ”΄ crowdsecurity/syslog-logs    β”œ s01-parse    |   β”” 🟒 crowdsecurity/nginx-logs (+22 ~2)    |       β”” update Stage : s01-parse -> s02-enrich    |       β”” create Parsed.body_bytes_sent : 136    |       β”” create Parsed.http_referer : -    |       β”” create Parsed.http_user_agent : Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko (Wallarm DirBuster)    |       β”” create Parsed.status : 404    |       β”” create Parsed.target_fqdn :     |       β”” create Parsed.proxy_alternative_upstream_name :     |       β”” create Parsed.request_time :     |       β”” create Parsed.time_local : 23/Oct/2017:10:46:06 +0200    |       β”” create Parsed.proxy_upstream_name :     |       β”” create Parsed.remote_user : -    |       β”” create Parsed.request : /admin/2019p1mnsa8q1ktU    |       β”” create Parsed.http_version : 1.1    |       β”” create Parsed.remote_addr : 10.42.42.42    |       β”” create Parsed.request_length :     |       β”” create Parsed.verb : GET    |       β”” update StrTime :  -> 23/Oct/2017:10:46:06 +0200    |       β”” create Meta.http_user_agent : Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko (Wallarm DirBuster)    |       β”” create Meta.service : http    |       β”” create Meta.source_ip : 10.42.42.42    |       β”” create Meta.http_verb : GET    |       β”” create Meta.log_type : http_access-log    |       β”” create Meta.http_path : /admin/2019p1mnsa8q1ktU    |       β”” create Meta.http_status : 404    β”œ s02-enrich    |   β”œ 🟒 crowdsecurity/dateparse-enrich (+1 ~1)    |       β”œ create Enriched.MarshaledTime : 2017-10-23T10:46:06+02:00

How it works#

Behind the scene, cscli explain relies on your local crowdsec setup to launch a crowdsec instance to process the given logs, and "simply" provides a more user-friendly representation of the debug information.