Skip to main content
Version: Next

Hub management

Hub management, via cscli allows you to install, upgrade, remove and view installed collections, parsers, scenarios etc.

Collections#

Install#

A collection contains parsers and scenarios to form a coherent ensemble. Most of the time, this is the only you will need to install.

Have nginx running ? cscli collections install crowdsecurity/nginx should do the trick !

Browse the hub for more collections.

sudo cscli collections install <collection_name>
Install crowdsecurity/whitelist-good-actors collection
sudo cscli collections install crowdsecurity/whitelist-good-actorsINFO[0000] crowdsecurity/seo-bots-whitelist : OK        INFO[0000] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rdns_seo_bots.txt' in '/var/lib/crowdsec/data/rdns_seo_bots.txt' INFO[0001] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rnds_seo_bots.regex' in '/var/lib/crowdsec/data/rdns_seo_bots.regex' INFO[0002] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/ip_seo_bots.txt' in '/var/lib/crowdsec/data/ip_seo_bots.txt' INFO[0002] crowdsecurity/cdn-whitelist : OK             INFO[0002] downloading data 'https://www.cloudflare.com/ips-v4' in '/var/lib/crowdsec/data/cloudflare_ips.txt' INFO[0003] crowdsecurity/rdns : OK                      INFO[0003] crowdsecurity/whitelist-good-actors : OK     INFO[0003] /etc/crowdsec/postoverflows/s01-whitelist doesn't exist, create INFO[0003] Enabled postoverflows : crowdsecurity/seo-bots-whitelist INFO[0003] Enabled postoverflows : crowdsecurity/cdn-whitelist INFO[0003] /etc/crowdsec/postoverflows/s00-enrich doesn't exist, create INFO[0003] Enabled postoverflows : crowdsecurity/rdns   INFO[0003] Enabled collections : crowdsecurity/whitelist-good-actors INFO[0003] Enabled crowdsecurity/whitelist-good-actors  INFO[0003] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec

List#

sudo cscli collections list
cscli collections list example
sudo cscli collections list   ------------------------------------------------------------------------------------------------------------- NAME                               ๐Ÿ“ฆ STATUS    VERSION  LOCAL PATH                                         ------------------------------------------------------------------------------------------------------------- crowdsecurity/nginx                โœ”๏ธ  enabled  0.1      /etc/crowdsec/collections/nginx.yaml                crowdsecurity/base-http-scenarios  โœ”๏ธ  enabled  0.1      /etc/crowdsec/collections/base-http-scenarios.yaml  crowdsecurity/sshd                 โœ”๏ธ  enabled  0.1      /etc/crowdsec/collections/sshd.yaml                 crowdsecurity/linux                โœ”๏ธ  enabled  0.2      /etc/crowdsec/collections/linux.yaml               -------------------------------------------------------------------------------------------------------------
tip

This will list only installed parsers.

Use --all to list available parsers.

Upgrade#

sudo cscli hub updatesudo cscli collections upgrade <collection_name>

Collection upgrade allows you to upgrade an existing collection (and its items) to the latest version.

Upgrade crowdsecurity/sshd collection
sudo cscli hub updateINFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli collections upgrade crowdsecurity/sshd  INFO[0000] crowdsecurity/sshd : up-to-date              WARN[0000] crowdsecurity/sshd-logs : overwrite          WARN[0000] crowdsecurity/ssh-bf : overwrite             WARN[0000] crowdsecurity/sshd : overwrite               INFO[0000] ๐Ÿ“ฆ crowdsecurity/sshd : updated               INFO[0000] Upgraded 1 items                             INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective.$ systemctl reload crowdsec

Monitor#

sudo cscli collections inspect <collection_name>

Collections inspect will give you detailed information about a given collection, including versioning information and runtime metrics (fetched from prometheus).

cscli collections inspect example
sudo cscli collections inspect crowdsecurity/sshd       type: collectionsname: crowdsecurity/sshdfilename: sshd.yamldescription: 'sshd support : parser and brute-force detection'author: crowdsecuritybelongs_to_collections:- crowdsecurity/linux- crowdsecurity/linuxremote_path: collections/crowdsecurity/sshd.yamlversion: "0.1"local_path: /etc/crowdsec/collections/sshd.yamllocalversion: "0.1"localhash: 21159aeb87529efcf1a5033f720413d5321a6451bab679a999f7f01a7aa972b3installed: truedownloaded: trueuptodate: truetainted: falselocal: falseparsers:- crowdsecurity/sshd-logsscenarios:- crowdsecurity/ssh-bf
Current metrics : 
 - (Scenario) crowdsecurity/ssh-bf: +---------------+-----------+--------------+--------+---------+| CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |+---------------+-----------+--------------+--------+---------+|             0 |         1 |            2 |     10 |       1 |+---------------+-----------+--------------+--------+---------+

Reference#

See more about collection here.

Parsers#

Install#

sudo cscli parsers install <parser_name>
Install crowdsecurity/iptables-logs parser
sudo cscli parsers install crowdsecurity/iptables-logs    INFO[0000] crowdsecurity/iptables-logs : OK             INFO[0000] Enabled parsers : crowdsecurity/iptables-logs INFO[0000] Enabled crowdsecurity/iptables-logs          INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. 

List#

sudo cscli parsers list

Parsers are yaml files in /etc/crowdsec/parsers/<STAGE>/parser.yaml.

List installed parsers
sudo cscli parsers list-------------------------------------------------------------------------------------------------------------- NAME                            ๐Ÿ“ฆ STATUS    VERSION  LOCAL PATH                                             -------------------------------------------------------------------------------------------------------------- crowdsecurity/whitelists        โœ”๏ธ  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml        crowdsecurity/dateparse-enrich  โœ”๏ธ  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml  crowdsecurity/iptables-logs     โœ”๏ธ  enabled  0.1      /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml      crowdsecurity/syslog-logs       โœ”๏ธ  enabled  0.1      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml          crowdsecurity/sshd-logs         โœ”๏ธ  enabled  0.1      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml          crowdsecurity/geoip-enrich      โœ”๏ธ  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml      crowdsecurity/http-logs         โœ”๏ธ  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml         crowdsecurity/nginx-logs        โœ”๏ธ  enabled  0.1      /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml        --------------------------------------------------------------------------------------------------------------

Upgrade#

sudo cscli hub updatesudo cscli parsers upgrade <parser_name>

Parsers upgrade allows you to upgrade an existing parser to the latest version.

Upgrade crowdsecurity/sshd-logs parser
sudo cscli hub updateINFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli parsers upgrade crowdsecurity/sshd-logs  INFO[0000] crowdsecurity/sshd : up-to-date              WARN[0000] crowdsecurity/sshd-logs : overwrite          WARN[0000] crowdsecurity/ssh-bf : overwrite             WARN[0000] crowdsecurity/sshd : overwrite               INFO[0000] ๐Ÿ“ฆ crowdsecurity/sshd : updated               INFO[0000] Upgraded 1 items                             INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective.

Monitor#

sudo cscli parsers inspect <parser_name>

Parsers inspect will give you detailed information about a given parser, including versioning information and runtime metrics (fetched from prometheus).

Inspect crowdsecurity/sshd-logs parser
sudo cscli parsers inspect crowdsecurity/sshd-logs     type: parsersstage: s01-parsename: crowdsecurity/sshd-logsfilename: sshd-logs.yamldescription: Parse openSSH logsauthor: crowdsecuritybelongs_to_collections:- crowdsecurity/sshdremote_path: parsers/s01-parse/crowdsecurity/sshd-logs.yamlversion: "0.1"local_path: /etc/crowdsec/parsers/s01-parse/sshd-logs.yamllocalversion: "0.1"localhash: ecd40cb8cd95e2bad398824ab67b479362cdbf0e1598b8833e2f537ae3ce2f93installed: truedownloaded: trueuptodate: truetainted: falselocal: false
Current metrics :
 - (Parser) crowdsecurity/sshd-logs:+-------------------+-------+--------+----------+|      PARSERS      | HITS  | PARSED | UNPARSED |+-------------------+-------+--------+----------+| /var/log/auth.log | 94138 |  42404 |    51734 |+-------------------+-------+--------+----------+

Reference#

See more details about parsers here.

Enrichers#

Enrichers are basically parsers that can rely on external methods to provide extra contextual information to the event. The enrichers are usually in the s02-enrich stage (after most of the parsing happened.md).

Enrichers functions should all accept a string as a parameter, and return an associative string array, that will be automatically merged into the Enriched map of the event.

caution

At the time of writing, enrichers plugin mechanism implementation is still ongoing (read: the list of available enrichment methods is currently hardcoded).

As an example let's look into the geoip-enrich parser/enricher :

It relies on the geolite2 data created by maxmind and the geoip2 golang module to provide the actual data.

It exposes three methods : GeoIpCity GeoIpASN and IpToRange that are used by the crowdsecurity/geoip-enrich. Enrichers can be installed as any other parsers with the following command:

sudo cscli parsers install crowdsecurity/geoip-enrich

Take a tour at the Hub to find them !

Reference#

See more about enrichers here.

Scenarios#

Install#

sudo cscli scenarios install <scenario_name>
Install crowdsecurity/http-bf-wordpress_bf scenario
sudo cscli scenarios install crowdsecurity/http-bf-wordpress_bfINFO[0000] crowdsecurity/http-bf-wordpress_bf : OK      INFO[0000] Enabled scenarios : crowdsecurity/http-bf-wordpress_bf INFO[0000] Enabled crowdsecurity/http-bf-wordpress_bf   INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec

List#

sudo cscli scenarios list
tip

This will list only installed parsers.

Use --all to list available parsers.

Scenario are yaml files in /etc/crowdsec/scenarios/.

List installed scenarios
sudo cscli scenarios list--------------------------------------------------------------------------------------------------------------------------- NAME                                       ๐Ÿ“ฆ STATUS    VERSION  LOCAL PATH                                               --------------------------------------------------------------------------------------------------------------------------- crowdsecurity/ssh-bf                       โœ”๏ธ  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml                       crowdsecurity/http-bf-wordpress_bf         โœ”๏ธ  enabled  0.1      /etc/crowdsec/scenarios/http-bf-wordpress_bf.yaml         crowdsecurity/http-crawl-non_statics       โœ”๏ธ  enabled  0.2      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml       crowdsecurity/http-probing                 โœ”๏ธ  enabled  0.1      /etc/crowdsec/scenarios/http-probing.yaml                 crowdsecurity/http-sensitive-files         โœ”๏ธ  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml         crowdsecurity/http-bad-user-agent          โœ”๏ธ  enabled  0.2      /etc/crowdsec/scenarios/http-bad-user-agent.yaml          crowdsecurity/http-path-traversal-probing  โœ”๏ธ  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml  crowdsecurity/http-sqli-probing            โœ”๏ธ  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml            crowdsecurity/http-backdoors-attempts      โœ”๏ธ  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml      crowdsecurity/http-xss-probing             โœ”๏ธ  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml            ---------------------------------------------------------------------------------------------------------------------------

Upgrade#

sudo cscli hub updatesudo cscli scenarios upgrade <scenario_name>

Scenarios upgrade allows you to upgrade an existing scenario to the latest version.

Upgrade crowdsecurity/http-bf-wordpress_bf scenario
sudo cscli hub updateINFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli scenarios upgrade crowdsecurity/ssh-bfINFO[0000] crowdsecurity/ssh-bf : up-to-date            WARN[0000] crowdsecurity/ssh-bf : overwrite             INFO[0000] ๐Ÿ“ฆ crowdsecurity/ssh-bf : updated             INFO[0000] Upgraded 1 items                             INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. 

Monitor#

sudo cscli scenarios inspect <scenario_name>

Scenarios inspect will give you detailed information about a given scenario, including versioning information and runtime metrics (fetched from prometheus).

Inspect crowdsecurity/http-bf-wordpress_bf scenario
sudo cscli scenarios inspect crowdsecurity/ssh-bf    type: scenariosname: crowdsecurity/ssh-bffilename: ssh-bf.yamldescription: Detect ssh bruteforceauthor: crowdsecurityreferences:- http://wikipedia.com/ssh-bf-is-badbelongs_to_collections:- crowdsecurity/sshdremote_path: scenarios/crowdsecurity/ssh-bf.yamlversion: "0.1"local_path: /etc/crowdsec/scenarios/ssh-bf.yamllocalversion: "0.1"localhash: 4441dcff07020f6690d998b7101e642359ba405c2abb83565bbbdcee36de280finstalled: truedownloaded: trueuptodate: truetainted: falselocal: false
Current metrics :
 - (Scenario) crowdsecurity/ssh-bf:+---------------+-----------+--------------+--------+---------+| CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |+---------------+-----------+--------------+--------+---------+|            14 |      5700 |         7987 |  42572 |    2273 |+---------------+-----------+--------------+--------+---------+

Reference#

See more about scenarios here.