Skip to main content

Troubleshooting Security Engine

Community support​

Please try to resolve your issue by reading the documentation. If you're unable to find a solution, don't hesitate to seek assistance in:

Health​

How to check the status​

sudo systemctl status crowdsec

How to check if LAPI is reachable​

cscli lapi status
Example output
time="27-11-2023 13:56:13" level=info msg="Loaded credentials from /etc/crowdsec/local_api_credentials.yaml"
time="27-11-2023 13:56:13" level=info msg="Trying to authenticate with username 6df9e06234fa428d9ba8e78f9b557c4axNHfFR9BmLcUlSfq on http://127.0.0.1:8080/"
time="27-11-2023 13:56:13" level=info msg="You can successfully interact with Local API (LAPI)"

This command can be run on child nodes to check if they can reach the LAPI of the parent node.

How to check if CAPI is reachable​

cscli capi status
Example output
time="2024-01-08 14:08:20" level=info msg="Loaded credentials from /etc/crowdsec/online_api_credentials.yaml"
time="2024-01-08 14:08:20" level=info msg="Trying to authenticate with username XXXXXXXXXXX on https://api.crowdsec.net/"
time="2024-01-08 14:08:22" level=info msg="You can successfully interact with Central API (CAPI)"

This command can should ONLY be run on the parent node.

Decisions​

How to list current decisions​

cscli decisions list

Additional filtering is possible please read cscli decisions list for more information.

How to remove a decision on a IP​

cscli decisions delete -i x.x.x.x

Additional filtering is possible please read cscli decisions delete for more information.

How to control granularity of decisions​

The Security Engine is designed to be as flexible as possible, and allows you to control the granularity of decisions.

Profiles allows you to control which decision will be applied to which alert.

How to add whitelists or prevent the Security Engine from banning a given IP​

There are many ways to prevent the Security Engine from banning a given IP, depending on your use case. You can see the whitelist section on how to achieve this.

I need to whitelist an event pattern instead of a single IP address​

You can whitelist a specific event pattern by following the expression section.

Scenarios​

Scenario X keeps triggering, it's a false trigger​

To address a particular scenario that is causing concern, you can choose from several options:

  • Activate simulation mode: In this mode, you'll receive alerts, but no actual decisions will be implemented.

  • Completely remove the scenario: By doing this, the scenario will be entirely disabled.

Set a custom/tainted scenario in simulation mode​

If you want to set a custom/tainted scenario in simulation mode, you need to provide the scenario's filename instead of its name.

For example, i have a scenario called crowdsecurity/my-custom-scenario located in {config_dir}/scenarios/my_custom_scenario.yaml.

Please see configuration section to see where the file will be located on your system.

To enable the simulation mode for this scenario, i need to run:

sudo cscli simulation enable my_custom_scenario.yaml

My scenario is triggered with less logs than the scenario capacity​

When you install CrowdSec, the CrowdSec Wizard runs automatically to find and add basic log files to the acquisition configuration. If you run the 'wizard.sh' script again after installing and you have common log files, they might be added multiple times to your acquisition configuration. This causes CrowdSec to read each log line as many times as the file is configured in the acquisition settings.

Please review your acquisition files and remove any duplicate log entires.

Is scenario X working on my logs ?​

You can replay old logs with replay mode, which will allow you to see which scenarios would have been triggered. If no scenario seem to trigger, you can have a closer look at potential parsing errors with cscli explain.

Configuration​

Where is configuration stored?​

Depending on your installation method and/or operating system, the configuration files are stored in different locations.

Here is a list of the most common locations:

  • Linux /etc/crowdsec/
  • Freebsd /usr/local/etc/crowdsec/
  • Windows C:\ProgramData\CrowdSec\
  • Kubernetes /etc/crowdsec/ can be overwriten in values.yaml.

Where is data stored?​

CrowdSec stores its data in /var/lib/crowdsec/data/ folder for UNIX and C:\ProgramData\CrowdSec\data\ for Windows.

This is where the default sqlite database and data files needed for scenarios are kept.

Which databases does the Security Engine support and how to switch ?​

Security Engine versions (after v1) supports SQLite (default), MySQL and PostgreSQL databases. See databases configuration for relevant configuration. Thanks to the Local API, distributed architectures are resolved even with sqlite database.

SQLite by default as it's suitable for standalone/single-engine setups.

Multi-server setup​

For multi-server setup please follow our guides on distributed architecture.

Logs​

Where are the logs stored?​

By default CrowdSec will log to the following locations depending on platform:

  • Linux /var/log/crowdsec.log
  • Freebsd /var/log/crowdsec.log
    • Opnsense /var/log/crowdsec/crowdsec.log
    • Pfsense /var/log/crowdsec/crowdsec.log
  • Windows C:\ProgramData\CrowdSec\log\crowdsec.log
  • Kubernetes kubectl logs -n crowdsec crowdsec-(agent|lapi)-*

This is the default log location if you are using a third party integration they may change the log location.

If you are unsure where CrowdSec is logging you can view the configuration file config.yaml and look for log_dir to see the configured location.

Filtering logs to only show errors​

You can use the os related commands to filter the logs to only show errors.

sudo grep -E "level=(error|fatal)" /var/log/crowdsec.log

Please make sure the log location matches your distribution.

Common Errors​

If it's a configuration file issue, the error message may log in a different location. As an example, if a parser/scenario file is invalid, the error message will be logged within the configured log file, however, if config.yaml is invalid, the error message will be logged in syslog instead.

Hash Mismatch​

  • error message might look like:
level=fatal msg="error while installing 'crowdsecurity/traefik': while downloading crowdsecurity/http-cve: while downloading crowdsecurity/http-cve to https://hub-cdn.crowdsec.net/master/collections/crowdsecurity/http-cve.yaml: hash mismatch: expected dd2e8debbba19d19646b9a8010baf9bd901a4eec84a53a0cb62294b8da3e91ef, got b974f20dc2cf23c292eede422c1c6e0008297f8b3397e3851bb6a988866f3e34"

A hash mismatch may occur when the local .index.json (hub index) is out of date with the current latest.

Solutions differ based on how you are running CrowdSec:

  • Bare Metal Install: Run cscli hub update on the machine and this will update the index file for you to run the previous install command.
  • Docker/Podman: The docker startup script will automatically try to update the hub index if there is an update pending.

Within the container the hub index will NOT be automatically updated if the modification time of the file is less than 24 hours from when the container starts so to force a index update you can manually run a temporary container

Temporary container to update index​

docker run --rm -it --entrypoint cscli -v $MountPath:/etc/crowdsec/ -v $MountPath:/var/lib/crowdsec/data/ crowdsecurity/crowdsec:latest hub update

In the example above I have used $MountPath as place holders for you to update where the mounts are if they are relative via your compose then launch the run within the compose directory with the values updated, if they are named volumes then replace with the named volumes. The names are volumes may have a prefix so check via docker volume ls for the actual names.

Once the temporary container has run and terminated you will be able to bring up the container as normal via docker compose up -d

If you are running a previous version of CrowdSec then please update the docker run command to us the same tag as the one deployed

Cannot bind to the configured port or IP​

  • error message might look like:
level=fatal msg="while serving local API: listen tcp 127.0.0.1:8080: bind: address already in use"
  • solution verify another service is not already using the port or ip address + port combination. If it is, you can edit the listen_uri in the configuration file config.yaml and update local_api_credentials.yaml to the same value. Then you can restart CrowdSec with sudo systemctl restart crowdsec.

How can I check to see if a service is already using a port?​

You can use the following os commands to check if a service is already using a port.

For example, to check if a service is using port 8080: (Our default port for the local API)

sudo netstat -tulpn | grep :8080

See prerequisites for all the ports used by the Security Engine.

Cannot authenticate to the local API​

  • error message might look like:
level=fatal msg="starting outputs error : authenticate watcher (fcb7303c4df44c03ada289dd7ec3dbe7cU3GaseSWdqUaVg6): API error: ent: machine not found"
  • solution regenerate the credentials via cscli machines command. If the local API is on the same machine you can run sudo cscli machines add -a (-a will automatically generate a random machine name and password). Then you can restart CrowdSec with sudo systemctl restart crowdsec.

Cannot connect to the local API​

  • error message might look like:
level=error msg="error while performing request: dial tcp 127.0.0.1:8080: connect: connection refused; 4 retries left"

There may be other variations of this error message just note if the message contains connection refused or connection reset by peer or no such host its likely a connection issue or configuration.

  • solution verify that the local API runs on the logged IP and port. If the logged IP and port is incorrect, you can update /etc/crowdsec/local_api_credentials.yaml to the correct IP and port (If local API is running on the same machine you can run grep listen_uri /etc/crowdsec/config.yaml to find it). Then you can restart CrowdSec with sudo systemctl restart crowdsec. If the logged IP and port is correct, verify that the local API is running via sudo systemctl status crowdsec.

Cannot start because of an invalid configuration file​

  • error message might look like:
level=fatal msg="/etc/crowdsec/config.yaml: yaml: unmarshal errors:\n  line 1: field test not found in type csconfig.Config"
  • solution CrowdSec will inform you which field or line is invalid. You can edit the configuration file and fix the error. Then you can restart CrowdSec with sudo systemctl restart crowdsec. If you are unsure what the configuration file should look like you can find the default configuration files here or examples via the documentation.

I receive few IPs in the community-blocklist​

The community-blocklist that you receive is based on your installed scenarios, if they are neither tainted nor custom.

For example, if your crowdsecurity/ssh-bf scenario is tainted, you will not receive IPs concerning this scenario in the community-blocklist.