Troubleshooting Security Engine
Community supportβ
Please try to resolve your issue by reading the documentation. If you're unable to find a solution, don't hesitate to seek assistance in:
Healthβ
How to check the statusβ
- Linux/Freebsd
- Windows
- Kubernetes
sudo systemctl status crowdsec
Get-Service crowdsec
kubectl get pods -n crowdsec
How to check if LAPI is reachableβ
cscli lapi status
Example output
time="27-11-2023 13:56:13" level=info msg="Loaded credentials from /etc/crowdsec/local_api_credentials.yaml"
time="27-11-2023 13:56:13" level=info msg="Trying to authenticate with username 6df9e06234fa428d9ba8e78f9b557c4axNHfFR9BmLcUlSfq on http://127.0.0.1:8080/"
time="27-11-2023 13:56:13" level=info msg="You can successfully interact with Local API (LAPI)"
How to check if CAPI is reachableβ
cscli capi status
Example output
time="2024-01-08 14:08:20" level=info msg="Loaded credentials from /etc/crowdsec/online_api_credentials.yaml"
time="2024-01-08 14:08:20" level=info msg="Trying to authenticate with username XXXXXXXXXXX on https://api.crowdsec.net/"
time="2024-01-08 14:08:22" level=info msg="You can successfully interact with Central API (CAPI)"
Decisionsβ
How to list current decisionsβ
cscli decisions list
Additional filtering is possible please read cscli decisions list for more information.
How to remove a decision on a IPβ
cscli decisions delete -i x.x.x.x
Additional filtering is possible please read cscli decisions delete for more information.
How to control granularity of decisionsβ
The Security Engine is designed to be as flexible as possible, and allows you to control the granularity of decisions.
Profiles allows you to control which decision will be applied to which alert.
How to add whitelists or prevent the Security Engine from banning a given IPβ
There are many ways to prevent the Security Engine from banning a given IP, depending on your use case. You can see the whitelist section on how to achieve this.
I need to whitelist an event pattern instead of a single IP addressβ
You can whitelist a specific event pattern by following the expression section.
Scenariosβ
Scenario X keeps triggering, it's a false triggerβ
To address a particular scenario that is causing concern, you can choose from several options:
-
Activate simulation mode: In this mode, you'll receive alerts, but no actual decisions will be implemented.
-
Completely remove the scenario: By doing this, the scenario will be entirely disabled.
Set a custom/tainted scenario in simulation modeβ
If you want to set a custom/tainted scenario in simulation mode, you need to provide the scenario's filename instead of its name.
For example, i have a scenario called crowdsecurity/my-custom-scenario
Β located in {config_dir}/scenarios/my_custom_scenario.yaml
.
To enable the simulation mode for this scenario, i need to run:
- Linux/Freebsd
- Windows
- Kubernetes
sudo cscli simulation enable my_custom_scenario.yaml
cscli.exe simulation enable my_custom_scenario.yaml
kubectl exec -it crowdsec-agent-* -n crowdsec -- cscli simulation enable my_custom_scenario.yaml
My scenario is triggered with less logs than the scenario capacityβ
When you install CrowdSec, the CrowdSec Wizard runs automatically to find and add basic log files to the acquisition configuration. If you run the 'wizard.sh' script again after installing and you have common log files, they might be added multiple times to your acquisition configuration. This causes CrowdSec to read each log line as many times as the file is configured in the acquisition settings.
Please review your acquisition files and remove any duplicate log entires.
Is scenario X working on my logs ?β
You can replay old logs with replay mode, which will allow you to see which scenarios would have been triggered. If no scenario seem to trigger, you can have a closer look at potential parsing errors with cscli explain
.
Configurationβ
Where is configuration stored?β
Depending on your installation method and/or operating system, the configuration files are stored in different locations.
Here is a list of the most common locations:
- Linux
/etc/crowdsec/
- Freebsd
/usr/local/etc/crowdsec/
- Windows
C:\ProgramData\CrowdSec\
- Kubernetes
/etc/crowdsec/
can be overwriten in values.yaml.
Where is data stored?β
CrowdSec stores its data in /var/lib/crowdsec/data/
folder for UNIX and C:\ProgramData\CrowdSec\data\
for Windows.
This is where the default sqlite database and data files needed for scenarios are kept.
Which databases does the Security Engine support and how to switch ?β
Security Engine versions (after v1) supports SQLite (default), MySQL and PostgreSQL databases. See databases configuration for relevant configuration. Thanks to the Local API, distributed architectures are resolved even with sqlite database.
SQLite by default as it's suitable for standalone/single-engine setups.
Multi-server setupβ
For multi-server setup please follow our guides on distributed architecture.
Logsβ
Where are the logs stored?β
By default CrowdSec will log to the following locations depending on platform:
- Linux
/var/log/crowdsec.log
- Freebsd
/var/log/crowdsec.log
- Opnsense
/var/log/crowdsec/crowdsec.log
- Pfsense
/var/log/crowdsec/crowdsec.log
- Opnsense
- Windows
C:\ProgramData\CrowdSec\log\crowdsec.log
- Kubernetes
kubectl logs -n crowdsec crowdsec-(agent|lapi)-*
This is the default log location if you are using a third party integration they may change the log location.
Filtering logs to only show errorsβ
You can use the os related commands to filter the logs to only show errors.
- Linux/Freebsd
- Windows
- Kubernetes
sudo grep -E "level=(error|fatal)" /var/log/crowdsec.log
- Powershell
- CMD
Select-String "level=(error|fatal)" C:\ProgramData\CrowdSec\log\crowdsec.log
findstr "level=error level=fatal" C:\ProgramData\CrowdSec\log\crowdsec.log
kubectl logs -n crowdsec crowdsec-agent-* | grep -E "level=(error|fatal)"
Please make sure the log location matches your distribution.
Common Errorsβ
Hash Mismatchβ
- error message might look like:
level=fatal msg="error while installing 'crowdsecurity/traefik': while downloading crowdsecurity/http-cve: while downloading crowdsecurity/http-cve to https://hub-cdn.crowdsec.net/master/collections/crowdsecurity/http-cve.yaml: hash mismatch: expected dd2e8debbba19d19646b9a8010baf9bd901a4eec84a53a0cb62294b8da3e91ef, got b974f20dc2cf23c292eede422c1c6e0008297f8b3397e3851bb6a988866f3e34"
A hash mismatch may occur when the local .index.json
(hub index) is out of date with the current latest.
Solutions differ based on how you are running CrowdSec:
- Bare Metal Install: Run
cscli hub update
on the machine and this will update the index file for you to run the previous install command. - Docker/Podman: The docker startup script will automatically try to update the hub index if there is an update pending.
Temporary container to update indexβ
docker run --rm -it --entrypoint cscli -v $MountPath:/etc/crowdsec/ -v $MountPath:/var/lib/crowdsec/data/ crowdsecurity/crowdsec:latest hub update
In the example above I have used $MountPath
as place holders for you to update where the mounts are if they are relative via your compose then launch the run within the compose directory with the values updated, if they are named volumes then replace with the named volumes. The names are volumes may have a prefix so check via docker volume ls
for the actual names.
Once the temporary container has run and terminated you will be able to bring up the container as normal via docker compose up -d
Cannot bind to the configured port or IPβ
- error message might look like:
level=fatal msg="while serving local API: listen tcp 127.0.0.1:8080: bind: address already in use"
- solution verify another service is not already using the port or ip address + port combination. If it is, you can edit the
listen_uri
in the configuration fileconfig.yaml
and updatelocal_api_credentials.yaml
to the same value. Then you can restart CrowdSec withsudo systemctl restart crowdsec
.
How can I check to see if a service is already using a port?β
You can use the following os commands to check if a service is already using a port.
For example, to check if a service is using port 8080: (Our default port for the local API)
- Linux/Freebsd
- Windows
sudo netstat -tulpn | grep :8080
Get-NetTCPConnection | where Localport -eq 8080 | select Localport,OwningProcess
See prerequisites for all the ports used by the Security Engine.
Cannot authenticate to the local APIβ
- error message might look like:
level=fatal msg="starting outputs error : authenticate watcher (fcb7303c4df44c03ada289dd7ec3dbe7cU3GaseSWdqUaVg6): API error: ent: machine not found"
- solution regenerate the credentials via cscli machines command. If the local API is on the same machine you can run
sudo cscli machines add -a
(-a
will automatically generate a random machine name and password). Then you can restart CrowdSec withsudo systemctl restart crowdsec
.
Cannot connect to the local APIβ
- error message might look like:
level=error msg="error while performing request: dial tcp 127.0.0.1:8080: connect: connection refused; 4 retries left"
- solution verify that the local API runs on the logged IP and port. If the logged IP and port is incorrect, you can update
/etc/crowdsec/local_api_credentials.yaml
to the correct IP and port (If local API is running on the same machine you can rungrep listen_uri /etc/crowdsec/config.yaml
to find it). Then you can restart CrowdSec withsudo systemctl restart crowdsec
. If the logged IP and port is correct, verify that the local API is running viasudo systemctl status crowdsec
.
Cannot start because of an invalid configuration fileβ
- error message might look like:
level=fatal msg="/etc/crowdsec/config.yaml: yaml: unmarshal errors:\n line 1: field test not found in type csconfig.Config"
- solution CrowdSec will inform you which field or line is invalid. You can edit the configuration file and fix the error. Then you can restart CrowdSec with
sudo systemctl restart crowdsec
. If you are unsure what the configuration file should look like you can find the default configuration files here or examples via the documentation.
I receive few IPs in the community-blocklistβ
The community-blocklist that you receive is based on your installed scenarios, if they are neither tainted nor custom.
For example, if your crowdsecurity/ssh-bf
scenario is tainted, you will not receive IPs concerning this scenario in the community-blocklist
.