Bot detection

Alpha feature

Bot detection is currently in alpha. It's ready to try and we'd love your feedback, but the configuration, helpers and shipped rules may still change between releases.

Bot detection blocks automated traffic before it reaches your application. Where the rest of the WAF reacts to what a user does (the payloads they send, the patterns they trigger) bot detection answers a different question: what a user is, a real browser or a script pretending to be one.

Ready to turn it on?

You can head straight to Enable bot detection for the install-and-verify steps, or read on for what it does for you.

Objectives

• Real users are unaffected. Legitimate browsers clear the check automatically and continue as usual. We try to avoid impacting their experience.
• Automation is stopped at the edge. Headless browsers, clients that don't run JavaScript, and known automation frameworks are filtered by the AppSec component before they reach the origin.
• Repeat offenders are blocked at the bouncer. Persistent probing is turned into CrowdSec decisions by the shipped behavioral scenarios, so the bouncer blocks them over time, not just the one request.
• Good bots are unaffected. Verified search-engine crawlers, uptime probes and the like are recognised and let past without a challenge.
• Every verdict is a CrowdSec signal. Detections show up as alerts in the console and in cscli alerts list, and the fingerprint verdict is available to your own scenarios and appsec-configs.

How it works

Bot detection interposes a browser-side proof-of-work + device-fingerprint check between a visitor and your application:

• The browser sends its proof-of-work and collected device fingerprint back to the CrowdSec WAF.
• If the PoW is correct and the fingerprint is accepted, the client receives a cookie granting it access.
• If the fingerprint is rejected, the PoW is incorrect, or no challenge is submitted, the client is filtered out.

For the full request flow, see How it works.

Prerequisites

• A working AppSec setup. If you don't have one yet, follow the general AppSec quickstart.
• A compatible bouncer. The currently compatible ones (look for the Bot Detection badge at the top of their page) are:
  • Nginx
  • OpenResty
  • HAProxy SPOA
  • Traefik
  • Envoy
• An engine host able to run the challenge runtime. Obfuscating the challenge JavaScript needs a WebAssembly runtime in compiler mode: arm64, or amd64 with SSE4.1 support. The host must also let CrowdSec map executable memory, which W^X hardening, a seccomp profile or a SELinux policy can deny. If either is missing, the AppSec datasource fails to load with wasm compiler mode unavailable or failed to create wasm runtime in compiler mode and CrowdSec does not start.
• Clients that run JavaScript and accept cookies, since the challenge is solved in the browser and its outcome is carried by a cookie. A browser with cookies disabled is shown an explicit error instead of being challenged over and over. Legitimate non-browser clients (API consumers, monitoring, …) must be excluded from the challenge or allowlisted.

Next steps

• Enable bot detection — install the collection, wire the acquisition, and verify it.
• See default collection — the request flow, the appsec-config, known-bot handling, and behavioral scenarios you just enabled.
• Customization & recipes — narrow the challenge to a path, allowlist a probe, and react to the bot signal in your own scenarios.
• Configuration — master secret, key rotation, cookie TTL, and JS obfuscation.
• Hooks reference — full list of helpers and the on_challenge / on_challenge_submit stages.
• Challenge protocol — the wire contract, for anyone adding challenge support to a remediation component.