AWS S3
This module allows the Security Engine
to acquire logs from a S3 bucket.
It supports reading plain text file and gzip file (detection is performed based on the file extension).
Configuration example
To monitor a S3 bucket detecting new objects from a SQS queue:
source: s3
polling_method: sqs
sqs_name: test-sqs-s3-acquis
labels:
type: foo
To monitor a S3 bucket detecting new objects by listing the bucket content:
source: s3
polling_method: list
bucket_name: my_bucket
polling_interval: 30
labels:
type: foo
Look at the configuration parameters
to view all supported options.
Parameters
polling_method
How to detect new files in a bucket. Must be one of sqs
or list
.
sqs
is the recommended mode.
polling_interval
How often in seconds the Security Engine will check for new objects in a bucket when using the list
polling method.
Defaults to 60.
sqs_name
Name of the SQS queue to poll.
Required when polling_method
is sqs
.
sqs_format
Format of the body inside the SQS messages.
Can be eventbridge
or s3notification
.
If not set, the Security Engine will automatically select the format based on the first valid event received from the queue.
bucket_name
Name of the bucket to poll.
Required when polling_method
is list
.
prefix
Only read objects matching this prefix when polling_method
is list
.
Optional, ignored when polling_method
is sqs
.
aws_profile
The AWS profile to use, relies on your ~/.aws/config/
.
Optional, the data source will automatically use the standard AWS env vars if present.
aws_region
The AWS region.
Optional, the data source will automatically use the standard AWS env vars if present.
aws_endpoint
Endpoint for AWS API. Optional, the data source will automatically use the standard AWS env vars if present.
Can be used to point the Security Engine to a S3-compatible object storage.
source
Must be s3
DSN and command-line
This module supports acquisition directly from the command line, to read files in one shot.
A single s3 URI is accepted with the -dsn
parameter, but you don't have to specify a specific object.
If no object is specified (either just a bucket, or a bucket and a prefix), the Security Engine will read all files matching the prefix.
If you don't specify an object, the path must end with /
.
crowdsec -type syslog -dsn s3://my_bucket/
crowdsec -type syslog -dsn s3://my_bucket/my_prefix/
crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log
You can specify the log_level
parameter to change the log level for the acquisition:
crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log?log_level=debug
AWS SDK behaviour can be configured with the standard AWS environment variables.