SQS Queues
In the following example, we specify that the compute function should be triggered whenever there are messages in the given SQS Queue.
The ARN for the queue can be specified as a string, the reference to the ARN of a resource by logical ID, or the import of an ARN that was exported by a different service or CloudFormation stack.
Note: The sqs event will hook up your existing SQS Queue to a Lambda function. Serverless won't create a new queue for you.
functions:
compute:
handler: handler.compute
events:
# These are all possible formats
- sqs: arn:aws:sqs:region:XXXXXX:MyFirstQueue
- sqs:
arn:
Fn::GetAtt:
- MySecondQueue
- Arn
- sqs:
arn:
Fn::ImportValue: MyExportedQueueArnId
- sqs:
arn:
Fn::Join:
- ':'
- - arn
- aws
- sqs
- Ref: AWS::Region
- Ref: AWS::AccountId
- MyOtherQueue
Setting the BatchSize
For the SQS event integration, you can set the batchSize, which effects how many SQS messages can be included in a single Lambda invocation. The default batchSize is 10. The max batchSize is 10000 for a standard queue, 10 for a FIFO queue.
You can also set maximumBatchingWindow to standard queues to specify the maximum amount of time in seconds to gather records before invoking the function. The max maximumBatchingWindow is 300 seconds.
You can set functionResponseType to ReportBatchItemFailures to let your function return a partial success result if one or more messages in the batch have failed.
Check AWS documentation for more details.
functions:
compute:
handler: handler.compute
events:
- sqs:
arn: arn:aws:sqs:region:XXXXXX:myQueue
batchSize: 10
maximumBatchingWindow: 60
functionResponseType: ReportBatchItemFailures
Setting filter patterns
This configuration allows customers to filter event before lambda invocation. It accepts up to 5 filter patterns by default and up to 10 with quota extension. If one event matches at least 1 pattern, lambda will process it.
For more details and examples of filter patterns, please see the AWS event filtering documentation
Note: Serverless only sets this property if you explicitly add it to the sqs configuration (see an example below). The following example will only process records where field a is equal to 1 or 2.
functions:
onlyOneOrTwo:
handler: handler.preprocess
events:
- sqs:
arn: arn:aws:sqs:region:XXXXXX:myQueue
filterPatterns:
- a: [1, 2]
Setting the Maximum Concurrency
The maximum concurrency setting limits the number of concurrent instances of the function that an Amazon SQS event source can invoke. The minimum limit of concurrent functions that the event source can invoke is 2, and the maximum is 1000. To turn off maximum concurrency, leave this field empty.
For more details, see the AWS SQS max concurrency documentation.
functions:
onlyOneOrTwo:
handler: handler.preprocess
events:
- sqs:
arn: arn:aws:sqs:region:XXXXXX:myQueue
maximumConcurrency: 250
Provisioned mode
Provisioned mode gives the event source mapping a dedicated pool of event pollers with minimum and maximum bounds, for faster scaling and higher throughput than the default on-demand polling. Configure it with provisionedPollers:
functions:
compute:
handler: handler.compute
events:
- sqs:
arn: arn:aws:sqs:region:XXXXXX:myQueue
provisionedPollers:
min: 2 # optional, 2-200, AWS default 2
max: 500 # optional, 2-10000, AWS default 200
Provisioned mode incurs additional AWS charges per event poller — the min value is an always-on cost. See AWS Lambda pricing.
provisionedPollers and maximumConcurrency are mutually exclusive scaling modes — setting both fails validation at package time. In provisioned mode, control concurrency through max (each SQS event poller drives up to 10 concurrent invocations).
Poller groups (group) are available for kafka and msk event sources only.
Both cross-field rules — provisionedPollers together with maximumConcurrency, and a min above max — fail at packaging time whatever configValidationMode is set to. A min or max outside the bounds above is a schema violation, so it follows configValidationMode: under the default warn it is reported as a warning and deployed as written, for AWS to reject; error stops packaging; off deploys it silently.
Disabling provisioned mode: removing provisionedPollers from your configuration does not disable it — provisioned mode stays active on the deployed event source mapping, so the pollers (and their cost) remain. The same applies to rollback to an older deployment. To actually disable it, deploy once with:
provisionedPollers: false
false clears provisioned mode on an existing mapping. It cannot be used on a mapping that is being created for the first time — for new mappings, simply omit the key.
Switching back to maximumConcurrency: set both keys for one deploy — false clears provisioned mode while the concurrency limit applies — then optionally remove the false line:
- sqs:
arn: arn:aws:sqs:region:XXXXXX:myQueue
maximumConcurrency: 250
provisionedPollers: false
IAM Permissions
The Serverless Framework will automatically configure the most minimal set of IAM permissions for you. However you can still add additional permissions if you need to. Read the official AWS documentation for more information about IAM Permissions for SQS events.
Deploying SQS queues
The examples above show how to consume messages from an existing SQS queue. To create an SQS queue in serverless.yml, you can either write custom CloudFormation, or you can use Lift.
Lift is a Serverless Framework plugin that simplifies deploying pieces of applications via "constructs". Lift can be installed via:
serverless plugin install -n serverless-lift
We can use the queue construct to deploy an SQS queue with its Lambda consumer:
constructs:
my-queue:
type: queue
worker:
handler: handler.compute
plugins:
- serverless-lift
The queue construct deploys:
- An SQS queue
- A
workerLambda function: this function processes messages sent to the queue. - An SQS "dead letter queue": this queue stores all the messages that failed to be processed.
Read the queue construct documentation to find a complete example with code, and to learn how to configure the batch size, retries and other options.