This sample project illustrates parallelization techniques described in this blog.
Programming languages offer diverse techniques and terminology for parallel processing. Java employs multi-threading and thread pools. Node.js, though single-threaded, provides event loop and promises (for async programming), as well as worker threads (for actual multi-threading). Python supports both multi-threading (subject to Global Interpreter Lock) and multi-processing. Concurrent routines are another technique gaining attention in the industry.
This sample is provided for illustration purposes only and uses a language-agnostic term “worker” to denote a unit of parallel processing. Your specific parallelization implementation depends on your choice of runtime language and frameworks. We highly recommend you use battle-tested frameworks like Powertools for AWS Lambda that implement batch processing parallelization when possible. Regardless of the programming language, it is crucial to ensure all background threads/workers/promises/routines/tasks spawned by the function handler are completed within the same invocation before the handler returns.
- Create an AWS account if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
- AWS CLI installed and configured
- Git Installed
- AWS Cloud Development Kit (AWS CDK) installed
- Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
git clone https://github.com/aws-samples/lambda-with-multithreading - Change directory to the pattern directory:
cd lambda-with-multithreading - From the command line, use AWS CDK to deploy the AWS resources for the pattern:
cdk deploy - Wait for CDK to complete the deployment.
Resources created as part of this sample deployment can have associated costs.
Delete the stack
cdk destroyThis library is licensed under the MIT-0 License. See the LICENSE file.