docs/sf/providers/aws/guide/python.md
Built‑in support to bundle Python dependencies from requirements.txt (or Pipenv/Poetry/uv projects) and make them available in your PYTHONPATH at runtime. No external plugin required.
Thanks to the original serverless-python-requirements maintainers at Capital One and all community contributors for pioneering this functionality. The Framework now ships with these capabilities built in; you can remove the plugin and use the built-in feature described below—just be sure to keep a custom.pythonRequirements block (an empty object is enough) so the bundled integration is enabled.
This feature is included by default in the Framework. There is nothing to install.
Migrating from the community plugin? Remove it from the plugins section of serverless.yml and from package.json. The configuration surface remains the same under custom.pythonRequirements, which must stay present (even as {}) to activate the built-in plugin.
Compiling non-pure-Python modules or fetching their manylinux wheels is
supported on non-linux OSs via the use of Docker and official AWS build images.
To enable docker usage, add the following to your serverless.yml:
custom:
pythonRequirements:
dockerizePip: true
The dockerizePip option supports a special case in addition to booleans of 'non-linux' which makes
it dockerize only on non-linux environments.
To utilize your own Docker container instead of the default, add the following to your serverless.yml:
custom:
pythonRequirements:
dockerImage: <image name>:tag
This must be the full image name and tag to use, including the runtime specific tag if applicable.
Alternatively, you can define your Docker image in your own Dockerfile and add the following to your serverless.yml:
custom:
pythonRequirements:
dockerFile: ./path/to/Dockerfile
With Dockerfile the path to the Dockerfile that must be in the current folder (or a subfolder).
Please note the dockerImage and the dockerFile are mutually exclusive.
To install requirements from private git repositories, add the following to your serverless.yml:
custom:
pythonRequirements:
dockerizePip: true
dockerSsh: true
The dockerSsh option will mount your $HOME/.ssh/id_rsa and $HOME/.ssh/known_hosts as a
volume in the docker container.
In case you want to use a different key, you can specify the path (absolute) to it through dockerPrivateKey option:
custom:
pythonRequirements:
dockerizePip: true
dockerSsh: true
dockerPrivateKey: /home/.ssh/id_ed25519
If your SSH key is password protected, you can use ssh-agent
because $SSH_AUTH_SOCK is also mounted & the env var is set.
It is important that the host of your private repositories has already been added in your
$HOME/.ssh/known_hosts file, as the install process will fail otherwise due to host authenticity
failure.
You can also pass environment variables to docker by specifying them in dockerEnv
option:
custom:
pythonRequirements:
dockerEnv:
- https_proxy
Requires pipenv in version 2022-04-08 or higher.
If you include a Pipfile and have pipenv installed, this will use pipenv to generate requirements instead of a requirements.txt. It is fully compatible with all options such as zip and
dockerizePip. If you don't want the Framework to generate it for you, set the following option:
custom:
pythonRequirements:
usePipenv: false
If you include a pyproject.toml and have poetry installed instead of a requirements.txt this will use
poetry export --without-hashes -f requirements.txt -o requirements.txt --with-credentials to generate them. It is fully compatible with all options such as zip and
dockerizePip. If you don't want the Framework to generate it for you, set the following option:
custom:
pythonRequirements:
usePoetry: false
Be aware that if no poetry.lock file is present, a new one will be generated on the fly. To help having predictable builds,
you can set the requirePoetryLockFile flag to true to throw an error when poetry.lock is missing.
custom:
pythonRequirements:
requirePoetryLockFile: false
If your Poetry configuration includes custom dependency groups, they will not be installed automatically. To include them in the deployment package, use the poetryWithGroups, poetryWithoutGroups and poetryOnlyGroups options which wrap poetry export's --with, --without and --only parameters.
custom:
pythonRequirements:
poetryWithGroups:
- internal_dependencies
- lambda_dependencies
Poetry by default generates the exported requirements.txt file with -e and that breaks pip with -t parameter
(used to install all requirements in a specific folder). In order to fix that we remove all -e from the generated file but,
for that to work you need to add the git dependencies in a specific way.
Instead of:
[tool.poetry.dependencies]
bottle = {git = "[email protected]/bottlepy/bottle.git", tag = "0.12.16"}
Use:
[tool.poetry.dependencies]
bottle = {git = "https://[email protected]/bottlepy/bottle.git", tag = "0.12.16"}
Or, if you have an SSH key configured:
[tool.poetry.dependencies]
bottle = {git = "ssh://[email protected]/bottlepy/bottle.git", tag = "0.12.16"}
uv projects behave just like Pipenv or Poetry projects. When a uv.lock file is present and custom.pythonRequirements.useUv is not disabled, the Framework will run uv export --no-dev --frozen --no-hashes to generate an intermediate requirements.txt in .serverless/requirements.txt before packaging. Set useUv: false if you prefer to manage that file yourself.
If you also want uv to drive the installation step (instead of invoking python -m pip), opt-in via custom.pythonRequirements.installer: uv:
custom:
pythonRequirements:
useUv: true # default
installer: uv
The uv CLI must be available wherever the build runs (your workstation or the Docker image when dockerizePip is enabled). When dockerizePip is enabled, installer: uv is configured, and the default AWS build images are used (the public.ecr.aws/sam/build-python... images selected by the Framework), uv is installed automatically inside the container before running uv pip install. If you configure a custom Docker image via dockerImage or dockerFile, you are responsible for ensuring uv is available in that image.
To help deal with potentially large dependencies (for example: numpy, scipy
and scikit-learn) the Framework supports compressing the libraries. This does
require a minor change to your code to decompress them. To enable this add the
following to your serverless.yml:
custom:
pythonRequirements:
zip: true
and add this to your handler module before any code that imports your deps:
try:
import unzip_requirements
except ImportError:
pass
Works on non 'win32' environments: Docker, WSL are included
To remove the tests, information and caches from the installed packages,
enable the slim option. This will: strip the .so files, remove __pycache__
and dist-info directories as well as .pyc and .pyo files.
custom:
pythonRequirements:
slim: true
To specify additional directories to remove from the installed packages,
define a list of patterns in the serverless config using the slimPatterns
option and glob syntax. These patterns will be added to the default ones (**/*.py[c|o], **/__pycache__*, **/*.dist-info*).
Note, the glob syntax matches against whole paths, so to match a file in any
directory, start your pattern with **/.
custom:
pythonRequirements:
slim: true
slimPatterns:
- '**/*.egg-info*'
To overwrite the default patterns set the option slimPatternsAppendDefaults to false (true by default).
custom:
pythonRequirements:
slim: true
slimPatternsAppendDefaults: false
slimPatterns:
- '**/*.egg-info*'
This will remove all folders within the installed requirements that match
the names in slimPatterns
In some cases, stripping binaries leads to problems like "ELF load command address/offset not properly aligned", even when done in the Docker environment. You can still slim down the package without *.so files with:
custom:
pythonRequirements:
slim: true
strip: false
Another method for dealing with large dependencies is to put them into a
Lambda Layer.
Simply add the layer option to the configuration.
custom:
pythonRequirements:
layer: true
The requirements will be zipped up and a layer will be created automatically. Now just add the reference to the functions that will use the layer.
functions:
hello:
handler: handler.hello
layers:
- Ref: PythonRequirementsLambdaLayer
If the layer requires additional or custom configuration, add them onto the layer option.
custom:
pythonRequirements:
layer:
name: ${self:provider.stage}-layerName
description: Python requirements lambda layer
compatibleRuntimes:
- python3.7
licenseInfo: GPLv3
allowedAccounts:
- '*'
You can omit a package from deployment with the noDeploy option. Note that
dependencies of omitted packages must explicitly be omitted too.
This example makes it instead omit pytest:
custom:
pythonRequirements:
noDeploy:
- pytest
You can enable two kinds of caching which are currently both ENABLED by default.
First, a download cache that will cache downloads that pip needs to compile the packages.
And second, a what we call "static caching" which caches output of pip after compiling everything for your requirements file.
Since generally requirements.txt files rarely change, you will often see large amounts of speed improvements when enabling the static cache feature.
These caches will be shared between all your projects if no custom cacheLocation is specified (see below).
Please note: This has replaced the previously recommended usage of "--cache-dir" in the pipCmdExtraArgs
custom:
pythonRequirements:
useDownloadCache: true
useStaticCache: true
There are two additional options related to caching.
You can specify where in your system to store the cache with the cacheLocation option.
By default it will figure out automatically where based on your username and your OS to store the cache via the appdirectory module.
Additionally, you can specify how many max static caches to store with staticCacheMaxVersions, as a simple attempt to limit disk space usage for caching.
This is DISABLED (set to 0) by default.
Example:
custom:
pythonRequirements:
useStaticCache: true
useDownloadCache: true
cacheLocation: '/home/user/.my_cache_goes_here'
staticCacheMaxVersions: 10
You can specify extra arguments supported by pip to be passed to pip like this:
custom:
pythonRequirements:
pipCmdExtraArgs:
- --compile
You can specify extra arguments to be passed to docker build during the build step, and docker run during the dockerized pip install step:
custom:
pythonRequirements:
dockerizePip: true
dockerBuildCmdExtraArgs: ['--build-arg', 'MY_GREAT_ARG=123']
dockerRunCmdExtraArgs: ['-v', '${env:PWD}:/my-app']
Some pip workflows involve using requirements files not named
requirements.txt.
To support these, the Framework has the following option:
custom:
pythonRequirements:
fileName: requirements-prod.txt
Note: this feature does not work with Pipenv/Poetry, it requires requirements.txt
files for your Python modules.
If you have different python functions, with different sets of requirements, you can avoid including all the unecessary dependencies of your functions by using the following structure:
├── serverless.yml
├── function1
│ ├── requirements.txt
│ └── index.py
└── function2
├── requirements.txt
└── index.py
With the content of your serverless.yml containing:
package:
individually: true
functions:
func1:
handler: index.handler
module: function1
func2:
handler: index.handler
module: function2
The result is 2 zip archives, with only the requirements for function1 in the first one, and only the requirements for function2 in the second one.
Quick notes on the config file:
module field must be used to tell the Framework where to find the requirements.txt file for
each function.handler field must not be prefixed by the folder name (already known through module) as
the root of the zip artifact is already the path to your function.Sometimes your Python executable isn't available on your $PATH as python2.7
or python3.6 (for example, windows or using pyenv).
To support this, the Framework has the following option:
custom:
pythonRequirements:
pythonBin: /opt/python3.6/bin/python
For certain libraries, default packaging produces too large an installation,
even when zipping. In those cases it may be necessary to tailor make a version
of the module. In that case you can store them in a directory and use the
vendor option, and the Framework will copy them along with all the other
dependencies to install:
custom:
pythonRequirements:
vendor: ./vendored-libraries
functions:
hello:
handler: hello.handler
vendor: ./hello-vendor # The option is also available at the function level
The .requirements and requirements.zip (if using zip support) files are left
behind to speed things up on subsequent deploys. To clean them up, run:
sls requirements clean
You can also create them (and unzip_requirements if
using zip support) manually with:
sls requirements install
The pip download/static cache is outside the serverless folder, and should be manually cleaned when i.e. changing python versions:
sls requirements cleanCache
If you are using your own Python library, you have to cleanup
.requirements on any update. You can use the following option to cleanup
.requirements everytime you package.
custom:
pythonRequirements:
invalidateCaches: true
Brew wilfully breaks the --target option with no seeming intention to fix it
which causes issues since this uses that option. There are a few easy workarounds for this:
pythonBin option.OR
OR
dockerizePip option.Also, brew seems to cause issues with pipenv, so make sure you install pipenv using pip.
dockerizePip notesFor usage of dockerizePip on Windows do Step 1 only if running serverless on windows, or do both Step 1 & 2 if running serverless inside WSL.
Some Python packages require extra OS dependencies to build successfully. To deal with this, replace the default image with a Dockerfile like:
FROM public.ecr.aws/sam/build-python3.9
# Install your dependencies
RUN yum -y install mysql-devel
Then update your serverless.yml:
custom:
pythonRequirements:
dockerFile: Dockerfile
Some Python packages require extra OS libraries (*.so files) at runtime. You need to manually include these files in the root directory of your Serverless package. The simplest way to do this is to use the dockerExtraFiles option.
For instance, the mysqlclient package requires libmysqlclient.so.1020. If you use the Dockerfile from the previous section, add an item to the dockerExtraFiles option in your serverless.yml:
custom:
pythonRequirements:
dockerExtraFiles:
- /usr/lib64/mysql57/libmysqlclient.so.1020
Then verify the library gets included in your package:
sls package
zipinfo .serverless/xxx.zip
If you can't see the library, you might need to adjust your package include/exclude configuration in serverless.yml.
If you wish to exclude most of the files in your project, and only include the source files of your lambdas and their dependencies you may well use an approach like this:
package:
individually: false
include:
- './src/lambda_one/**'
- './src/lambda_two/**'
exclude:
- '**'
This will be very slow. Serverless adds a default "**" include. If you are using the cacheLocation parameter, this will result in all of the cached files' names being loaded and then subsequently discarded because of the exclude pattern. To avoid this happening you can add a negated include pattern, as is observed in https://github.com/serverless/serverless/pull/5825.
Use this approach instead:
package:
individually: false
include:
- '!./**'
- './src/lambda_one/**'
- './src/lambda_two/**'
exclude:
- '**'
If you prefer to manage dependencies yourself (or run another bundler), set custom.pythonRequirements.enabled: false in your serverless.yml. You can also remove the custom.pythonRequirements block entirely; the built-in integration activates only when the block is present and not explicitly disabled.
custom:
pythonRequirements:
enabled: false
When disabled, the Framework leaves your artifacts untouched, so make sure another process prepares any required Python dependencies before deployment.