.. _broker-sqs:

================== Using Amazon SQS

.. _broker-sqs-installation:

Installation

For the Amazon SQS support you have to install additional dependencies. You can install both Celery and these dependencies in one go using the celery[sqs] :ref:bundle <bundles>:

.. code-block:: console

$ pip install "celery[sqs]"

.. _broker-sqs-configuration:

Configuration

You have to specify SQS in the broker URL::

broker_url = 'sqs://ABCDEFGHIJKLMNOPQRST:ZYXK7NiynGlTogH8Nj+P9nlE73sq3@'

where the URL format is:

.. code-block:: text

sqs://aws_access_key_id:aws_secret_access_key@

Please note that you must remember to include the @ sign at the end and encode the password so it can always be parsed correctly. For example:

.. code-block:: python

from kombu.utils.url import safequote

aws_access_key = safequote("ABCDEFGHIJKLMNOPQRST")
aws_secret_key = safequote("ZYXK7NiynG/TogH8Nj+P9nlE73sq3")

broker_url = "sqs://{aws_access_key}:{aws_secret_key}@".format(
    aws_access_key=aws_access_key, aws_secret_key=aws_secret_key,
)

.. warning::

Don't use this setup option with django's ``debug=True``.
It may lead to security issues within deployed django apps.

In debug mode django shows environment variables and the SQS URL
may be exposed to the internet including your AWS access and secret keys.
Please turn off debug mode on your deployed django application or
consider a setup option described below.

The login credentials can also be set using the environment variables :envvar:AWS_ACCESS_KEY_ID and :envvar:AWS_SECRET_ACCESS_KEY, in that case the broker URL may only be sqs://.

If you are using IAM roles on instances, you can set the BROKER_URL to: sqs:// and kombu will attempt to retrieve access tokens from the instance metadata.

Options

Region

The default region is us-east-1 but you can select another region by configuring the :setting:broker_transport_options setting::

broker_transport_options = {'region': 'eu-west-1'}

.. seealso::

An overview of Amazon Web Services regions can be found here:

    http://aws.amazon.com/about-aws/globalinfrastructure/

.. _sqs-visibility-timeout:

Visibility Timeout

The visibility timeout defines the number of seconds to wait for the worker to acknowledge the task before the message is redelivered to another worker. Also see caveats below.

This option is set via the :setting:broker_transport_options setting::

broker_transport_options = {'visibility_timeout': 3600}  # 1 hour.

The default visibility timeout is 30 minutes.

This option is used when creating the SQS queue and has no effect if using :ref:predefined queues <predefined-queues>.

Polling Interval

The polling interval decides the number of seconds to sleep between unsuccessful polls. This value can be either an int or a float. By default the value is one second: this means the worker will sleep for one second when there's no more messages to read.

You must note that more frequent polling is also more expensive, so increasing the polling interval can save you money.

The polling interval can be set via the :setting:broker_transport_options setting::

broker_transport_options = {'polling_interval': 0.3}

Very frequent polling intervals can cause busy loops, resulting in the worker using a lot of CPU time. If you need sub-millisecond precision you should consider using another transport, like RabbitMQ <broker-amqp>, or Redis <broker-redis>.

Long Polling

SQS Long Polling_ is enabled by default and the WaitTimeSeconds parameter of ReceiveMessage_ operation is set to 10 seconds.

The value of WaitTimeSeconds parameter can be set via the :setting:broker_transport_options setting::

broker_transport_options = {'wait_time_seconds': 15}

Valid values are 0 to 20. Note that newly created queues themselves (also if created by Celery) will have the default value of 0 set for the "Receive Message Wait Time" queue property.

.. _SQS Long Polling: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html .. _ReceiveMessage: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

Queue Prefix

By default Celery won't assign any prefix to the queue names, If you have other services using SQS you can configure it do so using the :setting:broker_transport_options setting::

broker_transport_options = {'queue_name_prefix': 'celery-'}

.. _predefined-queues:

Predefined Queues

If you want Celery to use a set of predefined queues in AWS, and to never attempt to list SQS queues, nor attempt to create or delete them, pass a map of queue names to URLs using the :setting:predefined_queues setting::

broker_transport_options = {
    'predefined_queues': {
        'my-q': {
            'url': 'https://ap-southeast-2.queue.amazonaws.com/123456/my-q',
            'access_key_id': 'xxx',
            'secret_access_key': 'xxx',
        }
    }
}

.. warning::

**Important:** When using ``predefined_queues``, do NOT use URL-encoded 
credentials (``safequote``) for the ``access_key_id`` and ``secret_access_key`` 
values. URL encoding should only be applied to credentials in the broker URL.

Using URL-encoded credentials in ``predefined_queues`` will cause signature 
mismatch errors like: "The request signature we calculated does not match 
the signature you provided."

Correct example combining broker URL and predefined queues:

.. code-block:: python

import os
from kombu.utils.url import safequote
from celery import Celery

# Raw credentials from environment
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")

# URL-encode ONLY for broker URL
aws_access_key_encoded = safequote(AWS_ACCESS_KEY_ID)
aws_secret_key_encoded = safequote(AWS_SECRET_ACCESS_KEY)

# Use encoded credentials in broker URL
broker_url = f"sqs://{aws_access_key_encoded}:{aws_secret_key_encoded}@"

celery_app = Celery("tasks", broker=broker_url)
celery_app.conf.broker_transport_options = {
    "region": "us-east-1",
    "predefined_queues": {
        "my-queue": {
            "url": "https://sqs.us-east-1.amazonaws.com/123456/my-queue",
            # Use RAW credentials here (NOT encoded)
            "access_key_id": AWS_ACCESS_KEY_ID,
            "secret_access_key": AWS_SECRET_ACCESS_KEY,
        },
    },
}

When using this option, the visibility timeout should be set in the SQS queue (in AWS) rather than via the :ref:visibility timeout <sqs-visibility-timeout> option.

Back-off policy

Back-off policy is using SQS visibility timeout mechanism altering the time difference between task retries. The mechanism changes message specific visibility timeout from queue Default visibility timeout to policy configured timeout. The number of retries is managed by SQS (specifically by the ApproximateReceiveCount message attribute) and no further action is required by the user.

Configuring the queues and backoff policy::

broker_transport_options = {
    'predefined_queues': {
        'my-q': {
            'url': 'https://ap-southeast-2.queue.amazonaws.com/123456/my-q',
            'access_key_id': 'xxx',
            'secret_access_key': 'xxx',
            'backoff_policy': {1: 10, 2: 20, 3: 40, 4: 80, 5: 320, 6: 640},
            'backoff_tasks': ['svc.tasks.tasks.task1']
        }
    }
}

backoff_policy dictionary where key is number of retries, and value is delay seconds between retries (i.e SQS visibility timeout) backoff_tasks list of task names to apply the above policy

The above policy:

STS token authentication

https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role.html

AWS STS authentication is supported by using the sts_role_arn and sts_token_timeout broker transport options. sts_role_arn is the assumed IAM role ARN we use to authorize our access to SQS. sts_token_timeout is the token timeout, defaults (and minimum) to 900 seconds. After the mentioned period, a new token will be created::

broker_transport_options = {
    'predefined_queues': {
        'my-q': {
            'url': 'https://ap-southeast-2.queue.amazonaws.com/123456/my-q',
            'access_key_id': 'xxx',
            'secret_access_key': 'xxx',
            'backoff_policy': {1: 10, 2: 20, 3: 40, 4: 80, 5: 320, 6: 640},
            'backoff_tasks': ['svc.tasks.tasks.task1']
        }
    },
'sts_role_arn': 'arn:aws:iam::<xxx>:role/STSTest', # optional
'sts_token_timeout': 900 # optional
}

.. _sqs-caveats:

Caveats

If a task isn't acknowledged within the visibility_timeout, the task will be redelivered to another worker and executed.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

So you have to increase the visibility timeout to match the time of the longest ETA you're planning to use.

Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of 'lost' tasks in the event of a power failure or forcefully terminated workers.

Periodic tasks won't be affected by the visibility timeout, as it is a concept separate from ETA/countdown.

The maximum visibility timeout supported by AWS as of this writing is 12 hours (43200 seconds)::
```
broker_transport_options = {'visibility_timeout': 43200}
```
SQS doesn't yet support worker remote control commands.
SQS doesn't yet support events, and so cannot be used with :program:celery events, :program:celerymon, or the Django Admin monitor.
With FIFO queues it might be necessary to set additional message properties such as MessageGroupId and MessageDeduplicationId when publishing a message.

Message properties can be passed as keyword arguments to :meth:~celery.app.task.Task.apply_async:

.. code-block:: python

message_properties = { 'MessageGroupId': '<YourMessageGroupId>', 'MessageDeduplicationId': '<YourMessageDeduplicationId>' } task.apply_async(**message_properties)
During :ref:shutdown <worker-stopping>, the worker will attempt to re-queue any unacknowledged messages with :setting:task_acks_late enabled. However, if the worker is terminated forcefully (:ref:cold shutdown <worker-cold-shutdown>), the worker might not be able to re-queue the tasks on time, and they will not be consumed again until the :ref:sqs-visibility-timeout has passed. This creates a problem when the :ref:sqs-visibility-timeout is very high and a worker needs to shut down just after it has received a task. If the task is not re-queued in such case, it will need to wait for the long visibility timeout to pass before it can be consumed again, leading to potentially very long delays in tasks execution.

The :ref:soft shutdown <worker-soft-shutdown> introduces a time-limited warm shutdown phase just before the :ref:cold shutdown <worker-cold-shutdown>. This time window significantly increases the chances of re-queuing the tasks during shutdown which mitigates the problem of long visibility timeouts.

To enable the :ref:soft shutdown <worker-soft-shutdown>, set the :setting:worker_soft_shutdown_timeout to a value greater than 0. The value must be an float describing the number of seconds. During this time, the worker will continue to process the running tasks until the timeout expires, after which the :ref:cold shutdown <worker-cold-shutdown> will be initiated automatically to terminate the worker gracefully.

If the :ref:REMAP_SIGTERM <worker-REMAP_SIGTERM> is configured to SIGQUIT in the environment variables, and the :setting:worker_soft_shutdown_timeout is set, the worker will initiate the :ref:soft shutdown <worker-soft-shutdown> when it receives the :sig:TERM signal (and the :sig:QUIT signal).

.. _sqs-results-configuration:

Results

Multiple products in the Amazon Web Services family could be a good candidate to store or publish results with, but there's no such result backend included at this point.

.. warning::

Don't use the ``amqp`` result backend with SQS.

It will create one queue for every task, and the queues will
not be collected. This could cost you money that would be better
spent contributing an AWS result store backend back to Celery :)