docs/getting-started/backends-and-brokers/rabbitmq.rst
.. _broker-rabbitmq:
.. contents:: :local:
RabbitMQ is the default broker so it doesn't require any additional dependencies or initial configuration, other than the URL location of the broker instance you want to use:
.. code-block:: python
broker_url = 'amqp://myuser:mypassword@localhost:5672/myvhost'
For a description of broker URLs and a full list of the
various broker configuration options available to Celery,
see :ref:conf-broker-settings, and see below for setting up the
username, password and vhost.
.. _installing-rabbitmq:
See Downloading and Installing RabbitMQ_ over at RabbitMQ's website. For macOS
see Installing RabbitMQ on macOS_.
.. _Downloading and Installing RabbitMQ: https://www.rabbitmq.com/download.html
.. note::
If you're getting `nodedown` errors after installing and using
:command:`rabbitmqctl` then this blog post can help you identify
the source of the problem:
http://www.somic.org/2009/02/19/on-rabbitmqctl-and-badrpcnodedown/
.. _rabbitmq-configuration:
To use Celery we need to create a RabbitMQ user, a virtual host and allow that user access to that virtual host:
.. code-block:: console
$ sudo rabbitmqctl add_user myuser mypassword
.. code-block:: console
$ sudo rabbitmqctl add_vhost myvhost
.. code-block:: console
$ sudo rabbitmqctl set_user_tags myuser mytag
.. code-block:: console
$ sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"
Substitute in appropriate values for myuser, mypassword and myvhost above.
See the RabbitMQ Admin Guide_ for more information about access control_.
.. _Admin Guide: https://www.rabbitmq.com/admin-guide.html
.. _access control: https://www.rabbitmq.com/access-control.html
.. _rabbitmq-macOS-installation:
The easiest way to install RabbitMQ on macOS is using Homebrew_ the new and
shiny package management system for macOS.
First, install Homebrew using the one-line command provided by the Homebrew documentation_:
.. code-block:: console
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Finally, we can install RabbitMQ using :command:brew:
.. code-block:: console
$ brew install rabbitmq
.. _Homebrew: https://github.com/mxcl/homebrew/
.. _Homebrew documentation: https://github.com/Homebrew/homebrew/wiki/Installation
.. _rabbitmq-macOS-system-hostname:
After you've installed RabbitMQ with :command:brew you need to add the following to
your path to be able to start and stop the broker: add it to the start-up file for your
shell (e.g., :file:.bash_profile or :file:.profile).
.. code-block:: bash
PATH=$PATH:/usr/local/sbin
Configuring the system host name
If you're using a DHCP server that's giving you a random host name, you need
to permanently configure the host name. This is because RabbitMQ uses the host name
to communicate with nodes.
Use the :command:`scutil` command to permanently set your host name:
.. code-block:: console
$ sudo scutil --set HostName myhost.local
Then add that host name to :file:`/etc/hosts` so it's possible to resolve it
back into an IP address::
127.0.0.1 localhost myhost myhost.local
If you start the :command:`rabbitmq-server`, your rabbit node should now
be `rabbit@myhost`, as verified by :command:`rabbitmqctl`:
.. code-block:: console
$ sudo rabbitmqctl status
Status of node rabbit@myhost ...
[{running_applications,[{rabbit,"RabbitMQ","1.7.1"},
{mnesia,"MNESIA CXC 138 12","4.4.12"},
{os_mon,"CPO CXC 138 46","2.2.4"},
{sasl,"SASL CXC 138 11","2.1.8"},
{stdlib,"ERTS CXC 138 10","1.16.4"},
{kernel,"ERTS CXC 138 10","2.13.4"}]},
{nodes,[rabbit@myhost]},
{running_nodes,[rabbit@myhost]}]
...done.
This is especially important if your DHCP server gives you a host name
starting with an IP address, (e.g., `23.10.112.31.comcast.net`). In this
case RabbitMQ will try to use `rabbit@23`: an illegal host name.
.. _rabbitmq-macOS-start-stop:
Starting/Stopping the RabbitMQ server
To start the server:
.. code-block:: console
$ sudo rabbitmq-server
you can also run it in the background by adding the -detached option
(note: only one dash):
.. code-block:: console
$ sudo rabbitmq-server -detached
Never use :command:kill (:manpage:kill(1)) to stop the RabbitMQ server,
but rather use the :command:rabbitmqctl command:
.. code-block:: console
$ sudo rabbitmqctl stop
When the server is running, you can continue reading Setting up RabbitMQ_.
.. _using-quorum-queues:
.. versionadded:: 5.5
.. warning::
Quorum Queues require disabling global QoS which means some features won't work as expected.
See `limitations`_ for details.
Celery supports Quorum Queues_ by setting the x-queue-type header to quorum like so:
.. code-block:: python
from kombu import Queue
task_queues = [Queue('my-queue', queue_arguments={'x-queue-type': 'quorum'})]
broker_transport_options = {"confirm_publish": True}
If you'd like to change the type of the default queue, set the :setting:task_default_queue_type setting to quorum.
Another way to configure Quorum Queues_ is by relying on default settings and using task_routes:
.. code-block:: python
task_default_queue_type = "quorum"
task_default_exchange_type = "topic"
task_default_queue = "my-queue"
broker_transport_options = {"confirm_publish": True}
task_routes = {
"*": {
"routing_key": "my-queue",
},
}
Celery automatically detects if quorum queues are used using the :setting:worker_detect_quorum_queues setting.
We recommend to keep the default behavior turned on.
To migrate from classic mirrored queues to quorum queues, please refer to RabbitMQ's documentation <https://www.rabbitmq.com/blog/2023/03/02/quorum-queues-migration>_ on the subject.
.. _Quorum Queues: https://www.rabbitmq.com/docs/quorum-queues
.. _limitations:
Disabling global QoS means that the per-channel QoS is now static. This means that some Celery features won't work when using Quorum Queues.
Autoscaling relies on increasing and decreasing the prefetch count whenever a new process is instantiated or terminated so it won't work when Quorum Queues are detected.
Similarly, the :setting:worker_enable_prefetch_count_reduction setting will be a no-op even when set to True
when Quorum Queues are detected.
In addition, :ref:ETA/Countdown <calling-eta> will block the worker when received until the ETA arrives since
we can no longer increase the prefetch count and fetch another task from the queue.
In order to properly schedule ETA/Countdown tasks we automatically detect if quorum queues are used
and in case they are, Celery automatically enables :ref:Native Delayed Delivery <native-delayed-delivery>.
.. _native-delayed-delivery:
Since tasks with ETA/Countdown will block the worker until they are scheduled for execution, we need to use RabbitMQ's native capabilities to schedule the execution of tasks.
The design is borrowed from NServiceBus. If you are interested in the implementation details, refer to their documentation_.
.. _documentation: https://docs.particular.net/transports/rabbitmq/delayed-delivery
Native Delayed Delivery is automatically enabled when quorum queues are detected.
By default the Native Delayed Delivery queues are quorum queues.
If you'd like to change them to classic queues you can set the :setting:broker_native_delayed_delivery_queue_type
to classic.