**Multi-agent setup:** ``N`` agents live in the environment and take actions computed by ``M`` policy networks.
The mapping from agent to policy is flexible and determined by a user-provided mapping function. Here, `agent_1`
and `agent_3` both map to `policy_1`, whereas `agent_2` maps to `policy_2`.

This paragraph describes RLlib's own :py:class:`~ray.rllib.env.multi_agent_env.MultiAgentEnv` API, which is the
recommended way of defining your own multi-agent environment logic. However, if you are already using a
third-party multi-agent API, RLlib offers wrappers for :ref:`Farama's PettingZoo API <farama-pettingzoo-api>` as well
as :ref:`DeepMind's OpenSpiel API <deepmind-openspiel-api>`.

from ray.rllib.env.multi_agent_env import MultiAgentEnv

class MyMultiAgentEnv(MultiAgentEnv):

    def __init__(self, config=None):
        super().__init__()
        ...

    def reset(self, *, seed=None, options=None):
        ...
        # return observation dict and infos dict.
        return {"agent_1": [obs of agent_1], "agent_2": [obs of agent_2]}, {}

    def step(self, action_dict):
        # return observation dict, rewards dict, termination/truncation dicts, and infos dict
        return {"agent_1": [obs of agent_1]}, {...}, ...

The number of agents in your environment and their IDs are entirely controlled by your :py:class:`~ray.rllib.env.multi_agent_env.MultiAgentEnv`
code. Your env decides, which agents start after an episode reset, which agents enter the episode at a later point, which agents
terminate the episode early, and which agents stay in the episode until the entire episode ends.

To define, which agent IDs might even show up in your episodes, set the `self.possible_agents` attribute to a list of
all possible agent ID.

.. code-block::

    def __init__(self, config=None):
        super().__init__()
        ...
        # Define all agent IDs that might even show up in your episodes.
        self.possible_agents = ["agent_1", "agent_2"]
        ...

In case your environment only starts with a subset of agent IDs and/or terminates some agent IDs before the end of the episode,
you also need to permanently adjust the `self.agents` attribute throughout the course of your episode.
If - on the other hand - all agent IDs are static throughout your episodes, you can set `self.agents` to be the same
as `self.possible_agents` and don't change its value throughout the rest of your code:

.. code-block::

    def __init__(self, config=None):
        super().__init__()
        ...
        # If your agents never change throughout the episode, set
        # `self.agents` to the same list as `self.possible_agents`.
        self.agents = self.possible_agents = ["agent_1", "agent_2"]
        # Otherwise, you will have to adjust `self.agents` in `reset()` and `step()` to whatever the
        # currently "alive" agents are.
        ...

Observation- and Action Spaces

import gymnasium as gym
import numpy as np

...

    def __init__(self, config=None):
        super().__init__()
        ...
        self.observation_spaces = {
            "agent_1": gym.spaces.Box(-1.0, 1.0, (4,), np.float32),
            "agent_2": gym.spaces.Box(-1.0, 1.0, (3,), np.float32),
        }
        self.action_spaces = {
            "agent_1": gym.spaces.Discrete(2),
            "agent_2": gym.spaces.Box(0.0, 1.0, (1,), np.float32),
        }
        ...

def get_observation_space(self, agent_id):
    if agent_id.startswith("robot_"):
        return gym.spaces.Box(0, 255, (84, 84, 3), np.uint8)
    elif agent_id.startswith("decision_maker"):
        return gym.spaces.Discrete(2)
    else:
        raise ValueError(f"bad agent id: {agent_id}!")

The remaining two things you need to implement in your custom :py:class:`~ray.rllib.env.multi_agent_env.MultiAgentEnv`
are the `reset()` and `step()` methods. Equivalently to a single-agent `gymnasium.Env <https://gymnasium.farama.org/_modules/gymnasium/core/#Env>`__,
you have to return observations and infos from `reset()`, and return observations, rewards, termination/truncation flags, and infos
from `step()`, however, instead of individual values, these all have to be dictionaries mapping agent IDs to the respective
individual agents' values.

Let's take a look at an example `reset()` implementation first:

.. code-block::

    def reset(self, *, seed=None, options=None):
        ...
        return {
            "agent_1": np.array([0.0, 1.0, 0.0, 0.0]),
            "agent_2": np.array([0.0, 0.0, 1.0]),
        }, {}  # <- empty info dict

Here, your episode starts with both agents in it, and both expected to compute and send actions
for the following `step()` call.

In general, the returned observations dict must contain those agents (and only those agents)
that should act next. Agent IDs that should NOT act in the next `step()` call must NOT have
their observations in the returned observations dict.

.. figure:: images/envs/multi_agent_episode_simultaneous.svg
    :width: 600
    :align: left

    **Env with simultaneously acting agents:** Both agents receive their observations at each
    time step, including right after `reset()`. Note that an agent must compute and send an action
    into the next `step()` call whenever an observation is present for that agent in the returned
    observations dict.

Note that the rule of observation dicts determining the exact order of agent moves doesn't equally apply to
either reward dicts nor termination/truncation dicts, all of which
may contain any agent ID at any time step regardless of whether that agent ID is expected to act or not
in the next `step()` call. This is so that an action taken by agent A may trigger a reward for agent B, even
though agent B currently isn't acting itself. The same is true for termination flags: Agent A may act in a way
that terminates agent B from the episode without agent B having acted itself.

.. note::
    Use the special agent ID `__all__` in the termination dicts and/or truncation dicts to indicate
    that the episode should end for all agent IDs, regardless of which agents are still active at that point.
    RLlib automatically terminates all agents in this case and ends the episode.

In summary, the exact order and synchronization of agent actions in your multi-agent episode is determined
through the agent IDs contained in (or missing from) your observations dicts.
Only those agent IDs that are expected to compute and send actions into the next `step()` call must be part of the
returned observation dict.

.. figure:: images/envs/multi_agent_episode_turn_based.svg
    :width: 600
    :align: left

    **Env with agents taking turns:** The two agents act by taking alternating turns. `agent_1` receives the
    first observation after the `reset()` and thus has to compute and send an action first. Upon receiving
    this action, the env responds with an observation for `agent_2`, who now has to act. After receiving the action
    for `agent_2`, a next observation for `agent_1` is returned and so on and so forth.

This simple rule allows you to design any type of multi-agent environment, from turn-based games to
environments where all agents always act simultaneously, to any arbitrarily complex combination of these two patterns:


.. figure:: images/envs/multi_agent_episode_complex_order.svg
    :width: 600
    :align: left

    **Env with a complex order of turns:** Three agents act in a seemingly chaotic order. `agent_1` and `agent_3` receive their
    initial observation after the `reset()` and thus has to compute and send actions first. Upon receiving
    these two actions, the env responds with an observation for `agent_1` and `agent_2`, who now have to act simultaneously.
    After receiving the actions for `agent_1` and `agent_2`, observations for `agent_2` and `agent_3` are returned and so on and
    so forth.


Let's take a look at two specific, complete :py:class:`~ray.rllib.env.multi_agent_env.MultiAgentEnv` example implementations,
one where agents always act simultaneously and one where agents act in a turn-based sequence.


Example: Environment with Simultaneously Stepping Agents

Let's now walk through another multi-agent env example implementation, but this time you
implement a turn-based game, in which you have two players (A and B), where A starts the game,
then B makes a move, then again A, and so on and so forth.

We implement the famous Tic-Tac-Toe game (with one slight aberration), played on a 3x3 field.
Each player adds one of their pieces to the field at a time. Pieces can't be moved once placed.
The player that first completes one row (horizontal, diagonal, or vertical) wins the game and
receives +1 reward. The losing player receives a -1 reward.
To make the implementation easier, the aberration from the original game is that trying to
place a piece on an already occupied field results in the board not changing at all, but the
moving player receiving a -5 reward as a penalty (in the original game, this move is
simply not allowed and therefore can never happen).

Here is your initial class scaffold for the Tic-Tac-Toe game:

.. literalinclude:: ../../../rllib/examples/envs/classes/multi_agent/tic_tac_toe.py
   :language: python
   :start-after: __sphinx_doc_1_begin__
   :end-before: __sphinx_doc_1_end__

In your constructor, make sure you define all possible agent IDs that can ever show up
in your game ("player1" and "player2"), the currently active agent IDs (same as all
possible agents), and each agent's observation- and action space.

.. literalinclude:: ../../../rllib/examples/envs/classes/multi_agent/tic_tac_toe.py
   :language: python
   :start-after: __sphinx_doc_2_begin__
   :end-before: __sphinx_doc_2_end__

Now let's implement your `reset()` method, in which you empty the board (set it to all 0s),
pick a random start player, and return this start player's first observation.
Note that you don't return the other player's observation as this player isn't
acting next.

.. literalinclude:: ../../../rllib/examples/envs/classes/multi_agent/tic_tac_toe.py
   :language: python
   :start-after: __sphinx_doc_3_begin__
   :end-before: __sphinx_doc_3_end__


From here on, in each `step()`, you always flip between the two agents (you use the
`self.current_player` attribute for keeping track) and return only the current agent's
observation, because that's the player you want to act next.

You also compute the both agents' rewards based on three criteria:
Did the current player win (the opponent lost)?
Did the current player place a piece on an already occupied field (gets penalized)?
Is the game done because the board is full (both agents receive 0 reward)?

.. literalinclude:: ../../../rllib/examples/envs/classes/multi_agent/tic_tac_toe.py
   :language: python
   :start-after: __sphinx_doc_4_begin__
   :end-before: __sphinx_doc_4_end__


Grouping Agents
~~~~~~~~~~~~~~~

It is common to have groups of agents in multi-agent RL, where each group is treated
like a single agent with Tuple action- and observation spaces (one item in the tuple
for each individual agent in the group).

Such a group of agents can then be assigned to a single policy for centralized execution,
or to specialized multi-agent policies that implement centralized training, but
decentralized execution.

You can use the :py:meth:`~ray.rllib.env.multi_agent_env.MultiAgentEnv.with_agent_groups`
method to define these groups:

.. literalinclude:: ../../../rllib/env/multi_agent_env.py
   :language: python
   :start-after: __grouping_doc_begin__
   :end-before: __grouping_doc_end__

For environments with multiple groups, or mixtures of agent groups and individual agents,
you can use grouping in conjunction with the policy mapping API described in prior sections.


Third Party Multi-Agent Env APIs
--------------------------------

Besides RLlib's own :py:class:`~ray.rllib.env.multi_agent_env.MultiAgentEnv` API, you can also use
various third-party APIs and libraries to implement custom multi-agent envs.


.. _farama-pettingzoo-api:

Farama PettingZoo
~~~~~~~~~~~~~~~~~

`PettingZoo <https://pettingzoo.farama.org>`__ offers a repository of over 50 diverse
multi-agent environments, directly compatible with RLlib through the built-in
:py:class:`~ray.rllib.env.wrappers.pettingzoo_env.PettingZooEnv` wrapper:


.. testcode::

    from pettingzoo.butterfly import pistonball_v6

    from ray.rllib.algorithms.ppo import PPOConfig
    from ray.rllib.env.wrappers.pettingzoo_env import PettingZooEnv
    from ray.tune.registry import register_env

    register_env(
        "pistonball",
        lambda cfg: PettingZooEnv(pistonball_v6.env(num_floors=cfg.get("n_pistons", 20))),
    )

    config = (
        PPOConfig()
        .environment("pistonball", env_config={"n_pistons": 30})
    )

See `this example script here <https://github.com/ray-project/ray/blob/master/rllib/examples/multi_agent/pettingzoo_parameter_sharing.py>`__
for an end-to-env example with the `water world env <https://pettingzoo.farama.org/environments/sisl/waterworld/>`__

Also, `see here for an example on the pistonball env with RLlib <https://github.com/Farama-Foundation/PettingZoo/blob/master/tutorials/Ray/rllib_pistonball.py>`__.


.. _deepmind-openspiel-api:

DeepMind OpenSpiel
~~~~~~~~~~~~~~~~~~

The `OpenSpiel API by DeepMind <https://github.com/google-deepmind/open_spiel>`__ is a comprehensive framework
designed for research and development in multi-agent reinforcement learning, game theory, and decision-making.
The API is directly compatible with RLlib through the built-in
:py:class:`~ray.rllib.env.wrappers.pettingzoo_env.PettingZooEnv` wrapper:


.. testcode::

    import pyspiel  # pip install open_spiel

    from ray.rllib.algorithms.ppo import PPOConfig
    from ray.rllib.env.wrappers.open_spiel import OpenSpielEnv
    from ray.tune.registry import register_env

    register_env(
        "open_spiel_env",
        lambda cfg: OpenSpielEnv(pyspiel.load_game("connect_four")),
    )

    config = PPOConfig().environment("open_spiel_env")


See here for an `end-to-end example with the Connect-4 env <https://github.com/ray-project/ray/blob/master/rllib/examples/multi_agent/self_play_with_open_spiel.py>`__
of OpenSpiel trained by an RLlib algorithm, using a self-play strategy.


Running actual Training Experiments with a MultiAgentEnv
--------------------------------------------------------

If all agents use the same algorithm class to train their policies, configure
multi-agent training as follows:

.. code-block:: python

    from ray.rllib.algorithm.ppo import PPOConfig
    from ray.rllib.core.rl_module.multi_rl_module import MultiRLModuleSpec
    from ray.rllib.core.rl_module.rl_module import RLModuleSpec

    config = (
        PPOConfig()
        .environment(env="my_multiagent_env")
        .multi_agent(
            policy_mapping_fn=lambda agent_id, episode, **kwargs: (
                "traffic_light" if agent_id.startswith("traffic_light_")
                else random.choice(["car1", "car2"])
            ),
            algorithm_config_overrides_per_module={
                "car1": PPOConfig.overrides(gamma=0.85),
                "car2": PPOConfig.overrides(lr=0.00001),
            },
        )
        .rl_module(
            rl_module_spec=MultiRLModuleSpec(rl_module_specs={
                "car1": RLModuleSpec(),
                "car2": RLModuleSpec(),
                "traffic_light": RLModuleSpec(),
            }),
        )
    )

    algo = config.build()
    print(algo.train())

To exclude certain policies from being updated, use the ``config.multi_agent(policies_to_train=[..])`` config setting.
This allows running in multi-agent environments with a mix of non-learning and learning policies:

.. code-block:: python

    def policy_mapping_fn(agent_id, episode, **kwargs):
        agent_idx = int(agent_id[-1])  # 0 (player1) or 1 (player2)
        return "learning_policy" if episode.id_ % 2 == agent_idx else "random_policy"

    config = (
        PPOConfig()
        .environment(env="two_player_game")
        .multi_agent(
            policy_mapping_fn=policy_mapping_fn,
            policies_to_train=["learning_policy"],
        )
        .rl_module(
            rl_module_spec=MultiRLModuleSpec(rl_module_specs={
                "learning_policy": RLModuleSpec(),
                "random_policy": RLModuleSpec(rl_module_class=RandomRLModule),
            }),
        )
    )

    algo = config.build()
    print(algo.train())

RLlib will create and route decisions to each policy based on the provided
``policy_mapping_fn``. Training statistics for each policy are reported
separately in the result-dict returned by ``train()``.

The example scripts `rock_paper_scissors_heuristic_vs_learned.py <https://github.com/ray-project/ray/blob/master/rllib/examples/multi_agent/rock_paper_scissors_heuristic_vs_learned.py>`__
and `rock_paper_scissors_learned_vs_learned.py <https://github.com/ray-project/ray/blob/master/rllib/examples/multi_agent/rock_paper_scissors_learned_vs_learned.py>`__
demonstrate competing policies with heuristic and learned strategies.


Scaling to Many MultiAgentEnvs per EnvRunner

Unlike for single-agent environments, multi-agent setups are not vectorizable yet.
The Ray team is working on a solution for this restriction by utilizing
`gymnasium >= 1.x` custom vectorization feature.

RLlib supports variable-sharing across policies.

See the `PettingZoo parameter sharing example <https://github.com/ray-project/ray/blob/master/rllib/examples/multi_agent/pettingzoo_parameter_sharing.py>`__ for details.