.. module:: optuna.samplers

optuna.samplers

The :mod:~optuna.samplers module defines a base class for parameter sampling as described extensively in :class:~optuna.samplers.BaseSampler. The remaining classes in this module represent child classes, deriving from :class:~optuna.samplers.BaseSampler, which implement different sampling strategies.

.. seealso:: :ref:pruning tutorial explains the overview of the sampler classes.

.. seealso:: :ref:user_defined_sampler tutorial could be helpful if you want to implement your own sampler classes.

.. seealso:: If you are unsure about which sampler to use, please consider using AutoSampler <https://hub.optuna.org/samplers/auto_sampler/>, which automatically selects a sampler during optimization. For more detail, see the article on AutoSampler <https://medium.com/optuna/autosampler-automatic-selection-of-optimization-algorithms-in-optuna-1443875fd8f9>.

+------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | | |AutoSampler|_ | :class:~optuna.samplers.RandomSampler | :class:~optuna.samplers.TPESampler | :class:~optuna.samplers.GPSampler | :class:~optuna.samplers.CmaEsSampler | :class:~optuna.samplers.NSGAIISampler | :class:~optuna.samplers.NSGAIIISampler | :class:~optuna.samplers.GridSampler | :class:~optuna.samplers.QMCSampler | :py:class:~optuna_integration.BoTorchSampler | :class:~optuna.samplers.BruteForceSampler | +====================================+=================================+=========================================+======================================+=====================================+========================================+===============================================================================+===============================================================================+=======================================+======================================+================================================+=================================================================================+ | Float parameters | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark (:math:\color{red}\times for infinite domain) | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Integer parameters | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\color{green}\checkmark | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Categorical parameters | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\blacktriangle | :math:\color{green}\checkmark | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Pruning | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\blacktriangle | :math:\color{red}\times (:math:\blacktriangle for single-objective) | :math:\color{red}\times (:math:\blacktriangle for single-objective) | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\color{green}\checkmark | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Multivariate optimization | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\blacktriangle | :math:\blacktriangle | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\blacktriangle | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Conditional search space | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\blacktriangle | :math:\blacktriangle | :math:\blacktriangle | :math:\blacktriangle | :math:\blacktriangle | :math:\blacktriangle | :math:\color{green}\checkmark | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Multi-objective optimization | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{red}\times | :math:\color{green}\checkmark (:math:\blacktriangle for single-objective) | :math:\color{green}\checkmark (:math:\blacktriangle for single-objective) | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Batch optimization | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Distributed optimization | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\blacktriangle | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{green}\checkmark | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Constrained optimization | :math:\color{green}\checkmark | :math:\color{red}\times | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{red}\times | :math:\color{green}\checkmark | :math:\color{green}\checkmark | :math:\color{red}\times | :math:\color{red}\times | :math:\color{green}\checkmark | :math:\color{red}\times | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Time complexity (per trial) (*) | N/A | :math:O(d) | :math:O(dn \log n) | :math:O(n^3) | :math:O(d^3) | :math:O(mp^2) (***) | :math:O(mp^2) (***) | :math:O(dn) | :math:O(dn) | :math:O(n^3) | :math:O(d) | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+ | Recommended budgets (#trials) (**) | as many as one likes | as many as one likes | 100–1000 | –500 | 1000–10000 | 100–10000 | 100–10000 | number of combinations | as many as one likes | 10–100 | number of combinations | +------------------------------------+---------------------------------+-----------------------------------------+--------------------------------------+-------------------------------------+----------------------------------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+---------------------------------------+--------------------------------------+------------------------------------------------+---------------------------------------------------------------------------------+

.. |AutoSampler| replace:: :class:AutoSampler .. _AutoSampler: https://hub.optuna.org/samplers/auto_sampler/

.. note:: :math:\color{green}\checkmark: Supports this feature. :math:\blacktriangle: Works, but inefficiently. :math:\color{red}\times: Causes an error, or has no interface.

(*): We assumes that :math:`d` is the dimension of the search space, :math:`n` is the number of finished trials, :math:`m` is the number of objectives, and :math:`p` is the population size (algorithm specific parameter).
This table shows the time complexity of the sampling algorithms. We may omit other terms that depend on the implementation in Optuna, including :math:`O(d)` to call the sampling methods and :math:`O(n)` to collect the completed trials.
This means that, for example, the actual time complexity of :class:`~optuna.samplers.RandomSampler` is :math:`O(d+n+d) = O(d+n)`.
From another perspective, with the exception of :class:`~optuna.samplers.NSGAIISampler` and :class:`~optuna.samplers.NSGAIIISampler`, all time complexity is written for single-objective optimization.

(**): (1) The budget depends on the number of parameters and the number of objectives. (2) This budget includes ``n_startup_trials`` if a sampler has ``n_startup_trials`` as one of its arguments.

(\*\*\*): This time complexity assumes that the number of population size :math:`p` and the number of parallelization are regular.
This means that the number of parallelization should not exceed the number of population size :math:`p`.

.. note:: Samplers initialize their random number generators by specifying seed argument at initialization. However, samplers reseed them when n_jobs!=1 of :func:optuna.study.Study.optimize to avoid sampling duplicated parameters by using the same generator. Thus we can hardly reproduce the optimization results with n_jobs!=1. For the same reason, make sure that use either seed=None or different seed values among processes with distributed optimization explained in :ref:distributed tutorial.

.. note:: For float, integer, or categorical parameters, see :ref:configurations tutorial.

For pruning, see :ref:`pruning` tutorial.

For multivariate optimization, see :class:`~optuna.samplers.BaseSampler`. The multivariate optimization is implemented as :func:`~optuna.samplers.BaseSampler.sample_relative` in Optuna. Please check the concrete documents of samplers for more details.

For conditional search space, see :ref:`configurations` tutorial and :class:`~optuna.samplers.TPESampler`. The ``group`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the conditional search space.

For multi-objective optimization, see :ref:`multi_objective` tutorial.

For batch optimization, see :ref:`Batch-Optimization` tutorial. Note that the ``constant_liar`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the batch optimization.

For distributed optimization, see :ref:`distributed` tutorial. Note that the ``constant_liar`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the distributed optimization.

For constrained optimization, see an `example <https://github.com/optuna/optuna-examples/blob/main/multi_objective/botorch_simple.py>`__.

.. autosummary:: :toctree: generated/ :nosignatures:

BaseSampler
RandomSampler
TPESampler
GPSampler
CmaEsSampler
NSGAIISampler
NSGAIIISampler
GridSampler
QMCSampler
BruteForceSampler
PartialFixedSampler

.. note:: The following :mod:optuna.samplers.nsgaii module defines crossover operations used by :class:~optuna.samplers.NSGAIISampler.

.. toctree:: :maxdepth: 1

nsgaii