Utilities

Configure logging

🤗 Datasets strives to be transparent and explicit about how it works, but this can be quite verbose at times. We have included a series of logging methods which allow you to easily adjust the level of verbosity of the entire library. Currently the default verbosity of the library is set to WARNING.

To change the level of verbosity, use one of the direct setters. For instance, here is how to change the verbosity to the INFO level:

import datasets
datasets.logging.set_verbosity_info()

You can also use the environment variable DATASETS_VERBOSITY to override the default verbosity, and set it to one of the following: debug, info, warning, error, critical:

DATASETS_VERBOSITY=error ./myprogram.py

All the methods of this logging module are documented below. The main ones are:

• [logging.get_verbosity] to get the current level of verbosity in the logger
• [logging.set_verbosity] to set the verbosity to the level of your choice

In order from the least to the most verbose (with their corresponding int values):

1. logging.CRITICAL or logging.FATAL (int value, 50): only report the most critical errors.
2. logging.ERROR (int value, 40): only report errors.
3. logging.WARNING or logging.WARN (int value, 30): only reports error and warnings. This the default level used by the library.
4. logging.INFO (int value, 20): reports error, warnings and basic information.
5. logging.DEBUG (int value, 10): report all information.

[[autodoc]] datasets.logging.get_verbosity

[[autodoc]] datasets.logging.set_verbosity

[[autodoc]] datasets.logging.set_verbosity_info

[[autodoc]] datasets.logging.set_verbosity_warning

[[autodoc]] datasets.logging.set_verbosity_debug

[[autodoc]] datasets.logging.set_verbosity_error

[[autodoc]] datasets.logging.disable_propagation

[[autodoc]] datasets.logging.enable_propagation

Configure progress bars

By default, tqdm progress bars will be displayed during dataset download and preprocessing. You can disable them globally by setting HF_DATASETS_DISABLE_PROGRESS_BARS environment variable. You can also enable/disable them using [~utils.enable_progress_bars] and [~utils.disable_progress_bars]. If set, the environment variable has priority on the helpers.

[[autodoc]] datasets.utils.enable_progress_bars

[[autodoc]] datasets.utils.disable_progress_bars

[[autodoc]] datasets.utils.are_progress_bars_disabled