Philosophy

🧨 Diffusers provides state-of-the-art pretrained diffusion models across multiple modalities. Its purpose is to serve as a modular toolbox for both inference and training.

We aim at building a library that stands the test of time and therefore take API design very seriously.

In a nutshell, Diffusers is built to be a natural extension of PyTorch. Therefore, most of our design choices are based on PyTorch's Design Principles. Let's go over the most important ones:

Usability over Performance

While Diffusers has many built-in performance-enhancing features (see Memory and Speed), models are always loaded with the highest precision and lowest optimization. Therefore, by default diffusion pipelines are always instantiated on CPU with float32 precision if not otherwise defined by the user. This ensures usability across different platforms and accelerators and means that no complex installations are required to run the library.
Diffusers aims to be a light-weight package and therefore has very few required dependencies, but many soft dependencies that can improve performance (such as accelerate, safetensors, onnx, etc...). We strive to keep the library as lightweight as possible so that it can be added without much concern as a dependency on other packages.
Diffusers prefers simple, self-explainable code over condensed, magic code. This means that short-hand code syntaxes such as lambda functions, and advanced PyTorch operators are often not desired.

Simple over easy

As PyTorch states, explicit is better than implicit and simple is better than complex. This design philosophy is reflected in multiple parts of the library:

We follow PyTorch's API with methods like DiffusionPipeline.to to let the user handle device management.
Raising concise error messages is preferred to silently correct erroneous input. Diffusers aims at teaching the user, rather than making the library as easy to use as possible.
Complex model vs. scheduler logic is exposed instead of magically handled inside. Schedulers/Samplers are separated from diffusion models with minimal dependencies on each other. This forces the user to write the unrolled denoising loop. However, the separation allows for easier debugging and gives the user more control over adapting the denoising process or switching out diffusion models or schedulers.
Separately trained components of the diffusion pipeline, e.g. the text encoder, the unet, and the variational autoencoder, each have their own model class. This forces the user to handle the interaction between the different model components, and the serialization format separates the model components into different files. However, this allows for easier debugging and customization. DreamBooth or Textual Inversion training is very simple thanks to Diffusers' ability to separate single components of the diffusion pipeline.

Tweakable, contributor-friendly over abstraction

For large parts of the library, Diffusers adopts an important design principle of the Transformers library, which is to prefer copy-pasted code over hasty abstractions. This design principle is very opinionated and stands in stark contrast to popular design principles such as Don't repeat yourself (DRY). In short, just like Transformers does for modeling files, Diffusers prefers to keep an extremely low level of abstraction and very self-contained code for pipelines and schedulers. Functions, long code blocks, and even classes can be copied across multiple files which at first can look like a bad, sloppy design choice that makes the library unmaintainable. However, this design has proven to be extremely successful for Transformers and makes a lot of sense for community-driven, open-source machine learning libraries because:

Machine Learning is an extremely fast-moving field in which paradigms, model architectures, and algorithms are changing rapidly, which therefore makes it very difficult to define long-lasting code abstractions.
Machine Learning practitioners like to be able to quickly tweak existing code for ideation and research and therefore prefer self-contained code over one that contains many abstractions.
Open-source libraries rely on community contributions and therefore must build a library that is easy to contribute to. The more abstract the code, the more dependencies, the harder to read, and the harder to contribute to. Contributors simply stop contributing to very abstract libraries out of fear of breaking vital functionality. If contributing to a library cannot break other fundamental code, not only is it more inviting for potential new contributors, but it is also easier to review and contribute to multiple parts in parallel.

At Hugging Face, we call this design the single-file policy which means that almost all of the code of a certain class should be written in a single, self-contained file. To read more about the philosophy, you can have a look at this blog post.

In Diffusers, we follow this philosophy for pipelines, schedulers, and models alike. Some older models predate this convention and are kept as-is; all new model architectures live in their own self-contained files. See the Models section below for details.

Great, now you should have generally understood why 🧨 Diffusers is designed the way it is 🤗. We try to apply these design principles consistently across the library. Nevertheless, there are some minor exceptions to the philosophy or some unlucky design choices. If you have feedback regarding the design, we would ❤️ to hear it directly on GitHub.

Design Philosophy in Details

Now for the nitty-gritty details of the design philosophy. Diffusers gives you two ways to compose models and schedulers into a runnable workflow: standard pipelines, which are monolithic with one task per pipeline class, and Modular Diffusers, which is composable and block-based. The sections below cover pipelines, modular pipelines, models, and schedulers in turn.

Pipelines

Pipelines, standard or modular, are intended only for inference. They're designed to be easy to use (so they don't follow Simple over easy 100%): readable, self-explanatory, easy to tweak, and best seen as examples of how to use models and schedulers. They aren't feature complete. To build feature-complete user interfaces on top of Diffusers, use Modular Diffusers.

The following design principles are followed:

Pipelines follow the single-file policy. All pipelines can be found in individual directories under src/diffusers/pipelines. One pipeline folder corresponds to one diffusion paper/project/release. Multiple pipeline files can be gathered in one pipeline folder, as it’s done for src/diffusers/pipelines/stable-diffusion. If pipelines share similar functionality, one can make use of the # Copied from mechanism.
Pipelines all inherit from [DiffusionPipeline].
Every pipeline consists of different model and scheduler components, that are documented in the model_index.json file, are accessible under the same name as attributes of the pipeline and can be shared between pipelines with DiffusionPipeline.components function.
Every pipeline should be loadable via the DiffusionPipeline.from_pretrained function.
Every pipeline should have one and only one way to run it via a __call__ method. The naming of the __call__ arguments should be consistent across all pipelines.
Pipelines should be named after the task they are intended to solve.

Modular Diffusers

Modular Diffusers is the composable alternative to standard pipelines. You build a workflow from reusable pipeline blocks that you can mix, match, swap, and share. Standard pipelines are loose reference examples of how to use models and schedulers. Modular Diffusers is the recommended path for building feature-complete user interfaces on top of Diffusers, and for the community to build and share new pipelines in a decentralized way.

The following design principles are followed:

Modular pipelines follow the single-file policy. Each one lives in its own folder under src/diffusers/modular_pipelines/, and the folder splits the workflow across one file per stage: encoders.py, before_denoise.py, denoise.py, and decoders.py. Two more files complete the pipeline. modular_blocks_<model>.py assembles the stages, and modular_pipeline.py defines the per-model [ModularPipeline] subclass. Modular pipelines don't cross-import each other.
Each modular pipeline is defined as a set of [ModularPipelineBlocks]. Leaf blocks live in the stage files (encoders.py, before_denoise.py, denoise.py, decoders.py), and modular_blocks_<model>.py assembles them into the full workflow with container classes like [SequentialPipelineBlocks] and [AutoPipelineBlocks]. This splits apart two concepts that [DiffusionPipeline combines]. A block is a pure definition. It declares inputs, outputs, and component dependencies, but holds no weights and can't run. A [ModularPipeline], created with .init_pipeline(repo_id), is the runnable counterpart. Keeping blocks stateless and weight-free is what makes them freely composable, swappable, and shareable across workflows.
To support a new task, write the task-specific blocks, compose them with existing ones, and register the workflow in _workflow_map on the top-level block assembly. A single ModularPipeline can support many workflows, such as text-to-image, image-to-image, and inpainting, whereas a DiffusionPipeline runs only one.

See the Modular Diffusers documentation for the full design and usage guide.

Models

Models are designed as configurable toolboxes that are natural extensions of PyTorch's Module class. They should follow the single-file policy. Some older models predate this convention and are kept as-is. Treat them as legacy exceptions, not patterns to follow for new models. For example, the original [UNet2DConditionModel] class was used for several UNet variations.

The following design principles are followed:

Each model architecture type lives in its own folder under src/diffusers/models, for example transformers/, autoencoders/, or unets/. Each model family has its own file within that folder, such as transformer_flux.py and transformer_wan.py.
Models follow the single-file policy. Each model file should be self-contained, except for a small number of standard modules that every model uses identically, such as timestep embeddings and normalization layers. Import those from embeddings.py and normalization.py.
Models intend to expose complexity, just like PyTorch's Module class, and give clear error messages.
Models all inherit from ModelMixin and ConfigMixin.
Models can be optimized for performance when it doesn’t demand major code changes, keeps backward compatibility, and gives significant memory or compute gain.
Models should by default have the highest precision and lowest performance setting.
To integrate a new model architecture that's similar to an existing one, copy the existing file as a starting point and adapt it. Use # Copied from annotations on layers that remain identical so make fix-copies keeps them in sync.

Schedulers

Schedulers are responsible to guide the denoising process for inference as well as to define a noise schedule for training. They are designed as individual classes with loadable configuration files and strongly follow the single-file policy.

The following design principles are followed:

All schedulers are found in src/diffusers/schedulers.
Schedulers are not allowed to import from large utils files and shall be kept very self-contained.
One scheduler Python file corresponds to one scheduler algorithm (as might be defined in a paper).
If schedulers share similar functionalities, we can make use of the # Copied from mechanism.
Schedulers all inherit from SchedulerMixin and ConfigMixin.
Schedulers can be easily swapped out with the ConfigMixin.from_config method as explained in detail here.
Every scheduler has to have a set_num_inference_steps, and a step function. set_num_inference_steps(...) has to be called before every denoising process, i.e. before step(...) is called.
Every scheduler exposes the timesteps to be "looped over" via a timesteps attribute, which is an array of timesteps the model will be called upon.
The step(...) function takes a predicted model output and the "current" sample (x_t) and returns the "previous", slightly more denoised sample (x_t-1).
Given the complexity of diffusion schedulers, the step function does not expose all the complexity and can be a bit of a "black box".
In almost all cases, novel schedulers shall be implemented in a new scheduling file.