dev/breeze/doc/01_installation.rst
.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
This document describes prerequisites for running Breeze and installation process.
The outline for this document in GitHub is available at top-right corner button (with 3-dots and 3 lines).
Docker Desktop <https://docs.docker.com/get-docker/>_
and make sure it is in your PATH. Breeze detects if you are using version that is too
old and warns you to upgrade.docker commands directly and not only via root user.
Your user should be in the docker group.
See Docker installation guide <https://docs.docker.com/install/>_ for details.Docker for Mac - Space <https://docs.docker.com/docker-for-mac/space>_ for details
on increasing disk space available for Docker on Mac.breeze cleanup command.
Also see pruning <https://docs.docker.com/config/pruning/>_ instructions from Docker.desktop-linux
docker context that uses docker socket created in user home directory. Older versions (and plain docker)
uses /var/run/docker.sock socket and default context. Breeze will attempt to detect if you have
desktop-linux context configured and will use it if it is available, but you can force the
context by adding --builder flag to the commands that build image or run the container and forward
the socket to inside the image.Here is an example configuration with more than 200GB disk space for Docker:
.. raw:: html
<div align="center">
</div>
docker/for-mac#6529 <https://github.com/docker/for-mac/issues/6529>_.Docker errors that may come while running breeze
If docker not running in python virtual environment:
sudo groupadd dockersudo usermod -aG docker $USERnewgrp dockerdocker run hello-world.. raw:: html
<div align="center">
</div>
Note: If you use Colima, please follow instructions in Colima section at:
Contributors Quick Start Guide </contributing-docs/03_contributors_quick_start.rst>__
Docker Compose <https://docs.docker.com/compose/install/>_
and add it to the PATH. Breeze detects if you are using version that is too old and warns you to upgrade.docker-compose command by your user.WSL 2 installation :
Install WSL 2 and a Linux Distro (e.g. Ubuntu) see
WSL 2 Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>_ for details.
Docker Desktop installation :
Install Docker Desktop for Windows. For Windows Home follow the
Docker Windows Home Installation Guide <https://docs.docker.com/docker-for-windows/install-windows-home>.
For Windows Pro, Enterprise, or Education follow the
Docker Windows Installation Guide <https://docs.docker.com/docker-for-windows/install/>.
Docker setting : WSL integration needs to be enabled
.. raw:: html
<div align="center">
</div>
WSL 2 Filesystem Performance :
Accessing the host Windows filesystem incurs a performance penalty,
it is therefore recommended to do development on the Linux filesystem.
E.g. Run cd ~ and create a development folder in your Linux distro home
and git pull the Airflow repo there.
WSL 2 Docker mount errors:
Another reason to use Linux filesystem, is that sometimes - depending on the length of
your path, you might get strange errors when you try start Breeze, such as
caused: mount through procfd: not a directory: unknown:. Therefore checking out
Airflow in Windows-mounted Filesystem is strongly discouraged.
WSL 2 Docker volume remount errors:
If you're experiencing errors such as ERROR: for docker-compose_airflow_run Cannot create container for service airflow: not a directory when starting Breeze
after the first time or an error like docker: Error response from daemon: not a directory. See 'docker run --help'. when running the prek tests, you may need to consider
installing Docker directly in WSL 2 <https://dev.to/bowmanjd/install-docker-on-windows-wsl-without-docker-desktop-34m9>_
instead of using Docker Desktop for Windows.
WSL 2 Memory Usage : WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim the memory after development you can:
sudo sysctl -w vm.drop_caches=3wsl --shutdownDeveloping in WSL 2:
You can use all the standard Linux command line utilities to develop on WSL 2.
Further VS Code supports developing in Windows but remotely executing in WSL.
If VS Code is installed on the Windows host system then in the WSL Linux Distro
you can run code . in the root directory of you Airflow repo to launch VS Code.
We are recommending to use the uv tool to manage your virtual environments and generally as a swiss-knife
of your Python environment (it supports installing various versions of Python, creating virtual environments,
installing packages, managing workspaces and running development tools.).
Installing uv is described in the uv documentation <https://docs.astral.sh/uv/getting-started/installation/>_.
We highly recommend using uv to manage your Python environments, as it is very comprehensive,
easy to use, it is faster than any of the other tools available (way faster!) and has a lot of features
that make it easier to work with Python.
gh cli needed for release managersThe gh GitHub CLI is a command line tool that allows you to interact with GitHub repositories, issues, pull
requests, and more. It is useful for release managers to automate tasks such as creating releases,
managing issues, and starting workflows (for example during documentation building). Release
managers should have gh installed (see gh installation guide <https://github.com/cli/cli>_) and they
should follow configuration steps to authorize gh in their local airflow repository (basically
running gh auth login command and following the instructions).
However, we do not want to be entirely dependent on uv as it is a software governed by a VC-backed vendor,
so we always want to provide open-source governed alternatives for our tools. If you can't or do not want to
use uv, we got you covered. Another tool you can use to manage development tools (and breeze development
environment) is Python-Software-Foundation managed pipx. The pipx tool is created by the creators
of pip from Python Packaging Authority <https://www.pypa.io/en/latest/>_
Note that pipx >= 1.4.1 should be used.
Install pipx
.. code-block:: bash
pip install --user "pipx>=1.4.1"
Breeze, is not globally accessible until your PATH is updated. Add <USER FOLDER>.local\bin as a variable environments. This can be done automatically by the following command (follow instructions printed).
.. code-block:: bash
pipx ensurepath
In case pipx is not in your PATH, you can run it with Python module:
.. code-block:: bash
python -m pipx ensurepath
Minimum 4GB RAM for Docker Engine is required to run the full Breeze environment.
On macOS, 2GB of RAM are available for your Docker containers by default, but more memory is recommended
(4GB should be comfortable). For details see
Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>_.
On Windows WSL 2 expect the Linux Distro and Docker containers to use 7 - 8 GB of RAM.
Minimum 40GB free disk space is required for your Docker Containers.
On Mac OS This might deteriorate over time so you might need to increase it or run breeze cleanup
periodically. For details see
Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>_.
On WSL2 you might want to increase your Virtual Hard Disk by following:
Expanding the size of your WSL 2 Virtual Hard Disk <https://docs.microsoft.com/en-us/windows/wsl/compare-versions#expanding-the-size-of-your-wsl-2-virtual-hard-disk>_
There is a command breeze ci resource-check that you can run to check available resources. See below
for details.
You may need to clean up your Docker environment occasionally. The images are quite big (1.5GB for both images needed for static code analysis and CI tests) and, if you often rebuild/update them, you may end up with some unused image data.
To clean up the Docker environment:
Stop Breeze with breeze down. (If Breeze is already running)
Run the breeze cleanup command.
Run docker images --all and docker ps --all to verify that your Docker is clean.
Both commands should return an empty list of images and containers respectively.
If you run into disk space errors, consider pruning your Docker images with the docker system prune --all
command. You may need to restart the Docker Engine before running this command.
In case of disk space errors on macOS, increase the disk space available for Docker. See
Prerequisites <#prerequisites>_ for details.
First, clone the Airflow repository, but make sure not to clone it into your home directory. Cloning it into your home directory will cause the following error:
Your Airflow sources are checked out in /Users/username/airflow, which is also your AIRFLOW_HOME where Airflow writes logs and database files. This setup is problematic because Airflow might overwrite or clean up your source code and .git repository.
.. code-block:: bash
git clone https://github.com/apache/airflow.git
Set your working directory to the root of this cloned repository.
.. code-block:: bash
cd airflow
Run this command to install Breeze (make sure to use -e flag) - you can choose uv (recommended) or
pipx:
.. code-block:: bash
uv tool install -e ./dev/breeze
.. code-block:: bash
pipx install -e ./dev/breeze
.. note:: Note for Windows users
The ``./dev/breeze`` in command about is a PATH to sub-folder where breeze source packages are.
If you are on Windows, you should use Windows way to point to the ``dev/breeze`` sub-folder
of Airflow either as absolute or relative path. For example:
.. code-block:: bash
uv tool install -e dev\breeze
or
.. code-block:: bash
pipx install -e dev\breeze
Once this is complete, you should have breeze binary on your PATH and available to run by breeze
command.
Those are all available commands for Breeze and details about the commands are described below:
.. image:: ./images/output-commands.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output-commands.svg :width: 100% :alt: Breeze commands
Breeze installed this way is linked to your checked out sources of Airflow, so Breeze will
automatically use latest version of sources from ./dev/breeze. Sometimes, when dependencies are
updated breeze commands with offer you to run self-upgrade.
You can always run such self-upgrade at any time:
.. code-block:: bash
breeze setup self-upgrade
If you have several checked out Airflow sources, Breeze will warn you if you are using it from a different source tree and will offer you to re-install from those sources - to make sure that you are using the right version.
You can skip Breeze's upgrade check by setting SKIP_BREEZE_UPGRADE_CHECK variable to non empty value.
By default Breeze works on the version of Airflow that you run it in - in case you are outside of the sources of Airflow and you installed Breeze from a directory - Breeze will be run on Airflow sources from where it was installed.
You can run breeze setup version command to see where breeze installed from and what are the current sources
that Breeze works on
.. warning:: Upgrading from earlier Python version
If you used Breeze with Python 3.8 and when running it, it will complain that it needs Python 3.10. In this
case you should force-reinstall Breeze with ``uv`` (or ``pipx``):
.. code-block:: bash
uv tool install --force -e ./dev/breeze
or
.. code-block:: bash
pipx install --force -e ./dev/breeze
.. note:: Note for Windows users
The ``./dev/breeze`` in command about is a PATH to sub-folder where breeze source packages are.
If you are on Windows, you should use Windows way to point to the ``dev/breeze`` sub-folder
of Airflow either as absolute or relative path. For example:
.. code-block:: bash
uv tool install --force -e dev\breeze
or
.. code-block:: bash
pipx install --force -e dev\breeze
.. note:: creating virtual env for ``apache-airflow-breeze`` with a specific python version
The ``uv tool install`` or ``pipx install`` use default system python version to create virtual
env for breeze. You can use a specific version by providing python version in ``uv`` or
python executable in ``pipx`` in ``--python``.
If you have breeze installed already with another Python version you can reinstall breeze with reinstall
command
.. code-block:: bash
uv tool install --python 3.10.16 ./dev/breeze --force
or
.. code-block:: bash
pipx install -e ./dev/breeze --python /Users/airflow/.pyenv/versions/3.10.16/bin/python --force
The First time you run Breeze, it pulls and builds a local version of Docker images.
It pulls the latest Airflow CI images from the
GitHub Container Registry <https://github.com/orgs/apache/packages?repo_name=airflow>_
and uses them to build your local Docker images. Note that the first run (per python) might take up to 10
minutes on a fast connection to start. Subsequent runs should be much faster.
Once you enter the environment, you are dropped into bash shell of the Airflow container and you can run tests immediately.
To use the full potential of breeze you should set up autocomplete. The breeze command comes
with a built-in bash/zsh/fish autocomplete setup command. After installing,
when you start typing the command, you can use <TAB> to show all the available switches and get
auto-completion on typical values of parameters that you can use.
You should set up the autocomplete option automatically by running:
.. code-block:: bash
breeze setup autocomplete
Breeze has tools that you can use to configure defaults and breeze behaviours and perform some maintenance operations that might be necessary when you add new commands in Breeze. It also allows to configure your host operating system for Breeze autocompletion.
These are all available flags of setup command:
.. image:: ./images/output_setup.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_setup.svg :width: 100% :alt: Breeze setup
Setting up autocompletion .........................
You get the auto-completion working when you re-enter the shell (follow the instructions printed). The command will warn you and not reinstall autocomplete if you already did, but you can also force reinstalling the autocomplete via:
.. code-block:: bash
breeze setup autocomplete --force
These are all available flags of setup autocomplete command:
.. image:: ./images/output_setup_autocomplete.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_setup_autocomplete.svg :width: 100% :alt: Breeze setup autocomplete
Breeze setup self-upgrade .........................
You can self-upgrade breeze automatically - which will reinstall all the latest dependencies of breeze. This
should generally happen automatically, but sometimes when your installation is broken you might need
to force the upgrade. These are all available flags of self-upgrade command:
.. image:: ./images/output_setup_self-upgrade.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_setup_self-upgrade.svg :width: 100% :alt: Breeze setup self-upgrade
Breeze setup version ....................
You can display Breeze version and with --verbose flag it can provide more information: where
Breeze is installed from and details about setup hashes.
These are all available flags of version command:
.. image:: ./images/output_setup_version.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_setup_version.svg :width: 100% :alt: Breeze version
Breeze setup config ....................
You can configure and inspect settings of Breeze command via this command: Python version, Backend used as well as backend versions.
Another part of configuration is enabling/disabling cheatsheet, asciiart. The cheatsheet and asciiart can be disabled - they are "nice looking" and cheatsheet contains useful information for first time users but eventually you might want to disable both if you find it repetitive and annoying.
With the config setting colour-blind-friendly communication for Breeze messages. By default we communicate
with the users about information/errors/warnings/successes via colour-coded messages, but we can switch
it off by passing --no-colour to config in which case the messages to the user printed by Breeze
will be printed using different schemes (italic/bold/underline) to indicate different kind of messages
rather than colours.
These are all available flags of setup config command:
.. image:: ./images/output_setup_config.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_setup_config.svg :width: 100% :alt: Breeze setup config
Breeze on POSIX-compliant systems (Linux, MacOS) can be automatically installed by running the
scripts/tools/setup_breeze bash script. This includes checking and installing uv, setting up
breeze with it and setting up autocomplete.
Since Breeze is installed with uv tool or pipx, you need to use the appropriate tool to uninstall it.
.. code-block:: bash
uv tool uninstall apache-airflow-breeze
This will also remove breeze from the folder: ${HOME}.local/bin/
.. code-block:: bash
pipx uninstall apache-airflow-breeze
Next step: Follow the Customizing <02_customizing.rst>_ guide to customize your environment.