Back to Numpy

Building from source

doc/source/building/index.rst

2.5.0.dev016.8 KB
Original Source

.. _building-from-source:

Building from source

.. This page is referenced from numpy/numpy/init.py. Please keep its location in sync with the link there.

.. note::

If you are only trying to install NumPy, we recommend using binaries - see Installation <https://numpy.org/install>__ for details on that.

Building NumPy from source requires setting up system-level dependencies (compilers, BLAS/LAPACK libraries, etc.) first, and then invoking a build. The build may be done in order to install NumPy for local usage, develop NumPy itself, or build redistributable binary packages. And it may be desired to customize aspects of how the build is done. This guide will cover all these aspects. In addition, it provides background information on how the NumPy build works, and links to up-to-date guides for generic Python build & packaging documentation that is relevant.

.. _system-level:

System-level dependencies

NumPy uses compiled code for speed, which means you need compilers and some other system-level (i.e, non-Python / non-PyPI) dependencies to build it on your system.

.. note::

If you are using Conda, you can skip the steps in this section - with the
exception of installing compilers for Windows or the Apple Developer Tools
for macOS. All other dependencies will be installed automatically by the
``mamba env create -f environment.yml`` command.

.. tab-set::

.. tab-item:: Linux :sync: linux

If you want to use the system Python and ``pip``, you will need:

* C and C++ compilers (typically GCC).

* Python header files (typically a package named ``python3-dev`` or
  ``python3-devel``)

* BLAS and LAPACK libraries. `OpenBLAS <https://github.com/OpenMathLib/OpenBLAS/>`__
  is the NumPy default; other variants include Apple Accelerate,
  `MKL <https://software.intel.com/en-us/intel-mkl>`__,
  `ATLAS <https://math-atlas.sourceforge.net/>`__ and
  `Netlib <https://www.netlib.org/lapack/index.html>`__ (or "Reference")
  BLAS and LAPACK.

* ``pkg-config`` for dependency detection.

* A Fortran compiler is needed only for running the ``f2py`` tests. The
  instructions below include a Fortran compiler, however you can safely
  leave it out.

.. tab-set::

  .. tab-item:: Debian/Ubuntu Linux

    To install NumPy build requirements, you can do::

      sudo apt install -y gcc g++ gfortran libopenblas-dev liblapack-dev pkg-config python3-pip python3-dev

    Alternatively, you can do::

      sudo apt build-dep numpy

    This command installs whatever is needed to build NumPy, with the
    advantage that new dependencies or updates to required versions are
    handled by the package managers.

  .. tab-item:: Fedora

    To install NumPy build requirements, you can do::

      sudo dnf install gcc-gfortran python3-devel openblas-devel lapack-devel pkgconfig

    Alternatively, you can do::

      sudo dnf builddep numpy

    This command installs whatever is needed to build NumPy, with the
    advantage that new dependencies or updates to required versions are
    handled by the package managers.

  .. tab-item:: CentOS/RHEL

    To install NumPy build requirements, you can do::

      sudo yum install gcc-gfortran python3-devel openblas-devel lapack-devel pkgconfig

    Alternatively, you can do::

      sudo yum-builddep numpy

    This command installs whatever is needed to build NumPy, with the
    advantage that new dependencies or updates to required versions are
    handled by the package managers.

  .. tab-item:: Arch

    To install NumPy build requirements, you can do::

      sudo pacman -S gcc-fortran openblas pkgconf

.. tab-item:: macOS :sync: macos

Install Apple Developer Tools. An easy way to do this is to
`open a terminal window <https://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line>`_,
enter the command::

    xcode-select --install

and follow the prompts. Apple Developer Tools includes Git, the Clang C/C++
compilers, and other development utilities that may be required.

Do *not* use the macOS system Python. Instead, install Python
with `the python.org installer <https://www.python.org/downloads/>`__ or
with a package manager like Homebrew, MacPorts or Fink.

On macOS >=13.3, the easiest build option is to use Accelerate, which is
already installed and will be automatically used by default.

On older macOS versions you need a different BLAS library, most likely
OpenBLAS, plus pkg-config to detect OpenBLAS. These are easiest to install
with `Homebrew <https://brew.sh/>`__::

    brew install openblas pkg-config gfortran

.. tab-item:: Windows :sync: windows

Compared to macOS and Linux, building NumPy on Windows is a little more
difficult, due to the need to set up compilers. First, install Microsoft
Visual Studio - the 2019 Community Edition or any newer version will work
(see the
`Visual Studio download site <https://visualstudio.microsoft.com/downloads/>`__).
This is needed to ensure you have the Windows Universal C Runtime. The
recommended version of the UCRT is >= 10.0.22621.0.

The MSVC installer does not put the compilers on the system path, and
the install location may change. To query the install location, MSVC
comes with a ``vswhere.exe`` command-line utility. And to make the
C/C++ compilers available inside the shell you are using, you need to
run a ``.bat`` file for the correct bitness and architecture (e.g., for
64-bit Intel CPUs, use ``vcvars64.bat``).

For detailed guidance, see `Use the Microsoft C++ toolset from the command line
<https://learn.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-170>`__.

If you don't need ``f2py``, MSVC alone is sufficient. For Fortran compiler
setup (needed for ``f2py``), see :ref:`F2PY and Windows <f2py-windows>`.

.. note::

    Compilers should be on the system path (i.e., the ``PATH`` environment
    variable should contain the directory in which the compiler executables
    can be found) in order to be found, with the exception of MSVC which
    will be found automatically if and only if there are no other compilers
    on the ``PATH``. You can use any shell (e.g., Powershell, ``cmd`` or
    Git Bash) to invoke a build.

.. warning::

    When using a conda environment it is possible that the environment
    creation will not work due to an outdated Fortran compiler. If that
    happens, remove the ``compilers`` entry from ``environment.yml`` and
    try again. The Fortran compiler should be installed as described in
    the :ref:`F2PY Windows documentation <f2py-windows>`.

.. tab-item:: Windows on ARM64 :sync: Windows on ARM64

In Windows on ARM64, the compiler options available for building NumPy are
limited. GCC and gfortran are not yet supported. Currently, the NumPy build
for Windows on ARM64 is supported with MSVC and LLVM toolchains.

First, install Microsoft Visual Studio - the 2022 Community Edition will
work (see the `Visual Studio download site <https://visualstudio.microsoft.com/downloads/>`__).
Ensure that you have installed necessary Visual Studio components for building NumPy
on WoA from `here <https://gist.github.com/Mugundanmcw/c3bb93018d5da9311fb2b222f205ba19>`__.

The MSVC installer does not put the compilers on the system path, and
the install location may change. To query the install location, MSVC
comes with a ``vswhere.exe`` command-line utility. And to make the
C/C++ compilers available inside the shell you are using, you need to
run a ``.bat`` file for the correct bitness and architecture (e.g., for
ARM64-based CPUs, use ``vcvarsarm64.bat``).

For detailed guidance, see `Use the Microsoft C++ toolset from the command line
<https://learn.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-170>`__.

If you don't need ``f2py``, MSVC alone is sufficient. For Fortran compiler
setup (needed for ``f2py``), see :ref:`F2PY and Windows <f2py-windows>`.

.. note::

    Compilers should be on the system path (i.e., the ``PATH`` environment
    variable should contain the directory in which the compiler executables
    can be found) in order to be found, with the exception of MSVC which
    will be found automatically if and only if there are no other compilers
    on the ``PATH``. You can use any shell (e.g., Powershell, ``cmd`` or
    Git Bash) to invoke a build.

.. warning::

    Currently, Conda environment is not yet supported officially on `Windows
    on ARM64 <https://github.com/conda-forge/conda-forge.github.io/issues/1940>`__.
    The present approach uses virtualenv for building NumPy from source on
    Windows on ARM64.

Building NumPy from source

If you want to only install NumPy from source once and not do any development work, then the recommended way to build and install is to use pip. Otherwise, conda is recommended.

.. note::

If you don't have a conda installation yet, we recommend using
Miniforge_; any conda flavor will work though.

Building from source to use NumPy


.. tab-set::

  .. tab-item:: Conda env
    :sync: conda

    If you are using a conda environment, ``pip`` is still the tool you use to
    invoke a from-source build of NumPy. It is important to always use the
    ``--no-build-isolation`` flag to the ``pip install`` command, to avoid
    building against a ``numpy`` wheel from PyPI. In order for that to work you
    must first install the remaining build dependencies into the conda
    environment::

      # Either install all NumPy dev dependencies into a fresh conda environment
      mamba env create -f environment.yml

      # Or, install only the required build dependencies
      mamba install python numpy cython compilers openblas meson-python pkg-config

      # To build the latest stable release:
      pip install numpy --no-build-isolation --no-binary numpy

      # To build a development version, you need a local clone of the NumPy git repository:
      git clone https://github.com/numpy/numpy.git
      cd numpy
      git submodule update --init
      pip install . --no-build-isolation

    .. warning::

        On Windows, the AR, LD, and LDFLAGS environment variables may be set,
        which will cause the pip install command to fail. These variables are only
        needed for flang and can be safely unset prior to running pip install.

  .. tab-item:: Virtual env or system Python
    :sync: pip

    ::

      # To build the latest stable release:
      pip install numpy --no-binary numpy

      # To build a development version, you need a local clone of the NumPy git repository:
      git clone https://github.com/numpy/numpy.git
      cd numpy
      git submodule update --init
      pip install .



.. _the-spin-interface:

Building from source for NumPy development

If you want to build from source in order to work on NumPy itself, first clone the NumPy repository::

  git clone https://github.com/numpy/numpy.git
  cd numpy
  git submodule update --init

Then you want to do the following:

  1. Create a dedicated development environment (virtual environment or conda environment),
  2. Install all needed dependencies (build, and also test, doc and optional dependencies),
  3. Build NumPy with the spin developer interface.

Step (3) is always the same, steps (1) and (2) are different between conda and virtual environments:

.. tab-set::

.. tab-item:: Conda env :sync: conda

To create a ``numpy-dev`` development environment with every required and
optional dependency installed, run::

    mamba env create -f environment.yml
    mamba activate numpy-dev

.. tab-item:: Virtual env or system Python :sync: pip

.. note::

   There are many tools to manage virtual environments, like ``venv``,
   ``virtualenv``/``virtualenvwrapper``, ``pyenv``/``pyenv-virtualenv``,
   Poetry, PDM, Hatch, and more. Here we use the basic ``venv`` tool that
   is part of the Python stdlib. You can use any other tool; all we need is
   an activated Python environment.

Create and activate a virtual environment in a new directory named ``venv`` (
note that the exact activation command may be different based on your OS and shell
- see `"How venvs work" <https://docs.python.org/3/library/venv.html#how-venvs-work>`__
in the ``venv`` docs).

.. tab-set::

  .. tab-item:: Linux
    :sync: linux

    ::

      python -m venv venv
      source venv/bin/activate

  .. tab-item:: macOS
    :sync: macos

    ::

      python -m venv venv
      source venv/bin/activate

  .. tab-item:: Windows
    :sync: windows

    ::

      python -m venv venv
      .\venv\Scripts\activate

  .. tab-item:: Windows on ARM64
    :sync: Windows on ARM64

    ::

      python -m venv venv
      .\venv\Scripts\activate

    .. note::

      Building NumPy with BLAS and LAPACK functions requires OpenBLAS
      library at Runtime. In Windows on ARM64, this can be done by setting
      up pkg-config for OpenBLAS dependency. The build steps for OpenBLAS
      for Windows on ARM64 can be found `here <http://www.openmathlib.org/OpenBLAS/docs/install/#windows-on-arm>`__.


Then install the Python-level dependencies from PyPI with::

   python -m pip install -r requirements/build_requirements.txt

To build NumPy in an activated development environment, run::

spin build

This will install NumPy inside the repository (by default in a build-install directory). You can then run tests (spin test), drop into IPython (spin ipython), or take other development steps like build the html documentation or running benchmarks. The spin interface is self-documenting, so please see spin --help and spin <subcommand> --help for detailed guidance.

.. warning::

In an activated conda environment on Windows, the AR, LD, and LDFLAGS
environment variables may be set, which will cause the build to fail.
These variables are only needed for flang and can be safely unset
for build.

.. _meson-editable-installs:

.. admonition:: IDE support & editable installs

While the ``spin`` interface is our recommended way of working on NumPy,
it has one limitation: because of the custom install location, NumPy
installed using ``spin`` will not be recognized automatically within an
IDE (e.g., for running a script via a "run" button, or setting breakpoints
visually). This will work better with an *in-place build* (or "editable
install").

Editable installs are supported. It is important to understand that **you
may use either an editable install or** ``spin`` **in a given repository clone,
but not both**. If you use editable installs, you have to use ``pytest``
and other development tools directly instead of using ``spin``.

To use an editable install, ensure you start from a clean repository (run
``git clean -xdf`` if you've built with ``spin`` before) and have all
dependencies set up correctly as described higher up on this page. Then
do::

    # Note: the --no-build-isolation is important!
    pip install -e . --no-build-isolation

    # To run the tests for, e.g., the `numpy.linalg` module:
    pytest numpy/linalg

When making changes to NumPy code, including to compiled code, there is no
need to manually rebuild or reinstall. NumPy is automatically rebuilt each
time NumPy is imported by the Python interpreter; see the meson-python_
documentation on editable installs for more details on how that works under
the hood.

When you run ``git clean -xdf``, which removes the built extension modules,
remember to also uninstall NumPy with ``pip uninstall numpy``.


.. warning::

    Note that editable installs are fundamentally incomplete installs.
    Their only guarantee is that ``import numpy`` works - so they are
    suitable for working on NumPy itself, and for working on pure Python
    packages that depend on NumPy. Headers, entrypoints, and other such
    things may not be available from an editable install.

Customizing builds

.. toctree:: :maxdepth: 1

compilers_and_options blas_lapack cpu_simd cross_compilation redistributable_binaries

Background information

.. toctree:: :maxdepth: 1

understanding_meson introspecting_a_build distutils_equivalents

.. _Miniforge: https://github.com/conda-forge/miniforge .. _meson-python: https://mesonbuild.com/meson-python/