>>> blas_dep = scipy.show_config(mode='dicts')['Build Dependencies']['blas']
>>> for key in blas_dep:
...     print(f"{key}:  {blas_dep[key]}")
...
name:  openblas
found:  True
version:  0.3.23
detection method:  pkgconfig
include directory:  /home/user/miniforge/envs/scipy-dev/include
lib directory:  /home/user/miniforge/envs/scipy-dev/lib
openblas configuration:  USE_64BITINT=0 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK=0 NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP=0 PRESCOTT MAX_THREADS=128
pc file directory:  /home/user/miniforge/envs/scipy-dev/lib/pkgconfig

$ python -m threadpoolctl -i scipy.linalg
[
  {
    "user_api": "blas",
    "internal_api": "openblas",
    "prefix": "libopenblas",
    "filepath": "/home/user/miniforge/envs/dev/lib/libopenblasp-r0.3.21.so",
    "version": "0.3.21",
    "threading_layer": "pthreads",
    "architecture": "SkylakeX",
    "num_threads": 24
  }
]

::

   $ ldd build/scipy/linalg/_fblas.cpython-*.so
   ...
   libopenblas.so.0 => /home/user/miniforge/envs/scipy-dev/lib/libopenblas.so.0 (0x0000780d6d000000)

::

   % otool -L build/scipy/linalg/_fblas.cpython-310-darwin.so
   build/scipy/linalg/_fblas.*.so:
       /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
       @rpath/libopenblas.0.dylib (compatibility version 0.0.0, current version 0.0.0)

::

    $ nm -gD ~/miniforge/envs/scipy-dev/lib/libblas.so | rg openblas_set_num_threads
    0000000000362990 T openblas_set_num_threads

::

    % nm ~/miniforge/envs/scipy-dev/lib/libblas.3.dylib | rg openblas_set_num_threads
    000000000015b6b0 T _openblas_set_num_threads

Perhaps the easiest way is to use the runtime switching abilities provided by
distros. For example, to create a conda environment with the latest SciPy
release installed and then switching between OpenBLAS, Netlib BLAS/LAPACK, and
MKL is as simple as::

    $ conda create -n blas-switch scipy threadpoolctl
    $ conda activate blas-switch
    $ python -m threadpoolctl -i scipy.linalg
    ...
        "user_api": "blas",
        "internal_api": "openblas",

    $ conda install "libblas=*=*netlib"
    ...
      libblas                         3.9.0-21_linux64_openblas --> 3.9.0-5_h92ddd45_netlib
      libcblas                        3.9.0-21_linux64_openblas --> 3.9.0-5_h92ddd45_netlib
      liblapack                       3.9.0-21_linux64_openblas --> 3.9.0-5_h92ddd45_netlib

    $ conda install "libblas=*=*mkl"
    ...
      libblas                           3.9.0-5_h92ddd45_netlib --> 3.9.0-21_linux64_mkl
      libcblas                          3.9.0-5_h92ddd45_netlib --> 3.9.0-21_linux64_mkl
      liblapack                         3.9.0-5_h92ddd45_netlib --> 3.9.0-21_linux64_mkl

    $ python -m threadpoolctl -i scipy.linalg
    ...
      "user_api": "blas",
      "internal_api": "mkl",

This can be done for development builds as well, when building via ``spin``
in the exact same way as in `SciPy's conda-forge build recipe
<https://github.com/conda-forge/scipy-feedstock/blob/main/recipe/build.sh>`__
(outputs omitted for brevity, they're similar to the ones above)::

    $ conda env create -f environment.yml
    $ conda activate scipy-dev
    $ conda install "libblas=*=*netlib"  # necessary, we need to build against blas/lapack
    $ spin build -S-Dblas=blas -S-Dlapack=lapack -S-Duse-g77-abi=true
    $ spin test -s linalg  # run tests to verify
    $ conda install "libblas=*=*mkl"
    $ spin test -s linalg
    $ conda install "libblas=*=*openblas"


Linux distro package managers

$ update-alternatives --config libblas.so.3
$ update-alternatives --config liblapack.so.3

`FlexiBLAS <https://github.com/mpimd-csc/flexiblas>`__ provides runtime
switching support (among other things) for all installed BLAS libraries that it
can detect. There are a few limitations at the time of writing (March 2024),
primarily: no support for Windows, no support for macOS Accelerate (the updated
version, with ``NEWLAPACK`` symbols). If those limitations don't matter for
you, FlexiBLAS can be a quite useful tool for efficient debugging, including
for versions of OpenBLAS and other BLAS libraries that you have to build from
source.

Once you have everything set up, the development experience is::

    $ spin build -S-Dblas=flexiblas -S-Dlapack=flexiblas
    $ FLEXIBLAS=NETLIB spin test -s linalg
    $ FLEXIBLAS=OpenBLAS spin test -s linalg
    # Or export the environment variable to make the selection stick:
    $ export FLEXIBLAS=OpenBLAS

You can also provide a path to a built BLAS library (e.g.,
``FLEXIBLAS="libbhlas_atlas.so"``) - see the `usage docs in its README
<https://github.com/mpimd-csc/flexiblas#selecting-the-backend-at-runtime>`__
for more details.

Unless you're on Fedora, you will likely have to build FlexiBLAS from source,
which is a bit of work. The good news is that this should work no matter if
you're on Linux or macOS, and use Python via virtualenvs, conda environments,
or in some other way. We'll go through how to build OpenBLAS and FlexiBLAS
from source, to allow debugging whether something in the latest OpenBLAS
version is different from Netlib BLAS/LAPACK (or MKL) or not.

The below should work in any environment where you can build SciPy itself; the
only additional tool we need is CMake (install with, for example, ``pip install
cmake``).

Clone each repository::

    $ cd ..  # starting from the root of the local `scipy` repo
    $ mkdir flexiblas-setup && cd flexiblas-setup
    $ git clone https://github.com/OpenMathLib/OpenBLAS.git openblas
    $ git clone https://github.com/mpimd-csc/flexiblas.git
    $ mkdir built-libs  # our local prefix to install everything to

Build OpenBLAS::

    $ cd openblas
    $ mkdir build && cd build
    $ cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=$PWD/../../built-libs
    $ cmake --build . -j
    $ cmake --install . --prefix $PWD/../../built-libs
    $ cd ../..

Build FlexiBLAS::

    $ cd flexiblas
    $ mkdir build && cd build
    $ # Note: this will also pick up the libraries in your system/env libdir
    $ cmake .. -DEXTRA="OpenBLAS" -DLAPACK_API_VERSION=3.9.0 \
        -DOpenBLAS_LIBRARY=$PWD/../../built-libs/lib/libopenblas.so \
        -DCMAKE_INSTALL_PREFIX=$PWD/../../built-libs
    $ cmake --build . -j
    $ cmake --install . --prefix $PWD/../../built-libs
    $ cd ../..

We're now ready to build SciPy against FlexiBLAS::

    $ export PKG_CONFIG_PATH=$PWD/flexiblas-setup/built-libs/lib/pkgconfig/
    $ cd scipy
    $ spin build -S-Dblas=flexiblas -S-Dlapack=flexiblas
    ...
    Run-time dependency flexiblas found: YES 3.4.2

Now we can run the tests. Note that the ``NETLIB`` option is built without
having to specify it; it's the default in FlexiBLAS and sources are included in
its repository::

    $ FLEXIBLAS=OpenBLAS spin test -s linalg
    $ FLEXIBLAS=NETLIB spin test -s linalg
    $ spin test -s linalg  # uses the default (NETLIB)

This backend switching can also be done inside a Python interpreter with
``threadpoolctl`` (see `its README
<https://github.com/joblib/threadpoolctl#switching-the-flexiblas-backend>`__
for details).

Other libraries available on the system can be inspected with::

    $ ./flexiblas-setup/built-libs/bin/flexiblas list

.. note::

    Using local builds of reference BLAS/LAPACK or BLIS is more difficult,
    because FlexiBLAS requires a single shared library which contains all
    needed symbols. It `may be feasible
    <https://github.com/mpimd-csc/flexiblas#setup-with-precompiled-reference-blas-and-lapack>`__
    to use a separate ``libblas`` and ``liblapack`` as the "system library",
    but this has proven to be more fragile and difficult to build (so this is
    YMMV). In case you do want to try:

    Build reference BLAS and LAPACK:

        $ git clone https://github.com/Reference-LAPACK/lapack.git
        $ cd lapack
        $ mkdir build && cd build
        $ cmake -DCBLAS=ON -DBUILD_SHARED_LIBS=OFF ..
        $ cmake --build . -j
        $ cmake --install . --prefix $PWD/../../built-libs

    Then add the following two lines to the ``cmake ..`` configure command for
    FlexiBLAS::

        -DSYS_BLAS_LIBRARY=$PWD/../../built-libs/lib/libblas.a \
        -DSYS_LAPACK_LIBRARY=$PWD/../../built-libs/lib/liblapack.a \


Creating reproducers in C or Fortran
------------------------------------

Our experience tells us that a large majority of bugs are inside SciPy rather
than in OpenBLAS or another BLAS library. If the testing with different BLAS
libraries tells us though that the problem is specific to a single BLAS library
(maybe even a single version of that library with a regression), the next step
is to produce a reproducer in C or Fortran; doing so is necessary for reporting
the bug upstream, and makes it much easier for the BLAS library developers to
address the problem.

To get from a Python reproducer which uses a ``scipy`` function with NumPy
arrays as input to a C/Fortran reproducer, it is necessary to find the code
path taken in SciPy and determine which exact BLAS or LAPACK function is
called, and with what inputs (note: the answer may be contained in the
``.pyf.src`` f2py signature files; looking into the generated
``_flapackmodule.c`` in the build directory may be useful too). This can then
be reproduced in C/Fortran by defining some integer/float variables and arrays
(typically small arrays with hardcoded numbers are enough).

Argument lists of BLAS and LAPACK functions can be looked up in for example
`the Netlib LAPACK docs <https://www.netlib.org/lapack/explore-html/>`__ or the
`Reference-LAPACK/lapack repository <https://github.com/Reference-LAPACK/lapack>`__.

Below a reproducer is shown for an issue in reference LAPACK, which was
reported as a SciPy issue in `scipy#11577
<https://github.com/scipy/scipy/issues/11577>`__. We'll name the file
``ggev_repro_gh_11577.c|f90``:

.. tab-set::

  .. tab-item:: C
    :sync: C

    .. literalinclude:: _code_examples/ggev_repro_gh_11577.c
      :language: C


  .. tab-item:: Fortran
    :sync: Fortran

    .. literalinclude:: _code_examples/ggev_repro_gh_11577.f90
      :language: fortran

Now we need to compile this reproducer locally and run it. If we're invoking a
compiler directly, we need to add the needed compile and link flags by hand.
The include path will depend on your local install, and the link flags will
depend on which library you're testing. For example, to test against a local
build of OpenBLAS:

.. tab-set::

  .. tab-item:: C
    :sync: C

    ::

        $ gcc ggev_repro_gh_11577.c \
          -I$PWD/../flexiblas-setup/built-libs/include/ \
          -L$PWD/../flexiblas-setup/built-libs/lib -lopenblas
        $ ./a.out  # to run the reproducer

  .. tab-item:: Fortran
    :sync: Fortran

    ::

        $ gfortran ggev_repro_gh_11577.f90 \
          -I/$PWD/../flexiblas-setup/built-libs/include/ \
          -L$PWD/../flexiblas-setup/built-libs/lib -lopenblas
        $ ./a.out  # to run the reproducer

For reference BLAS/LAPACK, the ``-lopenblas`` should be replaced with
``-lblas -llapack``.

Note that the explicit paths are only needed for libraries in non-standard
locations (like the ones we built in this guide). For building against a
package manager-installed library for which the shared library and headers are
on the normal compiler search path (e.g., in ``/usr/lib`` and ``/usr/include``,
or inside a conda env when using compilers from the same env), they can be left out:

.. tab-set::

  .. tab-item:: C
    :sync: C

    ::

        $ gcc ggev_repro_gh_11577.c -lopenblas
        $ ./a.out  # to run the reproducer

  .. tab-item:: Fortran
    :sync: Fortran

    ::

        $ gfortran ggev_repro_gh_11577.f90 -lopenblas
        $ ./a.out  # to run the reproducer

Alternatively (and probably a more robust way), use a small ``meson.build``
file to automate this and avoid the manual paths:

.. tab-set::

  .. tab-item:: C
    :sync: C

    .. literalinclude:: _code_examples/meson.build.c
      :language: meson

    To then build the test and run it::

        $ export PKG_CONFIG_PATH=~/code/tmp/flexiblas-setup/built-libs/lib/pkgconfig/
        $ meson setup build
        $ ninja -C build
        $ ./build/repro_c  # output may vary

        info = 0
        Re(eigv) = 4.000000 , 8.000000 , inf , -inf ,
        Im(eigv = 0.000000 , 0.000000 , -nan , -nan ,

  .. tab-item:: Fortran
    :sync: Fortran

    .. literalinclude:: _code_examples/meson.build.fortran
      :language: meson

    To then build the test and run it::

        $ export PKG_CONFIG_PATH=~/code/tmp/flexiblas-setup/built-libs/lib/pkgconfig/
        $ meson setup build
        $ ninja -C build
        $ ./build/repro_f90  # output may vary

        workspace query: lwork =           -1
        info =            0
        opt lwork =         156

        info =            0
        alphar =    1.0204501477442456        11.707793036240817        3.7423579363517347E-014  -1.1492523608519701E-014
        alphai =    0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000
        beta =   0.25511253693606051        1.4634741295300704        0.0000000000000000        0.0000000000000000

        Re(eigv) =    4.0000000000000142        8.0000000000001741                       Infinity                 -Infinity
        Im(eigv) =    0.0000000000000000        0.0000000000000000                            NaN                       NaN


.. warning::

   When you have multiple versions/builds of the same BLAS library on your
   machine, it's easy to accidentally pick up the wrong one during the build
   (remember: ``-lopenblas`` only says "link against *some*
   ``libopenblas.so``). If you're not sure, use ``ldd`` on the test executable
   you built to inspect which shared library it's linked against.


Debugging linalg issues with ``gdb``
------------------------------------

When debugging linalg issues, it is sometimes useful to step through both Python
and C code. You can use ``pdb`` for the former, and ``gdb`` for the latter.

First, prepare a small python reproducer, with a breakpoint. For example::

    $ cat chol.py
    import numpy as np
    from scipy import linalg
    n = 40
    rng = np.random.default_rng(1234)
    x = rng.uniform(size=n)
    a = x[:, None] @ x[None, :] + np.identity(n)

    breakpoint()      # note a breakpoint
    linalg.cholesky(a)

Then, you will need to run it under the ``gdb`` and add a C-level breakpoint at
the LAPACK function. This way, your execution will stop twice: first on the
Python breakpoint and then on the C breakpoint, and you will be able to step
through and inspect values of both Python and C variables.

To find out the LAPACK name, read the python source of the SciPy function and
use ``nm`` on the ``.so`` library to find out the exact name.
For the Cholesky factorization above, the LAPACK function is ``?potrf``, and the
C name on Ubuntu linux is ``dpotrf_`` (it may be spelled with or without the
trailing underscore, in uppper case or lower case, depending on the system).

Here is an example ``gdb`` session::

    $ gdb --args python chol.py
    ...
    (gdb) b dpotrf_     # this adds a C breakpoint (type "y" below)
    Function "dpotrf_" not defined.
    Make breakpoint pending on future shared library load? (y or [n]) y
    Breakpoint 1 (dpotrf_) pending.
    (gdb) run    # run the python script
    ...
    > /home/br/temp/chol/chol.py(12)<module>()
    -> linalg.cholesky(a)   # execution stopped at the python breakpoint
    (Pdb) p a.shape  # ... inspect the python state here
    (40, 40)
    (Pdb) c     # continue execution until the C breakpoint

    Thread 1 "python" hit Breakpoint 1, 0x00007ffff4c48820 in dpotrf_ ()
       from /home/br/miniforge/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/../../../../libcblas.so.3
    (gdb) s     # step through the C function
    Single stepping until exit from function dpotrf_,
    which has no line number information.
    f2py_rout__flapack_dpotrf (capi_self=<optimized out>, capi_args=<optimized out>,
        capi_keywds=<optimized out>, f2py_func=0x7ffff4c48820 <dpotrf_>)
        at scipy/linalg/_flapackmodule.c:63281
    ....
    (gdb) p lda    # inspect values of C variables
    $1 = 40

    # print out the C backtrace
    (gdb) bt
    #0  0x00007ffff3056b1e in f2py_rout.flapack_dpotrf ()
       from /path/to/site-packages/scipy/linalg/_flapack.cpython-311-x86_64-linux-gnu.so
    #1  0x0000555555734323 in _PyObject_MakeTpCall (tstate=0x555555ad0558 <_PyRuntime+166328>,
        callable=<fortran at remote 0x7ffff40ffc00>, args=<optimized out>, nargs=1,
        keywords=('lower', 'overwrite_a', 'clean'))
        at /usr/local/src/conda/python-3.11.8/Objects/call.c:214
    ...

Depending on your system, you may need to build SciPy with debug build type,
and unset CFLAGS/CXXFLAGS environment variables. See the `NumPy debugging guide
<https://numpy.org/devdocs/dev/development_environment.html#debugging>`__ for
more details.