We have a :class:`DataFrame` to which we want to apply a function row-wise.

.. ipython:: python

   df = pd.DataFrame(
       {
           "a": np.random.randn(1000),
           "b": np.random.randn(1000),
           "N": np.random.randint(100, 1000, (1000), dtype="int64"),
           "x": "x",
       }
   )
   df

Here's the function in pure Python:

.. ipython:: python

   def f(x):
       return x * (x - 1)


   def integrate_f(a, b, N):
       s = 0
       dx = (b - a) / N
       for i in range(N):
           s += f(a + i * dx)
       return s * dx

We achieve our result by using :meth:`DataFrame.apply` (row-wise):

.. ipython:: python

   %timeit df.apply(lambda x: integrate_f(x["a"], x["b"], x["N"]), axis=1)

Let's take a look and see where the time is spent during this operation
using the `prun ipython magic function <https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-prun>`__:

.. ipython:: python

   # most time consuming 4 calls
   %prun -l 4 df.apply(lambda x: integrate_f(x['a'], x['b'], x['N']), axis=1)

By far the majority of time is spend inside either ``integrate_f`` or ``f``,
hence we'll concentrate our efforts cythonizing these two functions.

.. _enhancingperf.plain:

Plain Cython

We can annotate the function variables and return types as well as use ``cdef``
and ``cpdef`` to improve performance:

.. ipython::

   In [3]: %%cython
      ...: cdef double f_typed(double x) except? -2:
      ...:     return x * (x - 1)
      ...: cpdef double integrate_f_typed(double a, double b, int N):
      ...:     cdef int i
      ...:     cdef double s, dx
      ...:     s = 0
      ...:     dx = (b - a) / N
      ...:     for i in range(N):
      ...:         s += f_typed(a + i * dx)
      ...:     return s * dx
      ...:

.. ipython:: python

   %timeit df.apply(lambda x: integrate_f_typed(x["a"], x["b"], x["N"]), axis=1)

Annotating the functions with C types yields an over ten times performance improvement compared to
the original Python implementation.

.. _enhancingperf.ndarray:

Using ndarray
~~~~~~~~~~~~~

When re-profiling, time is spent creating a :class:`Series` from each row, and calling ``__getitem__`` from both
the index and the series (three times for each row). These Python function calls are expensive and
can be improved by passing an ``np.ndarray``.

.. ipython:: python

   %prun -l 4 df.apply(lambda x: integrate_f_typed(x['a'], x['b'], x['N']), axis=1)

.. ipython::

   In [4]: %%cython
      ...: cimport numpy as np
      ...: import numpy as np
      ...: np.import_array()
      ...: cdef double f_typed(double x) except? -2:
      ...:     return x * (x - 1)
      ...: cpdef double integrate_f_typed(double a, double b, int N):
      ...:     cdef int i
      ...:     cdef double s, dx
      ...:     s = 0
      ...:     dx = (b - a) / N
      ...:     for i in range(N):
      ...:         s += f_typed(a + i * dx)
      ...:     return s * dx
      ...: cpdef np.ndarray[double] apply_integrate_f(np.ndarray col_a, np.ndarray col_b,
      ...:                                            np.ndarray col_N):
      ...:     assert (col_a.dtype == np.float64
      ...:             and col_b.dtype == np.float64 and col_N.dtype == np.dtype(int))
      ...:     cdef Py_ssize_t i, n = len(col_N)
      ...:     assert (len(col_a) == len(col_b) == n)
      ...:     cdef np.ndarray[double] res = np.empty(n)
      ...:     for i in range(len(col_a)):
      ...:         res[i] = integrate_f_typed(col_a[i], col_b[i], col_N[i])
      ...:     return res
      ...:


This implementation creates an array of zeros and inserts the result
of ``integrate_f_typed`` applied over each row. Looping over an ``ndarray`` is faster
in Cython than looping over a :class:`Series` object.

Since ``apply_integrate_f`` is typed to accept an ``np.ndarray``, :meth:`Series.to_numpy`
calls are needed to utilize this function.

.. ipython:: python

   %timeit apply_integrate_f(df['a'].to_numpy(), df['b'].to_numpy(), df['N'].to_numpy())

Performance has improved from the prior implementation by almost ten times.

.. _enhancingperf.boundswrap:

Disabling compiler directives

The ``@jit`` compilation will add overhead to the runtime of the function, so performance benefits may not be realized especially when using small data sets.
Consider `caching <https://numba.readthedocs.io/en/stable/developer/caching.html>`__ your function to avoid compilation overhead each time your function is run.

If Numba is installed, one can specify ``engine="numba"`` in select pandas methods to execute the method using Numba.
Methods that support ``engine="numba"`` will also have an ``engine_kwargs`` keyword that accepts a dictionary that allows one to specify
``"nogil"`` and ``"parallel"`` keys with boolean values to pass into the ``@jit`` decorator.
If ``engine_kwargs`` is not specified, it defaults to ``{"nogil": False, "parallel": False}`` unless otherwise specified.

.. note::

   In terms of performance, **the first time a function is run using the Numba engine will be slow**
   as Numba will have some function compilation overhead. However, the JIT compiled functions are cached,
   and subsequent calls will be fast. In general, the Numba engine is performant with
   a larger amount of data points (e.g. 1+ million).

   .. code-block:: ipython

      In [1]: data = pd.Series(range(1_000_000))  # noqa: E225

      In [2]: roll = data.rolling(10)

      In [3]: def f(x):
         ...:     return np.sum(x) + 5
      # Run the first time, compilation time will affect performance
      In [4]: %timeit -r 1 -n 1 roll.apply(f, engine='numba', raw=True)
      1.23 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)
      # Function is cached and performance will improve
      In [5]: %timeit roll.apply(f, engine='numba', raw=True)
      188 ms ± 1.93 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

      In [6]: %timeit roll.apply(f, engine='cython', raw=True)
      3.92 s ± 59 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

If your compute hardware contains multiple CPUs, the largest performance gain can be realized by setting ``parallel`` to ``True``
to leverage more than 1 CPU. Internally, pandas leverages numba to parallelize computations over the columns of a :class:`DataFrame`;
therefore, this performance benefit is only beneficial for a :class:`DataFrame` with a large number of columns.

.. code-block:: ipython

   In [1]: import numba

   In [2]: numba.set_num_threads(1)

   In [3]: df = pd.DataFrame(np.random.randn(10_000, 100))

   In [4]: roll = df.rolling(100)

   In [5]: %timeit roll.mean(engine="numba", engine_kwargs={"parallel": True})
   347 ms ± 26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

   In [6]: numba.set_num_threads(2)

   In [7]: %timeit roll.mean(engine="numba", engine_kwargs={"parallel": True})
   201 ms ± 2.97 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Custom Function Examples

Numba is best at accelerating functions that apply numerical functions to NumPy
arrays. If you try to ``@jit`` a function that contains unsupported `Python <https://numba.readthedocs.io/en/stable/reference/pysupported.html>`__
or `NumPy <https://numba.readthedocs.io/en/stable/reference/numpysupported.html>`__
code, compilation will revert `object mode <https://numba.readthedocs.io/en/stable/glossary.html#term-object-mode>`__ which
will mostly likely not speed up your function. For more on
troubleshooting Numba modes, see the `Numba troubleshooting page
<https://numba.readthedocs.io/en/stable/user/troubleshoot.html>`__.

Using ``parallel=True`` (e.g. ``@jit(parallel=True)``) may result in a ``SIGABRT`` if the threading layer leads to unsafe
behavior. You can first `specify a safe threading layer <https://numba.readthedocs.io/en/stable/user/threading-layer.html#selecting-a-threading-layer-for-safe-parallel-execution>`__
before running a JIT function with ``parallel=True``.

Generally if the you encounter a segfault (``SIGSEGV``) while using Numba, please report the issue
to the `Numba issue tracker. <https://github.com/numba/numba/issues/new/choose>`__

.. _enhancingperf.eval:

Expression evaluation via :func:`~pandas.eval`
----------------------------------------------

The top-level function :func:`pandas.eval` implements performant expression evaluation of
:class:`~pandas.Series` and :class:`~pandas.DataFrame`. Expression evaluation allows operations
to be expressed as strings and can potentially provide a performance improvement
by evaluate arithmetic and boolean expression all at once for large :class:`~pandas.DataFrame`.

.. note::

   You should not use :func:`~pandas.eval` for simple
   expressions or for expressions involving small DataFrames. In fact,
   :func:`~pandas.eval` is many orders of magnitude slower for
   smaller expressions or objects than plain Python. A good rule of thumb is
   to only use :func:`~pandas.eval` when you have a
   :class:`~pandas.DataFrame` with more than 10,000 rows.

Supported syntax

You must *explicitly reference* any local variable that you want to use in an
expression by placing the ``@`` character in front of the name. This mechanism is
the same for both :meth:`DataFrame.query` and :meth:`DataFrame.eval`. For example,

.. ipython:: python

   df = pd.DataFrame(np.random.randn(5, 2), columns=list("ab"))
   newcol = np.random.randn(len(df))
   df.eval("b + @newcol")
   df.query("b < @newcol")

If you don't prefix the local variable with ``@``, pandas will raise an
exception telling you the variable is undefined.

When using :meth:`DataFrame.eval` and :meth:`DataFrame.query`, this allows you
to have a local variable and a :class:`~pandas.DataFrame` column with the same
name in an expression.


.. ipython:: python

   a = np.random.randn()
   df.query("@a < a")
   df.loc[a < df["a"]]  # same as the previous expression

.. warning::

   :func:`pandas.eval` will raise an exception if you cannot use the ``@`` prefix because it
   isn't defined in that context.

   .. ipython:: python
      :okexcept:

      a, b = 1, 2
      pd.eval("@a + b")

   In this case, you should simply refer to the variables like you would in
   standard Python.

   .. ipython:: python

      pd.eval("a + b")


:func:`pandas.eval` parsers

There are two different expression engines.

The ``'numexpr'`` engine is the more performant engine that can yield performance improvements
compared to standard Python syntax for large :class:`DataFrame`. This engine requires the
optional dependency ``numexpr`` to be installed.

The ``'python'`` engine is generally *not* useful except for testing
other evaluation engines against it. You will achieve **no** performance
benefits using :func:`~pandas.eval` with ``engine='python'`` and may
incur a performance hit.

.. ipython:: python

   %timeit df1 + df2 + df3 + df4

.. ipython:: python

   %timeit pd.eval("df1 + df2 + df3 + df4", engine="python")


The :meth:`DataFrame.eval` method

:func:`pandas.eval` works well with expressions containing large arrays.

.. ipython:: python

   nrows, ncols = 20000, 100
   df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols)) for _ in range(4)]


:class:`DataFrame` arithmetic:

.. ipython:: python

   %timeit df1 + df2 + df3 + df4

.. ipython:: python

   %timeit pd.eval("df1 + df2 + df3 + df4")


:class:`DataFrame` comparison:

.. ipython:: python

   %timeit (df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)

.. ipython:: python

   %timeit pd.eval("(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)")


:class:`DataFrame` arithmetic with unaligned axes.

.. ipython:: python

   s = pd.Series(np.random.randn(50))
   %timeit df1 + df2 + df3 + df4 + s

.. ipython:: python

   %timeit pd.eval("df1 + df2 + df3 + df4 + s")

.. note::

   Operations such as

   .. code-block:: python

      1 and 2  # would parse to 1 & 2, but should evaluate to 2
      3 or 4  # would parse to 3 | 4, but should evaluate to 3
      ~1  # this is okay, but slower when using eval

   should be performed in Python. An exception will be raised if you try to
   perform any boolean/bitwise operations with scalar operands that are not
   of type ``bool`` or ``np.bool_``.

Here is a plot showing the running time of
:func:`pandas.eval` as function of the size of the frame involved in the
computation. The two lines are two different engines.

..
    The eval-perf.png figure below was generated with /doc/scripts/eval_performance.py

.. image:: ../_static/eval-perf.png

You will only see the performance benefits of using the ``numexpr`` engine with :func:`pandas.eval` if your :class:`~pandas.DataFrame`
has more than approximately 100,000 rows.

This plot was created using a :class:`DataFrame` with 3 columns each containing
floating point values generated using ``numpy.random.randn()``.

Expression evaluation limitations with ``numexpr``