10 minutes to pandas

  df2.dtypes
  df2.to_numpy()

For a :class:`DataFrame`, passing a single label selects a column and
yields a :class:`Series`:

.. ipython:: python

   df["A"]

If the label only contains letters, numbers, and underscores, you can
alternatively use the column name attribute:

.. ipython:: python

   df.A

Passing a list of column labels selects multiple columns, which can be useful
for getting a subset/rearranging:

.. ipython:: python

   df[["B", "A"]]

For a :class:`DataFrame`, passing a slice ``:`` selects matching rows:

.. ipython:: python

   df[0:3]
   df["20130102":"20130104"]

Selection by label

See more in :ref:`Selection by Position <indexing.integer>` using :meth:`DataFrame.iloc` or :meth:`DataFrame.iat`.

Select via the position of the passed integers:

.. ipython:: python

   df.iloc[3]

Integer slices acts similar to NumPy/Python:

.. ipython:: python

   df.iloc[3:5, 0:2]

Lists of integer position locations:

.. ipython:: python

   df.iloc[[1, 2, 4], [0, 2]]

For slicing rows explicitly:

.. ipython:: python

   df.iloc[1:3, :]

For slicing columns explicitly:

.. ipython:: python

   df.iloc[:, 1:3]

For getting a value explicitly:

.. ipython:: python

   df.iloc[1, 1]

For getting fast access to a scalar (equivalent to the prior method):

.. ipython:: python

   df.iat[1, 1]

Boolean indexing
~~~~~~~~~~~~~~~~

Select rows where ``df.A`` is greater than ``0``.

.. ipython:: python

   df[df["A"] > 0]

Selecting values from a :class:`DataFrame` where a boolean condition is met:

.. ipython:: python

   df[df > 0]

Using :func:`~Series.isin` method for filtering:

.. ipython:: python

   df2 = df.copy()
   df2["E"] = ["one", "one", "two", "three", "four", "three"]
   df2
   df2[df2["E"].isin(["two", "four"])]

Setting
~~~~~~~

Setting a new column automatically aligns the data by the indexes:

.. ipython:: python

   s1 = pd.Series(
      [1, 2, 3, 4, 5, 6],
      index=pd.date_range("20130102", periods=6))
   s1
   df["F"] = s1

Setting values by label:

.. ipython:: python

   df.at[dates[0], "A"] = 0

Setting values by position:

.. ipython:: python

   df.iat[0, 1] = 0

Setting by assigning with a NumPy array:

.. ipython:: python
   :okwarning:

   df.loc[:, "D"] = np.array([5] * len(df))

The result of the prior setting operations:

.. ipython:: python

   df

A ``where`` operation with setting:

.. ipython:: python

   df2 = df.copy()
   df2[df2 > 0] = -df2
   df2


Missing data
------------

For NumPy data types, ``np.nan`` represents missing data. It is by
default not included in computations. See the :ref:`Missing Data section
<missing_data>`.

Reindexing allows you to change/add/delete the index on a specified axis. This
returns a copy of the data:

.. ipython:: python

   df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ["E"])
   df1.loc[dates[0] : dates[1], "E"] = 1
   df1

:meth:`DataFrame.dropna` drops any rows that have missing data:

.. ipython:: python

   df1.dropna(how="any")

:meth:`DataFrame.fillna` fills missing data:

.. ipython:: python

   df1.fillna(value=5)

:func:`isna` gets the boolean mask where values are ``nan``:

.. ipython:: python

   pd.isna(df1)


Operations
----------

See the :ref:`Basic section on Binary Ops <basics.binop>`.

Stats
~~~~~

Operations in general *exclude* missing data.

Calculate the mean value for each column:

.. ipython:: python

   df.mean()

Calculate the mean value for each row:

.. ipython:: python

   df.mean(axis=1)

Operating with another :class:`Series` or :class:`DataFrame` with a different index or column
will align the result with the union of the index or column labels. In addition, pandas
automatically broadcasts along the specified dimension and will fill unaligned labels with ``np.nan``.

.. ipython:: python

   s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates).shift(2)
   s
   df.sub(s, axis="index")


User defined functions

See more at :ref:`Histogramming and Discretization <basics.discretization>`.

.. ipython:: python

   s = pd.Series(np.random.randint(0, 7, size=10))
   s
   s.value_counts()

String Methods

pandas provides various facilities for easily combining together :class:`Series` and
:class:`DataFrame` objects with various kinds of set logic for the indexes
and relational algebra functionality in the case of join / merge-type
operations.

See the :ref:`Merging section <merging>`.

Concatenating pandas objects together row-wise with :func:`concat`:

.. ipython:: python

   df = pd.DataFrame(np.random.randn(10, 4))
   df

   # break it into pieces
   pieces = [df[:3], df[3:7], df[7:]]

   pd.concat(pieces)

.. note::

   Adding a column to a :class:`DataFrame` is relatively fast. However, adding
   a row requires a copy, and may be expensive. We recommend passing a
   pre-built list of records to the :class:`DataFrame` constructor instead
   of building a :class:`DataFrame` by iteratively appending records to it.

Join
~~~~

:func:`merge` enables SQL style join types along specific columns. See the :ref:`Database style joining <merging.join>` section.

.. ipython:: python

   left = pd.DataFrame({"key": ["foo", "foo"], "lval": [1, 2]})
   right = pd.DataFrame({"key": ["foo", "foo"], "rval": [4, 5]})
   left
   right
   pd.merge(left, right, on="key")

:func:`merge` on unique keys:

.. ipython:: python

   left = pd.DataFrame({"key": ["foo", "bar"], "lval": [1, 2]})
   right = pd.DataFrame({"key": ["foo", "bar"], "rval": [4, 5]})
   left
   right
   pd.merge(left, right, on="key")

Grouping
--------

By "group by" we are referring to a process involving one or more of the
following steps:

* **Splitting** the data into groups based on some criteria
* **Applying** a function to each group independently
* **Combining** the results into a data structure

See the :ref:`Grouping section <groupby>`.

.. ipython:: python

   df = pd.DataFrame(
       {
           "A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],
           "B": ["one", "one", "two", "three", "two", "two", "one", "three"],
           "C": np.random.randn(8),
           "D": np.random.randn(8),
       }
   )
   df

Grouping by a column label, selecting column labels, and then applying the
:meth:`.DataFrameGroupBy.sum` function to the resulting
groups:

.. ipython:: python

   df.groupby("A")[["C", "D"]].sum()

Grouping by multiple columns label forms :class:`MultiIndex`.

.. ipython:: python

   df.groupby(["A", "B"]).sum()

Reshaping
---------

See the sections on :ref:`Hierarchical Indexing <advanced.hierarchical>` and
:ref:`Reshaping <reshaping.stacking>`.

Stack
~~~~~

.. ipython:: python

   arrays = [
      ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],
      ["one", "two", "one", "two", "one", "two", "one", "two"],
   ]
   index = pd.MultiIndex.from_arrays(arrays, names=["first", "second"])
   df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=["A", "B"])
   df2 = df[:4]
   df2

The :meth:`~DataFrame.stack` method "compresses" a level in the DataFrame's
columns:

.. ipython:: python

   stacked = df2.stack()
   stacked

With a "stacked" DataFrame or Series (having a :class:`MultiIndex` as the
``index``), the inverse operation of :meth:`~DataFrame.stack` is
:meth:`~DataFrame.unstack`, which by default unstacks the **last level**:

.. ipython:: python

   stacked.unstack()
   stacked.unstack(1)
   stacked.unstack(0)

Pivot tables

df = pd.DataFrame(
    {"id": [1, 2, 3, 4, 5, 6], "raw_grade": ["a", "b", "b", "a", "a", "e"]}
)

df["grade"] = df["raw_grade"].astype("category")
df["grade"]

new_categories = ["very good", "good", "very bad"]
df["grade"] = df["grade"].cat.rename_categories(new_categories)

df["grade"] = df["grade"].cat.set_categories(
    ["very bad", "bad", "medium", "good", "very good"]
)
df["grade"]

df.sort_values(by="grade")

df.groupby("grade", observed=False).size()

:ref:`Writing to a csv file: <io.store_in_csv>` using :meth:`DataFrame.to_csv`

.. ipython:: python

   df = pd.DataFrame(np.random.randint(0, 5, (10, 5)))
   df.to_csv("foo.csv")

:ref:`Reading from a csv file: <io.read_csv_table>` using :func:`read_csv`

.. ipython:: python

   pd.read_csv("foo.csv")

.. ipython:: python
   :suppress:

   import os

   os.remove("foo.csv")

Parquet

Reading and writing to :ref:`Excel <io.excel>`.

Writing to an excel file using :meth:`DataFrame.to_excel`:

.. ipython:: python

   df.to_excel("foo.xlsx", sheet_name="Sheet1")

Reading from an excel file using :func:`read_excel`:

.. ipython:: python

   pd.read_excel("foo.xlsx", "Sheet1", index_col=None, na_values=["NA"])

.. ipython:: python
   :suppress:

   os.remove("foo.xlsx")

Gotchas
-------

If you are attempting to perform a boolean operation on a :class:`Series` or :class:`DataFrame`
you might see an exception like:

.. ipython:: python
   :okexcept:

    if pd.Series([False, True, False]):
        print("I was true")

See :ref:`Comparisons<basics.compare>` and :ref:`Gotchas<gotchas>` for an explanation and what to do.