.. grid-item::
   :columns: 12 12 6 6

   *Dask is a Python library for parallel and distributed computing.*  Dask is:

   -  **Easy** to use and set up (it's just a Python library)
   -  **Powerful** at providing scale, and unlocking complex algorithms
   -  and **Fun** 🎉

.. grid-item::
   :columns: 12 12 6 6

   .. raw:: html

        <script src="https://fast.wistia.com/embed/medias/l9sgt2saht.jsonp" async></script><script src="https://fast.wistia.com/assets/external/E-v1.js" async></script><div class="wistia_responsive_padding" style="padding:75.0% 0 0 0;position:relative;"><div class="wistia_responsive_wrapper" style="height:100%;left:0;position:absolute;top:0;width:100%;"><div class="wistia_embed wistia_async_l9sgt2saht seo=true videoFoam=true" style="height:100%;position:relative;width:100%">&nbsp;</div></div></div>

.. tab-item:: Tasks

    Dask Futures parallelize arbitrary for-loop style Python code,
    providing:

    -  **Flexible** tooling allowing you to construct custom
       pipelines and workflows
    -  **Powerful** scaling techniques, processing several thousand
       tasks per second
    -  **Responsive** feedback allowing for intuitive execution,
       and helpful dashboards

    Dask futures form the foundation for other Dask work

    Learn more at :bdg-link-primary:`Futures Documentation <futures.html>`
    or see an example at :bdg-link-primary:`Futures Example <https://examples.dask.org/futures.html>`

    .. grid:: 1 1 2 2

        .. grid-item::
            :columns: 12 12 7 7

            .. code-block:: python

                from dask.distributed import LocalCluster
                client = LocalCluster().get_client()

                # Submit work to happen in parallel
                results = []
                for filename in filenames:
                    data = client.submit(load, filename)
                    result = client.submit(process, data)
                    results.append(result)

                # Gather results back to local computer
                results = client.gather(results)

        .. grid-item::
            :columns: 12 12 5 5

            .. figure:: images/futures-graph.png
               :align: center

.. tab-item:: DataFrames

    Dask Dataframes parallelize the popular pandas library, providing:

    -   **Larger-than-memory** execution for single machines, allowing you
        to process data that is larger than your available RAM
    -   **Parallel** execution for faster processing
    -   **Distributed** computation for terabyte-sized datasets

    Dask Dataframes are similar in this regard to Apache Spark, but use the
    familiar pandas API and memory model.  One Dask dataframe is simply a
    collection of pandas dataframes on different computers.

    Learn more at :bdg-link-primary:`DataFrame Documentation <dataframe.html>`
    or see an example at :bdg-link-primary:`DataFrame Example <https://examples.dask.org/dataframe.html>`

    .. grid:: 1 1 2 2

        .. grid-item::
            :columns: 12 12 7 7

            .. code-block:: python

                import dask.dataframe as dd

                # Read large datasets in parallel
                df = dd.read_parquet("s3://mybucket/data.*.parquet")
                df = df[df.value < 0]
                result = df.groupby(df.name).amount.mean()

                result = result.compute()  # Compute to get pandas result
                result.plot()

        .. grid-item::
            :columns: 12 12 5 5

            .. figure:: images/dask-dataframe.svg
               :align: center


.. tab-item:: Arrays

    Dask Arrays parallelize the popular NumPy library, providing:

    -   **Larger-than-memory** execution for single machines, allowing you
        to process data that is larger than your available RAM
    -   **Parallel** execution for faster processing
    -   **Distributed** computation for terabyte-sized datasets

    Dask Arrays allow scientists and researchers to perform intuitive and
    sophisticated operations on large datasets but use the
    familiar NumPy API and memory model.  One Dask array is simply a
    collection of NumPy arrays on different computers.

    Learn more at :bdg-link-primary:`Array Documentation <array.html>`
    or see an example at :bdg-link-primary:`Array Example <https://examples.dask.org/array.html>`

    .. grid:: 1 1 2 2

        .. grid-item::

            .. code-block:: python

                import dask.array as da

                x = da.random.random((10000, 10000))
                y = (x + x.T) - x.mean(axis=1)

                z = y.var(axis=0).compute()

        .. grid-item::
            :columns: 12 12 5 5

            .. figure:: images/dask-array.svg
               :align: center

    Xarray wraps Dask array and is a popular downstream project, providing
    labeled axes and simultaneously tracking many Dask arrays together,
    resulting in more intuitive analyses.  Xarray is popular and accounts
    for the majority of Dask array use today especially within geospatial
    and imaging communities.

    Learn more at :bdg-link-primary:`Xarray Documentation <https://docs.xarray.dev/en/stable/>`
    or see an example at :bdg-link-primary:`Xarray Example <https://examples.dask.org/xarray.html>`

    .. grid:: 1 1 2 2

        .. grid-item::

            .. code-block:: python

                import xarray as xr

                ds = xr.open_mfdataset("data/*.nc")
                da.groupby('time.month').mean('time').compute()


        .. grid-item::
            :columns: 12 12 5 5

            .. figure:: https://docs.xarray.dev/en/stable/_static/logos/Xarray_Logo_RGB_Final.png
               :align: center

.. tab-item:: Bags

    Dask Bags are simple parallel Python lists, commonly used to process
    text or raw Python objects.  They are ...

    -   **Simple** offering easy map and reduce functionality
    -   **Low-memory** processing data in a streaming way that minimizes memory use
    -   **Good for preprocessing** especially for text or JSON data prior
        ingestion into dataframes

    Dask bags are similar in this regard to Spark RDDs or vanilla
    Python data structures and iterators.  One Dask bag is simply a
    collection of Python iterators processing in parallel on different computers.

    Learn more at :bdg-link-primary:`Bag Documentation <bag.html>`
    or see an example at :bdg-link-primary:`Bag Example <https://examples.dask.org/bag.html>`

    .. code-block:: python

        import dask.bag as db

        # Read large datasets in parallel
        lines = db.read_text("s3://mybucket/data.*.json")
        records = (lines
            .map(json.loads)
            .filter(lambda d: d["value"] > 0)
        )
        df = records.to_dask_dataframe()

.. tab-item:: pip

   .. code-block::

      pip install "dask[complete]"

.. tab-item:: conda

   .. code-block::

      conda install dask

.. tab-item:: Local

   Dask can set itself up easily in your Python session if you create a
   ``LocalCluster`` object, which sets everything up for you.

   .. code-block:: python

      from dask.distributed import LocalCluster
      cluster = LocalCluster()
      client = cluster.get_client()

      # Normal Dask work ...

   Alternatively, you can skip this part, and Dask will operate within a
   thread pool contained entirely with your local process.

.. tab-item:: Cloud

    `Coiled <https://docs.coiled.io/user_guide/index.html?utm_source=dask-docs&utm_medium=homepage>`_
    is a commercial SaaS product that deploys Dask clusters on cloud platforms like AWS, GCP, and Azure.

    .. code-block:: python

        import coiled
        cluster = coiled.Cluster(
           n_workers=100,
           region="us-east-2",
           worker_memory="16 GiB",
           spot_policy="spot_with_fallback",
        )
        client = cluster.get_client()

    Learn more at :bdg-link-primary:`Coiled Documentation <https://docs.coiled.io/user_guide/index.html?utm_source=dask-docs&utm_medium=homepage>`


.. tab-item:: HPC

    The `Dask-Jobqueue project <https://jobqueue.dask.org>`_ deploys
    Dask clusters on popular HPC job submission systems like SLURM, PBS, SGE, LSF,
    Torque, Condor, and others.

    .. code-block:: python

        from dask_jobqueue import PBSCluster
        cluster = PBSCluster(
           cores=24,
           memory="100GB",
           queue="regular",
           account="my-account",
        )
        cluster.scale(jobs=100)
        client = cluster.get_client()

    Learn more at :bdg-link-primary:`Dask-Jobqueue Documentation <https://jobqueue.dask.org>`

.. tab-item:: Kubernetes

   The `Dask Kubernetes project <https://kubernetes.dask.org>`_ provides
   a Dask Kubernetes Operator for deploying Dask on Kubernetes clusters.

   .. code-block:: python

        from dask_kubernetes.operator import KubeCluster
        cluster = KubeCluster(
           name="my-dask-cluster",
           image="ghcr.io/dask/dask:latest",
           resources={"requests": {"memory": "2Gi"}, "limits": {"memory": "64Gi"}},
        )
        cluster.scale(10)
        client = cluster.get_client()


   Learn more at :bdg-link-primary:`Dask Kubernetes Documentation <https://kubernetes.dask.org>`