<center><h2>Scale your pandas workflows by changing one line of code</h2>

Exercise 1: How to use Modin

GOAL: Learn how to import Modin to accelerate and scale pandas workflows.

Modin is a drop-in replacement for pandas that distributes the computation across all of the cores in your machine or in a cluster. In practical terms, this means that you can continue using the same pandas scripts as before and expect the behavior and results to be the same. The only thing that needs to change is the import statement. Normally, you would change:

python

import pandas as pd

to:

python

import modin.pandas as pd

Changing this line of code will allow you to use all of the cores in your machine to do computation on your data. One of the major performance bottlenecks of pandas is that it only uses a single core for any given computation. Modin exposes an API that is identical to pandas, allowing you to continue interacting with your data as you would with pandas. There are no additional commands required to use Modin locally. Partitioning, scheduling, data transfer, and other related concerns are all handled by Modin under the hood.

<p style="text-align:left;"> <h1>pandas on a multicore laptop <span style="float:right;"> Modin on a multicore laptop </span> <div> </div>

Concept for exercise: setting Modin engine

Modin uses Ray as an execution engine by default so no additional action is required to start to use it. Alternatively, if you need to use another engine, it should be specified either by setting the Modin config or by setting Modin environment variable before the first operation with Modin as it is shown below. Also, note that the full list of Modin configs and corresponding environment variables can be found in the Modin Configuration Settings section of the Modin documentation.

One of the execution engines that Modin uses is Unidist. Currently, Modin only supports MPI through unidist, so it should be specified either by setting the Unidist config or by setting Unidist environment variable. The full list of Unidist configs and corresponding environment variables can be found in the Unidist Configuration Settings section of the Unidist documentation.

python

# Modin engine and Unidist backend can be specified either by config
import modin.config as modin_cfg
import unidist.config as unidist_cfg
modin_cfg.Engine.put("unidist")
unidist_cfg.Backend.put("mpi")

# or by setting the environment variable
# import os
# os.environ["MODIN_ENGINE"] = "unidist"
# os.environ["UNIDIST_BACKEND"] = "mpi"

Concept for exercise: Dataframe constructor

Often when playing around in pandas, it is useful to create a DataFrame with the constructor. That is where we will start.

python

import numpy as np
import pandas as pd

frame_data = np.random.randint(0, 100, size=(2**10, 2**5))
df = pd.DataFrame(frame_data)

When creating a dataframe from a non-distributed object, it will take extra time to partition the data. When this is happening, you will see this message:

UserWarning: Distributing <class 'numpy.ndarray'> object. This may take some time.

python

# Note: Do not change this code!
import numpy as np
import pandas
import sys
import modin

python

pandas.__version__

python

modin.__version__

python

# Implement your answer here. You are also free to play with the size
# and shape of the DataFrame, but beware of exceeding your memory!

# import pandas as pd
import pandas as pd

frame_data = np.random.randint(0, 100, size=(2**5, 2**5))
df = pd.DataFrame(frame_data)

# ***** Do not change the code below! It verifies that 
# ***** the exercise has been done correctly. *****

try:
    assert df is not None
    assert frame_data is not None
    assert isinstance(frame_data, np.ndarray)
except:
    raise AssertionError("Don't change too much of the original code!")
assert "modin.pandas" in sys.modules, "Not quite correct. Remember the single line of code change (See above)"

import modin.pandas
assert pd == modin.pandas, "Remember the single line of code change (See above)"
assert hasattr(df, "_query_compiler"), "Make sure that `df` is a modin.pandas DataFrame."

print("Success! You only need to change one line of code!")

Now that we have created a toy example for playing around with the DataFrame, let's print it out in different ways.

Concept for Exercise: Data Interaction and Printing

When interacting with data, it is very imporant to look at different parts of the data (e.g. df.head()). Here we will show that you can print the modin.pandas DataFrame in the same ways you would pandas.

python

# Print the first 10 lines.
df.head(10)

python

# Print the DataFrame.
df

python

# Free cell for custom interaction (Play around here!)
df.add_prefix("col")

python

df.count()

Please move on to Exercise 2 when you are ready