examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_4.ipynb
GOAL: Explore some of the experimental features being added to Modin.
Sometimes when running long functions on DataFrames, it can be hard to tell how much progress has been made, as well as how much longer the function will run. A progress bar allows users to see the estimated progress and completion time of each line they run, in environments such as a shell or Jupyter notebook.
To enable Modin's Progress Bar, add the following lines of code after importing modin.pandas:
from modin.config import ProgressBar
ProgressBar.enable()
In this exercise, we'll see how the progress bar can improve our experience running dataframe queries!
import modin.pandas as pd
import numpy as np
from modin.config import ProgressBar
ProgressBar.enable()
frame_data = np.random.randint(0, 100, size=(2**18, 2**8))
df = pd.DataFrame(frame_data).add_prefix("col")
On longer functions, its nice to be able to see an estimation of how much longer things will take!
df = df.applymap(lambda x: ~x)
df
For those who have worked with Excel, the Spreadsheet API will definitely feel familiar! The Spreadsheet API is a Jupyter notebook widget that allows us to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, and edit data as well as export the changes as reproducible code.
Let's look back at a subset of the 2015 NYC Taxi Data from Exercise 2, and see how the Spreadsheet API can make it easy to play with the data!
!jupyter nbextension enable --py --sys-prefix modin_spreadsheet
ProgressBar.disable()
import modin.experimental.spreadsheet as mss
s3_path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"
modin_df = pd.read_csv(s3_path, parse_dates=["tpep_pickup_datetime", "tpep_dropoff_datetime"], quoting=3, nrows=1000)
spreadsheet = mss.from_dataframe(modin_df)
spreadsheet