ContextQMD
Back to Whylogs

Getting Started

python/examples/basic/Getting_Started.ipynb

1.6.45.7 KB
Original Source

🚩 Create a free WhyLabs account to get more value out of whylogs!

Did you know you can store, visualize, and monitor whylogs profiles with the WhyLabs Observability Platform? Sign up for a free WhyLabs account to leverage the power of whylogs and WhyLabs together!

Getting Started

whylogs provides a standard to log any kind of data.

With whylogs, we will show how to log data, generating statistical summaries called profiles. These profiles can be used in a number of ways, like:

  • Data Visualization
  • Data Validation
  • Tracking changes in your datasets

Table of Content

In this example, we'll explore the basics of logging data with whylogs:

  • Installing whylogs
  • Profiling data
  • Interacting with the profile
  • Writing/Reading profiles to/from disk

Installing whylogs

whylogs is made available as a Python package. You can get the latest version from PyPI with pip install whylogs:

python
# Note: you may need to restart the kernel to use updated packages.
%pip install whylogs

Minimal requirements:

  • Python 3.7+ up to Python 3.10
  • Windows, Linux x86_64, and MacOS 10+

Loading a Pandas DataFrame

Before showing how we can log data, we first need the data itself. Let's create a simple Pandas DataFrame:

python
import pandas as pd
data = {
    "animal": ["cat", "hawk", "snake", "cat"],
    "legs": [4, 2, 0, 4],
    "weight": [4.3, 1.8, 1.3, 4.1],
}

df = pd.DataFrame(data)

Profiling with whylogs

To obtain a profile of your data, you can simply use whylogs' log call, and navigate through the result to a specific profile with profile():

python
import whylogs as why

results = why.log(df)
profile = results.profile()

Analyzing Profiles

Once you're done logging the data, you can generate a Profile View and inspect it in a Pandas Dataframe format:

python
prof_view = profile.view()
prof_df = prof_view.to_pandas()

prof_df

This will provide you with valuable statistics on a column (feature) basis, such as:

  • Counters, such as number of samples and null values
  • Inferred types, such as integral, fractional and boolean
  • Estimated Cardinality
  • Frequent Items
  • Distribution Metrics: min,max, median, quantile values

Writing to Disk

You can also store your profile in disk for further inspection:

python
why.write(profile, "profile.bin")

This will create a profile binary file in your local filesystem.

Reading from Disk

You can read the profile back into memory with:

python
n_prof = why.read("profile.bin")

Note: write expects a profile as parameter, while read returns a Profile View. That means that you can use the loaded profile for visualization purposes and merging, but not for further tracking and updates.

What's Next?

There's a lot you can do with the profiles you just created. Keep getting your hands dirty with the following examples!

  • Basic
  • Integrations
    • WhyLabs - Monitor your profiles continuously with the WhyLabs Observability Platform
    • Pyspark - Use whylogs with pyspark
    • Writing Profiles - See different ways and locations to output your profiles
    • Flask - See how you can create a Flask app with whylogs and WhyLabs integration
    • Feature Stores - Learn how to log features from your Feature Store with feast and whylogs
    • BigQuery - Profile data queried from a Google BigQuery table
    • MLflow - Log your whylogs profiles to an MLflow environment

Or go to the examples page for the complete list of examples!