Observability for Predibase with Opik - Opik

Predibase is a platform for fine-tuning and serving open-source Large Language Models (LLMs). It's built on top of open-source LoRAX.

Account Setup

Comet provides a hosted version of the Opik platform, simply create an account and grab your API Key.

You can also run the Opik platform locally, see the installation guide for more information.

Tracking your LLM calls

Predibase can be used to serve open-source LLMs and is available as a model provider in LangChain. We will leverage the Opik integration with LangChain to track the LLM calls made using Predibase models.

Getting Started

Installation

To use the Opik integration with Predibase, you'll need to have both the opik, predibase and langchain packages installed. You can install them using pip:

bash

pip install opik predibase langchain

Configuring Opik

Configure the Opik Python SDK for your deployment type. See the Python SDK Configuration guide for detailed instructions on:

CLI configuration: opik configure
Code configuration: opik.configure()
Self-hosted vs Cloud vs Enterprise setup
Configuration files and environment variables

Configuring Predibase

You will also need to set the PREDIBASE_API_TOKEN environment variable to your Predibase API token. You can set it as an environment variable:

bash

export PREDIBASE_API_TOKEN=<your-predibase-api-token>

Or set it programmatically:

python

import os
import getpass

if "PREDIBASE_API_TOKEN" not in os.environ:
    os.environ["PREDIBASE_API_TOKEN"] = getpass.getpass("Enter your Predibase API token: ")

Logging LLM calls

In order to log the LLM calls to Opik, you will need to wrap the Predibase model with the OpikTracer from the LangChain integration. When making calls with that wrapped model, all calls will be logged to Opik:

python

import os
from langchain_community.llms import Predibase
from opik.integrations.langchain import OpikTracer

os.environ["OPIK_PROJECT_NAME"] = "predibase-integration-demo"

# Create the Opik tracer
opik_tracer = OpikTracer(tags=["predibase", "langchain"])

# Create Predibase model
model = Predibase(
    model="mistral-7b",
    predibase_api_key=os.environ.get("PREDIBASE_API_TOKEN"),
)

# Test the model with Opik tracing
response = model.invoke(
    "Can you recommend me a nice dry wine?",
    config={
        "temperature": 0.5,
        "max_new_tokens": 1024,
        "callbacks": [opik_tracer]
    }
)
print(response)

In addition to passing the OpikTracer to the invoke method, you can also define it during the creation of the Predibase object:

python

model = Predibase(
    model="mistral-7b",
    predibase_api_key=os.environ.get("PREDIBASE_API_TOKEN"),
).with_config({"callbacks": [opik_tracer]})

<Tip> You can learn more about the Opik integration with LangChain in our [LangChain integration guide](/integrations/langchain). </Tip>

The trace will now be available in the Opik UI for further analysis.

Advanced Usage

SequentialChain Example

Now, let's create a more complex chain and run it with Opik tracing:

python

from langchain.chains import LLMChain, SimpleSequentialChain
from langchain_core.prompts import PromptTemplate

# Synopsis chain
template = """You are a playwright. Given the title of play, it is your job to write a synopsis for that title.

Title: {title}
Playwright: This is a synopsis for the above play:"""
prompt_template = PromptTemplate(input_variables=["title"], template=template)
synopsis_chain = LLMChain(llm=model, prompt=prompt_template)

# Review chain
template = """You are a play critic from the New York Times. Given the synopsis of play, it is your job to write a review for that play.

Play Synopsis:
{synopsis}
Review from a New York Times play critic of the above play:"""
prompt_template = PromptTemplate(input_variables=["synopsis"], template=template)
review_chain = LLMChain(llm=model, prompt=prompt_template)

# Overall chain
overall_chain = SimpleSequentialChain(
    chains=[synopsis_chain, review_chain], verbose=True
)

# Run the chain with Opik tracing
review = overall_chain.run("Tragedy at sunset on the beach", callbacks=[opik_tracer])
print(review)

Accessing Logged Traces

We can access the trace IDs collected by the Opik tracer:

python

traces = opik_tracer.created_traces()
print("Collected trace IDs:", [trace.id for trace in traces])

# Flush traces to ensure all data is logged
opik_tracer.flush()

Fine-tuned LLM Example

Finally, let's use a fine-tuned model with Opik tracing:

Note: In order to use a fine-tuned model, you will need to have access to the model and the correct model ID. The code below will return a NotFoundError unless the model and adapter_id are updated.

python

fine_tuned_model = Predibase(
    model="my-base-LLM",
    predibase_api_key=os.environ.get("PREDIBASE_API_TOKEN"),
    predibase_sdk_version=None,
    adapter_id="my-finetuned-adapter-id",
    adapter_version=1,
    **{
        "api_token": os.environ.get("HUGGING_FACE_HUB_TOKEN"),
        "max_new_tokens": 5,
    },
)

# Configure the Opik tracer
fine_tuned_model = fine_tuned_model.with_config({"callbacks": [opik_tracer]})

# Invoke the fine-tuned model
response = fine_tuned_model.invoke(
    "Can you help categorize the following emails into positive, negative, and neutral?",
    **{"temperature": 0.5, "max_new_tokens": 1024},
)
print(response)

# Final flush to ensure all traces are logged
opik_tracer.flush()

Tracking your fine-tuning training runs

If you are using Predibase to fine-tune an LLM, we recommend using Predibase's integration with Comet's Experiment Management functionality. You can learn more about how to set this up in the Comet integration guide in the Predibase documentation. If you are already using an Experiment Tracking platform, worth checking if it has an integration with Predibase.