Using Opik with LiteLLM

Lite allows you to call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]. You can learn more about LiteLLM here.

There are two main approaches to using LiteLLM, either using the litellm python library that will query the LLM API for you or by using the LiteLLM proxy server. In this cookbook we will focus on the first approach but you can learn more about using Opik with the LiteLLM proxy server in our documentation.

Creating an account on Comet.com

Comet provides a hosted version of the Opik platform, simply create an account and grab your API Key.

You can also run the Opik platform locally, see the installation guide for more information.

python

%pip install --upgrade opik litellm

python

import opik

opik.configure(use_local=False)

Preparing our environment

In order to use LiteLLM, we will configure the OpenAI API Key, if you are using any other providers you can replace this with the required API key:

python

import os
import getpass

if "OPENAI_API_KEY" not in os.environ:
    os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")

Logging traces

In order to log traces to Opik, you will need to set the opik callback:

python

from litellm.integrations.opik.opik import OpikLogger
from opik.opik_context import get_current_span_data
from opik import track
import litellm

os.environ["OPIK_PROJECT_NAME"] = "litellm-integration-demo"
opik_logger = OpikLogger()
litellm.callbacks = [opik_logger]

Every LiteLLM call will now be logged to Opik:

python

response = litellm.completion(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
    ],
)

print(response.choices[0].message.content)

The trace will now be viewable in the Opik platform:

Logging LLM calls within a tracked function

If you are using LiteLLM within a function tracked with the @track decorator, you will need to pass the current_span_data as metadata to the litellm.completion call:

python

@track
def streaming_function(input):
    messages = [{"role": "user", "content": input}]
    response = litellm.completion(
        model="gpt-3.5-turbo",
        messages=messages,
        metadata={
            "opik": {
                "current_span_data": get_current_span_data(),
                "tags": ["streaming-test"],
            },
        },
    )
    return response


response = streaming_function("Why is tracking and evaluation of LLMs important?")
chunks = list(response)