ContextQMD
Back to Litellm

LiteLLM Batch Completions Example

cookbook/LiteLLM_batch_completion.ipynb

1.84.0-dev.2980 B
Original Source

LiteLLM Batch Completions Example

python
!pip install litellm

Import Batch Completion

python
import os
from litellm import batch_completion

# set your API_KEY
os.environ['ANTHROPIC_API_KEY'] = ""

Calling litellm.batch_completion

In the batch_completion method, you provide a list of messages where each sub-list of messages is passed to litellm.completion(), allowing you to process multiple prompts efficiently in a single API call.

python
import os

os.environ['ANTHROPIC_API_KEY'] = ""


responses = batch_completion(
    model="claude-2",
    messages = [
        [
            {
                "role": "user",
                "content": "good morning? "
            }
        ],
        [
            {
                "role": "user",
                "content": "what's the time? "
            }
        ]
    ]
)
responses