ContextQMD
Back to Llama Index

LiteLLM

docs/examples/llm/litellm.ipynb

0.14.212.2 KB
Original Source

<a href="https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/llm/litellm.ipynb" target="_parent"></a>

LiteLLM

LiteLLM supports 100+ LLM APIs (Anthropic, Replicate, Huggingface, TogetherAI, Cohere, etc.). Complete List

Call complete with a prompt

If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

python
%pip install llama-index-llms-litellm
python
!pip install llama-index
python
import os
from llama_index.llms.litellm import LiteLLM
from llama_index.core.llms import ChatMessage

# set env variable
os.environ["OPENAI_API_KEY"] = "your-api-key"
os.environ["COHERE_API_KEY"] = "your-api-key"

message = ChatMessage(role="user", content="Hey! how's it going?")

# openai call
llm = LiteLLM("gpt-3.5-turbo")
chat_response = llm.chat([message])

# cohere call
llm = LiteLLM("command-nightly")
chat_response = llm.chat([message])
python
from llama_index.core.llms import ChatMessage
from llama_index.llms.litellm import LiteLLM

messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="Tell me a story"),
]
resp = LiteLLM("gpt-3.5-turbo").chat(messages)
python
print(resp)

Streaming

Using stream_complete endpoint

python
from llama_index.llms.litellm import LiteLLM

llm = LiteLLM("gpt-3.5-turbo")
resp = llm.stream_complete("Paul Graham is ")
python
for r in resp:
    print(r.delta, end="")
python
from llama_index.llms.litellm import LiteLLM

messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="Tell me a story"),
]

llm = LiteLLM("gpt-3.5-turbo")
resp = llm.stream_chat(messages)
python
for r in resp:
    print(r.delta, end="")

Async

python
from llama_index.llms.litellm import LiteLLM

llm = LiteLLM("gpt-3.5-turbo")
resp = await llm.acomplete("Paul Graham is ")
python
print(resp)