ContextQMD
Back to Llama Index

Anyscale

docs/examples/llm/anyscale.ipynb

0.14.211.5 KB
Original Source

<a href="https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/llm/anyscale.ipynb" target="_parent"></a>

Anyscale

If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

python
%pip install llama-index-llms-anyscale
python
!pip install llama-index
python
from llama_index.llms.anyscale import Anyscale
from llama_index.core.llms import ChatMessage

Call chat with ChatMessage List

You need to either set env var ANYSCALE_API_KEY or set api_key in the class constructor

python
# import os
# os.environ['ANYSCALE_API_KEY'] = '<your-api-key>'

llm = Anyscale(api_key="<your-api-key>")
python
message = ChatMessage(role="user", content="Tell me a joke")
resp = llm.chat([message])
print(resp)

Streaming

python
message = ChatMessage(role="user", content="Tell me a story in 250 words")
resp = llm.stream_chat([message])
for r in resp:
    print(r.delta, end="")

Call complete with Prompt

python
resp = llm.complete("Tell me a joke")
print(resp)
python
resp = llm.stream_complete("Tell me a story in 250 words")
for r in resp:
    print(r.delta, end="")

Model Configuration

python
llm = Anyscale(model="codellama/CodeLlama-34b-Instruct-hf")
python
resp = llm.complete("Show me the c++ code to send requests to HTTP Server")
print(resp)