Back to Llama Index

LlamaIndex Llms Integration: Text Generation Inference

llama-index-integrations/llms/llama-index-llms-text-generation-inference/README.md

0.14.21953 B
Original Source

LlamaIndex Llms Integration: Text Generation Inference

⚠️ This integration has been deprecated!

The TextGenerationInference is no longer maintained. Instead, you can use HuggingFaceInferenceAPI. The underlying Text Generation Inference SDK (tgi) has been deprecated in favor of huggingface_hub, which HuggingFaceInferenceAPI is built on top of.

Instead, use llama-index-llms-huggingface-api:

shell
pip install llama-index-llms-huggingface-api

Usage:

py
from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI

# access hugging face inference
hub_llm = HuggingFaceInferenceAPI(model="openai-community/gpt2")
# or with a local TGI server
tgi_llm = HuggingFaceInferenceAPI(model="http://localhost:8080")