import { Callout } from 'nextra/components' import Image from 'next/image' import { Steps } from 'nextra/components'

Setting Up Local Language Models for Your App

Setting up local language models for your app can significantly enhance its capabilities, enabling it to understand and generate text in multiple languages without relying on external APIs. By integrating local language models, you can improve privacy, reduce latency, and ensure continuous functionality even in offline environments. Here's a comprehensive guide on how to set up local language models for your application:

Steps:

For cloud version LLM change:

<Steps > ### Step 1 Visit the chat screen and you will be to see the default LLM selected. ### Step 2 Click on it and you will get a drop down of various LLM's available to choose. ### Step 3 Choose the LLM of your choice. </Steps>

Video Demo

For Open source llm change:

<Steps> ### Step 1 For open source version please edit `LLM_PROVIDER`, `LLM_NAME` and others in the .env file. Refer to [⚙️ App Configuration](/Deploying/DocsGPT-Settings) for more information. ### Step 2 Visit [☁️ Cloud Providers](/Models/cloud-providers) for the updated list of online models. Make sure you have the right API_KEY and correct LLM_PROVIDER. For self-hosted please visit [🖥️ Local Inference](/Models/local-inference). </Steps>

Fallback LLM

DocsGPT can automatically switch to a fallback LLM when the primary model fails, including mid-stream. This works with both streaming and non-streaming requests.

Fallback order:

Per-agent backup models (other models configured on the same agent)
Global fallback (FALLBACK_LLM_* env vars below)
Error returned if all fail

Setting	Description	Default
`FALLBACK_LLM_PROVIDER`	Provider name (e.g., `openai`, `anthropic`, `google`)	—
`FALLBACK_LLM_NAME`	Model name (e.g., `gpt-4o`, `claude-sonnet-4-20250514`)	—
`FALLBACK_LLM_API_KEY`	API key for the fallback provider	Falls back to `API_KEY`

All three (FALLBACK_LLM_PROVIDER, FALLBACK_LLM_NAME, and an API key) must resolve for the global fallback to activate.

env

FALLBACK_LLM_PROVIDER=anthropic
FALLBACK_LLM_NAME=claude-sonnet-4-20250514
FALLBACK_LLM_API_KEY=sk-ant-your-anthropic-key

<Callout type="info"> For maximum resilience, use a fallback provider from a different cloud than your primary. Each agent can also have multiple models configured — the other models are tried first before the global fallback. </Callout>