docs/content/features/runtime-settings.md
+++ disableToc = false title = "Runtime Settings" weight = 25 url = '/features/runtime-settings' +++
LocalAI provides a web-based interface for managing application settings at runtime. These settings can be configured through the web UI and are automatically persisted to a configuration file, allowing changes to take effect immediately without requiring a restart.
Navigate to the Settings page from the management interface at http://localhost:8080/manage. The settings page provides a comprehensive interface for configuring various aspects of LocalAI.
The watchdog monitors backend activity and can automatically stop idle or overly busy models to free up resources.
15m)5m)Changes to watchdog settings are applied immediately by restarting the watchdog service.
0 for unlimited, 1 for single-backend mode1s)Note: The "Single Backend" setting is deprecated. Use "Max Active Backends" set to
1for single-backend behavior.
By default, LocalAI will skip evicting models that have active API calls to prevent interrupting ongoing requests. When all models are busy and eviction is needed:
You can configure these settings via the web UI or through environment variables. See [VRAM Management]({{%relref "advanced/vram-management" %}}) for more details.
512)For multi-user authentication with roles, OAuth, and usage tracking, see [Authentication & Authorization]({{%relref "features/authentication" %}}).
Configure peer-to-peer networking for distributed inference:
Changes to P2P settings automatically restart the P2P stack with the new configuration.
Manage model and backend galleries:
url and name fieldsConfigure the built-in agent platform (see [Agents]({{%relref "features/agents" %}}) for full documentation):
granite-embedding-107m-multilingual)400)0)Note: Most agent pool settings require a restart to take effect.
All settings are automatically saved to runtime_settings.json in the LOCALAI_CONFIG_DIR directory (default: BASEPATH/configuration). This file is watched for changes, so modifications made directly to the file will also be applied at runtime.
Environment variables take precedence over settings configured via the web UI or configuration files. If a setting is controlled by an environment variable, it cannot be modified through the web interface. The settings page will indicate when a setting is controlled by an environment variable.
The precedence order is:
runtime_settings.json, api_keys.json)The runtime_settings.json file follows this structure:
{
"watchdog_enabled": true,
"watchdog_idle_enabled": true,
"watchdog_busy_enabled": false,
"watchdog_idle_timeout": "15m",
"watchdog_busy_timeout": "5m",
"max_active_backends": 0,
"force_eviction_when_busy": false,
"lru_eviction_max_retries": 30,
"lru_eviction_retry_interval": "1s",
"threads": 8,
"context_size": 2048,
"f16": false,
"debug": false,
"cors": true,
"csrf": false,
"cors_allow_origins": "*",
"p2p_token": "",
"p2p_network_id": "",
"federated": false,
"galleries": [
{
"url": "github:mudler/LocalAI/gallery/index.yaml@master",
"name": "localai"
}
],
"backend_galleries": [
{
"url": "github:mudler/LocalAI/backend/index.yaml@master",
"name": "localai"
}
],
"autoload_galleries": true,
"autoload_backend_galleries": true,
"api_keys": []
}
API keys can be managed through the runtime settings interface. Keys can be entered one per line or comma-separated.
Important Notes:
runtime_settings.jsonapi_keys.jsonThe runtime settings system supports dynamic configuration file watching. When LOCALAI_CONFIG_DIR is set, LocalAI monitors the following files for changes:
runtime_settings.json - Unified runtime settingsapi_keys.json - API keys (for backward compatibility)external_backends.json - External backend configurationsChanges to these files are automatically detected and applied without requiring a restart.
Use Environment Variables for Production: For production deployments, use environment variables for critical settings to ensure they cannot be accidentally changed via the web UI.
Backup Configuration Files: Before making significant changes, consider backing up your runtime_settings.json file.
Monitor Resource Usage: When enabling watchdog features, monitor your system to ensure the timeout values are appropriate for your workload.
Secure API Keys: API keys are sensitive information. Ensure proper file permissions on configuration files (they should be readable only by the LocalAI process).
Test Changes: Some settings (like watchdog timeouts) may require testing to find optimal values for your specific use case.
If settings are not being applied:
LOCALAI_CONFIG_DIR is set correctlyruntime_settings.jsonIf the watchdog is not functioning:
If P2P is not starting: