Helm Chart for LiteLLM

[!IMPORTANT] This is community maintained, Please make an issue if you run into a bug We recommend using Docker or Kubernetes for production deployments

Prerequisites

• Kubernetes 1.21+
• Helm 3.8.0+

If db.deployStandalone is used:

• PV provisioner support in the underlying infrastructure

If db.useStackgresOperator is used (not yet implemented):

• The Stackgres Operator must already be installed in the Kubernetes Cluster. This chart will not install the operator if it is missing.

Parameters

LiteLLM Proxy Deployment Settings

Example proxy_config ConfigMap from values (default):

proxyConfigMap:
  create: true
  key: "config.yaml"

proxy_config:
  general_settings:
    master_key: os.environ/PROXY_MASTER_KEY
  model_list:
    - model_name: gpt-3.5-turbo
      litellm_params:
        model: gpt-3.5-turbo
        api_key: eXaMpLeOnLy

Example using existing proxyConfigMap instead of creating it:

proxyConfigMap:
  create: false
  name: my-litellm-config
  key: config.yaml

# proxy_config is ignored in this mode

Example environmentSecrets Secret

apiVersion: v1
kind: Secret
metadata:
  name: litellm-envsecrets
data:
  AZURE_OPENAI_API_KEY: TXlTZWN1cmVLM3k=
type: Opaque

Database Settings

Example Postgres db.useExisting Secret

apiVersion: v1
kind: Secret
metadata:
  name: postgres
data:
  # Password for the "postgres" user
  postgres-password: <some secure password, base64 encoded>
  username: litellm
  password: <some secure password, base64 encoded>
type: Opaque

Examples for environmentSecrets and environemntConfigMaps

# Use config map for not-secret configuration data
apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-env-configmap
data:
  SOME_KEY: someValue
  ANOTHER_KEY: anotherValue

# Use secrets for things which are actually secret like API keys, credentials, etc
# Base64 encode the values stored in a Kubernetes Secret: $ pbpaste | base64 | pbcopy
# The --decode flag is convenient: $ pbpaste | base64 --decode

apiVersion: v1
kind: Secret
metadata:
  name: litellm-env-secret
type: Opaque
data:
  SOME_PASSWORD: cDZbUGVXeU5e0ZW # base64 encoded
  ANOTHER_PASSWORD: AAZbUGVXeU5e0ZB # base64 encoded

Source: GitHub Gist from troyharvey

Migration Job Settings

The migration job supports both ArgoCD and Helm hooks to ensure database migrations run at the appropriate time during deployments.

Accessing the Admin UI

When browsing to the URL published per the settings in ingress.*, you will be prompted for Admin Configuration. The Proxy Endpoint is the internal (from the litellm pod's perspective) URL published by the <RELEASE>-litellm Kubernetes Service. If the deployment uses the default settings for this service, the Proxy Endpoint should be set to http://<RELEASE>-litellm:4000.

The Proxy Key is the value specified for masterkey or, if a masterkey was not provided to the helm command line, the masterkey is a randomly generated string in the sk-... format stored in the <RELEASE>-litellm-masterkey Kubernetes Secret.

kubectl -n litellm get secret <RELEASE>-litellm-masterkey -o jsonpath="{.data.masterkey}"

Admin UI Limitations

At the time of writing, the Admin UI is unable to add models. This is because it would need to update the config.yaml file which is a exposed ConfigMap, and therefore, read-only. This is a limitation of this helm chart, not the Admin UI itself.