ContextQMD
Back to Feast

Guide 2 — Persistence

docs/how-to-guides/feast-operator/02-persistence.md

0.66.029.7 KB
Original Source

Guide 2 — Persistence

The offline store, online store, and registry each need a place to store data. The operator supports two persistence patterns for each:

  • File persistence — a path on a volume (ephemeral or PVC-backed)
  • DB/store persistence — an external database or managed store, wired via a Kubernetes Secret

The pattern is the same for all three services; only the store types differ.

Persistence patterns at a glance

PatternBest forData survives pod restart?
File — emptyDirDev / CINo
File — PVC (ref)Single-node prod or testingYes (if PVC is retained)
File — PVC (create)Operator-managed storageYes
DB storeProduction, HA, multi-podYes

File persistence

Ephemeral (emptyDir)

The default when no persistence block is set. Data lives on the pod's local disk and is lost on restart. Suitable for development only.

yaml
services:
  onlineStore:
    server: {}          # no persistence block → emptyDir

PVC — reference an existing PVC

When you already have a PVC provisioned (e.g. by your storage team):

yaml
services:
  onlineStore:
    persistence:
      file:
        path: online_store.db          # filename relative to mountPath
        pvc:
          ref:
            name: online-pvc           # name of the existing PVC
          mountPath: /data/online      # where the PVC is mounted in the pod

PVC — let the operator create one

yaml
services:
  offlineStore:
    persistence:
      file:
        type: duckdb                   # offline file type: file | dask | duckdb
        pvc:
          create:
            storageClassName: standard
            resources:
              requests:
                storage: 20Gi
          mountPath: /data/offline

Omitting storageClassName uses the cluster default StorageClass. Omitting create entirely creates a PVC with the operator's built-in defaults (1 Gi, default StorageClass).

yaml
registry:
  local:
    persistence:
      file:
        path: registry.db
        pvc:
          create: {}                   # operator defaults: 1 Gi, default StorageClass
          mountPath: /data/registry

DB / store persistence

For production, point the operator at an external database. The operator reads connection details from a Kubernetes Secret and writes them into feature_store.yaml.

Secret format

The Secret must contain one key per store component. The key name is the store type (e.g. postgres, sql, redis). The value is a YAML snippet identical to what you would write under the corresponding section in feature_store.yaml, minus the type: key (the operator inserts it from persistence.store.type).

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-data-stores
stringData:
  # Key for online store type "postgres"
  postgres: |
    host: postgres.feast.svc.cluster.local
    port: 5432
    database: feast
    db_schema: public
    user: feast
    password: feast

  # Key for registry type "sql" (SQLAlchemy URL)
  sql: |
    path: postgresql+psycopg://feast:[email protected]:5432/feast  #pragma: allowlist secret
    cache_ttl_seconds: 60
    sqlalchemy_config_kwargs:
      echo: false
      pool_pre_ping: true

Reference the Secret from the CR:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: postgres
        secretRef:
          name: feast-data-stores      # key "postgres" is read automatically
  registry:
    local:
      persistence:
        store:
          type: sql
          secretRef:
            name: feast-data-stores    # key "sql" is read automatically

Key lookup rule: the operator looks up the Secret key that matches persistence.store.type (e.g. type: postgres → key postgres). Keep all stores in one Secret or split across multiple — both work.

Injecting DB credentials into the server pod

The Secret key values in the example above hard-code passwords. For production, keep credentials in a separate Secret and inject them as environment variables using envFrom, then reference them with ${VAR} substitution in the data-stores Secret value:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: postgres-creds
stringData:
  POSTGRES_USER: feast
  POSTGRES_PASSWORD: s3cr3t
---
apiVersion: v1
kind: Secret
metadata:
  name: feast-data-stores
stringData:
  postgres: |
    host: postgres.feast.svc.cluster.local
    port: 5432
    database: feast
    user: ${POSTGRES_USER}
    password: ${POSTGRES_PASSWORD}
yaml
services:
  onlineStore:
    server:
      envFrom:
        - secretRef:
            name: postgres-creds      # expands ${POSTGRES_USER} / ${POSTGRES_PASSWORD}
    persistence:
      store:
        type: postgres
        secretRef:
          name: feast-data-stores

Store types by component

Online store

typeSecret keySDK docs
sqlitesqliteSQLite
redisredisRedis
postgrespostgresPostgres
cassandracassandraCassandra
hazelcasthazelcastHazelcast
hbasehbaseHBase
datastoredatastoreDatastore
dynamodbdynamodbDynamoDB
bigtablebigtableBigtable

For all store-specific YAML keys (connection options, pool sizes, etc.) see the linked SDK docs — the Secret value accepts the same keys.

Offline store

type (file)Notes
fileDefault pandas-based parquet offline store
daskDask-based parallel parquet
duckdbDuckDB in-process analytical engine

For external DB-backed offline stores (BigQuery, Snowflake, Spark, Trino, etc.), use persistence.store.type and a Secret with the matching key. See Offline Stores in the SDK docs.

Important — contrib store drivers require a custom image. The published quay.io/feastdev/feature-server image ships with feast[minimal] (aws, gcp, snowflake, redis, go, mysql, postgres-c, opentelemetry, grpcio, k8s, duckdb, mcp, milvus). Contrib offline stores such as Trino, Iceberg, Spark, Athena, ClickHouse, and others are not included in the base image. To use them, build a custom feature-server image that adds the required extras. See Building a custom feature-server image below.

Registry

typeSecret keyNotes
file(file persistence, no Secret)SQLite-backed file registry
sqlsqlSQLAlchemy URL — supports PostgreSQL, MySQL, SQLite
snowflake.registrysnowflake.registrySnowflake-backed registry

For sql, the Secret value is a path: (SQLAlchemy URL) plus optional cache_ttl_seconds and sqlalchemy_config_kwargs:

yaml
# feast-data-stores Secret, key "sql"
sql: |
  path: postgresql+psycopg://user:pass@host:5432/feast  #pragma: allowlist secret
  cache_ttl_seconds: 60

Common patterns

Redis online store

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-data-stores
stringData:
  redis: |
    connection_string: redis.feast.svc.cluster.local:6379
---
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
  name: sample-redis
spec:
  feastProject: my_project
  services:
    onlineStore:
      server: {}
      persistence:
        store:
          type: redis
          secretRef:
            name: feast-data-stores

Postgres for both online store and registry

yaml
services:
  onlineStore:
    persistence:
      store:
        type: postgres
        secretRef:
          name: feast-data-stores     # reads key "postgres"
  offlineStore:
    persistence:
      file:
        type: duckdb
        pvc:
          create: {}
          mountPath: /data/offline
  registry:
    local:
      persistence:
        store:
          type: sql
          secretRef:
            name: feast-data-stores   # reads key "sql"

Overriding the Secret key name

By default the operator looks up the Secret key that matches persistence.store.type (e.g. type: postgres → key postgres). To use a different key, set secretKeyName:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: postgres
        secretRef:
          name: feast-data-stores
        secretKeyName: my_custom_key    # reads key "my_custom_key" instead of "postgres"

This is useful when a single Secret holds configuration for multiple stores of the same type, or when you want a more descriptive key name.

Validation rules

The operator enforces these rules on Secret values at reconciliation time:

  1. The Secret key value must be valid YAML that deserializes to a map.
  2. If the YAML contains a type field, its value must match the CR's persistence.store.type. Otherwise the operator rejects it with an error. Best practice: omit type from the Secret.
  3. If the YAML contains a registry_type field (for registry stores), the same matching rule applies.
  4. The Secret must exist in the same namespace as the FeatureStore CR.
  5. Only one of file or store may be set under each persistence block (enforced by CRD validation).

Complete Secret examples by store type

Below are copy-paste-ready Secret YAML snippets for every operator-supported store type. Each snippet shows the Secret data key and the YAML value the operator expects.

Note: omit the type field from Secret values — the operator injects it from persistence.store.type. Including a matching type value is tolerated but not recommended.

Online store Secrets

Redis

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  redis: |
    connection_string: redis.feast.svc.cluster.local:6379

Redis Cluster with SSL:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  redis: |
    redis_type: redis_cluster
    connection_string: "redis1:6379,redis2:6379,ssl=true,password=my_password"

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: redis
        secretRef:
          name: feast-online-store

SDK reference: Redis

Postgres

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  postgres: |
    host: postgres.feast.svc.cluster.local
    port: 5432
    database: feast
    db_schema: public
    user: feast
    password: feast

With SSL:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  postgres: |
    host: postgres.feast.svc.cluster.local
    port: 5432
    database: feast
    db_schema: public
    user: feast
    password: feast
    sslmode: verify-ca
    sslrootcert_path: /path/to/server-ca.pem

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: postgres
        secretRef:
          name: feast-online-store

SDK reference: Postgres

Cassandra

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  cassandra: |
    hosts:
      - 192.168.1.1
      - 192.168.1.2
      - 192.168.1.3
    keyspace: KeyspaceName
    port: 9042
    username: user
    password: secret
    protocol_version: 5
    load_balancing:
      local_dc: datacenter1
      load_balancing_policy: TokenAwarePolicy(DCAwareRoundRobinPolicy)
    read_concurrency: 100
    write_concurrency: 100

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: cassandra
        secretRef:
          name: feast-online-store

SDK reference: Cassandra

Snowflake (online)

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  snowflake.online: |
    account: snowflake_deployment.us-east-1
    user: user_login
    password: user_password
    role: SYSADMIN
    warehouse: COMPUTE_WH
    database: FEAST

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: snowflake.online
        secretRef:
          name: feast-online-store

SDK reference: Snowflake

DynamoDB

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  dynamodb: |
    region: us-west-2
    batch_size: 100

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: dynamodb
        secretRef:
          name: feast-online-store

SDK reference: DynamoDB

Bigtable

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  bigtable: |
    project_id: my_gcp_project
    instance: my_bigtable_instance

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: bigtable
        secretRef:
          name: feast-online-store

SDK reference: Bigtable

Datastore

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  datastore: |
    project_id: my_gcp_project
    namespace: my_datastore_namespace

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: datastore
        secretRef:
          name: feast-online-store

SDK reference: Datastore

MySQL

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  mysql: |
    host: mysql.feast.svc.cluster.local
    port: 3306
    database: feast
    user: feast
    password: feast

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: mysql
        secretRef:
          name: feast-online-store

SDK reference: MySQL

Hazelcast

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  hazelcast: |
    cluster_name: dev
    cluster_members:
      - "localhost:5701"
    key_ttl_seconds: 36000

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: hazelcast
        secretRef:
          name: feast-online-store

SDK reference: Hazelcast

HBase

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-online-store
stringData:
  hbase: |
    host: hbase-thrift.feast.svc.cluster.local
    port: "9090"
    connection_pool_size: 4

CR snippet:

yaml
services:
  onlineStore:
    persistence:
      store:
        type: hbase
        secretRef:
          name: feast-online-store

SDK reference: HBase

Other supported online store types

The following types also use the same pattern (persistence.store.type + secretRef). Place the driver-specific YAML keys from the SDK docs under the matching Secret key:

typeSecret keySDK docs
sqlitesqliteSQLite
singlestoresinglestoreSingleStore
elasticsearchelasticsearchElasticsearch
qdrantqdrantQdrant
couchbase.onlinecouchbase.onlineCouchbase
milvusmilvusMilvus
mongodbmongodbMongoDB
hybridhybridHybrid

Offline store Secrets

Offline DB stores follow the same pattern. The type field tells the operator which store driver to use; the Secret value holds the connection parameters.

Snowflake (offline)

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-offline-store
stringData:
  snowflake.offline: |
    account: snowflake_deployment.us-east-1
    user: user_login
    password: user_password
    role: SYSADMIN
    warehouse: COMPUTE_WH
    database: FEAST
    schema: PUBLIC

CR snippet:

yaml
services:
  offlineStore:
    persistence:
      store:
        type: snowflake.offline
        secretRef:
          name: feast-offline-store

SDK reference: Snowflake

BigQuery

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-offline-store
stringData:
  bigquery: |
    dataset: feast_bq_dataset
    project_id: my_gcp_project

CR snippet:

yaml
services:
  offlineStore:
    persistence:
      store:
        type: bigquery
        secretRef:
          name: feast-offline-store

SDK reference: BigQuery

Postgres (offline)

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-offline-store
stringData:
  postgres: |
    host: postgres.feast.svc.cluster.local
    port: 5432
    database: feast
    db_schema: public
    user: feast
    password: feast

CR snippet:

yaml
services:
  offlineStore:
    persistence:
      store:
        type: postgres
        secretRef:
          name: feast-offline-store

SDK reference: Postgres

Other supported offline store types

typeSecret keySDK docs
redshiftredshiftRedshift
sparksparkSpark
trinotrinoTrino
athenaathenaAthena
mssqlmssqlMSSQL
couchbase.offlinecouchbase.offlineCouchbase
clickhouseclickhouseClickHouse
rayrayRay
oracleoracleOracle

Registry Secrets

SQL (SQLAlchemy) registry

The most common production registry. Uses a SQLAlchemy URL to connect to PostgreSQL, MySQL, or SQLite:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-registry-store
stringData:
  sql: |
    path: postgresql+psycopg://feast:[email protected]:5432/feast  #pragma: allowlist secret
    cache_ttl_seconds: 60
    sqlalchemy_config_kwargs:
      echo: false
      pool_pre_ping: true

CR snippet:

yaml
services:
  registry:
    local:
      persistence:
        store:
          type: sql
          secretRef:
            name: feast-registry-store

SDK reference: SQL Registry

Snowflake registry

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-registry-store
stringData:
  snowflake.registry: |
    account: snowflake_deployment.us-east-1
    user: user_login
    password: user_password
    role: SYSADMIN
    warehouse: COMPUTE_WH
    database: FEAST
    schema: PUBLIC
    cache_ttl_seconds: 60

CR snippet:

yaml
services:
  registry:
    local:
      persistence:
        store:
          type: snowflake.registry
          secretRef:
            name: feast-registry-store

SDK reference: Snowflake Registry

Multi-store Secret (single Secret for all components)

You can combine all store configurations into a single Secret:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: feast-data-stores
stringData:
  redis: |
    connection_string: redis.feast.svc.cluster.local:6379
  snowflake.offline: |
    account: snowflake_deployment.us-east-1
    user: user_login
    password: user_password
    role: SYSADMIN
    warehouse: COMPUTE_WH
    database: FEAST
    schema: PUBLIC
  sql: |
    path: postgresql+psycopg://feast:[email protected]:5432/feast  #pragma: allowlist secret
    cache_ttl_seconds: 60
---
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
  name: production-store
spec:
  feastProject: my_project
  services:
    onlineStore:
      persistence:
        store:
          type: redis
          secretRef:
            name: feast-data-stores
    offlineStore:
      persistence:
        store:
          type: snowflake.offline
          secretRef:
            name: feast-data-stores
    registry:
      local:
        persistence:
          store:
            type: sql
            secretRef:
              name: feast-data-stores

ConfigMap usage (batch engine)

The batchEngine is the only operator component that uses a ConfigMap rather than a Secret for its configuration. The ConfigMap must contain a YAML value under key config (default) or the key specified in configMapKey.

Unlike store Secrets, the batch engine ConfigMap value must include the type field:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: feast-batch-engine
data:
  config: |
    type: spark
    master: local
    spark_conf:
      spark.executor.memory: 4g
---
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
  name: sample
spec:
  feastProject: my_project
  batchEngine:
    configMapRef:
      name: feast-batch-engine
    # configMapKey: config    # optional, defaults to "config"

See Guide 6 — Batch & Jobs for full details.

Building a custom feature-server image

The published quay.io/feastdev/feature-server image includes a curated subset of Feast extras. Contrib store drivers (Trino, Iceberg, Spark, Athena, ClickHouse, etc.) and any additional Python packages your feature transformations depend on are not included. To use them, build a custom image that extends the base image with the packages you need.

You do not need to clone the Feast repository. Extend the published base image and install extra packages on top.

Writing the Dockerfile

Create a Dockerfile in your own infrastructure repository (or the repository that holds your Feast feature definitions):

dockerfile
FROM quay.io/feastdev/feature-server:0.65.0

RUN uv pip install --no-cache-dir \
    "feast[trino,redis,iceberg]==0.65.0"

Pin the Feast version in both the base image tag and the pip install command so the server and client libraries stay in sync. Add any other Python packages your feature transformations need (ML libraries, internal SDKs, etc.):

dockerfile
FROM quay.io/feastdev/feature-server:0.65.0

RUN uv pip install --no-cache-dir \
    "feast[trino,redis,iceberg,mlflow]==0.65.0" \
    "scikit-learn>=1.3,<2" \
    "my-internal-sdk==1.2.3"

Building and pushing via CI/CD

The image build and push should run entirely in CI/CD — never from a developer laptop in a production workflow.

yaml
# Example: GitHub Actions
name: Build Feast Feature Server
on:
  push:
    branches: [main]
    paths:
      - 'feast/Dockerfile'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      - uses: docker/build-push-action@v5
        with:
          context: feast/
          push: true
          tags: registry.example.com/feast/feature-server:0.65.0-custom

Referencing the custom image in the FeatureStore CR

Point the operator at the custom image using server.image on each service that needs it. Also set services.initImage so the init containers (feast-init for git clone/staging and feast-apply for registry updates) use the same custom image — otherwise they run with the default image which lacks the contrib drivers:

yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
  name: my-feature-store
spec:
  feastProject: my_project
  feastProjectDir:
    git:
      url: https://github.com/my-org/feast-feature-repo
      ref: <pinned-commit-sha>
  services:
    initImage: registry.example.com/feast/feature-server:0.65.0-custom
    offlineStore:
      server:
        image: registry.example.com/feast/feature-server:0.65.0-custom
      persistence:
        store:
          type: trino
          secretRef:
            name: feast-offline-store
    onlineStore:
      server:
        image: registry.example.com/feast/feature-server:0.65.0-custom
      persistence:
        store:
          type: redis
          secretRef:
            name: feast-online-store
    registry:
      local:
        server:
          image: registry.example.com/feast/feature-server:0.65.0-custom
        persistence:
          store:
            type: sql
            secretRef:
              name: feast-data-stores

The git section above points to your feature-definition repository (not Feast's GitHub repository). The operator clones that repository at deployment time to load feature_store.yaml and feature definitions.

Alternatively, set the image once for all services cluster-wide via the operator environment variable:

sh
kubectl set env deployment/feast-operator-controller-manager \
  RELATED_IMAGE_FEATURE_SERVER=registry.example.com/feast/feature-server:0.65.0-custom \
  -n feast-operator-system

See Guide 3 — Serving & Observability for the full image resolution priority chain.

Baking the feature repo into the custom image

If the cluster cannot clone a Git repository at runtime (air-gapped environments), combine the custom dependencies with the feature repository in one image:

dockerfile
FROM quay.io/feastdev/feature-server:0.65.0

RUN uv pip install --no-cache-dir \
    "feast[trino,redis,iceberg]==0.65.0"

COPY feature_repo/ /opt/feast/feature_repo/

Then use the packaged provisioning mode so the operator reads the baked-in repo:

yaml
spec:
  feastProject: my_project
  feastProjectDir:
    packaged:
      image: registry.example.com/feast/feature-server:0.65.0-custom
      featureRepoPath: /opt/feast/feature_repo
  services:
    runFeastApplyOnInit: false      # apply is handled separately
    offlineStore:
      server: {}
      persistence:
        store:
          type: trino
          secretRef:
            name: feast-offline-store

See Guide 1 — Project Provisioning for full packaged details.

Included extras in the base image

For reference, the base quay.io/feastdev/feature-server image installs feast[minimal], which expands to:

feast[aws, gcp, snowflake, redis, go, mysql, postgres-c, opentelemetry, grpcio, k8s, duckdb, mcp, milvus]

The minimal extra is defined in the Feast pyproject.toml.

Extras not included in minimal (require a custom image):

ExtraOffline store
trinoTrino
sparkSpark
athenaAthena
mssqlMSSQL
clickhouseClickHouse
oracleOracle
couchbaseCouchbase
icebergApache Iceberg
mlflowMLflow model registry

Troubleshooting

SymptomLikely causeFix
secret key X doesn't exist in secret YThe Secret key name doesn't match the store typeEither rename the Secret key to match type, or set secretKeyName in the CR
secret X contains invalid valueThe Secret value is not valid YAMLCheck indentation and quoting in the stringData value
contains tag named type with value XThe Secret includes a type field that doesn't match the CR's persistence.store.typeRemove type from the Secret value, or correct it to match
invalid secret X for offline storeThe referenced Secret doesn't existCreate the Secret in the same namespace as the FeatureStore CR
One selection required between file or storeBoth file and store are set under a persistence blockKeep only one — choose either file persistence or store (DB) persistence

See also