SparkApplication Compute Engine

Description

The SparkApplication compute engine runs Feast batch materialization on Kubernetes by creating a Kubeflow Spark Operator SparkApplication custom resource for each materialization job.

Unlike the in-process spark.engine compute engine (which uses a Spark session inside the Feast process), spark_application submits work to the Spark Operator. The operator starts a driver pod and executors from your configured image; Feast polls the SparkApplication until it completes.

Design

1. Feast creates a ConfigMap with job tasks and a driver copy of feature_store.yaml.
2. Feast creates a SparkApplication CR pointing at the driver entrypoint (main.py in the image).
3. Inside the pod, the batch engine type is rewritten to spark.engine so materialization uses the Spark session created by spark-submit (avoids recursive SparkApplication creation).
4. The driver writes features to your configured online store and updates the registry (same network backends as the server).

Requirements

• Kubeflow Spark Operator installed and watching the target namespace.
• A container image that includes the Feast SDK, PySpark, and clients for your stores. See the reference Dockerfile.
• Network-accessible online store, offline store, and registry. File-based backends are rejected because Spark pods have an ephemeral filesystem:

For distributed reads, configure offline_store.type: spark (or another store Spark can read efficiently).

Kubernetes / Feast Operator notes

When using the Feast Operator:

• Point spec.batchEngine.configMapRef at a ConfigMap whose type is spark_application (see Guide 6 — Batch Engine & Scheduled Jobs).
• The operator auto-creates RBAC for the spark_application batch engine (server and driver service accounts).
• Set spec.services.initImage if init / feast-apply containers need the Spark-capable image.

Example

{% code title="feature_store.yaml" %}

project: my_project
registry:
  registry_type: sql
  path: postgresql+psycopg://feast:****@postgres:5432/feast
online_store:
  type: redis
  connection_string: redis:6379
offline_store:
  type: spark
  spark_conf:
    spark.master: local[*]
batch_engine:
  type: spark_application
  image: my-registry.example.com/feast-spark-driver:latest
  namespace: feast
  spark_version: "4.0.1"
  driver_cores: 1
  driver_memory: "2g"
  executor_instances: 2
  executor_cores: 1
  executor_memory: "2g"
  spark_conf:
    spark.sql.shuffle.partitions: "100"

{% endcode %}

Feast Operator ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: feast-spark-batch-engine
  namespace: feast
data:
  config: |
    type: spark_application
    image: my-registry.example.com/feast-spark-driver:latest
    namespace: feast
    executor_instances: 2
    driver_memory: "2g"
    executor_memory: "2g"
---
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
  name: feast
  namespace: feast
spec:
  feastProject: my_project
  batchEngine:
    configMapRef:
      name: feast-spark-batch-engine
    configMapKey: config

Remote materialization

If the client uses a remote online store (online_store.type: remote), FeatureStore.materialize() delegates to the feature server HTTP API. The server runs the SparkApplication engine.

• Default (run_async=False): block until the server finishes sync materialization.
• run_async=True: accept asynchronously (?async=true); poll feature-view state in the registry for completion.
• force=True (with run_async=True): override stuck MATERIALIZING state on the server.

from datetime import datetime, timedelta
from feast import FeatureStore

store = FeatureStore(repo_path=".")  # client feature_store.yaml with online_store.type: remote

store.materialize(
    start_date=datetime.utcnow() - timedelta(days=1),
    end_date=datetime.utcnow(),
)

Configuration reference

Troubleshooting

Related

• Spark compute engine (in-process)
• Feast Operator — batch engine ConfigMap
• Creating a custom compute engine