metadata-ingestion/docs/sources/databricks/unity-catalog_post.md
Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.
Unity Catalog Metric Views are first-class semantic layer assets that expose dimensions and measures via a YAML specification. DataHub ingests them as datasets with subtype Metric View when you opt in with include_metric_views: true.
source:
type: unity-catalog
config:
include_metric_views: true
metric_view_pattern:
allow:
- my_catalog\.analytics\..*
When enabled, each metric view emits:
Metric View subtype so it is distinguishable from regular tables and views in the UI.ViewProperties aspect carrying the raw YAML body with viewLanguage: YAML. The materialized flag is set when the YAML contains materialization: materialized (v0.1 string form) or any materialization: object (v1.1 form).comment when present (falls back to the underlying Unity Catalog table comment otherwise).source and joins[].source fields. Both 3-part (catalog.schema.table) and 2-part (schema.table, resolved against the metric view's own catalog) identifiers are supported, as are backtick-quoted parts (`db.with.dots`.schema.table). Nested join hierarchies (snowflake-style joins) are walked recursively — every join target along the chain becomes an upstream, and every alias along the chain becomes resolvable in dimension and measure expressions.dimensions[].expr / measures[].expr using the Databricks SQL dialect (requires include_column_lineage: true, which is the default). Unqualified columns map to the source table; <join_name>.column references map through the join's source.MEASURE(name) composable measure references. A measure expressed as MEASURE(total_revenue) / MEASURE(order_count) emits two upstream edges to the total_revenue and order_count measures within the same metric view dataset. Matching is case-insensitive; the emitted URN uses the canonical case from the spec.Dimension tag on schema fields matching a YAML dimensions[].name, and a Measure tag on those matching measures[].name. Measures with a non-empty window: block get an additional Window Measure tag alongside Measure.dimensions[].comment / measures[].comment when present, with description accepted as a v0.1 fallback (falls back to the underlying Unity Catalog column comment otherwise).view.sqlConfig.spark.* keys (~150 entries per view) is dropped, since it is identical across views in a workspace and crowds the UI. All other source-table properties are preserved unchanged.The following spec-level properties are surfaced on the metric view dataset so the spec is inspectable without re-reading the YAML body:
metric_view.spec_version — the spec version (e.g. 1.1).metric_view.filter — the top-level filter: expression.metric_view.joins — the entire joins: hierarchy as a JSON string, preserving on: predicates, using: shorthand, and any nested joins.metric_view.materialization.schedule / metric_view.materialization.mode / metric_view.materialization.materialized_views — when materialization: is a v1.1 object, each subkey lands as a queryable property.Per-field agent metadata (Databricks Runtime 17.3+, YAML 1.1) is exposed as dataset-level custom properties keyed metric_view.field.<name>.*:
display_name — human-readable label for the field. Truncated to 255 characters; truncation events are counted in the ingestion report.synonyms — comma-joined alternative names. Each entry is limited to 255 characters and the list is capped at 10 entries per field (per the Databricks v1.1 spec). Per-item truncations, invalid-type drops, and the 10-cap are each recorded in the ingestion report.format.type and its subkeys (for dimensions and measures) — number, currency, percentage, byte, date, and date_time formats are supported. Each known subkey is exposed individually (including nested decimal_places.type and decimal_places.places); unknown subkeys are dropped and counted in the ingestion report.window.order / window.range / window.semiadditive for single-entry window measures, or window as a JSON string for multi-entry windows. Window properties are emitted on measures only.If a metric view's source is a SQL subquery, or if it uses a 1-part identifier that DataHub can't resolve, the YAML lineage path is skipped and DataHub falls back to the Unity Catalog table-lineage REST API for upstream resolution.
include_metric_views is false by default for backwards compatibility — when the flag is off (or when the installed databricks-sdk predates TableType.METRIC_VIEW), metric views continue to be emitted as plain Table entities with no view body.
Usage is enabled by default (include_usage_statistics: true). Choose how query history is read with usage_data_source:
AUTO (default) — system tables when warehouse_id is set; otherwise the REST API.SYSTEM_TABLES — system.query.history only (requires warehouse_id).API — REST API only.On the system-tables path, query history is joined to system.access.table_lineage on statement_id. When lineage rows exist, dataset references come from lineage; otherwise queries are parsed with sqlglot. Set skip_sqlglot_when_system_table_lineage_missing: true to skip queries with no lineage rows instead of parsing them.
include_operational_stats (default true) — when false, only SELECT statements are fetched.include_queries / include_query_usage_statistics — emit Query entities and per-query popularity (system-tables path only).include_column_usage_stats (default false) — when true, force full sqlglot parsing of every usage query so column-level usage statistics (fieldCounts) are produced. This bypasses the faster preparsed system-table lineage path and is slower; it also overrides push_down_database_pattern_access_history and skip_sqlglot_when_system_table_lineage_missing.push_down_database_pattern_access_history: true applies catalog_pattern filtering in system.access.table_lineage and semi-joins query history to statements that have lineage in the configured time window. Statements without lineage rows are not fetched (even when catalog_pattern allows all catalogs).
:::warning Coverage vs. speed tradeoff
skip_sqlglot_when_system_table_lineage_missing and push_down_database_pattern_access_history trade usage coverage for speed, not just parsing time. Databricks only records a system.access.table_lineage row for statements that emit a lineage event (typically a minority of warehouse/serverless queries) — CREATE, DESCRIBE, SET, and most other statements have no lineage row at all. Enabling either option therefore drops usage and operations for every statement that lacks lineage in the time window, which is usually the large majority of activity. They are off by default for this reason; leave them off unless you specifically want to restrict usage to the lineage-bearing subset in exchange for faster, lighter ingestion.
The default preparsed path emits table-level usage only (no column fieldCounts). Set include_column_usage_stats: true to regain column-level usage statistics via full sqlglot parsing at the cost of speed.
:::
When emit_siblings is enabled (the default), the connector emits sibling relationships between Unity Catalog external tables and their corresponding delta-lake platform entities for tables stored on S3 or other object storage. This means you may see a second dataset entity for each external Delta table — one under the databricks platform and one under delta-lake — linked as siblings in DataHub. Set emit_siblings: false in your recipe to disable this behavior if you don't need cross-platform linkage.
DataHub detects Unity Catalog foreign catalogs (Lakehouse Federation) and links their tables to the external source dataset each one mirrors (PostgreSQL, SQL Server, MySQL, Snowflake, Redshift, BigQuery, Oracle, Teradata, another Databricks workspace, or Glue/Hive).
include_federation_lineage (default true) emits an upstream COPY lineage edge from each foreign-catalog table to the external source dataset it mirrors. Column-level lineage is added when include_column_lineage is set. Set it to false to skip the cross-platform link.emit_federation_structured_properties (default true) marks the foreign catalog with structured properties (platform, remote_database, connection, catalog_type) so federated catalogs are facetable in the UI.include_federation_column_backfill (default true) fills in a foreign-catalog table's columns from the external source when Unity Catalog has not synced them yet (structure only — governance is not copied).platform_instance and case-folding must match. Use federation_connection_details (keyed by Unity Catalog connection name) to align them:source:
type: unity-catalog
config:
include_federation_lineage: true
federation_connection_details:
pg_conn:
platform_instance: prod-pg
env: PROD
:::caution Dangling lineage to an un-ingested external source
The upstream lineage edge only resolves if the external source is also ingested into DataHub as its own recipe, using the exact same platform_instance (and convert_urns_to_lowercase) that you set in federation_connection_details. If the external source is never ingested, or is ingested with a different platform_instance or case-folding setting, the edge points at a dataset URN that DataHub never creates — a dangling external dataset that never reconciles with the real one.
:::
The emit_siblings option described under Delta Lake External Tables above is unrelated: it governs only the Delta Lake (S3 external table) sibling path, not Lakehouse Federation.
If you have multiple databricks workspaces that point to the same Unity Catalog metastore, our suggestion is to use separate recipes for ingesting the workspace-specific Hive Metastore catalog and Unity Catalog metastore's information schema.
To ingest Hive metastore information schema
catalogs: ["hive_metastore"]To ingest Unity Catalog information schema
include_hive_metastore: FalseModule behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.
Check that you meet the Unity Catalog lineage requirements.
Also check the Unity Catalog limitations to make sure that lineage would be expected to exist in this case.
Unity Catalog REST API requires one call per table (table lineage) and one call per column (column lineage). To improve performance, disable column lineage with include_column_lineage: false.
Similarly, include_table_constraints: true adds one tables.get() call per non-Hive table to fetch primary key and foreign key constraints. For workspaces with thousands of tables this adds latency; leave the flag disabled (the default) if Primary Key / Foreign Key metadata is not needed.
system.access.table_lineage are parsed with sqlglot unless skip_sqlglot_when_system_table_lineage_missing: true.push_down_database_pattern_access_history: true, only statements with lineage in the time window are fetched. Disable pushdown or relax catalog_pattern if usage looks incomplete.