ContextQMD
Back to Datahub

Odcs Pre

metadata-ingestion/docs/sources/odcs/odcs_pre.md

1.7.06.2 KB
Original Source

Overview

The odcs module ingests Open Data Contract Standard (ODCS) v3.0 and v3.1 YAML files from a path, directory, or glob, and models each contract as a logical dataset on the odcs platform: dataset properties, canonical schema metadata, ownership, top-level and column-level tags, a link to the source document, and the contract's quality[] rules as Assertions attached to the logical dataset. When a schema[] entry resolves to a physical dataset (derived from the contract's typed servers[]), the source also emits a logicalParent link from the physical dataset to the logical one. ODCS is governed by the Linux Foundation under the Bitol project; see bitol.io and the open-data-contract-standard repository.

:::info Looking for the deprecated *.dhub.dc.yaml CLI?

The datahub datacontract upsert flow that consumes *.dhub.dc.yaml files is unrelated to this source and is being phased out. To ingest ODCS YAML, use source.type: odcs via datahub ingest -c <recipe.yml> instead.

:::

Prerequisites

A single ODCS file may describe multiple tables. Each schema[] entry becomes its own logical odcs Dataset. Contract-level metadata (description, owners, top-level tags) applies to every logical dataset by default. See the Concept Mapping table on the platform Overview above for per-row semantics, and the Limitations section below for how to disable replication.

  • ODCS v3.0 and v3.1 are supported (any 3.0.x patch level validates against the same v3.0.2 JSON Schema; 3.1.0 against the v3.1 schema). Contracts whose apiVersion reports v2.x — or any value outside odcs_versions — are skipped with a warning.
  • Filter which datasets are ingested with dataset_pattern. The allow/deny regexes match the composed logical dataset name ({contract_id}.{schema_name}, per logical_dataset_name_template). A non-matching schema[] entry is skipped along with its assertions and logicalParent link, and recorded under report.filtered. Deny takes precedence over allow. This is orthogonal to path globs, which filter by file location rather than contract content.
  • Assertions always attach to the logical dataset. Quality rules and the schema-compliance assertion are emitted whether or not a physical table exists yet, so contract-first workflows keep their expectations. Propagation of those expectations onto bound physical datasets is handled by DataHub via the PhysicalInstanceOf relationship — not by this source.
  • Physical binding is derived from the contract itself. The spec requires servers[].type; the source maps supported types (postgres, mysql, snowflake, bigquery, redshift, databricks, sqlserver, trino) to DataHub platforms and composes fully-qualified dataset names from the server's own fields (e.g. database.schema.table). Use server_overrides to refine env / platform_instance, or physical_urn_overrides (keyed by contract id, then schema[] entry name) for explicit URNs. Binding affects only the logicalParent link.
  • Derived physical URNs are verified by default. With a DataHub graph available (datahub-rest sink), a derived URN that does not exist in DataHub is left unbound with a warning instead of creating a stub dataset (verify_physical_urns_exist: false opts out). With a file sink there is no graph, and links are emitted without verification.
  • Logical Models are in private beta. The logical odcs datasets this source emits — and the assertions attached to them — render in the UI only when LOGICAL_MODELS_ENABLED is enabled (off by default). The metadata is ingested either way. The recommended workflow is: ingest the physical platform source (postgres / snowflake / …), ingest ODCS, and enable LOGICAL_MODELS_ENABLED to see the logical models, their assertions, and their physical links.
  • Files are loaded leniently by default (strict_validation: false) so that contracts with extra or non-conformant fields are accepted with warnings. Spec-valid fields the source does not map (SLA, support, pricing, relationships, …) are summarized once per file as an info; genuinely unknown fields warn individually. Set strict_validation: true to fail on JSON Schema violations — recommended for multi-tenant or untrusted directories.
  • Stale-metadata removal uses standard stateful ingestion. Enable it via the stateful_ingestion block (server-side checkpoints; requires a DataHub graph). When you remove a schema[] entry from a contract file, the corresponding logical odcs Dataset and Assertion URNs are marked removed on the next run. Physical datasets and their logicalParent links are never marked removed — those are owned by their platform-of-record source. The fail_safe_threshold guard (default 75%) blocks mass deletions caused by config or naming changes.
  • Symlinks are not followed by default (follow_symlinks: false). If your directory layout organises contracts using symlinks, set follow_symlinks: true deliberately. The default is conservative because following symlinks in a shared directory can disclose files outside the configured root.
  • Files larger than max_input_file_bytes (default 5 MB) are skipped with a warning before parsing.
  • Contracts can live in object stores, over HTTP, or in a Git repository. In addition to local paths, path accepts s3:// / gs:// object-store URIs (single file or glob), http(s):// URLs to a single file, and any mix of these in a list. S3 URIs require an aws_connection block and GCS URIs a gcs_connection block (both validated up front). For authenticated http(s):// URLs, set an http_connection block with either a bearer token or basic-auth username/password (the two are mutually exclusive), plus an optional verify_ssl: false toggle for trusted hosts with self-signed certificates (disabling it emits a warning). Public URLs need no http_connection. Set git_info to shallow-clone a repository (using an SSH deploy key) and scan it; each non-URI path entry is then resolved relative to the checkout (e.g. path: contracts/ or path: '**/*.odcs.yaml'). Install the extra dependencies with pip install 'acryl-datahub[odcs]'.