metadata-ingestion/docs/sources/matillion-dpc/matillion-dpc_post.md
Optional configuration to map OpenLineage namespace URIs to DataHub platform information. Without this, the connector extracts platform type from URIs with default environment.
Fields:
platform_instance: Platform instance identifier (must match source ingestion)database / schema: Defaults for incomplete dataset names from OpenLineage
database.schema.tableschema.tableconvert_urns_to_lowercase: Normalize URNs to lowercase (use true for Snowflake)env: Environment tag (PROD, DEV, etc.)Fallback behavior: Unmapped namespaces extract platform type from the URI (e.g., postgresql://... → postgres) without platform instance assignment.
Enable parse_sql_for_lineage: true to parse SQL queries from OpenLineage events for additional column-level lineage.
Requirements:
Snowflake: Use convert_urns_to_lowercase: true in namespace mapping
BigQuery: 3-tier naming (project.dataset.table). Set database: project-id, schema: dataset-name
MySQL / 2-tier: 2-tier naming (schema.table). Set schema only
Postgres / Redshift: 3-tier naming (database.schema.table). Set both database and schema
Pipelines and their components are organized into a browsable container hierarchy that mirrors their path in Matillion:
Project › Environment › <folder> › … › Pipeline › Component
The folder levels come from the pipeline's path (e.g. ingest/staging/orders/load.orch.yaml
yields ingest › staging › orders folders), so the browse tree lines up with the paths
you match on in pipeline_patterns. Components (DataJobs) live in their pipeline's folder and browse
directly under the pipeline.
The environment and folder levels are always built. extract_projects_to_containers (default true)
only controls whether the top-level Project container is added to the hierarchy; set it to false
to hang environments (and their folders) directly at the root instead of under a project.
The connector supports flexible regex-based filtering to control what metadata is ingested.
project_patterns:
allow: ["^prod-.*", "^staging-.*"]
deny: [".*-deprecated$"]
environment_patterns:
allow: ["^production$", "^staging$"]
deny: ["^sandbox.*"]
pipeline_patterns:
allow: [".*"]
deny: ["^test_.*", ".*_backup$"]
streaming_pipeline_patterns:
allow: ["^cdc_.*"]
deny: [".*_test$"]
All patterns are case-insensitive by default and support full regex syntax. Deny patterns take precedence over allow patterns.
The connector automatically detects and tracks when pipelines call other pipelines (via "Run Pipeline" components). This creates step-level dependency relationships in DataHub, showing:
No configuration needed — this feature is automatic when execution history is ingested.
Child pipelines that only appear in lineage events (and were not discovered as project pipelines themselves) are, by default, created as their own DataFlows/DataJobs so the full dependency graph is captured. To suppress these lineage-only dependencies and keep ingestion scoped to discovered pipelines, disable:
include_dependent_pipelines: false
Lineage is still emitted for discovered pipelines when this is disabled — only the lineage-only dependent pipelines are skipped.
The connector can discover pipelines from two sources:
/published-pipelines API)/pipeline-executions API)By default, both types are ingested. To only ingest published pipelines:
include_unpublished_pipelines: false
This is useful when:
Run history — per-execution DataProcessInstance entities with status and timing — is produced from the pipeline-executions API. Each execution surfaces a run on the pipeline (DataFlow) itself as well as on each of its components (DataJobs), so the "Runs" tab is populated at both levels. When include_unpublished_pipelines: true, this happens automatically as part of discovery.
When include_unpublished_pipelines: false, discovery only lists published pipelines and does not fetch executions, so no runs are emitted by default. To get run history for your published pipelines without also ingesting unpublished ones, enable:
include_unpublished_pipelines: false
extract_run_history: true
This fetches executions within the configured time window and attaches runs to the matching published pipelines. It is off by default because it calls the pipeline-executions and per-execution steps APIs, which are slower and degrade over wider time windows — pair it with a narrow start_time / end_time and stateful ingestion.
Enabling it has a second benefit: lineage often references unpublished child orchestrations (e.g. SRC_*_ORCH pipelines invoked by a published schedule). The OpenLineage namespace only carries an opaque environment UUID, so when such a pipeline is neither published nor seen in executions, its environment cannot be resolved and the connector skips it rather than placing it in a pipeline with no environment. Because executions report the environment name, enabling extract_run_history (or include_unpublished_pipelines) lets these child orchestrations resolve their environment and nest correctly under it.
Pipeline-execution discovery and lineage are bounded by start_time / end_time. If you do not set them, end_time defaults to now and start_time defaults to the start (00:00 UTC) of the previous day — i.e. at least the last 24 hours of jobs. Set start_time to backfill more history on the first run, especially if your pipelines do not run daily:
start_time: "2024-01-01T00:00:00Z" # absolute
# start_time: "-30d" # or relative to end_time
Enable stateful ingestion so subsequent runs only fetch new lineage instead of re-reading the whole window:
stateful_ingestion:
enabled: true
Lineage endpoint performance: the Matillion lineage events API paginates by offset and gets progressively slower the further back it reads. Wide time windows therefore both increase total runtime and make individual requests more likely to time out. Lineage requests are automatically split into sub-windows of at most 31 days (the API's hard limit per request), but each sub-window still pages through its full result set. Prefer a narrower window plus stateful ingestion over a single very large backfill, and only raise api_config.request_timeout_sec when a genuinely large window is unavoidable (a high timeout multiplies the worst-case wait, since failed requests are retried).
include_external_urls: true to emit them. Pipeline names and the pipeline link use the pipeline file name — the only name the API exposes; a different display name set inside the Maia editor is not retrievable. The pipeline link opens the observability dashboard pre-filtered by that file name (there is no per-pipeline deep-link), so the pipeline must have run within the dashboard's time window to appear.Processing OpenLineage event messagesEnable parse_sql_for_lineage: true (requires DataHub graph connection).
start_time to query further back in time if neededThe lineage events endpoint paginates by offset and slows down the further back in time it reads, so wide windows are the most common cause of slow runs and request_timeout_sec timeouts. In order of preference:
start_time (e.g., last 7 days instead of 30) and enable stateful_ingestion so later runs stay incremental.project_patterns to filter projectsenvironment_patterns to filter environmentspipeline_patterns to filter pipelinesstreaming_pipeline_patterns to filter streaming pipelinesinclude_streaming_pipelines if not needed.api_config.request_timeout_sec. Keep it as low as practical — failed requests are retried, so a very high timeout multiplies the worst-case wait on a slow endpoint.