Back to Datahub

Source Name

metadata-ingestion/source-docs-template.md

1.5.0.36.8 KB
Original Source

Source Name

<!-- Set Support Status -->

Integration Details

<!-- Plain-language description of what this integration is meant to do. --> <!-- Include details about where metadata is extracted from (ie. logs, source API, manifest, etc.) -->

Concept Mapping

<!-- This should be a manual mapping of concepts from the source to the DataHub Metadata Model --> <!-- Authors should provide as much context as possible about how this mapping was generated, including assumptions made, known shortcuts, & any other caveats -->

This ingestion source maps the following Source System Concepts to DataHub Concepts:

<!-- Remove all unnecessary/irrevant DataHub Concepts -->
Source ConceptDataHub ConceptNotes
Data Platform
Dataset
Data Job
Data Flow
Chart
Dashboard
User (a.k.a CorpUser)
CorpGroup
Domain
Container
Tag
GlossaryTerm
GlossaryNode
Assertion
DataProcess
MlFeature
MlFeatureTable
MlModel
MlModelDeployment
MlPrimaryKey
SchemaField
DataHubPolicy
DataHubIngestionSource
DataHubSecret
DataHubExecutionRequest
DataHubREtention

Supported Capabilities

<!-- This should be an auto-generated table of supported DataHub features/functionality --> <!-- Each capability should link out to a feature guide -->
CapabilityStatusNotes
Data ContainerEnabled by default
Detect Deleted EntitiesRequires recipe configuration
Data DomainRequires transformer
Dataset ProfilingRequires acryl-datahub[source-usage-name]
Dataset UsageRequires acryl-datahub[source-usage-name]
Extract DescriptionsEnabled by default
Extract LineageEnabled by default
Extract OwnershipEnabled by default
Extract TagsRequires transformer
Partition SupportNot applicable to source
Platform InstanceNot applicable to source
...

Metadata Ingestion Quickstart

Prerequisites

In order to ingest metadata from [Source Name], you will need:

  • eg. Python version, source version, source access requirements
  • eg. Steps to configure source access
  • ...

Install the Plugin(s)

Run the following commands to install the relevant plugin(s):

pip install 'acryl-datahub[source-name]'

pip install 'acryl-datahub[source-usage-name]'

Configure the Ingestion Recipe(s)

Use the following recipe(s) to get started with ingestion.

For general pointers on writing and running a recipe, see our main recipe guide.

'acryl-datahub[source-name]'

yml
source:
  type: source_name
  config:
    # Required fields
    option1: value1

sink:
  # sink configs
<details> <summary>View All Recipe Configuartion Options</summary>
FieldRequiredDefaultDescription
field1default_valueA required field with a default value
field2default_valueAn optional field with a default value
field3An optional field without a default value
...
</details>

'acryl-datahub[source-usage-name]'

yml
source:
  type: source-usage-name
  config:
    # Required Fields
    option1: value1

    # Options
    top_n_queries: 10

sink:
  # sink configs
<details> <summary>View All Recipe Configuartion Options</summary>
FieldRequiredDefaultDescription
field1default_valueA required field with a default value
field2default_valueAn optional field with a default value
field3An optional field without a default value
...
</details>

Troubleshooting

[Common Issue]

[Provide description of common issues with this integration and steps to resolve]