metadata-ingestion/source-docs-template.md
This ingestion source maps the following Source System Concepts to DataHub Concepts:
<!-- Remove all unnecessary/irrevant DataHub Concepts -->| Source Concept | DataHub Concept | Notes |
|---|---|---|
| Data Platform | ||
| Dataset | ||
| Data Job | ||
| Data Flow | ||
| Chart | ||
| Dashboard | ||
| User (a.k.a CorpUser) | ||
| CorpGroup | ||
| Domain | ||
| Container | ||
| Tag | ||
| GlossaryTerm | ||
| GlossaryNode | ||
| Assertion | ||
| DataProcess | ||
| MlFeature | ||
| MlFeatureTable | ||
| MlModel | ||
| MlModelDeployment | ||
| MlPrimaryKey | ||
| SchemaField | ||
| DataHubPolicy | ||
| DataHubIngestionSource | ||
| DataHubSecret | ||
| DataHubExecutionRequest | ||
| DataHubREtention |
| Capability | Status | Notes |
|---|---|---|
| Data Container | ✅ | Enabled by default |
| Detect Deleted Entities | ✅ | Requires recipe configuration |
| Data Domain | ❌ | Requires transformer |
| Dataset Profiling | ✅ | Requires acryl-datahub[source-usage-name] |
| Dataset Usage | ✅ | Requires acryl-datahub[source-usage-name] |
| Extract Descriptions | ✅ | Enabled by default |
| Extract Lineage | ✅ | Enabled by default |
| Extract Ownership | ✅ | Enabled by default |
| Extract Tags | ❌ | Requires transformer |
| Partition Support | ❌ | Not applicable to source |
| Platform Instance | ❌ | Not applicable to source |
| ... |
In order to ingest metadata from [Source Name], you will need:
Run the following commands to install the relevant plugin(s):
pip install 'acryl-datahub[source-name]'
pip install 'acryl-datahub[source-usage-name]'
Use the following recipe(s) to get started with ingestion.
For general pointers on writing and running a recipe, see our main recipe guide.
'acryl-datahub[source-name]'source:
type: source_name
config:
# Required fields
option1: value1
sink:
# sink configs
| Field | Required | Default | Description |
|---|---|---|---|
field1 | ✅ | default_value | A required field with a default value |
field2 | ❌ | default_value | An optional field with a default value |
field3 | ❌ | An optional field without a default value | |
| ... |
'acryl-datahub[source-usage-name]'source:
type: source-usage-name
config:
# Required Fields
option1: value1
# Options
top_n_queries: 10
sink:
# sink configs
| Field | Required | Default | Description |
|---|---|---|---|
field1 | ✅ | default_value | A required field with a default value |
field2 | ❌ | default_value | An optional field with a default value |
field3 | ❌ | An optional field without a default value | |
| ... |
[Provide description of common issues with this integration and steps to resolve]