Back to Datahub

Hex Post

metadata-ingestion/docs/sources/hex/hex_post.md

1.5.0.42.2 KB
Original Source

Capabilities

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Limitations

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

Currently, the Hex API has some limitations that affect the completeness of the extracted metadata:

  1. Projects and Components Relationship: The API does not support fetching the many-to-many relationship between Projects and their Components.

  2. Metadata Access: There is no direct method to retrieve metadata for Collections, Status, or Categories. This information is only available indirectly through references within Projects and Components.

Please keep these limitations in mind when working with the Hex connector.

For the Dataset - Hex Project lineage, the connector relies on the Hex query metadata feature. Therefore, in order to extract lineage information, the required setup must include:

  • A separated warehouse ingestor (eg BigQuery, Snowflake, Redshift, ...) with use_queries_v2 enabled in order to fetch Queries. This will ingest the queries into DataHub as Query entities and the ones triggered by Hex will include the corresponding Hex query metadata.
  • A DataHub server with version >= SaaS 0.3.10 or > OSS 1.0.0 so the Query entities are properly indexed by source (Hex in this case) and so fetched and processed by the Hex ingestor in order to emit the Dataset - Project lineage.

Please note:

  • Lineage is only captured for scheduled executions of the Project.
  • In cases where queries are handled by hextoolkit, Hex query metadata is not injected, which prevents capturing lineage.

Troubleshooting

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.