Back to Genai Toolbox

dataplex-check-data-quality

docs/en/integrations/knowledge-catalog/tools/knowledge-catalog-check-data-quality.md

1.5.03.0 KB
Original Source

About

A dataplex-check-data-quality tool triggers a new Data Quality scan to evaluate rules (e.g. non-null, value range limits, custom SQL assertions) against table rows.

Since scan template creation is asynchronous, this tool returns an LRO name. You must poll dataplex-get-operation with this ID until it is done, extract the scanId, and poll dataplex-get-run-status with the scanId until the job is SUCCEEDED before calling dataplex-get-data-quality-results to fetch results.

Compatible Sources

{{< compatible-sources >}}

Requirements

IAM Permissions

Knowledge Catalog uses Identity and Access Management (IAM) to control user and group access to Knowledge Catalog resources. Toolbox will use your Application Default Credentials (ADC) to authorize and authenticate when interacting with [Knowledge Catalog][dataplex-docs].

In addition to setting the ADC for your server, you need to ensure the IAM identity has been given the correct IAM permissions for the tasks you intend to perform. See Knowledge Catalog IAM permissions and Knowledge Catalog IAM roles for more information on applying IAM permissions and roles to an identity.

Parameters

The dataplex-check-data-quality tool accepts the following parameters:

fieldtyperequireddescription
resourcePathstringtrueThe resource path of the target BigQuery table (format: projects/{project}/datasets/{dataset}/tables/{table}).
locationstringtrueThe Google Cloud region where the scan should be executed (e.g. us-central1).
publishbooleanfalseIf true, publishes the quality results directly to the Dataplex Universal Catalog. Defaults to false.
specJSONstringtrueA raw JSON string defining the quality checks rules (e.g. {"rules": [{"column": "age", "nonNullExpectation": {}}]}, maps directly to dataplexpb.DataQualitySpec).

Example

yaml
kind: tool
name: check_data_quality
type: dataplex-check-data-quality
source: my-dataplex-source
description: Trigger a new data quality scan.

Reference

fieldtyperequireddescription
typestringtrueMust be "dataplex-check-data-quality".
sourcestringtrueName of the source the tool should execute on.
descriptionstringtrueDescription of the tool that is passed to the LLM.