Configuring Your BigQuery Connector to DataHub

Now that you have created a Service Account and Service Account Key in BigQuery in the prior step, it's now time to set up a connection via the DataHub UI.

Configure Secrets

1. Within DataHub, navigate to the Ingestion tab in the top, right corner of your screen

<p align="center"> </p>

:::note If you do not see the Ingestion tab, please contact your DataHub admin to grant you the correct permissions :::

2. Navigate to the Secrets tab and click Create new secret

<p align="center"> </p>

3. Create a Private Key secret

This will securely store your BigQuery Service Account Private Key within DataHub

• Enter a name like BIGQUERY_PRIVATE_KEY - we will use this later to refer to the secret
• Copy and paste the private_key value from your Service Account Key
• Optionally add a description
• Click Create

<p align="center"> </p>

4. Create a Private Key ID secret

This will securely store your BigQuery Service Account Private Key ID within DataHub

• Click Create new secret again
• Enter a name like BIGQUERY_PRIVATE_KEY_ID - we will use this later to refer to the secret
• Copy and paste the private_key_id value from your Service Account Key
• Optionally add a description
• Click Create

<p align="center"> </p>

Configure Recipe

5. Navigate to the Sources tab and click Create new source

<p align="center"> </p>

6. Select BigQuery

<p align="center"> </p>

7. Fill out the BigQuery Recipe

You can find the following details in your Service Account Key file:

• Project ID
• Client Email
• Client ID

Populate the Secret Fields by selecting the Private Key and Private Key ID secrets you created in steps 3 and 4.

<p align="center"> </p>

8. Click Test Connection

This step will ensure you have configured your credentials accurately and confirm you have the required permissions to extract all relevant metadata.

<p align="center"> </p>

After you have successfully tested your connection, click Next.

Schedule Execution

Now it's time to schedule a recurring ingestion pipeline to regularly extract metadata from your BigQuery instance.

9. Decide how regularly you want this ingestion to run-- day, month, year, hour, minute, etc. Select from the dropdown

<p align="center"> </p>

10. Ensure you've configured your correct timezone

<p align="center"> </p>

11. Click Next when you are done

Finish Up

12. Name your ingestion source, then click Save and Run

<p align="center"> </p>

You will now find your new ingestion source running

<p align="center"> </p>

Validate Ingestion Runs

13. View the latest status of ingestion runs on the Ingestion page

<p align="center"> </p>

14. Click the plus sign to expand the full list of historical runs and outcomes; click Details to see the outcomes of a specific run

<p align="center"> </p>

15. From the Ingestion Run Details page, pick View All to see which entities were ingested

<p align="center"> </p>

16. Pick an entity from the list to manually validate if it contains the detail you expected

<p align="center"> </p>

Congratulations! You've successfully set up BigQuery as an ingestion source for DataHub!