.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

.. _howto/operator:AzureBlobStorageToTeradataOperator:

================================== AzureBlobStorageToTeradataOperator

The purpose of AzureBlobStorageToTeradataOperator is to define tasks involving CSV, JSON and Parquet format data transfer from an Azure Blob Storage to Teradata table. Use the :class:AzureBlobStorageToTeradataOperator <airflow.providers.teradata.transfers.azure_blob_to_teradata> to transfer data from an Azure Blob Storage to Teradata.This operator leverages the Teradata READ_NOS <https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide-17.20/Welcome-to-Native-Object-Store>_ feature to import data in CSV, JSON, and Parquet formats from Azure Blob Storage into Teradata. This operator accesses data directly from the object store and generates permanent tables within the database using READ_NOS and CREATE TABLE AS functionalities with below SQL statement.

.. code-block:: sql

CREATE MULTISET TABLE multiset_table_name AS (
  SELECT *
  FROM (
    LOCATION='YOUR-OBJECT-STORE-URI'
    AUTHORIZATION=authorization_object
  ) AS d
) WITH DATA;

It facilitates data loading from both public and private object storage. For private object storage, access to the object store can be granted via either Teradata Authorization database object or Object Store Login and Object Store Key defined with Azure Blob Storage connection in Airflow. Conversely, for data transfer from public object storage, no authorization or access credentials are required.

Teradata Authorization database object access type can be used with teradata_authorization_name parameter of AzureBlobStorageToTeradataOperator
Object Store Access Key ID and Access Key Secret access type can be used with azure_conn_id parameter of S3ToTeradataOperator

https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide-17.20/Setting-Up-Access/Setting-Access-Privileges

.. note:: Teradata Authorization database object takes precedence if both access types defined.

Transferring data from public Azure Blob Storage to Teradata

An example usage of the AzureBlobStorageToTeradataOperator to transfer CSV data format from public Azure Blob Storage to teradata table is as follows:

.. exampleinclude:: /../../teradata/tests/system/teradata/example_azure_blob_to_teradata_transfer.py :language: python :start-after: [START azure_blob__to_teradata_transfer_operator_howto_guide_transfer_data_public_blob_to_teradata_csv] :end-before: [END azure_blob__to_teradata_transfer_operator_howto_guide_transfer_data_public_blob_to_teradata_csv]

Transferring data from private Azure Blob Storage to Teradata with AWS connection

An example usage of the AzureBlobStorageToTeradataOperator to transfer CSV data format from private S3 object store to teradata with AWS credentials defined as AWS connection:

.. exampleinclude:: /../../teradata/tests/system/teradata/example_azure_blob_to_teradata_transfer.py :language: python :start-after: [START azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_access_blob_to_teradata_csv] :end-before: [END azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_access_blob_to_teradata_csv]

Transferring data from private Azure Blob Storage to Teradata with Teradata Authorization Object

Teradata authorization database object is used to control who can access an external object store. Teradata authorization database object should exists in Teradata database to use it in transferring data from S3 to Teradata. Refer Authentication for External Object Stores in Teradata <https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide-17.20/Authentication-for-External-Object-Stores>_

An example usage of the AzureBlobStorageToTeradataOperator to transfer CSV data format from private S3 object store to teradata with Authorization database object defined in Teradata.

.. exampleinclude:: /../../teradata/tests/system/teradata/example_azure_blob_to_teradata_transfer.py :language: python :start-after: [START azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_authorization_blob_to_teradata_csv] :end-before: [END azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_authorization_blob_to_teradata_csv]

Transferring data in CSV format from Azure Blob Storage to Teradata

An example usage of the AzureBlobStorageToTeradataOperator to transfer CSV data format from Azure Blob Storage to teradata table is as follows:

Transferring data in JSON format from Azure Blob Storage to Teradata

An example usage of the AzureBlobStorageToTeradataOperator to transfer JSON data format from Azure Blob Storage to teradata table is as follows:

.. exampleinclude:: /../../teradata/tests/system/teradata/example_azure_blob_to_teradata_transfer.py :language: python :start-after: [START azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_blob_to_teradata_json] :end-before: [END azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_blob_to_teradata_json]

Transferring data in PARQUET format from Azure Blob Storage to Teradata

An example usage of the AzureBlobStorageToTeradataOperator to transfer PARQUET data format from Azure Blob Storage to teradata table is as follows:

.. exampleinclude:: /../../teradata/tests/system/teradata/example_azure_blob_to_teradata_transfer.py :language: python :start-after: [START azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_blob_to_teradata_parquet] :end-before: [END azure_blob_to_teradata_transfer_operator_howto_guide_transfer_data_blob_to_teradata_parquet]

The complete `AzureBlobStorageToTeradataOperator` Operator Dag

When we put everything together, our Dag should look like this:

.. exampleinclude:: /../../teradata/tests/system/teradata/example_azure_blob_to_teradata_transfer.py :language: python :start-after: [START azure_blob_to_teradata_transfer_operator_howto_guide] :end-before: [END azure_blob_to_teradata_transfer_operator_howto_guide]

AzureBlobStorageToTeradataOperator