.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Usage Guide

The Informatica provider enables automatic lineage tracking for Airflow tasks that define inlets and outlets.

How It Works

The Informatica plugin automatically detects tasks with lineage support and sends inlet/outlet information to Informatica EDC when tasks succeed. No additional configuration is required beyond defining inlets and outlets in your tasks.

Key Features

• Automatic Lineage Detection: Plugin automatically detects tasks with lineage support
• EDC Integration: Native REST API integration with Informatica Enterprise Data Catalog
• Transparent Operation: No code changes required beyond inlet/outlet definitions
• Error Handling: Robust error handling for API failures and invalid objects
• Configurable: Extensive configuration options for different environments

Architecture

The provider consists of several key components:

Hooks InformaticaEDCHook provides low-level EDC API access for authentication, object retrieval, and lineage creation.

Extractors InformaticaLineageExtractor handles lineage data extraction and conversion to Airflow-compatible formats.

Plugins InformaticaProviderPlugin registers listeners that monitor task lifecycle events and trigger lineage operations.

Listeners Event-driven listeners that respond to task success/failure events and process lineage information.

Requirements

• Apache Airflow 3.0+
• Access to Informatica Enterprise Data Catalog instance
• Valid EDC credentials with API access permissions

Quick Start

1. Install the provider:

   .. code-block:: bash

   pip install apache-airflow-providers-informatica

2. Configure connection:

   Create an HTTP connection in Airflow UI with EDC server details and security domain in extras.

3. Add lineage to tasks:

   Define inlets and outlets in your tasks using EDC object URIs.

4. Run your DAG:

   The provider automatically handles lineage extraction when tasks succeed.

Example DAG

.. code-block:: python

from airflow import DAG from airflow.providers.standard.operators.python import PythonOperator from datetime import datetime

def my_python_task(**kwargs): print("Hello Informatica Lineage!")

with DAG( dag_id="example_informatica_lineage_dag", start_date=datetime(2024, 1, 1), schedule=None, catchup=False, ) as dag: python_task = PythonOperator( task_id="my_python_task", python_callable=my_python_task, inlets=[{"dataset_uri": "edc://object/source_table_abc123"}], outlets=[{"dataset_uri": "edc://object/target_table_xyz789"}], )

When this task succeeds, the provider automatically creates a lineage link between the source and target objects in EDC.

Hooks

InformaticaEDCHook ^^^^^^^^^^^^^^^^^^

The hook provides low-level access to Informatica EDC API.

.. code-block:: python

from airflow.providers.informatica.hooks.edc import InformaticaEDCHook

hook = InformaticaEDCHook(informatica_edc_conn_id="my_connection") object_data = hook.get_object("edc://object/table_123") result = hook.create_lineage_link("source_id", "target_id")

Plugins and Listeners

The InformaticaProviderPlugin automatically registers listeners that:

• Monitor task success events
• Extract inlet/outlet information from tasks
• Resolve object IDs using EDC API
• Create lineage links between resolved objects

No manual intervention is required. The plugin works transparently with any task that defines inlets and outlets.

Supported Inlet/Outlet Formats

Inlets and outlets can be defined as:

• String URIs: "edc://object/table_name"
• Dictionary with dataset_uri: {"dataset_uri": "edc://object/table_name"}

The plugin automatically handles both formats and resolves them to EDC object IDs.

Support

• Documentation: See the guides section for detailed usage and configuration
• Issues: Report bugs on the Apache Airflow GitHub repository
• Community: Join the Airflow community for discussions and support

License

Licensed under the Apache License, Version 2.0. See LICENSE file for details.