AVEVA PI System - Tdengine

import Enterprise from '../../assets/resources/_enterprise.mdx';

This section describes how to create data migration tasks through the Explorer interface, migrating data from the PI system to the current TDengine cluster.

Feature Overview

The PI system is a software product suite used for data collection, retrieval, analysis, transmission, and visualization, serving as the infrastructure for enterprise-level systems managing real-time data and events. taosX can extract real-time or historical data from the PI system using the PI connector plugin.

From the perspective of data timeliness, PI data source tasks are divided into two categories: real-time tasks and backfill tasks. In the task type dropdown list, these two categories correspond to the names: PI and PI backfill.

From the data model perspective, PI data source tasks are divided into single-column model tasks and multi-column model tasks:

Single-column model tasks map one PI Point to one table in TDengine
Multi-column model tasks map one PI AF element to one table

Regarding the type of connected data source, PI data source tasks are further divided into Archive Server data sources and AF Server data sources. For Archive Server data sources, only the single-column model can be used. For AF Server data sources, both single-column model and multi-column model can be chosen.

Users configure the data mapping rules from PI to TDengine through a CSV file, referred to as the model configuration file:

For tasks using the AF Server single-column model, taosX automatically identifies which attributes of the element are referencing PI Point data, mapping one PI Point attribute to one table.
For tasks using the AF Server multi-column model, one element corresponds to one table. taosX by default maps PI Point attributes to TDengine Metric columns and other attributes to TDengine tag columns.

Creating Tasks

Add Data Source

In the data writing page, click the +Add Data Source button to enter the add data source page.

Basic Configuration

Enter the task name in Name, such as "test";

Select PI or PI backfill from the Type dropdown list.

If the taosX service is running on or can directly connect to the server where the PI system is located (dependent on PI AF SDK), Proxy is not necessary; otherwise, configure Proxy: select the specified proxy from the dropdown, or click the +Create New Proxy button on the right to create a new proxy and follow the prompts to configure the proxy. That is, taosX or its proxy needs to be deployed on a host that can directly connect to the PI system.

Select a target database from the Target Database dropdown list, or click the +Create Database button on the right to create a new database.

Connection Configuration

The PI connector supports two connection methods:

PI Data Archive Only: Does not use AF mode. In this mode, directly fill in the PI Service Name (server address, usually using the hostname).
PI Data Archive and Asset Framework (AF) Server: Uses AF SDK. In addition to configuring the service name, this mode also requires configuring the PI system (AF Server) name (hostname) and AF database name.

Click the Connectivity Check button to verify if the data source is available.

Data Model Configuration

This part has two tabs, corresponding to the configuration of the single-column model and the multi-column model. If this is your first configuration, whether you choose a single-column model or a multi-column model, be sure to click the "Download Default Configuration" button. This action will trigger the generation of the default model configuration file and also download the model configuration file to your local machine, which you can view or edit. After editing, you can also upload it again to overwrite the default configuration.

If you want to synchronize all points or all template elements, then the default configuration is sufficient. If you want to filter specific naming patterns of points or element templates, you need to fill in the filter conditions before clicking "Download Default Configuration".

Multi-column Model Configuration File

Below is an example of a multi-column model configuration file. This configuration file includes configurations for two supertables: one is the metertemplate table, which receives data from elements of the MeterTemplate template; the other is the farm table, which receives data from elements of the Farm template.

csv

SuperTable,metertemplate
SubTable,${element_name}_${element_id}
Template,MeterTemplate
Filter,
ts,KEY,TIMESTAMP,$ts
voltage,COLUMN,DOUBLE,$voltage
voltage_status,COLUMN,INT,$voltage_status
current,COLUMN,DOUBLE,$current
current_status,COLUMN,INT,$current_status
element_id,tag,VARCHAR(100),$element_id
element_name,tag,VARCHAR(100),$element_name
path,tag,VARCHAR(100),$path
categories,tag,VARCHAR(100),$categories

SuperTable,farm
SubTable,${element_name}_${element_id}
Template,Farm
Filter,
ts,KEY,TIMESTAMP,$ts
wind_speed,COLUMN,FLOAT,$wind_speed
wind_speed_status,COLUMN,INT,$wind_speed_status
power_production,COLUMN,FLOAT,$power_production
power_production_status,COLUMN,INT,$power_production_status
lost_power,COLUMN,FLOAT,$lost_power
lost_power_status,COLUMN,INT,$lost_power_status
farm_lifetime_production__weekly_,COLUMN,FLOAT,$farm_lifetime_production__weekly_
farm_lifetime_production__weekly__status,COLUMN,INT,$farm_lifetime_production__weekly__status
farm_lifetime_production__hourly_,COLUMN,FLOAT,$farm_lifetime_production__hourly_
farm_lifetime_production__hourly__status,COLUMN,INT,$farm_lifetime_production__hourly__status
element_id,tag,VARCHAR(100),$element_id
element_name,tag,VARCHAR(100),$element_name
path,tag,VARCHAR(100),$path
categories,tag,VARCHAR(100),$categories

The multi-column model configuration file consists of one or more supertable definitions. Each supertable configuration includes:

Correspondence between supertables and templates
Correspondence between attributes and TDengine Metric columns
Correspondence between attributes and TDengine tag columns
Source data filtering conditions
For each column, whether it is a Metrics column or a tag column, a mapping rule can be configured, see Zero-code third-party data access "Data extraction, filtering, and transformation" section

Single-column model configuration file

Below is an example of a single-column model configuration file.

csv

SuperTable,volt_float32
SubTable,${point_name}
Filter,
ts,KEY,TIMESTAMP,$ts
value,COLUMN,FLOAT,$value
status,COLUMN,INT,$status
path,tag,VARCHAR(200),$path
point_name,tag,VARCHAR(100),$point_name
ptclassname,tag,VARCHAR(100),$ptclassname
sourcetag,tag,VARCHAR(100),$sourcetag
tag,tag,VARCHAR(100),$tag
descriptor,tag,VARCHAR(100),$descriptor
exdesc,tag,VARCHAR(100),$exdesc
engunits,tag,VARCHAR(100),$engunits
pointsource,tag,VARCHAR(100),$pointsource
step,tag,VARCHAR(100),$step
future,tag,VARCHAR(100),$future
element_paths,tag,VARCHAR(512),`$element_paths.replace("\\", ".")`

SuperTable,milliampere_float32
SubTable,${point_name}
Filter,
ts,KEY,TIMESTAMP,$ts
value,COLUMN,FLOAT,$value
status,COLUMN,INT,$status
path,tag,VARCHAR(200),$path
point_name,tag,VARCHAR(100),$point_name
ptclassname,tag,VARCHAR(100),$ptclassname
sourcetag,tag,VARCHAR(100),$sourcetag
tag,tag,VARCHAR(100),$tag
descriptor,tag,VARCHAR(100),$descriptor
exdesc,tag,VARCHAR(100),$exdesc
engunits,tag,VARCHAR(100),$engunits
pointsource,tag,VARCHAR(100),$pointsource
step,tag,VARCHAR(100),$step
future,tag,VARCHAR(100),$future
element_paths,tag,VARCHAR(512),`$element_paths.replace("\\", ".")`

Meter_1000004_Voltage,POINT,volt_float32
Meter_1000004_Current,POINT,milliampere_float32
Meter_1000001_Voltage,POINT,volt_float32
Meter_1000001_Current,POINT,milliampere_float32
Meter_1000474_Voltage,POINT,volt_float32
Meter_1000474_Current,POINT,milliampere_float32

The single-column model configuration file is divided into two parts. The first part, like the multi-column model configuration file, consists of several supertable definitions. The second part is the point list, which configures the mapping between points and supertables. The default configuration maps points with the same UOM and data type to the same supertable.

Backfill Configuration

For PI tasks, you can configure the "restart compensation time." If the task is unexpectedly interrupted, configuring this parameter when restarting is very useful as it allows taosX to automatically backfill data for a period.
For PI backfill tasks, you must configure the start and end times of the backfill.

Advanced Options

The advanced options vary for different types of tasks. Common advanced options include:

Connector log level
Batch size for connector queries and data sending
Maximum delay for a single read

For real-time tasks of the multi-column model, there are also the following switch options:

Whether to synchronize newly added elements. If enabled, the PI connector will listen for newly added elements under the template and automatically synchronize the data of the newly added elements without needing to restart the task.
Whether to synchronize changes in static attributes. If enabled, the PI connector will synchronize all changes in static attributes (non-PI Point attributes). That is, if a static attribute value of an element in the PI AF Server is modified, the corresponding tag value in the TDengine table will also be modified.
Whether to synchronize the deletion of elements. If enabled, the PI connector will listen for events of element deletions under the configured template and synchronize the deletion of the corresponding subtable in TDengine.
Whether to synchronize the deletion of historical data. If enabled, for the time-series data of an element, if data at a certain time is deleted in PI, the corresponding column data at that time in TDengine will be set to null.
Whether to synchronize the modification of historical data. If enabled, for the time-series data of an element, if historical data is modified in PI, the corresponding data at that time in TDengine will also be updated.