import ChangeLog from '../changelog/connector-maxcompute.md';

Maxcompute

Maxcompute sink connector

Description

Used to read data from Maxcompute.

Key features

• exactly-once

Options

accessId [string]

accessId Your Maxcompute accessId which cloud be access from Alibaba Cloud.

accesskey [string]

accesskey Your Maxcompute accessKey which cloud be access from Alibaba Cloud.

endpoint [string]

endpoint Your Maxcompute endpoint start with http.

project [string]

project Your Maxcompute project which is created in Alibaba Cloud.

table_name [string]

table_name Target Maxcompute table name eg: fake.

partition_spec [string]

partition_spec This spec of Maxcompute partition table eg:ds='20220101'.

overwrite [boolean]

overwrite Whether to overwrite the table or partition, default: false.

save_mode_create_template

We use templates to automatically create MaxCompute tables, which will create corresponding table creation statements based on the type of upstream data and schema type, and the default template can be modified according to the situation. Only work on multi-table mode at now.

Default template:

CREATE TABLE IF NOT EXISTS `${table}` (
${rowtype_fields}
) COMMENT '${comment}';

If a custom field is filled in the template, such as adding an id field

CREATE TABLE IF NOT EXISTS `${table}`
(   
    id,
    ${rowtype_fields}
) COMMENT '${comment}';

The connector will automatically obtain the corresponding type from the upstream to complete the filling, and remove the id field from rowtype_fields. This method can be used to customize the modification of field types and attributes.

You can use the following placeholders

• database: Used to get the database in the upstream schema
• table_name: Used to get the table name in the upstream schema
• rowtype_fields: Used to get all the fields in the upstream schema, we will automatically map to the field description of MaxCompute
• rowtype_primary_key: Used to get the primary key in the upstream schema (maybe a list)
• rowtype_unique_key: Used to get the unique key in the upstream schema (maybe a list)
• comment: Used to get the table comment in the upstream schema

schema_save_mode [Enum]

Before the synchronous task is turned on, different treatment schemes are selected for the existing surface structure of the target side.
Option introduction：
RECREATE_SCHEMA ：Will create when the table does not exist, delete and rebuild when the table is saved. If the partition_spec is set, the partition will be deleted and rebuilt.
CREATE_SCHEMA_WHEN_NOT_EXIST ：Will Created when the table does not exist, skipped when the table is saved. If the partition_spec is set, the partition will be created.
ERROR_WHEN_SCHEMA_NOT_EXIST ：Error will be reported when the table does not exist
IGNORE ：Ignore the treatment of the table

data_save_mode [Enum]

Before the synchronous task is turned on, different processing schemes are selected for data existing data on the target side.
Option introduction：
DROP_DATA： Preserve database structure and delete data
APPEND_DATA：Preserve database structure, preserve data
CUSTOM_PROCESSING：User defined processing
ERROR_WHEN_DATA_EXISTS：When there is data, an error is reported

custom_sql [String]

When data_save_mode selects CUSTOM_PROCESSING, you should fill in the CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be executed. SQL will be executed before synchronization tasks.

datetime_format [String]

User-defined format string used to convert LocalDateTime fields to strings.

Use this option when you want to specify a custom datetime format that matches one of the predefined values in DateTimeUtils.Formatter (e.g. yyyy-MM-dd HH:mm:ss, yyyyMMddHHmmss, etc.).

Example values:

• yyyy-MM-dd HH:mm:ss
• yyyy-MM-dd HH:mm:ss.SSSSSS
• yyyy.MM.dd HH:mm:ss
• yyyy/MM/dd HH:mm:ss
• yyyy/M/d HH:mm
• yyyy-M-d HH:mm
• yyyy/M/d HH:mm:ss
• yyyy-M-d HH:mm:ss
• yyyyMMddHHmmss

Default: yyyy-MM-dd HH:mm:ss

tunnel_endpoint [String]

Specifies the custom endpoint URL for the MaxCompute Tunnel service.

By default, the endpoint is automatically inferred from the configured region.

This option allows you to override the default behavior and use a custom Tunnel endpoint. If not specified, the connector will use the region-based default Tunnel endpoint.

In general, you do not need to set tunnel_endpoint. It is only needed for custom networking, debugging, or local development.

Example values:

• https://dt.cn-hangzhou.maxcompute.aliyun.com
• https://dt.ap-southeast-1.maxcompute.aliyun.com
• http://maxcompute:8080

Default: Not set (auto-inferred from region)

insert_strategy [string]

If insert_strategy is set to upload, insert operations use an upload session. If set to upsert, insert operations use an upsert session. Upsert sessions require a primary key.

Note: Using upload sessions for insert operations alongside update or delete operations may cause insert records to appear in the table later than expected. When a primary key is present, it is recommended to set insert_strategy to upsert to ensure consistent upsert behavior.

common options

Sink plugin common parameters, please refer to Sink Common Options for details.

Examples

sink {
  Maxcompute {
    accessId="<your access id>"
    accesskey="<your access Key>"
    endpoint="<http://service.odps.aliyun.com/api>"
    project="<your project>"
    table_name="<your table name>"
    #partition_spec="<your partition spec>"
    #overwrite = false
  }
}

Changelog