docs/en/faq/loading/Stream_load_faq.md
Stream Load does not support identifying column names held in the first few rows of a CSV-formatted file. Stream Load considers the first few rows to be normal data like the other rows.
In v2.5 and earlier, Stream Load does not support skipping the first few rows of a CSV file during data reading. If the first few rows of the CSV file you want to load hold column names, take one of the following actions:
sed -i '1d' filename to delete the first few rows of the CSV file.-H "where: <column_name> != '<column_name>'" to filter out the first few rows of the CSV file. <column_name> is any of the column names held in the first few rows. Note that StarRocks first transforms and then filters the source data. Therefore, if the column names in the first few rows fail to be transformed into their matching destination data types, NULL values are returned for them. This means the destination StarRocks table cannot contain columns that are set to NOT NULL.-H "max_filter_ratio:0.01" to set a maximum error tolerance that is 1% or lower but can tolerate a few error rows, thereby allowing StarRocks to ignore the data transformation failures in the first few rows. In this case, the Stream Load job can still succeed even if ErrorURL is returned to indicate error rows. Do not set max_filter_ratio to a large value. If you set max_filter_ratio to a large value, some important data quality issues may be missed.From v3.0 onwards, Stream Load supports the skip_header parameter, which specifies whether to skip the first few rows of a CSV file. For more information,see CSV parameters.
StarRocks supports transforming data at loading. For more information, see Transform data at loading.
Suppose that you want to load a CSV-formatted file named TEST and the file consists of four columns, NO, DATE, VERSION, and PRICE, among which the data from the DATE column is in a non-standard format such as 202106.00. If you want to use DATE as the partition column in StarRocks, you need to first create a StarRocks table, for example, one that consists of the following four columns: NO, VERSION, PRICE, and DATE. Then, you need to specify the data type of the DATE column of the StarRocks table as DATE, DATETIME, or INT. Finally, when you create a Stream Load job, you need to specify the following setting in the load command or statement to transform data from the source DATE column's data type to the destination column's data type:
-H "columns: NO,DATE_1, VERSION, PRICE, DATE=LEFT(DATE_1,6)"
In the preceding example, DATE_1 can be considered to be a temporarily named column mapping the destination DATE column, and the final results loaded into the destination DATE column are computed by the left() function. Note that you must first list the temporary names of the source columns and then use functions to transform data. The functions supported are scalar functions, including non-aggregate functions and window functions.
The size of the source data file exceeds 10 GB, which is the maximum file size supported by Stream Load. Take one of the following actions:
seq -w 0 n to split the source data file into smaller files.curl -XPOST http://be_host:http_port/api/update_config?streaming_load_max_mb=<file_size> to adjust the value of the BE configuration item streaming_load_max_mb to increase the maximum file size.Use the replace function:
-H "columns: pk, temp, pd_type=replace(temp,'NULL',NULL)"
role is a reserved keyword. Use backticks to wrap the reserved keyword when you use it in names.
Example:
-H $'columns:k1,`role`'