Data Backup and Restoration

You can back up the data in your TDengine cluster and restore it in the event that data is lost or damaged.

Data Backup and Restoration Using taosdump

taosdump is an open-source tool that supports backing up data from a running TDengine cluster and restoring the backed-up data to the same or another running TDengine cluster. taosdump can back up the database as a logical data unit or back up data records within a specified time period in the database. When using taosdump, you can specify the directory path for data backup. If no directory path is specified, taosdump will default to backing up the data in the current directory.

Back Up Data with taosdump

Below is an example of using taosdump to perform data backup.

taosdump -h localhost -P 6030 -D dbname -o /file/path

After executing the above command, taosdump will connect to the TDengine cluster at localhost:6030, query all data in the database dbname, and back up the data to /file/path.

When using taosdump, if the specified storage path already contains data files, taosdump will prompt the user and exit immediately to avoid data overwriting. This means the same storage path can only be used for one backup. If you see related prompts, please operate carefully to avoid accidental data loss.

Restore Data with taosdump

To restore data files from a specified local file path to a running TDengine cluster, you can execute the taosdump command by specifying command-line parameters and the data file path. Below is an example code for taosdump performing data restoration.

taosdump -i /file/path -h localhost -P 6030

After executing the above command, taosdump will connect to the TDengine cluster at localhost:6030 and restore the data files from /file/path to the TDengine cluster.

Data Backup and Restoration in TDengine Enterprise

TDengine Enterprise implements incremental backup and recovery of data by using data subscription. The backup and recovery functions of TDengine Enterprise include the following concepts:

1. Incremental data backup: Based on TDengine's data subscription function, all data changes of the backup object (including: addition, modification, deletion, metadata change, etc.) are recorded to generate a backup file.
2. Data recovery: Use the backup file generated by incremental data backup to restore the backup object to a specified point in time.
3. Backup object: The object that the user backs up can be a database or a supertable.
4. Backup plan: The user creates a periodic backup task for the backup object. The backup plan starts at a specified time point and periodically executes the backup task at intervals of the backup cycle. Each backup task generates a backup point .
5. Backup point: Each time a backup task is executed, a set of backup files is generated. They correspond to a time point, called a backup point . The first backup point is called the initial backup point .
6. Restore task: The user selects a backup point in the backup plan and creates a restore task. The restore task starts from the initial backup point and plays back the data changes in the backup file one by one until the specified backup point ends.

Incremental Backup Example

1. The user creates a backup plan to execute the backup task every 1 day starting from 2024-08-27 00:00:00 .
2. The first backup task was executed at 2024-08-27 00:00:00, generating an initial backup point .
3. After that, the backup task is executed every 1 day, and multiple backup points are generated .
4. Users can select a backup point and create a restore task .
5. The restore task starts from the initial backup point, applies the backup points one by one, and restores to the specified backup point.

Back Up Data in TDengine Enterprise

1. In a web browser, open the taosExplorer interface for TDengine. This interface is located on port 6060 on the hostname or IP address running TDengine.
2. In the main menu on the left, click Management and open the Backup tab.
3. Under Backup Plan, click Create New Backup to define your backup plan.
   1. Database: Select the database that you want to backup.
   2. Super Table: (Optional) Select the supertable that you want to backup. If you do not select a supertable, all data in the database is backed up.
   3. Next execution time: Enter the date and time when you want to perform the initial backup for this backup plan. If you specify a date and time in the past, the initial backup is performed immediately.
   4. Backup Cycle: Specify how often you want to perform incremental backups. The value of this field must be less than the value of WAL_RETENTION_PERIOD for the specified database.
   5. Retry times: Enter how many times you want to retry a backup task that has failed, provided that the specific failure might be resolved by retrying.
   6. Retry interval: Enter the delay in seconds between retry attempts.
   7. Directory: Enter the full path of the directory in which you want to store backup files.
   8. Backup file max size: Enter the maximum size of a single backup file. If the total size of your backup exceeds this number, the backup is split into multiple files.
   9. Compression level: Select fastest for the fastest performance but lowest compression ratio, best for the highest compression ratio but slowest performance, or balanced for a combination of performance and compression.
4. Users can enable S3 dumping to upload backup files to the S3 storage service. To enable S3 dumping, the following information needs to be provided:
   1. Endpoint: The address of the S3 endpoint.
   2. Access Key ID: The access key ID for authentication.
   3. Secret Access Key: The secret access key for authentication.
   4. Bucket: The name of the target bucket.
   5. Region: The region where the bucket is located.
   6. Object Prefix: A prefix for backup file objects, similar to a directory path on S3.
   7. Backup Retention Period: The retention duration for local backups. All files older than current time - backup_retention_period must be uploaded to S3.
   8. Backup Retention Count: The number of local backups to retain. Only the latest backup_retention_size backup files are kept locally.
5. Click Confirm to create the backup plan.

You can view your backup plans and modify, clone, or delete them using the buttons in the Operation columns. Click Refresh to update the status of your plans. Note that you must stop a backup plan before you can delete it. You can also click View in the Backup File column to view the backup record points and files created by each plan.

Restore Data in TDengine Enterprise

1. Locate the backup plan containing data that you want to restore and click View in the Backup File column.
2. Determine the backup record point to which you want to restore and click the Restore icon in the Operation column.
3. Select the backup file timestamp and target database and click Confirm.

Troubleshooting

Port Access Exception

A port access exception is indicated by the following error:

Error: tmq to td task exec error

Caused by:
    [0x000B] Unable to establish connection

If you encounter this error, check whether the data source FQDN is connected and whether port 6030 is listening and accessible.

Connection Issues

A connection issue is indicated by the task failing to start and reporting the following error:

Error: tmq to td task exec error

Caused by:
    0: WebSocket internal error: IO error: failed to lookup address information: Temporary failure in name resolution
    1: IO error: failed to lookup address information: Temporary failure in name resolution
    2: failed to lookup address information: Temporary failure in name resolution

The following are some possible errors for WebSocket connections:

• "Temporary failure in name resolution": DNS resolution error. Check whether the specified IP address or FQDN can be accessed normally.
• "IO error: Connection refused (os error 111)": Port access failed. Check whether the port is configured correctly and is enabled and accessible.
• "IO error: received corrupt message": Message parsing failed. This may be because SSL was enabled using the WSS method, but the source port is not supported.
• "HTTP error: *": Confirm that you are connecting to the correct taosAdapter port and that your LSB/Nginx/Proxy has been configured correctly.
• "WebSocket protocol error: Handshake not finished": WebSocket connection error. This is typically caused by an incorrectly configured port.

WAL Configuration

A WAL configuration issue is indicated by the task failing to start and reporting the following error:

Error: tmq to td task exec error

Caused by:
    [0x038C] WAL retention period is zero

To resolve this error, modify the WAL retention period for the affected database:

ALTER DATABASE test WAL_RETENTION_PERIOD 3600;