docs/en/14-reference/05-connector/30-python.md
import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import RequestId from "../../assets/resources/_request_id.mdx";
taospy is the official TDengine python connector, which provides multiple access interfaces for database writing, querying, subscribing, etc.
The installation command is as follows:
# Native connection
pip3 install taospy
# WebSocket connection, optional installation
pip3 install taos-ws-py
The connector code is open sourced and hosted on Github Taos Connector Python.
taospy provides two connection methods, and we recommend using WebSocket connection.
For a detailed introduction of the connection method, please refer to: Connection Method
In addition to encapsulating Native interface, taospy also provides compliance with the Python Data Access Specification (PEP 249) The programming interface.
This makes it easy to integrate taospy with many third-party tools, such as SQLAlchemy and pandas.
The method of establishing a connection directly with the server using the native interface provided by the client driver is referred to as "Native Connection" in the following text; The method of establishing a connection with the server using the WebSocket interface provided by the taosAdapter is referred to as a "WebSocket connection" in the following text.
:::note
:::
Supports Python 3.0 and above.
-The platforms supported by native connections are consistent with those supported by the TDengine client driver. -WebSocket connections support all platforms that can run Python.
Python Connector historical versions (it is recommended to use the latest version of 'taopsy'):
| Python Connector Version | Major Changes | TDengine Version |
|---|---|---|
| 2.8.8 | Support TOTP authentication and token authentication | - |
| 2.8.6 | Support for pandas' read_Sql_table, to_Sql, and read_Sql interface calls | - |
| 2.8.5 | Support the SQLAlchemy feature of taos-ws-py | - |
| 2.8.4 | Support DBUtils connection pool. | - |
| 2.8.3 | Support BLOB data type. | - |
| 2.8.2 | The connection parameter settings support cross-platform compatibility. | - |
| 2.8.1 | Add two functions to set the connect property | - |
| 2.8.0 | Remove Apache Superset Driver | - |
| 2.7.23 | Supports DECIMAL data type | - |
| 2.7.22 | Supports Python 3.12 and above | - |
| 2.7.21 | Supports native STMT2 writing | - |
| 2.7.19 | Supports Apache Superset connection to TDengine Cloud data source | - |
| 2.7.18 | Supports Apache SuperSet BI Tools | - |
| 2.7.16 | Adds subscription configuration (session.timeout.ms, Max.roll.interval.ms) | - |
| 2.7.15 | Supports VARBINARY and GEOMETRY data types | - |
| 2.7.14 | Fixes known issues | - |
| 2.7.13 | Adds tmq synchronous submission offset interface | - |
| 2.7.12 | 1. Adds support for VARBINARY type (STMT currently does not support VARBINARY) | |
| 2 Improves query performance (thanks to contributor hadrianl ) | 3.1.1.2 and higher | |
| 2.7.9 | Data subscription supports obtaining and resetting consumption progress | 3.0.2.6 and higher |
| 2.7.8 | Adds 'executioner_many' | 3.0.0.0 and higher |
WebSocket Connector Historical Versions:
| WebSocket Connector Version | Major Changes | TDengine Version |
|---|---|---|
| 0.6.5 | Support TOTP authentication and token authentication | - |
| 0.6.4 | Support reporting connector version information | - |
| 0.6.3 | Support configuring the response timeout for WebSocket connections (excluding data subscription) | - |
| 0.6.2 | Fixes memory leaks | - |
| 0.6.1 | 1. Support BLOB data type |
The Python connector may generate 4 types of exceptions:
| Error Type | Description | Suggested Actions |
|---|---|---|
| InterfaceError | taosc version too low, does not support the used interface | Please check the TDengine client version |
| ConnectionError | Database connection error | Please check the TDengine server status and connection parameters |
| DatabaseError | Database error | Please check the TDengine server version and upgrade the Python connector to the latest version |
| OperationalError | Operation error | API usage error, please check your code |
| ProgrammingError | Interface call error | Please check if the submitted data is correct |
| StatementError | stmt related exception | Please check if the binding parameters match the SQL |
| ResultError | Operation data error | Please check if the operation data matches the data type in the database |
| SchemalessError | schemaless related exception | Please check the data format and corresponding protocol type are correct |
| TmqError | tmq related exception | Please check if the Topic and consumer configuration are correct |
In Python, exceptions are usually handled using try-expect. For more on exception handling, refer to Python Errors and Exceptions Documentation.
For other TDengine module errors, please refer to Error Codes
All database operations in the Python Connector, if an exception occurs, will be thrown directly. The application is responsible for handling exceptions. For example:
{{#include docs/examples/python/handle_exception.py}}
TDengine currently supports timestamp, numeric, character, boolean types, and the corresponding Python type conversions are as follows:
| TDengine DataType | Python DataType |
|---|---|
| TIMESTAMP | datetime |
| INT | int |
| BIGINT | int |
| FLOAT | float |
| DOUBLE | int |
| SMALLINT | int |
| TINYINT | int |
| BOOL | bool |
| BINARY | str |
| NCHAR | str |
| JSON | str |
| GEOMETRY | bytearray |
| VARBINARY | bytearray |
| DECIMAL | Decimal |
| BLOB | bytearray |
| Example Program Link | Example Program Content |
|---|---|
| bind_multi.py | Parameter binding, bind multiple rows at once |
| bind_row.py | Parameter binding, bind one row at a time |
| insert_lines.py | InfluxDB line protocol writing |
| json_tag.py | Using JSON type tags |
| tmq_consumer.py | tmq subscription |
| native_all_type_query.py | Example supporting all types |
| native_all_type_stmt.py | Parameter binding example supporting all types |
| test_stmt2.py | Example of STMT2 writing |
| Example program source code can be found at: |
Due to the current imperfect support for nanoseconds in Python (see the links below), the current implementation returns an integer when nanosecond precision is used, rather than the datetime type returned for ms and us. Application developers need to handle this themselves, and it is recommended to use pandas' to_datetime(). If Python officially fully supports nanoseconds in the future, the Python connector may modify the relevant interfaces.
Feel free to ask questions or report issues.
[+<protocol>]://[<username>:<password>@][<host1>:<port1>[,...<hostN>:<portN>]][/<database>][?<key1>=<value1>[&...<keyN>=<valueN>]]
|-----------|---|----------|-----------|-------------------------------------|------------|--------------------------------------|
| protocol | | username | password | addresses | database | params |
ws://localhost:6041 establishes a connection using the WebSocket protocol.
[::1] or [2001:db8:1234:5678::1]) to avoid port number parsing conflicts.
ws://host1:6041,host2:6041 or ws:// (equivalent to ws://localhost:6041).token: Authentication for the TDengine TSDB cloud service.timezone: Time zone, IANA format (e.g., Asia/Shanghai), defaults to the local time zone.compression: Whether to enable data compression, defaults to false.conn_retries: Maximum number of retries when a connection fails, defaults to 5.retry_backoff_ms: Initial wait time (milliseconds) when a connection fails, defaults to 200. This value increases exponentially with consecutive failures until the maximum wait time is reached.retry_backoff_max_ms: Maximum wait time (milliseconds) when a connection fails, defaults to 2000.read_timeout: WebSocket connection response timeout (seconds), excluding data subscription, defaults to 300 (5 minutes).fn connect(dsn: Option<&str>, args: Option<&PyDict>) -> PyResult<Connection>
dsn: Optional parameter, of type Option<&str>. Data Source Name (DSN), used to specify the database connection information, including protocol, username, password, host, port, database name and parameters, etc.args: Optional parameter, of type Option<&PyDict>. Provided as a Python dictionary, where all values are strings. Can be used to configure the following options:
user: Username for the databasepassword: Password for the databasehost: Host addressport: Port numberdatabase: Database nametimezone: Time zone, IANA format (e.g., Asia/Shanghai), defaults to the local time zone.compression: Whether to enable data compression, defaults to false.conn_retries: Maximum number of retries when a connection fails, defaults to 5.retry_backoff_ms: Initial wait time (milliseconds) when a connection fails, defaults to 200. This value increases exponentially with consecutive failures until the maximum wait time is reached.retry_backoff_max_ms: Maximum wait time (milliseconds) when a connection fails, defaults to 2000.read_timeout: WebSocket connection response timeout (seconds), excluding data subscription, defaults to 300 (5 minutes).totp_code: Used for Time-Based One-Time Password (TOTP) authenticationbearer_token: Used for token authenticationConnectionError exception on operation failure.fn cursor(&self) -> PyResult<Cursor>
ConnectionError exception on operation failure.fn execute(&self, sql: &str) -> PyResult<i32>
sql: SQL statement to be executed.QueryError exception on operation failure.fn execute_with_req_id(&self, sql: &str, req_id: u64) -> PyResult<i32>
sql: SQL statement to be executed.reqId: Used for issue tracking.QueryError exception on operation failure.fn query(&self, sql: &str) -> PyResult<TaosResult>
sql: SQL statement to be executed.TaosResult data set object.QueryError exception on operation failure.fn query_with_req_id(&self, sql: &str, req_id: u64) -> PyResult<TaosResult>
sql: SQL statement to be executed.reqId: Used for issue tracking.TaosResult data set object.QueryError exception on operation failure.TaosResult objects can be accessed by iterating over them to retrieve the queried data.
fn fields(&self) -> Vec<TaosField>
Vec<TaosField> Array of field information.fn field_count(&self) -> i32
i32 The number of fields.fn __iter__(slf: PyRef<Self>) -> PyRef<Self>
TaosResult object to support the iterator protocol, returning the iterator itself. Each row of data in the result set can be traversed using a for loop.PyRef<Self> iterator object itself.fn __next__(mut slf: PyRefMut<Self>) -> PyResult<Option<PyObject>>
Option<PyObject> tuple containing the values of one row of fields.fn schemaless_insert(&self, lines: Vec<String>, protocol: PySchemalessProtocol, precision: PySchemalessPrecision, ttl: i32, req_id: u64) -> PyResult<()>
lines: Array of data to be written, for specific data format refer to Schemaless Insert.protocol: Protocol type
PySchemalessProtocol::Line: InfluxDB Line Protocol.PySchemalessProtocol::Telnet: OpenTSDB Text Line Protocol.PySchemalessProtocol::Json: JSON Protocol format.precision: Time precision
PySchemalessPrecision::Hour: HourPySchemalessPrecision::Minute: MinutePySchemalessPrecision::Second SecondPySchemalessPrecision::Millisecond: MillisecondPySchemalessPrecision::Microsecond: MicrosecondPySchemalessPrecision::Nanosecond: Nanosecondttl: Table expiration time in days.reqId: Used for issue tracking.DataError or OperationalError on failure.Since version 0.5.1, an interface for binding parameters of stmt2 has been provided to achieve efficient writing.
fn stmt2_statement(&self) -> PyResult<TaosStmt2>
ConnectionError on failure.fn prepare(&mut self, sql: &str) -> PyResult<()>
sql: Precompiled SQL statement.ProgrammingError on failure.fn bind(&mut self, params: Vec<PyStmt2BindParam>) -> PyResult<()>
params: Data to bind.ProgrammingError on failure.fn execute(&mut self) -> PyResult<usize>
QueryError on failure.fn result_set(&mut self) -> PyResult<TaosResult>
TaosResult data set object.QueryError on failure.fn affect_rows(&mut self) -> PyResult<usize>
fn close(&self) -> PyResult<()>
The interface for binding parameters of the standard Stmt.
fn statement(&self) -> PyResult<TaosStmt>
ConnectionError on failure.fn prepare(&mut self, sql: &str) -> PyResult<()>
sql: Precompiled SQL statement.ProgrammingError on failure.fn set_tbname(&mut self, table_name: &str) -> PyResult<()>
tableName: Table name, to specify a database, use db_name.table_name.ProgrammingError on failure.fn set_tags(&mut self, tags: Vec<PyTagView>) -> PyResult<()>
paramsArray: Tags data.ProgrammingError on failure.fn bind_param(&mut self, params: Vec<PyColumnView>) -> PyResult<()>
paramsArray: Data to bind.ProgrammingError on failure.fn add_batch(&mut self) -> PyResult<()>
ProgrammingError on failure.fn execute(&mut self) -> PyResult<usize>
QueryError on failure.fn affect_rows(&mut self) -> PyResult<usize>
fn close(&self) -> PyResult<()>
Supported properties list for creating consumers:
fn Consumer(conf: Option<&PyDict>, dsn: Option<&str>) -> PyResult<Self>
conf: Type Option<&PyDict> optional, provided in the form of a Python dictionary, see the properties list for specific configurations.dsn: Type Option<&str> optional, Data Source Name (DSN), used to specify the location and authentication information of the database to connect to.ConsumerException on operation failure.fn subscribe(&mut self, topics: &PyList) -> PyResult<()>
topics: List of topics to subscribe to.ConsumerException on operation failure.fn unsubscribe(&mut self)
ConsumerException on operation failure.fn poll(&mut self, timeout: Option<f64>) -> PyResult<Option<Message>>
timeoutMs: Indicates the polling timeout in milliseconds.Message data corresponding to each topic.ConsumerException on operation failure.fn commit(&mut self, message: &mut Message) -> PyResult<()>
message: Type Message, the offset of the currently processed message.ConsumerException on operation failure.fn assignment(&mut self) -> PyResult<Option<Vec<TopicAssignment>>>
Vec<TopicAssignment>, i.e., all partitions currently assigned to the consumer.ConsumerException on operation failure.fn seek(&mut self, topic: &str, vg_id: i32, offset: i64) -> PyResult<()>
topic: Subscribed topic.vg_id: vgroupid.offset: The offset to be set.ConsumerException on operation failure.fn committed(&mut self, topic: &str, vg_id: i32) -> PyResult<i64>
topic: Subscribed topic.vg_id: vgroupid.i64, the last committed offset of the partition.ConsumerException on operation failure.fn position(&mut self, topic: &str, vg_id: i32) -> PyResult<i64>
topic: Subscribed topic.vg_id: vgroupid.i64, the last committed offset of the partition.ConsumerException on operation failure.fn close(&mut self)
ConsumerException on operation failure.def connect(*args, **kwargs):
kwargs: Provided in the form of a Python dictionary, can be used to set
user: Username for the databasepassword: Password for the database.host: Host addressport: Port numberdatabase: Database nametimezone: Time zonetotp_code: Used for Time-Based One-Time Password (TOTP) authenticationbearer_token: Used for token authenticationTaosConnection connection object.AttributeError or ConnectionError if operation fails.def cursor(self)
def execute(self, operation, req_id: Optional[int] = None)
operation: SQL statement to be executed.reqId: Used for issue tracking.ProgrammingError if operation fails.def query(self, sql: str, req_id: Optional[int] = None) -> TaosResult
sql: SQL statement to be executed.reqId: Used for issue tracking.TaosResult data set object.ProgrammingError if operation fails.TaosResult object can be iterated over to retrieve queried data.
def fields(&self)
TaosFields list of field information.def field_count(&self)
def fetch_all_into_dict(self)
def __iter__(self)
TaosResult object to support the iterator protocol, returning the iterator itself. Each row of data in the result set can be traversed using a for loop.def __next__(self)
def schemaless_insert(&self, lines: List[str], protocol: SmlProtocol, precision: SmlPrecision, req_id: Optional[int] = None, ttl: Optional[int] = None) -> int:
lines: Array of data to be inserted, specific data format for schemaless can refer to Schemaless Insertion.protocol: Protocol type
SmlProtocol.LINE_PROTOCOL: InfluxDB Line Protocol.SmlProtocol.TELNET_PROTOCOL: OpenTSDB text line protocol.SmlProtocol.JSON_PROTOCOL: JSON protocol formatprecision: Time precision
SmlPrecision.Hour: HourSmlPrecision.Minute: MinuteSmlPrecision.Second SecondSmlPrecision.Millisecond: MillisecondSmlPrecision.Microsecond: MicrosecondSmlPrecision.Nanosecond: Nanosecondttl: Table expiration time in days.reqId: Used for issue tracking.SchemalessError if operation fails.def statement2(self, sql=None, option=None)
sql: The bound SQL statement will call the prepare function if it is not emptyoption Pass in TaoStmt2Option class instanceConnectionError on failuredef prepare(self, sql)
sql: Precompiled SQL statementStatementError on failuredef bind_param(self, tbnames, tags, datas)
tbnames:Bind table name array, data type is listtags: Bind tag column value array, data type is listdatas: Bind data column value array, data type of listStatementError on failuredef bind_param_with_tables(self, tables)
tables: BindTable Independent table object arrayStatementError on failuredef execute(self) -> int:
QueryError on failuredef result(self)
def close(self)
def Consumer(configs)
configs: Provided in the form of a Python dictionary, see the properties list for specific configurations.TmqError exception on failure.def subscribe(self, topics)
topics: List of topics to subscribe to.TmqError exception on failure.def unsubscribe(self)
TmqError exception on failure.def poll(self, timeout: float = 1.0)
timeout: Polling timeout in milliseconds.Message data for each topic.TmqError exception on failure.def commit(self, message: Message = None, offsets: [TopicPartition] = None)
message: Type Message, the offset of the currently processed message.offsets: Type [TopicPartition], submits the offset of a batch of messages.TmqError exception on failure.def assignment(self)
[TopicPartition], all partitions currently assigned to the consumer.TmqError exception on failure.def seek(self, partition)
partition: Offset to be set.
topic: Subscribed topicpartition: Partitionoffset: OffsetTmqError exception on failure.def committed(self, partitions)
partition: Offset to be set.
topic: Subscribed topicpartition: Partitionpartition, the last committed offset of the partition.TmqError exception on failure.def position(self, partitions)
partition: Offset to be set.
topic: Subscribed topicpartition: Partitionpartition, the last committed offset of the partition.TmqError exception on failure.def close(self)
TmqError exception on failure.