From 98a7d5fbe00958f136851569555716945f36ba27 Mon Sep 17 00:00:00 2001 From: Leto_b Date: Wed, 6 May 2026 15:10:40 +0800 Subject: [PATCH 1/2] fix slow query of audit log version --- .../Release-history_timecho.md | 1 - .../Table/User-Manual/Audit-Log_timecho.md | 23 ++++++++-------- .../Release-history_timecho.md | 1 - .../Tree/User-Manual/Audit-Log_timecho.md | 27 ++++++++++--------- .../Release-history_timecho.md | 1 - .../User-Manual/Audit-Log_timecho.md | 22 ++++++++------- .../Release-history_timecho.md | 1 - .../latest/User-Manual/Audit-Log_timecho.md | 24 +++++++++-------- .../Release-history_timecho.md | 1 - .../Table/User-Manual/Audit-Log_timecho.md | 22 ++++++++------- .../Release-history_timecho.md | 1 - .../Tree/User-Manual/Audit-Log_timecho.md | 24 +++++++++-------- .../Release-history_timecho.md | 1 - .../User-Manual/Audit-Log_timecho.md | 22 ++++++++------- .../Release-history_timecho.md | 1 - .../latest/User-Manual/Audit-Log_timecho.md | 22 ++++++++------- 16 files changed, 100 insertions(+), 94 deletions(-) diff --git a/src/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md b/src/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md index 1331615f1..d1537c182 100644 --- a/src/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md +++ b/src/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ Version 2.0.8.2 adds support for modifying the full path of time series in the t - Storage Module: The tree model supports [modifying the full name of time series](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称) and [changing the data type of time series](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型). - Storage Module: The table model supports [modifying column data types](../Basic-Concept/Table-Management_timecho.md#_1-5-修改表) and [customizing the Time column name](../Basic-Concept/Table-Management_timecho.md#_1-1-创建表). - Interface Module: Adds support for the [ODBC Driver](../API/Programming-ODBC_timecho.md); the Python SessionDataset supports fetching DataFrames in batches; the MQTT service is externalized, and a new system table named Services is added for service queries. -- System Module: The [audit log](../User-Manual/Audit-Log_timecho.md) now supports statistics on the number of slow requests. - AI Node: The table model supports adaptive [covariate inference](../AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理). - Stream Processing Module: The tree model data synchronization PIPE statement supports specifying multiple precise paths. diff --git a/src/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md b/src/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md index d62a08f68..c2b4fcaa2 100644 --- a/src/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md +++ b/src/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md @@ -52,8 +52,7 @@ Edit the `iotdb-system.properties` file to enable audit logging using the follow | `audit_log_batch_interval_in_ms` | Batch write interval for audit logs | Long | 1000 | Hot Reload | | `audit_log_batch_max_queue_bytes` | Maximum byte size of the queue for batch processing audit logs. Subsequent write operations will be blocked when this threshold is exceeded. | Long | 268435456 | Hot Reload | -* V2.0.8.2 - +* V2.0.9.2 | Parameter Name | Description | Data Type | Default Value | Activation Method | |-------------------------------------------|------------------------------------------------------------------------------------------------------------|-----------|-------------------------------|-------------------| @@ -150,15 +149,17 @@ It costs 0.011s ``` -* Query audit event records with types 'slow operation' and 'login' +* Query audit event records with types 'slow operation' ```SQL -IoTDB:__audit> select * from audit_log where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN' limit 1 -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result|database|sql_string| log| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -|2026-01-23T11:47:42.136+08:00| node_1| u_none| user1| 127.0.0.1| LOGIN| CONTROL| null| GLOBAL| false| | |User user1 (ID=-1) login failed with code: 804, Authentication failed.| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -Total line number = 1 -It costs 0.033s +IoTDB:__audit> select * from audit_log where audit_event_type='SLOW_OPERATION' limit 3 ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result| database| sql_string| log| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|2026-05-06T14:57:57.468+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true| | show databases| SLOW_QUERY: cost 10 ms, show databases| +|2026-05-06T14:58:38.149+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| DML| [INSERT]| OBJECT| true|database1|INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17')|Execution: INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17') cost 329 ms, with status code: TSStatus(code:200, message:)| +|2026-05-06T14:58:45.534+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true|database1| select * from table1| SLOW_QUERY: cost 121 ms, select * from table1| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +Total line number = 3 +It costs 0.026s ``` \ No newline at end of file diff --git a/src/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md b/src/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md index faa4ba63f..d8d39def5 100644 --- a/src/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md +++ b/src/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ Version 2.0.8.2 adds support for modifying the full path of time series in the t - Storage Module: The tree model supports [modifying the full name of time series](../Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称) and [changing the data type of time series](../Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型). - Storage Module: The table model supports [modifying column data types](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-5-修改表) and [customizing the Time column name](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-1-创建表). - Interface Module: Adds support for the [ODBC Driver](../API/Programming-ODBC_timecho.md); the Python SessionDataset supports fetching DataFrames in batches; the MQTT service is externalized, and a new system table named Services is added for service queries. -- System Module: The [audit log](../User-Manual/Audit-Log_timecho.md) now supports statistics on the number of slow requests. - AI Node: The table model supports adaptive [covariate inference](../../latest-Table/AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理). - Stream Processing Module: The tree model data synchronization PIPE statement supports specifying multiple precise paths. diff --git a/src/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md b/src/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md index 2264aaca2..f63be18b4 100644 --- a/src/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md +++ b/src/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md @@ -38,7 +38,6 @@ Audit logs serve as the record credentials of a database, enabling tracking of v Edit the `iotdb-system.properties` file to enable audit logging using the following parameters: - * V2.0.8.1 | Parameter Name | Description | Data Type | Default Value | Activation Method | @@ -52,8 +51,7 @@ Edit the `iotdb-system.properties` file to enable audit logging using the follow | `audit_log_batch_interval_in_ms` | Batch write interval for audit logs | Long | 1000 | Hot Reload | | `audit_log_batch_max_queue_bytes` | Maximum byte size of the queue for batch processing audit logs. Subsequent write operations will be blocked when this threshold is exceeded. | Long | 268435456 | Hot Reload | -* V2.0.8.2 - +* V2.0.9.2 | Parameter Name | Description | Data Type | Default Value | Activation Method | |-------------------------------------------|------------------------------------------------------------------------------------------------------------|-----------|-------------------------------|-------------------| @@ -74,6 +72,7 @@ Edit the `iotdb-system.properties` file to enable audit logging using the follow - When the parameters `auditable_dml_event_type`, `auditable_ddl_event_type`, `auditable_query_event_type`, or `auditable_control_event_type` are set to `OBJECT_AUTHENTICATION`, the corresponding event types will be recorded in the audit log. - When the parameters `auditable_dml_event_type`, `auditable_ddl_event_type`, `auditable_query_event_type`, or `auditable_control_event_type` are set to `SLOW_OPERATION`, only the corresponding event types whose execution time exceeds the value of the `slow_query_threshold` parameter (default: 3000 ms) will be recorded in the audit log. The value of the `slow_query_threshold` parameter can be configured in the `iotdb-system.properties` file. + ## 3. Access Methods Supports direct reading of audit logs via SQL. @@ -150,15 +149,17 @@ Total line number = 4 It costs 0.024s ``` -* Query audit records for user 'u_0' on node 'node_1' with event types 'SLOW_OPERATION' and 'LOGIN' +* Query audit records for user 'u_0' on node 'node_1' with event types 'SLOW_OPERATION' ```SQL -IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN' limit 1 align by device -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -| Time| Device|result|privilege_level|privilege_type|database|operation_type| log|sql_string|audit_event_type|cli_hostname|username| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -|2026-01-23T11:42:23.636+08:00|root.__audit.log.node_1.u_0| true| GLOBAL| null| | CONTROL|IoTDB: Login status: Login successfully. User root (ID=0), opens Session-1-root:127.0.0.1:51308| | LOGIN| 127.0.0.1| root| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -Total line number = 1 -It costs 0.021s -``` \ No newline at end of file +IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' align by device ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +| Time| Device|result|privilege_level|privilege_type|database|operation_type| log| sql_string|audit_event_type|cli_hostname|username| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +|2026-05-06T14:43:55.088+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [READ_DATA]| | QUERY| SLOW_QUERY: cost 60 ms, select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device|select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:08.715+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML| Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2') cost 290 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2')| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:11.684+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML|Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4') cost 6 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4')| SLOW_OPERATION| 127.0.0.1| root| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +Total line number = 3 +It costs 0.010s +``` diff --git a/src/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md b/src/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md index 1331615f1..d1537c182 100644 --- a/src/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md +++ b/src/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ Version 2.0.8.2 adds support for modifying the full path of time series in the t - Storage Module: The tree model supports [modifying the full name of time series](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称) and [changing the data type of time series](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型). - Storage Module: The table model supports [modifying column data types](../Basic-Concept/Table-Management_timecho.md#_1-5-修改表) and [customizing the Time column name](../Basic-Concept/Table-Management_timecho.md#_1-1-创建表). - Interface Module: Adds support for the [ODBC Driver](../API/Programming-ODBC_timecho.md); the Python SessionDataset supports fetching DataFrames in batches; the MQTT service is externalized, and a new system table named Services is added for service queries. -- System Module: The [audit log](../User-Manual/Audit-Log_timecho.md) now supports statistics on the number of slow requests. - AI Node: The table model supports adaptive [covariate inference](../AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理). - Stream Processing Module: The tree model data synchronization PIPE statement supports specifying multiple precise paths. diff --git a/src/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md b/src/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md index 4020434ef..c2b4fcaa2 100644 --- a/src/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md +++ b/src/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md @@ -52,7 +52,7 @@ Edit the `iotdb-system.properties` file to enable audit logging using the follow | `audit_log_batch_interval_in_ms` | Batch write interval for audit logs | Long | 1000 | Hot Reload | | `audit_log_batch_max_queue_bytes` | Maximum byte size of the queue for batch processing audit logs. Subsequent write operations will be blocked when this threshold is exceeded. | Long | 268435456 | Hot Reload | -* V2.0.8.2 +* V2.0.9.2 | Parameter Name | Description | Data Type | Default Value | Activation Method | |-------------------------------------------|------------------------------------------------------------------------------------------------------------|-----------|-------------------------------|-------------------| @@ -149,15 +149,17 @@ It costs 0.011s ``` -* Query audit event records with types 'slow operation' and 'login' +* Query audit event records with types 'slow operation' ```SQL -IoTDB:__audit> select * from audit_log where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN' limit 1 -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result|database|sql_string| log| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -|2026-01-23T11:47:42.136+08:00| node_1| u_none| user1| 127.0.0.1| LOGIN| CONTROL| null| GLOBAL| false| | |User user1 (ID=-1) login failed with code: 804, Authentication failed.| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -Total line number = 1 -It costs 0.033s +IoTDB:__audit> select * from audit_log where audit_event_type='SLOW_OPERATION' limit 3 ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result| database| sql_string| log| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|2026-05-06T14:57:57.468+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true| | show databases| SLOW_QUERY: cost 10 ms, show databases| +|2026-05-06T14:58:38.149+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| DML| [INSERT]| OBJECT| true|database1|INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17')|Execution: INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17') cost 329 ms, with status code: TSStatus(code:200, message:)| +|2026-05-06T14:58:45.534+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true|database1| select * from table1| SLOW_QUERY: cost 121 ms, select * from table1| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +Total line number = 3 +It costs 0.026s ``` \ No newline at end of file diff --git a/src/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md b/src/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md index faa4ba63f..d8d39def5 100644 --- a/src/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md +++ b/src/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ Version 2.0.8.2 adds support for modifying the full path of time series in the t - Storage Module: The tree model supports [modifying the full name of time series](../Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称) and [changing the data type of time series](../Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型). - Storage Module: The table model supports [modifying column data types](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-5-修改表) and [customizing the Time column name](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-1-创建表). - Interface Module: Adds support for the [ODBC Driver](../API/Programming-ODBC_timecho.md); the Python SessionDataset supports fetching DataFrames in batches; the MQTT service is externalized, and a new system table named Services is added for service queries. -- System Module: The [audit log](../User-Manual/Audit-Log_timecho.md) now supports statistics on the number of slow requests. - AI Node: The table model supports adaptive [covariate inference](../../latest-Table/AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理). - Stream Processing Module: The tree model data synchronization PIPE statement supports specifying multiple precise paths. diff --git a/src/UserGuide/latest/User-Manual/Audit-Log_timecho.md b/src/UserGuide/latest/User-Manual/Audit-Log_timecho.md index d6006290d..f63be18b4 100644 --- a/src/UserGuide/latest/User-Manual/Audit-Log_timecho.md +++ b/src/UserGuide/latest/User-Manual/Audit-Log_timecho.md @@ -51,7 +51,7 @@ Edit the `iotdb-system.properties` file to enable audit logging using the follow | `audit_log_batch_interval_in_ms` | Batch write interval for audit logs | Long | 1000 | Hot Reload | | `audit_log_batch_max_queue_bytes` | Maximum byte size of the queue for batch processing audit logs. Subsequent write operations will be blocked when this threshold is exceeded. | Long | 268435456 | Hot Reload | -* V2.0.8.2 +* V2.0.9.2 | Parameter Name | Description | Data Type | Default Value | Activation Method | |-------------------------------------------|------------------------------------------------------------------------------------------------------------|-----------|-------------------------------|-------------------| @@ -149,15 +149,17 @@ Total line number = 4 It costs 0.024s ``` -* Query audit records for user 'u_0' on node 'node_1' with event types 'SLOW_OPERATION' and 'LOGIN' +* Query audit records for user 'u_0' on node 'node_1' with event types 'SLOW_OPERATION' ```SQL -IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN' limit 1 align by device -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -| Time| Device|result|privilege_level|privilege_type|database|operation_type| log|sql_string|audit_event_type|cli_hostname|username| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -|2026-01-23T11:42:23.636+08:00|root.__audit.log.node_1.u_0| true| GLOBAL| null| | CONTROL|IoTDB: Login status: Login successfully. User root (ID=0), opens Session-1-root:127.0.0.1:51308| | LOGIN| 127.0.0.1| root| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -Total line number = 1 -It costs 0.021s -``` \ No newline at end of file +IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' align by device ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +| Time| Device|result|privilege_level|privilege_type|database|operation_type| log| sql_string|audit_event_type|cli_hostname|username| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +|2026-05-06T14:43:55.088+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [READ_DATA]| | QUERY| SLOW_QUERY: cost 60 ms, select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device|select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:08.715+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML| Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2') cost 290 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2')| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:11.684+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML|Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4') cost 6 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4')| SLOW_OPERATION| 127.0.0.1| root| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +Total line number = 3 +It costs 0.010s +``` diff --git a/src/zh/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md b/src/zh/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md index f63940394..828968616 100644 --- a/src/zh/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md +++ b/src/zh/UserGuide/Master/Table/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ V2.0.8.2 版本新增树模型修改序列全名功能，表模型支持自定 - 存储模块：树模型支持[修改序列全名](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称)，支持[更改序列数据类型](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型) - 存储模块：表模型支持[更改列数据类型](../Basic-Concept/Table-Management_timecho.md#_1-5-修改表)，支持[自定义 Time 列列名](../Basic-Concept/Table-Management_timecho.md#_1-1-创建表) - 接口模块：支持 [ODBC Driver](../API/Programming-ODBC_timecho.md), Python SessionDataset 支持分批获取 DataFrame，MQTT 服务外置并新增系统表 Services 提供服务查询 -- 系统模块：[审计日志](../User-Manual/Audit-Log_timecho.md)支持慢请求个数统计 - AINode：表模型支持自适应[协变量推理](../AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理) - 流处理模块：树模型数据同步 pipe 语句中支持填写多个精确路径的 path diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md b/src/zh/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md index 2e6615ed2..0f0df7463 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Audit-Log_timecho.md @@ -53,7 +53,7 @@ | `audit_log_batch_max_queue_bytes` | 用于批量处理审计日志的队列最大字节数。当队列大小超过此值时，后续的写入操作将被阻塞。 | Long | 268435456 | 热加载 | -* V2.0.8.2 +* V2.0.9.2 | 参数名称 | 参数描述 | 数据类型 | 默认值 | 生效方式 | |-----------------------------------|------------------------------------------------------------------------------------------------------| ---------- | ------------------------ | ---------- | @@ -148,14 +148,16 @@ Total line number = 1 It costs 0.011s ``` -* 查询审计事件类型为慢操作和登录的记录 +* 设置 slow_query_threshold = 1 (ms)，查询审计事件类型为慢操作的记录 ```SQL -IoTDB:__audit> select * from __audit.audit_log where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN' limit 1 -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result|database|sql_string| log| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -|2026-01-23T11:47:42.136+08:00| node_1| u_none| user1| 127.0.0.1| LOGIN| CONTROL| null| GLOBAL| false| | |User user1 (ID=-1) login failed with code: 804, Authentication failed.| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -Total line number = 1 -It costs 0.033s +IoTDB:__audit> select * from audit_log where audit_event_type='SLOW_OPERATION' limit 3 ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result| database| sql_string| log| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|2026-05-06T14:57:57.468+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true| | show databases| SLOW_QUERY: cost 10 ms, show databases| +|2026-05-06T14:58:38.149+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| DML| [INSERT]| OBJECT| true|database1|INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17')|Execution: INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17') cost 329 ms, with status code: TSStatus(code:200, message:)| +|2026-05-06T14:58:45.534+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true|database1| select * from table1| SLOW_QUERY: cost 121 ms, select * from table1| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +Total line number = 3 +It costs 0.026s ``` \ No newline at end of file diff --git a/src/zh/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md b/src/zh/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md index 554c77b0f..2ebe2f1df 100644 --- a/src/zh/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md +++ b/src/zh/UserGuide/Master/Tree/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ V2.0.8.2 版本新增树模型修改序列全名功能，表模型支持自定 - 存储模块：树模型支持[修改序列全名](../Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称)，支持[更改序列数据类型](../Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型) - 存储模块：表模型支持[更改列数据类型](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-5-修改表)，支持[自定义 Time 列列名](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-1-创建表) - 接口模块：支持 [ODBC Driver](../API/Programming-ODBC_timecho.md), Python SessionDataset 支持分批获取 DataFrame，MQTT 服务外置并新增系统表 Services 提供服务查询 -- 系统模块：[审计日志](../User-Manual/Audit-Log_timecho.md)支持慢请求个数统计 - AINode：表模型支持自适应[协变量推理](../../latest-Table/AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理) - 流处理模块：树模型数据同步 pipe 语句中支持填写多个精确路径的 path diff --git a/src/zh/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md b/src/zh/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md index cfc2e0fe0..14e0f28cb 100644 --- a/src/zh/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md +++ b/src/zh/UserGuide/Master/Tree/User-Manual/Audit-Log_timecho.md @@ -53,7 +53,7 @@ | `audit_log_batch_max_queue_bytes` | 用于批量处理审计日志的队列最大字节数。当队列大小超过此值时，后续的写入操作将被阻塞。 | Long | 268435456 | 热加载 | -* V2.0.8.2 +* V2.0.9.2 | 参数名称 | 参数描述 | 数据类型 | 默认值 | 生效方式 | |-----------------------------------|------------------------------------------------------------------------------------------------------| ---------- | ------------------------ | ---------- | @@ -150,15 +150,17 @@ Total line number = 4 It costs 0.024s ``` -* 查询某个用户在某个数据节点上审计事件类型为慢操作和登录的记录 +* 设置 slow_query_threshold = 1 (ms)， 查询某个用户在某个数据节点上审计事件类型为慢操作的记录 ```SQL -IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -| Time| Device|result|privilege_level|privilege_type|database|operation_type| log|sql_string|audit_event_type|cli_hostname|username| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -|2026-01-23T11:42:23.636+08:00|root.__audit.log.node_1.u_0| true| GLOBAL| null| | CONTROL|IoTDB: Login status: Login successfully. User root (ID=0), opens Session-1-root:127.0.0.1:51308| | LOGIN| 127.0.0.1| root| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -Total line number = 1 -It costs 0.021s -``` +IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' align by device ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +| Time| Device|result|privilege_level|privilege_type|database|operation_type| log| sql_string|audit_event_type|cli_hostname|username| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +|2026-05-06T14:43:55.088+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [READ_DATA]| | QUERY| SLOW_QUERY: cost 60 ms, select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device|select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:08.715+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML| Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2') cost 290 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2')| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:11.684+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML|Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4') cost 6 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4')| SLOW_OPERATION| 127.0.0.1| root| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +Total line number = 3 +It costs 0.010s +``` \ No newline at end of file diff --git a/src/zh/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md b/src/zh/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md index f63940394..828968616 100644 --- a/src/zh/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md +++ b/src/zh/UserGuide/latest-Table/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ V2.0.8.2 版本新增树模型修改序列全名功能，表模型支持自定 - 存储模块：树模型支持[修改序列全名](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称)，支持[更改序列数据类型](../../latest/Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型) - 存储模块：表模型支持[更改列数据类型](../Basic-Concept/Table-Management_timecho.md#_1-5-修改表)，支持[自定义 Time 列列名](../Basic-Concept/Table-Management_timecho.md#_1-1-创建表) - 接口模块：支持 [ODBC Driver](../API/Programming-ODBC_timecho.md), Python SessionDataset 支持分批获取 DataFrame，MQTT 服务外置并新增系统表 Services 提供服务查询 -- 系统模块：[审计日志](../User-Manual/Audit-Log_timecho.md)支持慢请求个数统计 - AINode：表模型支持自适应[协变量推理](../AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理) - 流处理模块：树模型数据同步 pipe 语句中支持填写多个精确路径的 path diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md b/src/zh/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md index 2e6615ed2..0f0df7463 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Audit-Log_timecho.md @@ -53,7 +53,7 @@ | `audit_log_batch_max_queue_bytes` | 用于批量处理审计日志的队列最大字节数。当队列大小超过此值时，后续的写入操作将被阻塞。 | Long | 268435456 | 热加载 | -* V2.0.8.2 +* V2.0.9.2 | 参数名称 | 参数描述 | 数据类型 | 默认值 | 生效方式 | |-----------------------------------|------------------------------------------------------------------------------------------------------| ---------- | ------------------------ | ---------- | @@ -148,14 +148,16 @@ Total line number = 1 It costs 0.011s ``` -* 查询审计事件类型为慢操作和登录的记录 +* 设置 slow_query_threshold = 1 (ms)，查询审计事件类型为慢操作的记录 ```SQL -IoTDB:__audit> select * from __audit.audit_log where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN' limit 1 -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result|database|sql_string| log| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -|2026-01-23T11:47:42.136+08:00| node_1| u_none| user1| 127.0.0.1| LOGIN| CONTROL| null| GLOBAL| false| | |User user1 (ID=-1) login failed with code: 804, Authentication failed.| -+-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+--------+----------+----------------------------------------------------------------------+ -Total line number = 1 -It costs 0.033s +IoTDB:__audit> select * from audit_log where audit_event_type='SLOW_OPERATION' limit 3 ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| time|node_id|user_id|username|cli_hostname|audit_event_type|operation_type|privilege_type|privilege_level|result| database| sql_string| log| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|2026-05-06T14:57:57.468+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true| | show databases| SLOW_QUERY: cost 10 ms, show databases| +|2026-05-06T14:58:38.149+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| DML| [INSERT]| OBJECT| true|database1|INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17')|Execution: INSERT INTO table1(region, plant_id, device_id, model_id, maintenance, time, temperature, humidity, status, arrival_time) VALUES ('北京', '1001', '100', 'A', '180', '2024-11-26 13:37:00', 90.0, 35.1, true, '2024-11-26 13:37:34'), ('北京', '1001', '100', 'A', '180', '2024-11-26 13:38:00', 90.0, 35.1, true, '2024-11-26 13:38:25'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:38:00', NULL, 35.1, true, '2024-11-27 16:37:01'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:39:00', 85.0, 35.3, NULL, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:40:00', 85.0, NULL, NULL, '2024-11-27 16:37:03'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:41:00', 85.0, NULL, NULL, '2024-11-27 16:37:04'), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:42:00', NULL, 35.2, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:43:00', NULL, Null, false, Null), ('北京', '1001', '101', 'B', '180', '2024-11-27 16:44:00', NULL, Null, false, '2024-11-27 16:37:08'), ('上海', '3001', '100', 'C', '90', '2024-11-28 08:00:00', 85.0, Null, NULL, '2024-11-28 08:00:09'), ('上海', '3001', '100', 'C', '90', '2024-11-28 09:00:00', NULL, 40.9, true, NULL), ('上海', '3001', '100', 'C', '90', '2024-11-28 10:00:00', 85.0, 35.2, NULL, '2024-11-28 10:00:11'), ('上海', '3001', '100', 'C', '90', '2024-11-28 11:00:00', 88.0, 45.1, true, '2024-11-28 11:00:12'), ('上海', '3001', '101', 'D', '360', '2024-11-29 10:00:00', 85.0, NULL, NULL, '2024-11-29 10:00:13'), ('上海', '3002', '100', 'E', '180', '2024-11-29 11:00:00', NULL, 45.1, true, NULL), ('上海', '3002', '100', 'E', '180', '2024-11-29 18:30:00', 90.0, 35.4, true, '2024-11-29 18:30:15'), ('上海', '3002', '101', 'F', '360', '2024-11-30 09:30:00', 90.0, 35.2, true, NULL), ('上海', '3002', '101', 'F', '360', '2024-11-30 14:30:00', 90.0, 34.8, true, '2024-11-30 14:30:17') cost 329 ms, with status code: TSStatus(code:200, message:)| +|2026-05-06T14:58:45.534+08:00| node_1| u_0| root| 127.0.0.1| SLOW_OPERATION| QUERY| [SELECT]| OBJECT| true|database1| select * from table1| SLOW_QUERY: cost 121 ms, select * from table1| ++-----------------------------+-------+-------+--------+------------+----------------+--------------+--------------+---------------+------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +Total line number = 3 +It costs 0.026s ``` \ No newline at end of file diff --git a/src/zh/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md b/src/zh/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md index 554c77b0f..2ebe2f1df 100644 --- a/src/zh/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md +++ b/src/zh/UserGuide/latest/IoTDB-Introduction/Release-history_timecho.md @@ -34,7 +34,6 @@ V2.0.8.2 版本新增树模型修改序列全名功能，表模型支持自定 - 存储模块：树模型支持[修改序列全名](../Basic-Concept/Operate-Metadata_timecho.md#_2-4-修改时间序列名称)，支持[更改序列数据类型](../Basic-Concept/Operate-Metadata_timecho.md#_2-3-修改时间序列数据类型) - 存储模块：表模型支持[更改列数据类型](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-5-修改表)，支持[自定义 Time 列列名](../../latest-Table/Basic-Concept/Table-Management_timecho.md#_1-1-创建表) - 接口模块：支持 [ODBC Driver](../API/Programming-ODBC_timecho.md), Python SessionDataset 支持分批获取 DataFrame，MQTT 服务外置并新增系统表 Services 提供服务查询 -- 系统模块：[审计日志](../User-Manual/Audit-Log_timecho.md)支持慢请求个数统计 - AINode：表模型支持自适应[协变量推理](../../latest-Table/AI-capability/AINode_Upgrade_timecho.md#_4-1-模型推理) - 流处理模块：树模型数据同步 pipe 语句中支持填写多个精确路径的 path diff --git a/src/zh/UserGuide/latest/User-Manual/Audit-Log_timecho.md b/src/zh/UserGuide/latest/User-Manual/Audit-Log_timecho.md index cfc2e0fe0..d65afcc7c 100644 --- a/src/zh/UserGuide/latest/User-Manual/Audit-Log_timecho.md +++ b/src/zh/UserGuide/latest/User-Manual/Audit-Log_timecho.md @@ -53,7 +53,7 @@ | `audit_log_batch_max_queue_bytes` | 用于批量处理审计日志的队列最大字节数。当队列大小超过此值时，后续的写入操作将被阻塞。 | Long | 268435456 | 热加载 | -* V2.0.8.2 +* V2.0.9.2 | 参数名称 | 参数描述 | 数据类型 | 默认值 | 生效方式 | |-----------------------------------|------------------------------------------------------------------------------------------------------| ---------- | ------------------------ | ---------- | @@ -150,15 +150,17 @@ Total line number = 4 It costs 0.024s ``` -* 查询某个用户在某个数据节点上审计事件类型为慢操作和登录的记录 +* 设置 slow_query_threshold = 1 (ms)， 查询某个用户在某个数据节点上审计事件类型为慢操作的记录 ```SQL -IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -| Time| Device|result|privilege_level|privilege_type|database|operation_type| log|sql_string|audit_event_type|cli_hostname|username| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -|2026-01-23T11:42:23.636+08:00|root.__audit.log.node_1.u_0| true| GLOBAL| null| | CONTROL|IoTDB: Login status: Login successfully. User root (ID=0), opens Session-1-root:127.0.0.1:51308| | LOGIN| 127.0.0.1| root| -+-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------+----------+----------------+------------+--------+ -Total line number = 1 -It costs 0.021s +IoTDB> select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' align by device ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +| Time| Device|result|privilege_level|privilege_type|database|operation_type| log| sql_string|audit_event_type|cli_hostname|username| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +|2026-05-06T14:43:55.088+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [READ_DATA]| | QUERY| SLOW_QUERY: cost 60 ms, select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device|select * from root.__audit.log.node_1.u_0 where audit_event_type='SLOW_OPERATION' or audit_event_type='LOGIN'limit 1 align by device| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:08.715+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML| Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2') cost 290 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) values (2, false, 'v2')| SLOW_OPERATION| 127.0.0.1| root| +|2026-05-06T14:44:11.684+08:00|root.__audit.log.node_1.u_0| true| OBJECT| [WRITE_DATA]| | DML|Execution: insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4') cost 6 ms, with status code: TSStatus(code:200, message:)| insert into root.ln.wf02.wt02(timestamp, status, hardware) VALUES (3, false, 'v3'),(4, true, 'v4')| SLOW_OPERATION| 127.0.0.1| root| ++-----------------------------+---------------------------+------+---------------+--------------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+--------+ +Total line number = 3 +It costs 0.010s ``` From 0d6b00464f9546f74b29207288817608dd7b0f01 Mon Sep 17 00:00:00 2001 From: Leto_b Date: Mon, 11 May 2026 17:18:38 +0800 Subject: [PATCH 2/2] support obejct import(import-data) and export(tsfile-backup and pipe sql) from 2092 --- .../Tools-System/Data-Export-Tool_timecho.md | 86 +++++++++++- .../Tools-System/Data-Import-Tool_timecho.md | 72 ++++++++-- .../Table/User-Manual/Data-Sync_timecho.md | 128 +++++++++++++++-- .../Tools-System/Data-Export-Tool_timecho.md | 86 +++++++++++- .../Tools-System/Data-Import-Tool_timecho.md | 70 ++++++++-- .../User-Manual/Data-Sync_timecho.md | 128 +++++++++++++++-- .../Tools-System/Data-Export-Tool_timecho.md | 82 ++++++++++- .../Tools-System/Data-Import-Tool_timecho.md | 68 +++++++-- .../Table/User-Manual/Data-Sync_timecho.md | 129 ++++++++++++++++-- .../Tools-System/Data-Export-Tool_timecho.md | 82 ++++++++++- .../Tools-System/Data-Import-Tool_timecho.md | 68 +++++++-- .../User-Manual/Data-Sync_timecho.md | 129 ++++++++++++++++-- 12 files changed, 1013 insertions(+), 115 deletions(-) diff --git a/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md b/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md index 47d38c9f6..01d9eb9b7 100644 --- a/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md @@ -1,7 +1,11 @@ # Data Export ## 1. Function Overview -The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows: + +IoTDB supports two methods for data export: + +* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory. It can export the query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) files. +* PIPE Framework-based TsFileBackup: `tsfile-backup.sh/bat` is located in the `tools` directory. It can export specified data files into TsFile format using the PIPE framework. @@ -19,15 +23,18 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an - + + + + +
File containing custom SQL statements.
TsFileTsFile Open-source time-series file format.
tsfile-backup.sh/batAn open-source time-series data file format,and this script supports the Object data type.
-- Does not support the Object data type. -## 2. Detailed Features +## 2. Data Export Tool ### 2.1 Common Parameters | Short | Full Parameter | Description | Required | Default | |----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |-----------------------------------------------| @@ -91,7 +98,7 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an > export-data.sh -ft csv -sql_dialect table -t /path/export/dir -q "select * from table1" Parse error: Missing required option: db ``` -## 2.3 SQL Format +### 2.3 SQL Format #### 2.3.1 Command ```Shell # Unix/OS X @@ -168,3 +175,72 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + + +## 3. TsFileBackup Based on PIPE Framework +Since **V2.0.9.2**, IoTDB supports the `tsfile-backup.sh/bat` script. This script can automatically generate and send the `CREATE PIPE` SQL command to the server, exporting specified data files to TsFile format. + +**Notes:** +1. **To use this script, contact the Timecho Team to obtain the customized installation package `timechodb--extension`.** +2. **This script supports exporting Object-type data to TsFile files.** + + +### 3.1 Execution Commands +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] + +# Windows +> tools\windows\tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + + +### 3.2 Script Parameters +| Abbreviation | Full Name | Description | Required | Default | +| ------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- | -------- | --------------- | +| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - | +| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` | +| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` | +| `-u` | `--user` | Username for IoTDB authentication. | No | `root` | +| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` | +| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - | +| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` | +| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` | +| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - | +| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - | +| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - | +| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - | +| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - | +| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` | +| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - | +| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - | +| `-help` | `--help` | Show help information. | No | - | + + +### 3.3 Execution Examples + +Example 1: SCP Remote Export (Send Data to Another Server) + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +Example 2: Remote Object Data Export with Rate Limiting + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +Example 3: Specify Pipe Plugin JAR Directory + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` diff --git a/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md b/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md index 7c18a1e54..b2c8cfe81 100644 --- a/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md @@ -37,7 +37,8 @@ IoTDB supports three methods for data import:
-- Does not support the Object data type. +- The table model TsFile import currently only supports local import. +- Since version V2.0.9.2, the import-data.sh/bat script supports the Object data type when importing TsFile files. ## 2. Data Import Tool ### 2.1 Common Parameters @@ -184,7 +185,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -197,18 +198,19 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x and later versions > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 TsFile-Specific Parameters -| Short | Full Parameter | Description | Required | Default | -| ----------- | ----------------------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- | --------------------------- | -| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | -| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | -| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | -| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | -| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| Short | Full Parameter | Description | Required | Default | +|---------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|-----------------------| +| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | +| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | +| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | +| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | +| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| `-o` | `--object-file-paths` | Storage path for Object files.
Default mode: If this parameter is not specified, the script automatically identifies and imports Object files located in the subdirectory with the same name as the TsFile.
Absolute path mode: Explicitly specifies the external storage root directory for Object files; the tool creates an associated data index based on this path.
Note: This parameter is supported since V2.0.9.2 | No | | #### 2.4.3 Examples @@ -221,6 +223,54 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` + +**Object Type Import** + +1. Import Directory Structure + +* Default Mode + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TsFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* Specified Object Directory + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. Command Line Examples + +* Basic Import (automatically identifies Object files in the TsFile-named directory) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* Batch Directory Import (specify concurrent threads and post-success action) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* Table Model Associated Import (specify external Object storage path and target database) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile Auto-Loading This feature enables IoTDB to automatically monitor a specified directory for new TsFiles and load them into the database without manual intervention. @@ -281,7 +331,7 @@ load '' with ( |--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|--------------------------------|-------------------|----------------------------| | `database-level` | When the database corresponding to the TsFile does not exist, the database hierarchy level can be specified via the ` database-level` parameter. The default is the level set in `iotdb-common.properties`. For example, setting level=1 means the prefix path of level 1 in all time series in the TsFile will be used as the database. | Integer | `[1: Integer.MAX_VALUE]` | No | 1 | | `on-success` | Action for successfully loaded TsFiles: `delete` (delete the TsFile after successful import) or `none` (retain the TsFile in the source folder). | String | `delete / none` | No | delete | -| `model` | Specifies whether the TsFile uses the `table` model or `tree` model. This parameter becomes invalid starting from V2.0.2.1. The system automatically identifies whether the data model is tree-based or table-based. | String | `tree / table` | No | Aligns with `-sql_dialect` | +| `model` | Specifies whether the TsFile uses the `table` model or `tree` model. This parameter becomes invalid starting from V2.0.2.1. The system automatically identifies whether the data model is tree-based or table-based. | String | `tree / table` | No | Aligns with `-sql_dialect` | | `database-name` | Table model only: Target database for import. Automatically created if it does not exist. The database-name must not include the `root.` prefix (an error will occur if included). | String | `-` | No | null | | `convert-on-type-mismatch` | Whether to perform type conversion during loading if data types in the TsFile mismatch the target schema. | Boolean | `true / false` | No | true | | `verify` | Whether to validate the schema before loading the TsFile. | Boolean | `true / false` | No | true | diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index 17774e0ed..9628b6d30 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - Data synchronization between IoTDB of 1. x series version and IoTDB of 2. x and above series versions is not supported. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. -- Does not support the Object data type. +- Object-type data export is supported since version V2.0.9.2. ## 2. Usage Instructions @@ -227,17 +227,18 @@ Example Output: ```SQL IoTDB> SHOW PIPEPLUGINS -**+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): @@ -263,8 +264,8 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param - - + + @@ -288,6 +289,14 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param + + + + + + + +
Default processor plugin that does not process incoming data.
sink PluginSupportedsink PluginSupported do-nothing-sink Does not process outgoing data.
opc-ua-sink An OPC UA protocol data transfer plugin for IoTDB (V2.0.2 and above), supporting both Client/Server and Pub/Sub communication modes.
tsfile-local-sinkUsed in IoTDB (V2.0.9.2 and later) to support exporting Object data to the local file system where the IoTDB server resides.
tsfile-remote-sinkUsed in IoTDB (V2.0.9.2 and later) to support sending Object data to a remote server via the SSH/SCP protocol.
@@ -514,6 +523,77 @@ WITH SINK ( ) ``` +### 3.9 Object-Type Data Export +Since version V2.0.9.2, IoTDB supports exporting Object-type data. The following two methods are supported by configuring sink parameters: + +* **Local Mode**: Exports data to the local file system where the IoTDB server resides. +* **SCP Mode**: Sends data to a remote server via the SSH/SCP protocol. + +**Example 1: Local Export** + +You can directly use the built-in `tsfile-local-sink` plugin to create a PIPE statement for data export. For example: + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- Required, specifies the Sink type + 'sink.local.target-path' = '/data/backup/export_2024' -- Target export path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Example 2: Remote Transfer** + +1. Contact the Timecho Team to obtain the JAR package related to the `tsfile-remote-sink` plugin, such as `tsfile-remote-sink--jar-with-dependencies.jar`, and place it in a path accessible to IoTDB (e.g., all Data Node hosts). +2. Register the plugin using the following statement: + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink--jar-with-dependencies.jar'; +``` + +3. Create the PIPE statement: + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- Specifies SCP mode + 'sink.scp.host' = '192.168.1.100', -- Remote host IP + 'sink.scp.port' = '22', -- SSH port + 'sink.scp.user' = 'backup_user', -- SSH username + 'sink.scp.password' = 'ComplexPass123!', -- SSH password + 'sink.scp.remote-path' = '/remote/archive/', -- Remote storage path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Sink Exported TSFile and Object Format:** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TSFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + ## Reference: Notes You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: @@ -689,3 +769,23 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.user | User for OPC UA, specified in the configuration | String | No | root | | sink.password | Password for OPC UA, specified in the configuration | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | | sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | Optional | "null" | + + +#### tsfile-local-sink +| Parameter | Description | Value Range | Required | Default | +|-----------------------------------|-----------------------------------------------------------------------------|------------------------|----------|---------| +| sink | Component name | String: tsfile-local-sink | Yes | - | +| sink.local.target-path | Local target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Rate limit threshold (unit: bytes/second). Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + +#### tsfile-remote-sink +| Parameter | Description | Value Range | Required | Default | +|------------------------------------|-----------------------------------------------------------------------------|-------------------------|----------|---------| +| sink | Component name | String: tsfile-remote-sink | Yes | - | +| sink.scp.host | Remote host IP | String | Yes | - | +| sink.scp.port | Remote SSH port | Long | No | 22 | +| sink.scp.user | Remote SSH user | String | Yes | - | +| sink.scp.password | Remote SSH password | String | Yes | - | +| sink.scp.remote-path | Remote target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Unit: bytes/second. Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + diff --git a/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md b/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md index 47d38c9f6..01d9eb9b7 100644 --- a/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md @@ -1,7 +1,11 @@ # Data Export ## 1. Function Overview -The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows: + +IoTDB supports two methods for data export: + +* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory. It can export the query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) files. +* PIPE Framework-based TsFileBackup: `tsfile-backup.sh/bat` is located in the `tools` directory. It can export specified data files into TsFile format using the PIPE framework. @@ -19,15 +23,18 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an - + + + + +
File containing custom SQL statements.
TsFileTsFile Open-source time-series file format.
tsfile-backup.sh/batAn open-source time-series data file format,and this script supports the Object data type.
-- Does not support the Object data type. -## 2. Detailed Features +## 2. Data Export Tool ### 2.1 Common Parameters | Short | Full Parameter | Description | Required | Default | |----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |-----------------------------------------------| @@ -91,7 +98,7 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an > export-data.sh -ft csv -sql_dialect table -t /path/export/dir -q "select * from table1" Parse error: Missing required option: db ``` -## 2.3 SQL Format +### 2.3 SQL Format #### 2.3.1 Command ```Shell # Unix/OS X @@ -168,3 +175,72 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + + +## 3. TsFileBackup Based on PIPE Framework +Since **V2.0.9.2**, IoTDB supports the `tsfile-backup.sh/bat` script. This script can automatically generate and send the `CREATE PIPE` SQL command to the server, exporting specified data files to TsFile format. + +**Notes:** +1. **To use this script, contact the Timecho Team to obtain the customized installation package `timechodb--extension`.** +2. **This script supports exporting Object-type data to TsFile files.** + + +### 3.1 Execution Commands +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] + +# Windows +> tools\windows\tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + + +### 3.2 Script Parameters +| Abbreviation | Full Name | Description | Required | Default | +| ------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- | -------- | --------------- | +| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - | +| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` | +| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` | +| `-u` | `--user` | Username for IoTDB authentication. | No | `root` | +| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` | +| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - | +| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` | +| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` | +| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - | +| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - | +| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - | +| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - | +| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - | +| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` | +| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - | +| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - | +| `-help` | `--help` | Show help information. | No | - | + + +### 3.3 Execution Examples + +Example 1: SCP Remote Export (Send Data to Another Server) + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +Example 2: Remote Object Data Export with Rate Limiting + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +Example 3: Specify Pipe Plugin JAR Directory + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` diff --git a/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md b/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md index 6f8263103..b2c8cfe81 100644 --- a/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md @@ -37,7 +37,8 @@ IoTDB supports three methods for data import:
-- Does not support the Object data type. +- The table model TsFile import currently only supports local import. +- Since version V2.0.9.2, the import-data.sh/bat script supports the Object data type when importing TsFile files. ## 2. Data Import Tool ### 2.1 Common Parameters @@ -184,7 +185,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -197,18 +198,19 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x and later versions > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 TsFile-Specific Parameters -| Short | Full Parameter | Description | Required | Default | -| ----------- | ----------------------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- | --------------------------- | -| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | -| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | -| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | -| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | -| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| Short | Full Parameter | Description | Required | Default | +|---------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|-----------------------| +| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | +| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | +| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | +| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | +| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| `-o` | `--object-file-paths` | Storage path for Object files.
Default mode: If this parameter is not specified, the script automatically identifies and imports Object files located in the subdirectory with the same name as the TsFile.
Absolute path mode: Explicitly specifies the external storage root directory for Object files; the tool creates an associated data index based on this path.
Note: This parameter is supported since V2.0.9.2 | No | | #### 2.4.3 Examples @@ -221,6 +223,54 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` + +**Object Type Import** + +1. Import Directory Structure + +* Default Mode + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TsFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* Specified Object Directory + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. Command Line Examples + +* Basic Import (automatically identifies Object files in the TsFile-named directory) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* Batch Directory Import (specify concurrent threads and post-success action) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* Table Model Associated Import (specify external Object storage path and target database) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile Auto-Loading This feature enables IoTDB to automatically monitor a specified directory for new TsFiles and load them into the database without manual intervention. diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 17774e0ed..9628b6d30 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - Data synchronization between IoTDB of 1. x series version and IoTDB of 2. x and above series versions is not supported. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. -- Does not support the Object data type. +- Object-type data export is supported since version V2.0.9.2. ## 2. Usage Instructions @@ -227,17 +227,18 @@ Example Output: ```SQL IoTDB> SHOW PIPEPLUGINS -**+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): @@ -263,8 +264,8 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param - - + + @@ -288,6 +289,14 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param + + + + + + + +
Default processor plugin that does not process incoming data.
sink PluginSupportedsink PluginSupported do-nothing-sink Does not process outgoing data.
opc-ua-sink An OPC UA protocol data transfer plugin for IoTDB (V2.0.2 and above), supporting both Client/Server and Pub/Sub communication modes.
tsfile-local-sinkUsed in IoTDB (V2.0.9.2 and later) to support exporting Object data to the local file system where the IoTDB server resides.
tsfile-remote-sinkUsed in IoTDB (V2.0.9.2 and later) to support sending Object data to a remote server via the SSH/SCP protocol.
@@ -514,6 +523,77 @@ WITH SINK ( ) ``` +### 3.9 Object-Type Data Export +Since version V2.0.9.2, IoTDB supports exporting Object-type data. The following two methods are supported by configuring sink parameters: + +* **Local Mode**: Exports data to the local file system where the IoTDB server resides. +* **SCP Mode**: Sends data to a remote server via the SSH/SCP protocol. + +**Example 1: Local Export** + +You can directly use the built-in `tsfile-local-sink` plugin to create a PIPE statement for data export. For example: + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- Required, specifies the Sink type + 'sink.local.target-path' = '/data/backup/export_2024' -- Target export path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Example 2: Remote Transfer** + +1. Contact the Timecho Team to obtain the JAR package related to the `tsfile-remote-sink` plugin, such as `tsfile-remote-sink--jar-with-dependencies.jar`, and place it in a path accessible to IoTDB (e.g., all Data Node hosts). +2. Register the plugin using the following statement: + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink--jar-with-dependencies.jar'; +``` + +3. Create the PIPE statement: + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- Specifies SCP mode + 'sink.scp.host' = '192.168.1.100', -- Remote host IP + 'sink.scp.port' = '22', -- SSH port + 'sink.scp.user' = 'backup_user', -- SSH username + 'sink.scp.password' = 'ComplexPass123!', -- SSH password + 'sink.scp.remote-path' = '/remote/archive/', -- Remote storage path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Sink Exported TSFile and Object Format:** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TSFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + ## Reference: Notes You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: @@ -689,3 +769,23 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.user | User for OPC UA, specified in the configuration | String | No | root | | sink.password | Password for OPC UA, specified in the configuration | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | | sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | Optional | "null" | + + +#### tsfile-local-sink +| Parameter | Description | Value Range | Required | Default | +|-----------------------------------|-----------------------------------------------------------------------------|------------------------|----------|---------| +| sink | Component name | String: tsfile-local-sink | Yes | - | +| sink.local.target-path | Local target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Rate limit threshold (unit: bytes/second). Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + +#### tsfile-remote-sink +| Parameter | Description | Value Range | Required | Default | +|------------------------------------|-----------------------------------------------------------------------------|-------------------------|----------|---------| +| sink | Component name | String: tsfile-remote-sink | Yes | - | +| sink.scp.host | Remote host IP | String | Yes | - | +| sink.scp.port | Remote SSH port | Long | No | 22 | +| sink.scp.user | Remote SSH user | String | Yes | - | +| sink.scp.password | Remote SSH password | String | Yes | - | +| sink.scp.remote-path | Remote target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Unit: bytes/second. Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + diff --git a/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md b/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md index c5f6499a8..3d3056dd7 100644 --- a/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md @@ -2,7 +2,10 @@ ## 1. 功能概述 -数据导出工具 `export-data.sh/bat` 位于 `tools` 目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile（开源时间序列文件格式）格式。具体功能如下： +IoTDB 支持两种方式进行数据导出： + +* 数据导出工具 ：`export-data.sh/bat` 位于 `tools `目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile （开源时间序列文件格式）格式。 +* 基于 PIPE 框架的 TsFileBackup：`tsfile-backup.sh/bat`位于 `tools `目录下，能够使用 PIPE 将指定的数据文件导出为 TsFile 格式。 @@ -20,15 +23,18 @@ - + + + + +
包含自定义 SQL 语句的文件
TsFileTsFile 开源时序数据文件格式
tsfile-backup.sh/bat开源时序数据文件格式，支持 Object 数据类型
-- 不支持 Object 数据类型。 -## 2. 功能详解 +## 2. 数据导出工具 ### 2.1 公共参数 @@ -178,3 +184,71 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. 基于 PIPE 框架的 TsFileBackup + +IoTDB 自 **V2.0.9.2** 版本起支持 `tsfile-backup.sh/bat` 脚本，该脚本能够自动生成并向服务端发送 `CREATE PIPE` SQL 指令，将指定的数据文件导出为 TsFile 格式。 + +**注意：** + +1. ​**使用该脚本需联系天谋团队获取定制安装包 ​`timechodb--extension`**​**。** +2. **该脚本支持 Object 类型数据导出为 TsFile 文件。** + +### 3.1 运行命令 + +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +# Windows +> tools\windows>tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + +### 3.2 脚本参数 + +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +| -------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------- | -------------- | -------------------- | +| `-sql_dialect` | `--sql_dialect` | 指定数据模型类型，可选值：`tree`(树模型) 或`table`(表模型)。 | 是 | - | +| `-h` | `--host` | 本地主机地址。指当前数据所在的 IoTDB 实例 IP。 | 否 | `127.0.0.1` | +| `-p` | `--port` | 端口号，IoTDB RPC 服务端口。 | 否 | `6667` | +| `-u` | `--user` | 用户名，用于登录 IoTDB 验证。 | 否 | `root` | +| `-pw` | `--password` | 密码，对应用户的IoTDB密码，支持隐藏输入。 | 否 | `root` | +| `-t` | `--target` | 导出目标目录。在 SCP 模式下，此路径指远程服务器上的绝对物理路径。TsFile 和关联的 Object 目录将导出至此。 | 是 | - | +| `-db` | `--database` | 数据库名称 (表模型可选) | 否 | `.*` | +| `-table` | `--table` | 表名 (表模型可选) | 否 | `.*` | +| `-s` | `--start_time` | 起始时间。支持 ISO8601 格式（如 2026-01-01T00:00:00）或毫秒时间戳。仅导出该时间点及之后的数据。 | 否 | - | +| `-e` | `--end_time` | 截止时间。格式同上。仅导出该时间点之前的数据。 | 否 | - | +| `-th` | `--target_host` | 远程目标主机 IP，默认自动识别启动脚本的IP。指定此参数后，脚本将自动配置 Pipe 使用 SCP 模式进行数据传输。 | 否 | - | +| `-tu` | `--target_host_user` | 远程主机用户名。用于 SSH/SCP 登录目标服务器。 | 否 | - | +| `-tpw` | `--target_host_pw` | 远程主机密码。用于远程身份验证，支持隐藏输入。 | 否 | - | +| `-tp` | `--target_host_port` | 远程 SSH 端口。 | 否 | `22` | +| `--rate_limit` | `--rate_limit` | 发送速率限制。单位：字节/秒 (Bytes/s)。防止导出任务占用过多网络带宽。 | 否 | - | +| `--plugin_jar` | `--plugin_jar` | 指定 Pipe 插件的Jar包路径 | 否 | - | +| `-help` | `--help` | 查看帮助 | 否 | - | + +### 3.3 运行示例 + +示例一：SCP 远程导出（将数据发送到另一台服务器） + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +示例二：带限速的远程 Object 数据导出 + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +示例三：指定 Pipe jar 目录 + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` \ No newline at end of file diff --git a/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md b/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md index 6341f0649..bc94826d1 100644 --- a/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md @@ -38,7 +38,7 @@ IoTDB 支持三种方式进行数据导入：
- **表模型 TsFile 导入暂时只支持本地导入。** -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起，import-data.sh/bat 脚本导入 tsfile 文件时支持 Object 数据类型。 ## 2. 数据导入工具 @@ -189,7 +189,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -202,19 +202,20 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x 版本及之后 > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 私有参数 -| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | -| ---------- | ------------------------ |----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | -| -os| --on\_succcess| 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || -| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| -| -of| --on\_fail| 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || -| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | -| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +|------|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | +| -os | --on\_succcess | 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || +| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| +| -of | --on\_fail | 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || +| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | +| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| -o | --object-file-paths | Object 文件存储路径。
默认模式：若不指定此参数，脚本将自动识别并导入位于 `/` 同名子目录下的 Object 文件。
绝对路径模式：显式指定 Object 文件的外部存储根目录，工具将基于此路径建立数据的关联索引。
注意：该参数自 V2.0.9.2 版本起支持 | 否 | | #### 2.4.3 运行示例 @@ -228,6 +229,53 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` +**Object 类型导入** + +1. 导入格式 + +* 默认 + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* 指定 Object 目录 + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. 命令行示例 + +* 基础导入（自动识别 TsFile 同名目录下的 Object 文件） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* 批量导入目录（指定并发线程数与成功后的处理动作） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* 表模型关联导入（指定外部 Object 存储路径与目标数据库） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile 自动加载功能 本功能允许 IoTDB 主动监听指定目录下的新增 TsFile，并将 TsFile 自动加载至 IoTDB 中。通过此功能，IoTDB 能自动检测并加载 TsFile，无需手动执行任何额外的加载操作。 diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index 2968a8301..99d523cdc 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ - 不支持 1.x 系列版本 IoTDB 与 2.x 以及以上系列版本的 IoTDB 之间进行数据同步。 - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起支持 Object 类型数据导出。 ## 2. 使用说明 @@ -223,17 +223,18 @@ SHOW PIPEPLUGINS ```SQL IoTDB> SHOW PIPEPLUGINS -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` 预置插件详细介绍如下(各插件的详细参数可参考本文[参数说明](#参考参数说明))： @@ -260,8 +261,8 @@ IoTDB> SHOW PIPEPLUGINS - - + + @@ -285,6 +286,14 @@ IoTDB> SHOW PIPEPLUGINS + + + + + + + +
默认的 processor 插件，不对传入的数据做任何的处理
sink 插件支持sink 插件支持 do-nothing-sink 不对发送出的数据做任何的处理
opc-ua-sink 用于 IoTDB （V2.0.2 及以上）支持OPC UA协议的数据传输插件，支持Client/Server 和 Pub/Sub 两种通信模式。
tsfile-local-sink用于 IoTDB (V2.0.9.2及以上）支持将 Object 数据导出到 IoTDB 服务器所在的本地文件系统。
tsfile-remote-sink用于 IoTDB (V2.0.9.2及以上）支持通过 SSH/SCP 协议将 Object 数据发送到远程服务器。
@@ -507,6 +516,79 @@ with sink ( ) ``` +### 3.9 Object 类型数据导出 + +IoTDB 自 V2.0.9.2 版本起支持导出 Object 类型数据，通过配置 sink 参数支持如下两种方式： + +* Local 模式（本地导出）：将数据导出到 IoTDB 服务器所在的本地文件系统。 +* SCP 模式（远程传输）：通过 SSH/SCP 协议将数据发送到远程服务器。 + +**示例一：本地导出** + +可直接使用系统内置的 `tsfile-local-sink `插件创建 PIPE 语句导出数据，例如： + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- 必填，指定 Sink 类型 + 'sink.local.target-path' = '/data/backup/export_2024' -- 导出目标路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**示例二：远程传输** + +1. 联系天谋团队获取 `tsfile-remote-sink` 插件相关的 jar 包，如 `tsfile-remote-sink--jar-with-dependencies.jar`，并放至 IoTDB 可访问的路径（例如所有数据节点主机）。 +2. 使用如下语句注册插件 + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink-<版本号>-jar-with-dependencies.jar'; +``` + +3. 创建 PIPE 语句 + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- 指定为 SCP 模式 + 'sink.scp.host' = '192.168.1.100', -- 远程主机 IP + 'sink.scp.port' = '22', -- SSH 端口 + 'sink.scp.user' = 'backup_user', -- SSH 用户名 + 'sink.scp.password' = 'ComplexPass123!', -- SSH 密码 + 'sink.scp.remote-path' = '/remote/archive/', -- 远程存放路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**Sink 导出 TSFile 与 Object 格式：** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + + ## 参考：注意事项 可通过修改 IoTDB 配置文件（`iotdb-system.properties`）以调整数据同步的参数，如同步数据存储目录等。完整配置如下：： @@ -683,3 +765,22 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.password | 密码，这里指 OPC UA 的允许密码 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | | sink.opcua.placeholder | 当ID列的值出现null时，用于替代null映射路径的占位字符串 | String | 选填 | "null" | +#### tsfile-local-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|-----------------------------------|-----------------------------------------|---------------------------|------|---------| +| sink | 组件名称 | String: tsfile-local-sink | 是 | - | +| sink.local.target-path | 本地目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 限速阈值。单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 | + +#### tsfile-remote-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|------------------------------------|-----------------------------------|--------------------------|----------|---------| +| sink | 组件名称 | String: tsfile-remote-sink | 是 | - | +| sink.scp.host | 远程主机 IP | String | 是 | - | +| sink.scp.port | 远程 SSH 端口 | Long | 否 | 22 | +| sink.scp.user | 远程 SSH 用户 | String | 是 | - | +| sink.scp.password | 远程 SSH 密码 | String | 是 | - | +| sink.scp.remote-path | 远程目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 | diff --git a/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md b/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md index c5f6499a8..3d3056dd7 100644 --- a/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md @@ -2,7 +2,10 @@ ## 1. 功能概述 -数据导出工具 `export-data.sh/bat` 位于 `tools` 目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile（开源时间序列文件格式）格式。具体功能如下： +IoTDB 支持两种方式进行数据导出： + +* 数据导出工具 ：`export-data.sh/bat` 位于 `tools `目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile （开源时间序列文件格式）格式。 +* 基于 PIPE 框架的 TsFileBackup：`tsfile-backup.sh/bat`位于 `tools `目录下，能够使用 PIPE 将指定的数据文件导出为 TsFile 格式。 @@ -20,15 +23,18 @@ - + + + + +
包含自定义 SQL 语句的文件
TsFileTsFile 开源时序数据文件格式
tsfile-backup.sh/bat开源时序数据文件格式，支持 Object 数据类型
-- 不支持 Object 数据类型。 -## 2. 功能详解 +## 2. 数据导出工具 ### 2.1 公共参数 @@ -178,3 +184,71 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. 基于 PIPE 框架的 TsFileBackup + +IoTDB 自 **V2.0.9.2** 版本起支持 `tsfile-backup.sh/bat` 脚本，该脚本能够自动生成并向服务端发送 `CREATE PIPE` SQL 指令，将指定的数据文件导出为 TsFile 格式。 + +**注意：** + +1. ​**使用该脚本需联系天谋团队获取定制安装包 ​`timechodb--extension`**​**。** +2. **该脚本支持 Object 类型数据导出为 TsFile 文件。** + +### 3.1 运行命令 + +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +# Windows +> tools\windows>tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + +### 3.2 脚本参数 + +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +| -------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------- | -------------- | -------------------- | +| `-sql_dialect` | `--sql_dialect` | 指定数据模型类型，可选值：`tree`(树模型) 或`table`(表模型)。 | 是 | - | +| `-h` | `--host` | 本地主机地址。指当前数据所在的 IoTDB 实例 IP。 | 否 | `127.0.0.1` | +| `-p` | `--port` | 端口号，IoTDB RPC 服务端口。 | 否 | `6667` | +| `-u` | `--user` | 用户名，用于登录 IoTDB 验证。 | 否 | `root` | +| `-pw` | `--password` | 密码，对应用户的IoTDB密码，支持隐藏输入。 | 否 | `root` | +| `-t` | `--target` | 导出目标目录。在 SCP 模式下，此路径指远程服务器上的绝对物理路径。TsFile 和关联的 Object 目录将导出至此。 | 是 | - | +| `-db` | `--database` | 数据库名称 (表模型可选) | 否 | `.*` | +| `-table` | `--table` | 表名 (表模型可选) | 否 | `.*` | +| `-s` | `--start_time` | 起始时间。支持 ISO8601 格式（如 2026-01-01T00:00:00）或毫秒时间戳。仅导出该时间点及之后的数据。 | 否 | - | +| `-e` | `--end_time` | 截止时间。格式同上。仅导出该时间点之前的数据。 | 否 | - | +| `-th` | `--target_host` | 远程目标主机 IP，默认自动识别启动脚本的IP。指定此参数后，脚本将自动配置 Pipe 使用 SCP 模式进行数据传输。 | 否 | - | +| `-tu` | `--target_host_user` | 远程主机用户名。用于 SSH/SCP 登录目标服务器。 | 否 | - | +| `-tpw` | `--target_host_pw` | 远程主机密码。用于远程身份验证，支持隐藏输入。 | 否 | - | +| `-tp` | `--target_host_port` | 远程 SSH 端口。 | 否 | `22` | +| `--rate_limit` | `--rate_limit` | 发送速率限制。单位：字节/秒 (Bytes/s)。防止导出任务占用过多网络带宽。 | 否 | - | +| `--plugin_jar` | `--plugin_jar` | 指定 Pipe 插件的Jar包路径 | 否 | - | +| `-help` | `--help` | 查看帮助 | 否 | - | + +### 3.3 运行示例 + +示例一：SCP 远程导出（将数据发送到另一台服务器） + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +示例二：带限速的远程 Object 数据导出 + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +示例三：指定 Pipe jar 目录 + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` \ No newline at end of file diff --git a/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md b/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md index 6341f0649..d48186fb5 100644 --- a/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md @@ -38,7 +38,7 @@ IoTDB 支持三种方式进行数据导入：
- **表模型 TsFile 导入暂时只支持本地导入。** -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起，import-data.sh/bat 脚本导入 tsfile 文件时支持 Object 数据类型。 ## 2. 数据导入工具 @@ -189,7 +189,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -202,19 +202,20 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x 版本及之后 > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 私有参数 -| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | -| ---------- | ------------------------ |----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | -| -os| --on\_succcess| 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || -| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| -| -of| --on\_fail| 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || -| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | -| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +|------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | +| -os | --on\_succcess | 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || +| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| +| -of | --on\_fail | 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || +| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | +| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| -o | --object-file-paths | Object 文件存储路径。
默认模式：若不指定此参数，脚本将自动识别并导入位于 `/` 同名子目录下的 Object 文件。
绝对路径模式：显式指定 Object 文件的外部存储根目录，工具将基于此路径建立数据的关联索引。
注意：该参数自 V2.0.9.2 版本起支持 | 否 | | #### 2.4.3 运行示例 @@ -228,6 +229,53 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` +**Object 类型导入** + +1. 导入格式 + +* 默认 + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* 指定 Object 目录 + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. 命令行示例 + +* 基础导入（自动识别 TsFile 同名目录下的 Object 文件） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* 批量导入目录（指定并发线程数与成功后的处理动作） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* 表模型关联导入（指定外部 Object 存储路径与目标数据库） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile 自动加载功能 本功能允许 IoTDB 主动监听指定目录下的新增 TsFile，并将 TsFile 自动加载至 IoTDB 中。通过此功能，IoTDB 能自动检测并加载 TsFile，无需手动执行任何额外的加载操作。 diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 2968a8301..99d523cdc 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ - 不支持 1.x 系列版本 IoTDB 与 2.x 以及以上系列版本的 IoTDB 之间进行数据同步。 - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起支持 Object 类型数据导出。 ## 2. 使用说明 @@ -223,17 +223,18 @@ SHOW PIPEPLUGINS ```SQL IoTDB> SHOW PIPEPLUGINS -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` 预置插件详细介绍如下(各插件的详细参数可参考本文[参数说明](#参考参数说明))： @@ -260,8 +261,8 @@ IoTDB> SHOW PIPEPLUGINS - - + + @@ -285,6 +286,14 @@ IoTDB> SHOW PIPEPLUGINS + + + + + + + +
默认的 processor 插件，不对传入的数据做任何的处理
sink 插件支持sink 插件支持 do-nothing-sink 不对发送出的数据做任何的处理
opc-ua-sink 用于 IoTDB （V2.0.2 及以上）支持OPC UA协议的数据传输插件，支持Client/Server 和 Pub/Sub 两种通信模式。
tsfile-local-sink用于 IoTDB (V2.0.9.2及以上）支持将 Object 数据导出到 IoTDB 服务器所在的本地文件系统。
tsfile-remote-sink用于 IoTDB (V2.0.9.2及以上）支持通过 SSH/SCP 协议将 Object 数据发送到远程服务器。
@@ -507,6 +516,79 @@ with sink ( ) ``` +### 3.9 Object 类型数据导出 + +IoTDB 自 V2.0.9.2 版本起支持导出 Object 类型数据，通过配置 sink 参数支持如下两种方式： + +* Local 模式（本地导出）：将数据导出到 IoTDB 服务器所在的本地文件系统。 +* SCP 模式（远程传输）：通过 SSH/SCP 协议将数据发送到远程服务器。 + +**示例一：本地导出** + +可直接使用系统内置的 `tsfile-local-sink `插件创建 PIPE 语句导出数据，例如： + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- 必填，指定 Sink 类型 + 'sink.local.target-path' = '/data/backup/export_2024' -- 导出目标路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**示例二：远程传输** + +1. 联系天谋团队获取 `tsfile-remote-sink` 插件相关的 jar 包，如 `tsfile-remote-sink--jar-with-dependencies.jar`，并放至 IoTDB 可访问的路径（例如所有数据节点主机）。 +2. 使用如下语句注册插件 + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink-<版本号>-jar-with-dependencies.jar'; +``` + +3. 创建 PIPE 语句 + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- 指定为 SCP 模式 + 'sink.scp.host' = '192.168.1.100', -- 远程主机 IP + 'sink.scp.port' = '22', -- SSH 端口 + 'sink.scp.user' = 'backup_user', -- SSH 用户名 + 'sink.scp.password' = 'ComplexPass123!', -- SSH 密码 + 'sink.scp.remote-path' = '/remote/archive/', -- 远程存放路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**Sink 导出 TSFile 与 Object 格式：** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + + ## 参考：注意事项 可通过修改 IoTDB 配置文件（`iotdb-system.properties`）以调整数据同步的参数，如同步数据存储目录等。完整配置如下：： @@ -683,3 +765,22 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.password | 密码，这里指 OPC UA 的允许密码 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | | sink.opcua.placeholder | 当ID列的值出现null时，用于替代null映射路径的占位字符串 | String | 选填 | "null" | +#### tsfile-local-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|-----------------------------------|-----------------------------------------|---------------------------|------|---------| +| sink | 组件名称 | String: tsfile-local-sink | 是 | - | +| sink.local.target-path | 本地目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 限速阈值。单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 | + +#### tsfile-remote-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|------------------------------------|-----------------------------------|--------------------------|----------|---------| +| sink | 组件名称 | String: tsfile-remote-sink | 是 | - | +| sink.scp.host | 远程主机 IP | String | 是 | - | +| sink.scp.port | 远程 SSH 端口 | Long | 否 | 22 | +| sink.scp.user | 远程 SSH 用户 | String | 是 | - | +| sink.scp.password | 远程 SSH 密码 | String | 是 | - | +| sink.scp.remote-path | 远程目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 |