https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/ Recent content on Smartchart开发手册 Hugo -- gohugo.io https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/ConnectAPIData/ Mon, 01 Jan 0001 00:00:00 +0000 https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/ConnectAPIData/ Introduction # SmartPip supports ingesting third-party data via API. Define a standard function and call it in the DAG using the apistarrocks driver. This is suitable for scenarios requiring data from external systems. Setup Flow # Step Action 1 Define _get_api_data(param) function 2 Configure ##apiConn=function_name in API settings 3 Use apistarrocks driver in DAG Define the Standard Function # # Assuming API returns: {'data': [{'a':1, 'b':2}, ...]} def _get_api_data(param): import json, requests param = json. https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/DataXExtract/ Mon, 01 Jan 0001 00:00:00 +0000 https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/DataXExtract/ Introduction # DataX incremental extraction achieves fine-grained data synchronization using max ID or time conditions from the target database. This is suitable for scenarios requiring incremental sync rather than full overwrite. Incremental Methods # Method Use Case Complexity Standard Single-table incremental Low Dataset Complex conditions Medium Custom Dynamic logic High Usage # Standard Method # Add to DataX sync config: ##incColumn = incremental_field ##incDB = target_query_DB [default: starrocks] Dataset Method # Create a query SQL in SmartChart datasets, e. https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/FlinkCDCSync/ Mon, 01 Jan 0001 00:00:00 +0000 https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/FlinkCDCSync/ Introduction # FlinkCDC (Change Data Capture) is SmartPip’s real-time data sync solution, supporting database change capture, auto table creation, and schema change detection. Configure the flinkcdc driver to achieve real-time sync from source to target database. FlinkCDC Features # Feature Description Auto table creation Automatically creates tables matching source schema Schema change detection Automatically detects and syncs source table structure changes Full database sync Sync an entire database in one task Batch routing Custom table name mapping and transformation rules Sync Task Notes # Fully automatic table creation with schema change detection Supports full database sync — one task per data source is recommended After starting, scheduled tasks automatically monitor To modify a task, cancel it in Flink Web UI first, then resubmit Simple Full Database Sync # Source database training auto-creates matching database and tables in target: https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/MessageSend/ Mon, 01 Jan 0001 00:00:00 +0000 https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/Components/MessageSend/ Introduction # SmartPip supports sending email or WeCom (Enterprise WeChat) notifications within DAG tasks, supporting both fixed content and dynamic data. This is suitable for result notifications after task completion. Notification Methods # Method Driver Use Case Email #send_mail Formal reports, detailed data WeCom #diy + dataset Real-time alerts, short notifications Usage # Fixed Content # msg = ['Report refresh succeeded', '<h1>Refresh Successful!!</h1>'] maillist = 'xxx@smartchart.cn' #send_mail reportmail msg maillist Dynamic Data # maillist = 'xxx@smartchart.