https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/CustomTasks/ Recent content on Smartchart开发手册 Hugo -- gohugo.io https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/CustomTasks/CommonFunctions/ Mon, 01 Jan 0001 00:00:00 +0000 https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/CustomTasks/CommonFunctions/ Introduction # SmartPip provides rich built-in functions supporting file upload, data retrieval, email sending, shell execution, SQL execution, Kettle tasks, and more. Call custom functions via the #diy driver for flexible task logic. Built-in Functions Quick Reference # Function Description Notes File upload smart_upload(csvfilepath) Get data get_dataset(id, param={}) Send email dash_mail(title, content, maillist) Execute shell run_bash(cmdStr) Execute datax run_datax('DAG_name/task_name', para_dict) Execute SQL run_sql_file('DAG_name/task_name', db_connect='starrocks', para_dict=None) Execute Kettle run_kettle('DAG_name/task_name.ktr', para_str='') Execute SP run_sp(sp_name, sp_para=None, dev='', db_connect='oracle') Note: https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/CustomTasks/CustomLoopExtract/ Mon, 01 Jan 0001 00:00:00 +0000 https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/CustomTasks/CustomLoopExtract/ Introduction # When a database contains time-suffixed partitioned tables (e.g., order_202001, order_202002) that need to be consolidated into a single target table, use the custom loop extraction feature. Combine the #diy driver with run_datax() for flexible multi-table loop sync. Implementation Steps # Step Action 1 Create a DataX extraction task with parameter placeholders 2 Create a DIY task with loop logic 3 Orchestrate task dependencies in DAG Usage # SmartPip’s built-in datax component only supports single-table extraction without custom logic. https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/CustomTasks/GetExecParams/ Mon, 01 Jan 0001 00:00:00 +0000 https://help.smartchart.cn/en/docs/12.DataPlatform/smartpip/CustomTasks/GetExecParams/ Introduction # SmartPip is built on Airflow and supports time-based task backfilling. Through built-in parameters like execution_date, you can obtain task execution time information for time-dimensional data processing. Common Time Parameters # Parameter Format Description ds YYYY-MM-DD Task execution date ds_nodash YYYYMMDD Date without separators yesterday_ds YYYY-MM-DD Yesterday’s date tomorrow_ds YYYY-MM-DD Tomorrow’s date prev_ds YYYY-MM-DD Previous execution date Usage # Airflow passes the following parameters to tasks: {'ds': '2022-07-13', 'ds_nodash': '20220713', 'execution_date': DateTime(2022, 7, 13, 2, 4, 33, 244294, tzinfo=Timezone('+00:00')), 'next_ds': '2022-07-13', 'tomorrow_ds': '2022-07-14', 'yesterday_ds': '2022-07-12', 'ts': '2022-07-13T02:04:33.