Introduction #
SmartPip is built on Airflow and supports time-based task backfilling. Through built-in parameters like execution_date, you can obtain task execution time information for time-dimensional data processing.
Common Time Parameters #
| Parameter | Format | Description |
|---|---|---|
ds |
YYYY-MM-DD | Task execution date |
ds_nodash |
YYYYMMDD | Date without separators |
yesterday_ds |
YYYY-MM-DD | Yesterday’s date |
tomorrow_ds |
YYYY-MM-DD | Tomorrow’s date |
prev_ds |
YYYY-MM-DD | Previous execution date |
Usage #
Airflow passes the following parameters to tasks:
{'ds': '2022-07-13',
'ds_nodash': '20220713',
'execution_date': DateTime(2022, 7, 13, 2, 4, 33, 244294, tzinfo=Timezone('+00:00')),
'next_ds': '2022-07-13',
'tomorrow_ds': '2022-07-14',
'yesterday_ds': '2022-07-12',
'ts': '2022-07-13T02:04:33.244294+00:00',
...}
Use in DAG configuration (DIY example):
def get_execution_date(**kwargs):
execution_date = kwargs['execution_date']
zymd = execution_date.strftime('%Y%m%d')
para_dict = {'zymd': zymd}
run_sql_file('DAG_name/task_name.sql', 'starrocks', para_dict)
#diy job_get_execution_date get_execution_date