Introduction #

DataX incremental extraction achieves fine-grained data synchronization using max ID or time conditions from the target database. This is suitable for scenarios requiring incremental sync rather than full overwrite.

Incremental Methods #

Usage #

Standard Method #

Add to DataX sync config:

##incColumn = incremental_field
##incDB = target_query_DB [default: starrocks]

Dataset Method #

• Create a query SQL in SmartChart datasets, e.g.:

select 
 max(id) as maxid, max(event_day) as event_day 
from targettablename

• Pass the dataset ID as a parameter in DAG (e.g., dataset ID 455):

#datax  jobname   455   -- sync table

• Use the dataset results as parameters in the extraction SQL:

select .... from xxxx
where id > $maxid  and event_day > '$event_day'

Custom Method #

• Define a custom function to process dataset data:

def get_targettablename_id():
    result = get_dataset(455)['data']
    maxid = result[1][0]
    event_day = result[1][1]
    return {'maxid': maxid, 'event_day': event_day}

• Pass the function as parameter:

#datax  jobname   get_targettablename_id   -- sync table