Introduction #

SmartPip supports ingesting third-party data via API. Define a standard function and call it in the DAG using the apistarrocks driver. This is suitable for scenarios requiring data from external systems.

Setup Flow #

Define the Standard Function #

# Assuming API returns: {'data': [{'a':1, 'b':2}, ...]}
def _get_api_data(param):
    import json, requests
    param = json.loads(param)
    res = requests.post(url=url, json=param).json()['data']
    res = json.dumps(res).encode()
    return res

API Configuration #

##apiConn=get_api_data
##param={"p1": xxx, "p2": "xxxxx"}
##table=xxxx    -- target table name

--------- Optional common parameters ----------
##columns=a,b

--------- Optional JSON format ----------
##format=json
##jsonpaths=a,b
##strip_outer_array=true

For more settings, refer to kafkastarrocks configuration.