在smartpip中配置flinkcdc即可实现,数据库实时同步

同步任务说明 #

• 全自动建表并能自动感知源表结构变化,自动变化
• 支持整库同步, 因此建议针对一个数据源只需启动一个任务即可
• 实时同步任务启动后, 定时任务为自动监控任务
• 如果修改任务后需在flink web中先取消任务后再可提交生效

简单的全库同步设定 #

比如源库为training, 则会自动在目标源建相同名称的库和表

##template = test_starrocks_flink
##name = test_starrocks2
##tables = training.\.*
##config =

批量路由 #

##template = test_starrocks_flink
##name = test_starrocks2
##tables = source_db.\.*

##config=
route:
  - source-table: source_db.\.*
    sink-table: sink_db.<>
    replace-symbol: <>
    description: route all tables in source_db to sink_db

自定义路由或转化 #

##template = test_starrocks_flink
##name = test_starrocks2
##tables = adb.\.*, bdb.user_table_[0-9]+, [app|web]_order_\.*

##config=
transform:
  - source-table: mydb.web_order
    projection: \*, UPPER(product_name) as product_name
    description: project fields with wildcard character from source table
  - source-table: mydb.app_order
    projection: UPPER(product_name) as product_name, *
    description: project fields with wildcard character from source table
  - source-table: mydb.web_order
    projection: id, order_id, UPPER(province) as new_province 
    filter: new_province = 'SHANGHAI'
    description: filtering rows based on computed columns

route:
  - source-table: app_db.orders
    sink-table: ods_db.ods_orders
  - source-table: app_db.shipments
    sink-table: ods_db.ods_shipments