Introduction #
FlinkCDC (Change Data Capture) is SmartPip’s real-time data sync solution, supporting database change capture, auto table creation, and schema change detection. Configure the flinkcdc driver to achieve real-time sync from source to target database.
FlinkCDC Features #
| Feature | Description |
|---|---|
| Auto table creation | Automatically creates tables matching source schema |
| Schema change detection | Automatically detects and syncs source table structure changes |
| Full database sync | Sync an entire database in one task |
| Batch routing | Custom table name mapping and transformation rules |
Sync Task Notes #
- Fully automatic table creation with schema change detection
- Supports full database sync — one task per data source is recommended
- After starting, scheduled tasks automatically monitor
- To modify a task, cancel it in Flink Web UI first, then resubmit
Simple Full Database Sync #
Source database training auto-creates matching database and tables in target:
##template = test_starrocks_flink
##name = test_starrocks2
##tables = training.\.*
##config =
Batch Routing #
##template = test_starrocks_flink
##name = test_starrocks2
##tables = source_db.\.*
##config=
route:
- source-table: source_db.\.*
sink-table: sink_db.<>
replace-symbol: <>
description: route all tables in source_db to sink_db
Custom Routing/Transformation #
##template = test_starrocks_flink
##name = test_starrocks2
##tables = adb.\.*, bdb.user_table_[0-9]+, [app|web]_order_\.*
##config=
transform:
- source-table: mydb.web_order
projection: \*, UPPER(product_name) as product_name
- source-table: mydb.app_order
projection: UPPER(product_name) as product_name, *
- source-table: mydb.web_order
projection: id, order_id, UPPER(province) as new_province
filter: new_province = 'SHANGHAI'
route:
- source-table: app_db.orders
sink-table: ods_db.ods_orders
- source-table: app_db.shipments
sink-table: ods_db.ods_shipments