Vector Database | Smartchart开发手册

Introduction #

Vector database is used for storing and retrieving text embeddings, supporting semantic similarity-based knowledge base Q&A scenarios. SmartChart uses StarRocks as the vector compute backend, combined with custom data source functions, to implement the full vector ingestion and retrieval pipeline. Enterprise vector solution requires Pro edition.

Core Component	Description
`SmartVectorDB`	Vector database operations class, supports incremental and full-text retrieval
`Text2VecEmbeddingFunction`	Local model text-to-vector
`Text2VecDashscopeFunction`	Alibaba Dashscope cloud text-to-vector
`insert_dataset()`	Batch ingest documents and vectors
`dataset()`	Retrieve most similar results by query text

Setup #

Create the vector table in StarRocks:

create database smartdb;
CREATE TABLE IF NOT EXISTS vectors(
    collection String not null,
    document String,
    id bigint AUTO_INCREMENT,
    sr String,
    embedding Array<Float>,
    answer string,
    c String,
    m String,
    owner varchar(50),
    update_time DATETIME DEFAULT CURRENT_TIMESTAMP
)
DUPLICATE KEY (collection) comment 'smart embeddings'
DISTRIBUTED BY HASH(collection)

Custom Connector #

from smart_chart.common.smartvector import SmartVectorDB, Text2VecEmbeddingFunction, Text2VecDashscopeFunction

text_vector = Text2VecEmbeddingFunction()  # Local model
# text_vector = Text2VecDashscopeFunction()  # Alibaba Dashscope

def dataset(*args, **kwargs):
    sqlList = args[0]
    config = args[1]
    vdb = SmartVectorDB(config, text_vector)
    result = vdb.search(sqlList[0], collection=sqlList[1] if len(sqlList) > 1 else 'default')
    return result

def insert_dataset(*args, **kwargs):
    contents = args[0]
    config = args[3]
    vdb = SmartVectorDB(config, text_vector)
    vdb.insert(contents)