Skip to main content
POST
Create or update a pipeline configuration

Authorizations

x-api-key
string
header
required

Optional API key for authentication (enabled via application.yaml)

Body

application/json
name
string
required

Pipeline name

source
object

Source configuration (fileAttributes, schemaProperties)

preprocessor
object

Optional REST endpoint preprocessor (endpoint, async, bearerToken, apiKey, timeoutMs, batchSize). batchSize defaults to 0 (one call carrying every row, whose response data replaces the payload). With batchSize > 0, row-based payloads are POSTed in batches of that many rows — each request also carrying top-level batch (1-based) and ofBatches — and every response's data.rows are stitched, in call order, into the payload passed downstream.

dataQuality
object

Data quality rules (AI rules, header validation, schema validation)

transformation
object

Transformation rules (AI transformation, deduplication, row functions).

rowFunctions is an array of { "function", "parameters" }. For "function": "restEndpoint" the positional parameters are [url, mode, timeoutMs, bearerToken, apiKey, batchSize]: mode is "row" (default; one call per row carrying {pipelineName, pipelineToken, row}, a null row in the response removes it) or "batch" (calls carry {pipelineName, pipelineToken, rows}). In batch mode the optional sixth parameter batchSize (default 0) sets the rows per call: 0 makes one call carrying every row; > 0 sends ceil(rows / batchSize) calls of that many rows, in source order, and stitches the responses in call order. Only url is required.

destination
object

Destination configuration (database, objectStore, kafka, activeMQ, restEndpoint, qdrant, weaviate, milvus, chroma, pgvector).

The database destination selects its engine with exactly one of usePostgres, useMongoDB, useSnowflake, or useDatabricks. Snowflake and Databricks additionally take credentialsSecret (a Platform secret) and warehouse (Snowflake warehouse name / Databricks SQL warehouse ID); for Databricks, dbName names the Unity Catalog catalog.

objectStore supports two providers via the provider field: minio (default, uses the built-in MinIO credentials) and s3 (AWS S3). For provider: s3, set destinationBucketOverride (required), credentialsSecret (Vault secret with accessKey/secretKey/region and optional sessionToken), and optionally endpoint (must be https://). Region lives in the credentials secret, not on the objectStore config.

fileFormat accepts parquet (default), orc, or iceberg; with iceberg, writeMode may also be merge, which upserts on the keyFields array (each a destination schema column).

restEndpoint takes endpoint, async, bearerToken, apiKey, timeoutMs and an optional batchSize (default 0 = one call carrying every row). With batchSize > 0, row-based payloads are POSTed in batches of that many rows, each request also carrying top-level batch (1-based) and ofBatches.

Response

Pipeline created/updated