Pipeline Configuration API

Manage pipeline configurations through CRUD operations.

Get a Pipeline

GET /api/v1/pipeline?pipeline={name}

Parameters:

Parameter	Type	Required	Description
`pipeline`	query	Yes	Pipeline name

Example:

curl "http://localhost:8080/api/v1/pipeline?pipeline=sales_data"

Response: 200 OK with the full PipelineConfig JSON.

Unity Catalog sync state

GET /api/v1/pipelines/{name}/unity-catalog

The last Unity Catalog sync for a pipeline that sets unityCatalog.enabled (or that the install default DATRIS_UNITY_CATALOG_DEFAULT=enabled turned on): the metadata push and lineage for a Databricks pipeline (see Databricks), or the table registration for an object store Iceberg pipeline (see Register Iceberg tables). Example:

curl http://localhost:8080/api/v1/pipelines/orders_pipeline/unity-catalog

Response: 200 OK. enabled says whether Unity Catalog is on for the pipeline and enabledBy says who decided: pipeline (the pipeline’s own unityCatalog block, including an "enabled": false opt-out), default (the install default turned on a Databricks pipeline with no block), or null (off, no block). lineageEnabled follows the same resolution. syncSwitchedOff is true while DATRIS_UNITY_CATALOG_SYNC=false: nothing is written to Unity Catalog even when enabled is true. coordinates gives the Unity Catalog table (Databricks destinations only). state is never (no sync yet), synced, or error (with lastError holding the warehouse’s message, or when the last catalogMode: rest run was refused; see restRefusedReason). lastSyncAt and lastRunId identify the last sync. lineage is the lineage publish status: off (unityCatalog.lineage is false, or the push is off), never, published, or error (the failure is a lastError line starting with uc-lineage:); lastLineageAt is when lineage was last published and lineageHash identifies the published definition (see Lineage in Unity Catalog). For an object store Iceberg pipeline, coordinates names the registered table (catalog.schema.table, with kind: "iceberg" and the table’s location), registerEnabled says whether registration is on, and register is off, never, registered, stale (Unity Catalog still points at an older metadata file) or error (the failure is a lastError line starting with uc-register:); registeredMetadataLocation and lastRegisterAt record the last registration. catalogMode is register, rest or managed (see Keep the catalog current and Databricks: catalogMode managed); in managed mode register is managed when the last run committed through the catalog, restMetadataLocation is under the location the catalog chose, and coordinates.location is that location once known; in rest mode register is rest when the last run committed through the catalog and refused when it wrote by path instead, restMetadataLocation and lastRestCommitAt record the last catalog commit, and restRefusedReason says why a run was refused. restCreatedTable names a table the catalog created for the pipeline at a location Datris refused (a Databricks managed table): it was never written to, and an admin has to drop it. 404 if the pipeline does not exist.

{
  "pipeline": "orders_pipeline",
  "enabled": true,
  "enabledBy": "pipeline",
  "syncSwitchedOff": false,
  "coordinates": { "catalog": "datris", "schema": "default", "table": "orders", "qualified": "datris.default.orders" },
  "state": "synced",
  "lastSyncAt": "2026-09-29T12:00:00Z",
  "lastRunId": "...",
  "lineageEnabled": true,
  "lineage": "published",
  "lineageHash": "...",
  "lastLineageAt": "2026-09-29T12:00:01Z",
  "registerEnabled": false,
  "register": "off"
}

For an object store Iceberg pipeline:

{
  "pipeline": "orders_lake",
  "enabled": true,
  "coordinates": { "catalog": "analytics", "schema": "default", "table": "orders_lake", "qualified": "analytics.default.orders_lake", "kind": "iceberg", "location": "s3a://my-lake/orders" },
  "registerEnabled": true,
  "register": "registered",
  "registeredMetadataLocation": "s3a://my-lake/orders/metadata/00001-....metadata.json",
  "lastRegisterAt": "2026-09-29T12:00:02Z"
}

List All Pipelines

GET /api/v1/pipelines

Example:

curl http://localhost:8080/api/v1/pipelines

Response: 200 OK with an array of all PipelineConfig objects.

Create or Update a Pipeline

POST /api/v1/pipeline
Content-Type: application/json

Body: Full PipelineConfig JSON (see Pipeline Configuration). Parameters:

Parameter	Type	Required	Description
`changeNote`	query	No	Note recorded on the saved pipeline version

Example:

curl -X POST http://localhost:8080/api/v1/pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "name": "sales_data",
    "source": {
      "schemaProperties": {
        "fields": [
          {"name": "id", "type": "int"},
          {"name": "amount", "type": "double"}
        ]
      },
      "fileAttributes": {
        "csvAttributes": {"delimiter": ",", "header": true, "encoding": "UTF-8"}
      }
    },
    "destination": {
      "database": {
        "dbName": "mydb",
        "schema": "public",
        "table": "sales",
        "usePostgres": true
      }
    }
  }'

Response: 200 OK

{"warnings": []}

warnings is advisory: the pipeline is saved either way. It is empty unless something about the configuration will not work as written, for example an object store Iceberg pipeline whose Unity Catalog secret points at a Databricks workspace with catalogMode register or rest (Databricks cannot serve either; the warning says to set catalogMode managed or use the databricks destination). Each warning is also written to the server log. Validation: The configuration is validated before saving. Invalid configurations return 400 with {"error": "<message>"} and nothing is saved; 500 means the server itself failed. See Pipeline Configuration for all constraints.

Delete a Pipeline

DELETE /api/v1/pipeline?pipeline={name}

Parameters:

Parameter	Type	Required	Description
`pipeline`	query	Yes	Pipeline name
`deleteData`	query	No	Delete destination data (default: `true`). Only takes effect together with `deleteConfig=false`: deleting the configuration always deletes the data too, so `deleteData=false` on its own still removes the data.
`deleteConfig`	query	No	Delete pipeline configuration (default: `true`). Set to `false` to delete data only while keeping the pipeline config.

There is no way to delete a pipeline’s configuration and keep its destination data; a pipeline with no configuration would leave tables, collections, and files behind that nothing owns. Examples:

# Delete pipeline and all data
curl -X DELETE "http://localhost:8080/api/v1/pipeline?pipeline=sales_data"

# Delete data only, keep pipeline configuration
curl -X DELETE "http://localhost:8080/api/v1/pipeline?pipeline=sales_data&deleteData=true&deleteConfig=false"

Response: 200 OK with a JSON body:

{ "warnings": [] }

warnings lists anything left for you to handle. For an object-store Iceberg pipeline whose table was committed through Unity Catalog (catalogMode: rest), it says Unity Catalog still holds <catalog>.<schema>.<table>; have an admin drop it, because Datris removes the files but never drops the catalog table (see Unity Catalog). For a catalogMode: managed table it says

Unity Catalog owns <catalog>.<schema>.<table> and its files (catalogMode managed); Datris deleted nothing there; drop <catalog>.<schema>.<table> in Unity Catalog to remove the data

AI Schema Generation

To generate a pipeline configuration automatically from an uploaded file, see the AI Schema Generation API.

Authentication

If useApiKeys is enabled, all requests must include the x-api-key header:

curl -H "x-api-key: your-api-key" http://localhost:8080/api/v1/pipelines

API keys are stored in Vault under {environment}/api-keys (oss/api-keys on a default install). secrets.apiKeysSecretName in application.yaml tells the validator where to read them and should point at that same path — the Keys UI and multi-tenant resolution always write to {environment}/api-keys.

API Documentation

Pipelines

Ingestion

AI

Status

Query

Search

Health

Metadata

Configuration

Secrets

Taps

Code repository

Versions

Auth

Agent Policy

Recovery Agent

Audit Log

API Keys

Assistant

Chat

MCP

Tap Prompts

Provenance

Internal callbacks

Pipeline Configuration API

Get a Pipeline

Unity Catalog sync state

List All Pipelines

Create or Update a Pipeline

Delete a Pipeline

AI Schema Generation

Authentication

​Get a Pipeline

​Unity Catalog sync state

​List All Pipelines

​Create or Update a Pipeline

​Delete a Pipeline

​AI Schema Generation

​Authentication

Get a Pipeline

Unity Catalog sync state

List All Pipelines

Create or Update a Pipeline

Delete a Pipeline

AI Schema Generation

Authentication