Data Ingestion API

Upload a File

Upload a data file for processing by a configured pipeline.

POST /api/v1/pipeline/upload
Content-Type: multipart/form-data

Parameters:

Parameter	Type	Required	Description
`file`	form-data	Yes	The file to upload
`pipeline`	form-data	Yes	Target pipeline name
`publishertoken`	form-data	No	Publisher identifier for tracking

Behavior:

Compressed files (.zip, .gz, .tar, .jar): Extracted in-process. For a CSV pipeline with several files in the archive, the files are concatenated (headers on files 2+ skipped) and submitted as one batch job; otherwise each extracted file is submitted as its own job
Uncompressed files: Processed immediately in-memory
Schema evolution: If a CSV file contains new columns not in the pipeline schema, they are automatically added (as string type) and the destination table is altered. See Schema Evolution.

Example:

curl -X POST http://localhost:8080/api/v1/pipeline/upload \
  -F "file=@/path/to/data.csv" \
  -F "pipeline=sales_data" \
  -F "publishertoken=batch-001"

Response: 200 OK with the pipeline token (for uncompressed files):

pt-abc12345-6789-...

For compressed files, the response is 200 OK with the single pipeline token of the batch job (CSV batch mode, or a single-file archive), or the plain-text summary N file(s) submitted when each extracted file was submitted as its own job.

Generate Pipeline Schema

Upload a CSV file to automatically infer the schema and generate a partial pipeline configuration.

POST /api/v1/pipeline/generate
Content-Type: multipart/form-data

Parameters:

Parameter	Type	Required	Description
`file`	form-data	Yes	Data file to analyze (CSV, JSON, or XML)
`pipeline`	form-data	No	Pipeline name (auto-derived from filename if omitted)
`delimiter`	form-data	No	CSV delimiter (default: `,`)
`header`	form-data	No	Whether file has header row
`allStrings`	form-data	No	If `true`, all fields are typed as `string` (default: `false`)

Example:

curl -X POST http://localhost:8080/api/v1/pipeline/generate \
  -F "file=@/path/to/sample.csv" \
  -F "pipeline=my_pipeline"

Response: 200 OK with a partial PipelineConfig JSON:

{
  "name": "my_pipeline",
  "source": {
    "fileAttributes": {
      "csvAttributes": {
        "delimiter": ",",
        "header": true
      }
    },
    "schemaProperties": {
      "fields": [
        {"name": "id", "type": "int"},
        {"name": "name", "type": "string"},
        {"name": "amount", "type": "double"},
        {"name": "created_at", "type": "string"}
      ]
    }
  },
  "destination": {
    "database": {
      "dbName": "DATABASE_NAME",
      "schema": "SCHEMA_NAME",
      "table": "TABLE_NAME",
      "usePostgres": true
    }
  }
}

Inferred types: boolean, int, bigint, float, double, string, date, timestamp Note: For CSV files, the AI analyzes the content and infers types. For JSON and XML files, a default config is generated with a single _json or _xml string field. When allStrings is true, all fields are set to string (used by the MCP server for reliable ingestion). The destination block carries DATABASE_NAME / SCHEMA_NAME / TABLE_NAME placeholders; replace them before registering the config with POST /pipeline.

Profile Data

Upload a data file for AI-powered profiling — get summary statistics, quality issues, and suggested validation rules.

POST /api/v1/pipeline/profile
Content-Type: multipart/form-data

Parameters:

Parameter	Type	Required	Description
`file`	form-data	Yes	Data file to profile
`delimiter`	form-data	No	CSV delimiter (default: `,`)
`header`	form-data	No	Whether file has header row (default: `true`)
`sampleSize`	form-data	No	Number of rows to sample (default: `200`)

Example:

curl -X POST http://localhost:8080/api/v1/pipeline/profile \
  -F "file=@/path/to/data.csv" \
  -F "sampleSize=200"

Response: 200 OK with a JSON profile including summary statistics, quality issues, recommendations, and suggested data quality rules. See AI Data Profiling for the full response format.

API Documentation

Pipelines

Ingestion

AI

Status

Query

Search

Health

Metadata

Configuration

Secrets

Taps

Code repository

Versions

Auth

Agent Policy

Recovery Agent

Audit Log

API Keys

Assistant

Chat

MCP

Tap Prompts

Provenance

Internal callbacks

Data Ingestion API

Upload a File

Generate Pipeline Schema

Profile Data

​Upload a File

​Generate Pipeline Schema

​Profile Data

Upload a File

Generate Pipeline Schema

Profile Data