AI Schema Generation API

The pipeline can automatically generate a ready-to-use pipeline configuration from an uploaded file using an AI model. The generated config can be pasted directly into POST /api/v1/pipeline to register the pipeline without writing any JSON by hand.

How It Works

Uploaded file
  |
  v
File type detection (CSV / JSON / XML)
  |
  +-- JSON / XML --> Fixed schema (_json or _xml field)
  |                  No AI call needed
  |
  +-- CSV / other --> First 100 lines sent to AI model
                      AI infers column names and data types
                      Returns JSON array of field definitions
  |
  v
Config builder assembles full PipelineConfig JSON
  - source.schemaProperties.fields  (AI-inferred or fixed)
  - source.fileAttributes            (csvAttributes / jsonAttributes / xmlAttributes)
  - destination                      (Postgres for CSV, MongoDB for JSON/XML)
  |
  v
Response: complete PipelineConfig JSON ready to register

The response is not registered automatically — it is returned to the caller so you can review it, fill in the placeholder values, and then POST it to /api/v1/pipeline.

Endpoint

POST /api/v1/pipeline/generate
Content-Type: multipart/form-data

Parameters:

Parameter	Type	Required	Description
`file`	form-data (file)	Yes	The file to analyze
`pipeline`	query	No	Pipeline name. If omitted, derived from the filename (lowercased, non-alphanumeric characters replaced with `_`)
`delimiter`	query	No	Column delimiter for delimited files. Defaults to `,`
`header`	query	No	Whether the file has a header row (default `false`; pass `true` for a file with a header row)
`allStrings`	query	No	If `true`, all fields are typed as `string` (default: `false`)
`x-api-key`	header	No	API key (required if `useApiKeys: true`)

Schema Rules by File Type

File type	Schema	Default destination
CSV / delimited	AI infers field names and types from file content	PostgreSQL (`usePostgres: true`)
JSON (`.json`)	Single field: `_json` (type `string`)	MongoDB (`useMongoDB: true`)
XML (`.xml`)	Single field: `_xml` (type `string`)	MongoDB (`useMongoDB: true`)

JSON and XML files use a fixed schema because the pipeline stores them as raw documents — no column inference is needed. Valid AI-inferred types: boolean, int, bigint, float, double, string, date, timestamp

When the model declines

An AI provider’s safety classifier can decline to look at a file, most often one that looks like personal data. For a CSV or other delimited file the endpoint then still returns 200 with every field typed string. With header=true and a first line that reads as column names, the fields keep the header’s names exactly as allStrings=true writes them, except that an empty header cell (empty, plain spaces or a byte-order mark only; an empty or newline-only file included), a cell that only renders blank (such as a Hangul filler), or a name repeated ignoring case and surrounding spaces, is named column_N for its position, so no field name is blank or duplicated. A cell holding only other characters that are not letters (a non-breaking or zero-width space, for example) makes line 1 read as data: every column is then numbered and csvAttributes.header is false. When header is false (the default when the parameter is omitted) or line 1 looks like data, the columns are named column_1, column_2, … and the config sets csvAttributes.header to false. The response carries a top-level "aiDeclined": true, and the pipeline wizard shows a one-line note; set the types by hand. aiDeclined is a response-only flag: it is never saved with a pipeline. The provider’s own text is not included.

Provider family	Recognised as a decline
Anthropic (direct or AWS Bedrock)	`stop_reason: "refusal"`
OpenAI, Azure OpenAI, Grok and Ollama (chat completions, streamed or not)	`finish_reason: "content_filter"`, or a non-empty `message.refusal` (`delta.refusal` when streamed)
OpenAI Responses API models	an output item or message content part of type `refusal`, or `incomplete_details.reason: "content_filter"`

Only response-side declines are recognised. Azure OpenAI’s prompt-side content filter answers with HTTP 400 (code: content_filter) and is still returned as an error. Nothing else counts as a decline: a missing or wrong provider key, a timeout, a rate limit or any other provider error is still returned as an error, so a misconfiguration is not hidden. Other AI features (search answers, CodeGen, tap generation, fix suggestions and the rest) have no local fallback: a decline there, including a content_filter finish that used to return the cut-off text, is reported as an error. JSON Schema and XSD generation (POST /api/v1/config/generate-schema) have no local fallback: a decline there is still returned as an error, with the message “The model declined this request (…)”.

Example: CSV File

curl -X POST http://localhost:8080/api/v1/pipeline/generate \
  -H "x-api-key: your-api-key" \
  -F "file=@./stock_price.csv" \
  -F "pipeline=stock_price" \
  -F "header=true"

Response:

{
  "name": "stock_price",
  "source": {
    "schemaProperties": {
      "fields": [
        { "name": "symbol",    "type": "string" },
        { "name": "date",      "type": "string" },
        { "name": "open",      "type": "double" },
        { "name": "high",      "type": "double" },
        { "name": "low",       "type": "double" },
        { "name": "close",     "type": "double" },
        { "name": "volume",    "type": "int"    },
        { "name": "adj_close", "type": "double" }
      ]
    },
    "fileAttributes": {
      "csvAttributes": { "delimiter": ",", "header": true, "encoding": "UTF-8" }
    }
  },
  "destination": {
    "database": {
      "dbName": "DATABASE_NAME",
      "schema": "SCHEMA_NAME",
      "table": "TABLE_NAME",
      "usePostgres": true
    }
  }
}

Replace DATABASE_NAME, SCHEMA_NAME, and TABLE_NAME with real values, then register the pipeline:

curl -X POST http://localhost:8080/api/v1/pipeline \
  -H "Content-Type: application/json" \
  -d '<paste response here>'

Example: JSON File

curl -X POST http://localhost:8080/api/v1/pipeline/generate \
  -H "x-api-key: your-api-key" \
  -F "file=@./events.json"

Response:

{
  "name": "events",
  "source": {
    "schemaProperties": {
      "fields": [
        { "name": "_json", "type": "string" }
      ]
    },
    "fileAttributes": {
      "jsonAttributes": { "everyRowContainsObject": false, "encoding": "UTF-8" }
    }
  },
  "destination": {
    "database": {
      "dbName": "DATABASE_NAME",
      "table": "TABLE_NAME",
      "useMongoDB": true
    }
  }
}

Configuration

AI schema generation is disabled by default. To enable it: Schema generation uses the codegen AI slot. As of v1.5.6, AI configuration is split into three independent self-describing Vault secrets — see AI Configuration for the full picture. application.yaml (or docker/config/application.yaml for Docker deployments):

ai:
  enabled: "true"
  aiPrimary:
    secretName: "oss/ai-primary"
  codegen:
    secretName: "oss/codegen"
  embedding:
    secretName: "oss/embedding"

Vault secret (each secret is self-describing — provider, endpoint, model, apiKey, and optionally version, all inline):

vault kv put secret/oss/codegen \
  provider="anthropic" \
  endpoint="https://api.anthropic.com/v1/messages" \
  model="claude-opus-4-8" \
  apiKey="sk-ant-..." \
  version="2023-06-01"

For OpenAI:

vault kv put secret/oss/codegen \
  provider="openai" \
  endpoint="https://api.openai.com/v1/chat/completions" \
  model="gpt-5.4" \
  apiKey="sk-..."

The Vault secret keys:

Key	Description
`provider`	`anthropic`, `openai`, `azure`, `bedrock`, `grok`, or `ollama`
`endpoint`	The AI provider API URL
`model`	The model name to use
`apiKey`	The API key for authentication (omit for local Ollama)
`version`	Optional API version header (e.g. Anthropic’s `anthropic-version`)

The pipeline reads the secret at startup. If ai.enabled: true and the codegen secret is missing or malformed, startup will fail with a descriptive error. docker/vault-init.sh seeds this automatically from ANTHROPIC_API_KEY or OPENAI_API_KEY in .env.

Supported Providers

Provider	`provider` value	Auth header
Anthropic Claude	`anthropic`	`x-api-key` + `anthropic-version: 2023-06-01`
OpenAI	`openai`	`Authorization: Bearer`
Azure OpenAI	`azure`	API key or Entra ID (keyless)
Amazon Bedrock	`bedrock`	AWS credentials
Grok (xAI)	`grok`	`Authorization: Bearer`
Ollama (local)	`ollama`	none

Any other value for provider will cause startup to fail with an unsupported provider error.

API Documentation

Pipelines

Ingestion

AI

Status

Query

Search

Health

Metadata

Configuration

Secrets

Taps

Code repository

Versions

Auth

Agent Policy

Recovery Agent

Audit Log

API Keys

Assistant

Chat

MCP

Tap Prompts

Provenance

Internal callbacks

AI Schema Generation API

How It Works

Endpoint

Schema Rules by File Type

When the model declines

Example: CSV File

Example: JSON File

Configuration

Supported Providers

​How It Works

​Endpoint

​Schema Rules by File Type

​When the model declines

​Example: CSV File

​Example: JSON File

​Configuration

​Supported Providers

How It Works

Endpoint

Schema Rules by File Type

When the model declines

Example: CSV File

Example: JSON File

Configuration

Supported Providers