# Approval gates for AI agents writing to a database
Source: https://docs.datris.ai/agent-policy

Decide per action whether an agent runs it unattended, waits for a person to approve it, or is refused. Enforced by the platform on every MCP client and API key, not by a prompt.

The agent policy is the boundary between agents being useful and agents being safe. For every kind of change an agent can make through Datris — create a tap, run it, delete a pipeline, migrate a destination table's column types — an administrator decides one of three things:

| Mode | What happens when an agent tries it |
| - | - |
| **Auto** | It runs, exactly as today. |
| **Approve** | It does **not** run. The request is parked, the agent is told it is waiting for approval, and a person approves or rejects it under **Activity → Approvals**. On approval the platform performs the original request on the approver's behalf. |
| **Deny** | It is refused with a clear error the agent can read. |

Anything you have not set is **Auto**, so turning the feature on changes nothing until you set a rule. People using the Datris UI are never gated — they are the approvers.

This is infrastructure, not a system prompt. An agent cannot talk its way past it, and it applies to every client — the in-platform Assistant, Claude Desktop, Cursor, the CLI, or any script holding an API key.

## Enabling it

Off by default — existing installs are unchanged until you opt in.

1. Add the flag to your `.env`:

   ```bash theme={null}
   USE_AGENT_POLICY=true
   ```

2. Recreate the Datris container:

   ```bash theme={null}
   docker compose up -d --force-recreate datris
   ```

3. Open **Configuration → Agent Policy** (admins only when user authentication is on). Set the actions you care about, or click **Use recommended** for a starting point that pauses deletes, job kills and destination-type migrations and refuses secret, code-repository and platform-configuration writes (`secret:write`, `code-repo:write`, `config:write`).

The policy applies within a few seconds of saving; no restart.

## What counts as an agent

| Caller | Gated? |
| - | - |
| A person clicking in the Datris UI | No |
| The in-platform Assistant, Ops or Catalog chat acting for a logged-in user | **Yes** — attributed to the user, but it is still the agent proposing |
| The Configuration assistant, after you click Confirm | No — you are the approver |
| An external MCP client (Claude Desktop, Cursor, Claude Code, …) or a script with an API key | **Yes** |
| A scheduled tap run, Kafka or object-store trigger | No — these are the platform's own work |
| A tap script reading platform data during a run | No |

## Actions

The actions you can set are exactly the ones the [capability model](/api-keys) already recognizes — `pipeline:create`, `tap:delete`, `job:kill`, `secret:write`, and so on — plus two refinements: `pipeline:update:dest-types`, the destination column-type migration, which shares a capability with an ordinary pipeline save but rewrites a landed table, and `tap:run:enable-ordered`, which enables and runs every tap of a catalog in dependency order as one job (one approval for the whole job). A sub-action inherits its parent's mode unless set; `resource:*` sets every action on a resource.

Reads, queries, searches and metadata lookups are never gated. What a key may read is governed by its [capabilities](/api-keys).

Two rules are not part of the policy and cannot be changed by it: an agent can never edit the policy, and an agent can never approve or reject an approval — whatever capabilities its key holds.

### Per-resource overrides

An override tightens the policy for one pipeline or tap: `tap:prices → tap:run = approve` pauses runs of that one tap while every other tap runs freely. Overrides can only make an action stricter, never looser.

## Approvals

When an agent's request hits an **Approve** rule, the agent receives a `pending_approval` response with an approval id instead of a result. The Assistant will tell you plainly that the action is waiting, and it can poll for the decision. The request itself — method, path and body, with secret-looking values masked — is shown on the approval card so you see exactly what will run.

Approve, and the platform performs the original request, attributed to you. The audit log then holds two linked entries: the agent's queued request (with the reason it gave, if any) and your approval executing it. Reject, and the agent is told so on its next poll. Approvals not decided within the configured time (24 hours by default) expire on their own.

An approval executes what was proposed, not what the agent would propose now: if the pipeline or tap changed in the meantime, the approval is refused as stale and the agent has to propose again.

Some requests cannot be parked — file uploads, and bodies over 256 KB. If such an action is set to **Approve**, the agent is told to ask you to do it in the UI instead.

## For agents

Three MCP tools support the flow:

| Tool | Purpose |
| - | - |
| `get_agent_policy` | Read the policy before acting, to know whether a delete or migration will run or queue |
| `list_pending_approvals` | The actions this agent has queued, with their state |
| `get_approval` | Poll one approval by id until it is executed, rejected or expired |

Every mutating tool also accepts an optional `reason` — one line stating why the agent wants the change. It is stored in the [audit log](/audit-log) and shown on the approval card.

## Limits

| Setting | Default | Meaning |
| - | - | - |
| Pending approval TTL | 24 hours | Undecided approvals expire after this |
| Max pending per agent | 50 | An agent with this many undecided approvals is told to wait |

## Recovery mode

The policy document also carries the [recovery agent](/incidents)'s dial — `recovery.mode` (`off` / `propose` / `autopilot`) and its limits — plus per-resource `recovery` overrides. The recovery agent's every action passes through this same policy gate: under `propose`, everything it wants to do queues here for your approval.

## Related

* [API Keys](/api-keys) — what a key *may* do; the policy decides what it may do *unattended*
* [Audit Log](/audit-log) — where queued requests, decisions and executions are recorded


# Give your coding agent the Datris skill
Source: https://docs.datris.ai/agent-skill

Install the datris-platform skill in a project so Claude Code, Codex, Hermes, OpenClaw and other coding agents use Datris for data work instead of writing their own loaders, schedulers and vector stores.

Connecting an agent to the [MCP server](/mcp-server) gives it the tools. The **datris-platform skill** gives it the judgment: when to reach for Datris during ordinary project work, in what order to access it, and the rules that keep every run verifiable.

A skill is a folder with a `SKILL.md` file that an agent loads when a task matches its description. This one triggers whenever the work involves getting data in from a file, an API, a website, a database or a document set; validating, transforming or landing it; building a semantic-search index; scheduling a recurring pull; querying data the project already landed; or tracing where a dataset came from. The agent does not need to be told to use Datris each time.

The skill lives in the Datris repository at [`skills/datris-platform`](https://github.com/datris/datris-platform-oss/tree/main/skills/datris-platform).

## What the skill tells the agent

* **Access order.** Use the MCP tools for every action the agent takes itself. Use the REST API only for code it writes into the project, which runs later without an agent, or when a capability has no MCP tool. Never shell out to the CLI; it is a tool for people at a terminal.
* **If Datris isn't installed.** How to tell that no instance is reachable, to offer the user a local install and wait for a yes, the install command, and the two public endpoints that show the server is up. The full steps stay on [Install for Agents](/install-for-agents).
* **First contact.** Confirm the MCP connection, read the server version and budgets, read the [agent policy](/agent-policy), pick one [Data Catalog](/data-catalog) for the project, and look for existing pipelines, taps and secrets before creating anything.
* **What to use Datris for.** A table mapping project needs (load a file, pull from an API on a schedule, store a credential, validate rows, land data in a warehouse, build a document index, preview rows without keeping them, find and query existing data, trace provenance) to the Datris capability that does it.
* **Working rules.** Verify completion by polling the job rollup rather than trusting a response body, read a tap run's `persisted` flag before anything else, test a tap before running or scheduling it, schedule inside Datris, keep configs to source plus destination, respect approval gates, and only act on what the user asked for in the current turn.

Reference files alongside the skill carry the full tool catalog, the REST endpoints a service needs, a destination chooser, step-by-step workflows and setup. The agent reads those on demand, so the skill itself stays short.

## Install

Copy the folder into the project where the agent looks for skills. Pick the lines for your agent.

```bash theme={null}
git clone --depth 1 https://github.com/datris/datris-platform-oss.git /tmp/datris-oss

# Claude Code (project)
mkdir -p .claude/skills && cp -r /tmp/datris-oss/skills/datris-platform .claude/skills/

# Claude Code (every project on this machine)
cp -r /tmp/datris-oss/skills/datris-platform ~/.claude/skills/

# OpenAI Codex, Hermes, Cursor and other agents that read the .agents convention (project)
mkdir -p .agents/skills && cp -r /tmp/datris-oss/skills/datris-platform .agents/skills/

# Hermes (user-wide), or install straight from GitHub without cloning
cp -r /tmp/datris-oss/skills/datris-platform ~/.hermes/skills/
hermes skills install datris/datris-platform-oss/skills/datris-platform

# OpenClaw (user-wide)
cp -r /tmp/datris-oss/skills/datris-platform ~/.openclaw/workspace/skills/
```

Codex also reads `AGENTS.md` at the project root. One line there makes the skill discoverable before any task triggers it:

```
Datris is this project's data platform. Read .agents/skills/datris-platform/SKILL.md before any data ingestion, storage, search or scheduling work.
```

### Agents without a skills loader

For an agent that accepts project or system instructions but has no skills directory, paste the body of `SKILL.md` (everything below the frontmatter) into those instructions and keep the `references/` folder in the repository so the agent can read the tool catalog and workflows when it needs them.

## Before the agent starts

When no Datris instance is reachable, the skill tells the agent to say so and offer to install one locally, and to install only if the user agrees. To have everything ready before the agent starts:

1. [Install Datris](/installation) (from an agent or CI job, see [Install for Agents](/install-for-agents)) and connect the agent: [Claude Desktop and Claude Code](/configuring-claude), [OpenClaw](/configuring-openclaw), or the generic snippet on the [MCP server](/mcp-server#configuring-claude) page.
2. For any shared install, turn on [API keys](/api-keys), the [agent policy](/agent-policy) and the [audit log](/audit-log), and issue the agent its own scoped key. The [Claude Code to PostgreSQL recipe](/recipes/claude-code-postgres) walks through this in about fifteen minutes.

## Keeping the skill current

The skill's tool catalog mirrors the [MCP server tool list](/mcp-server#available-tools). Re-copy the folder when you upgrade Datris so the agent learns about new tools. If you edit the skill for your project, keep your copy under version control alongside the project.


# AI Configuration
Source: https://docs.datris.ai/ai-configuration

Configure AI features

The data platform integrates with AI providers for these features:

* **Schema generation** — upload a file to `POST /api/v1/pipeline/generate` and receive a complete pipeline configuration with inferred field names and types
* **Data quality rules** — describe validation logic in plain English using `aiRule` instead of writing code
* **Transformations** — describe data transformations in plain English using `aiTransformation`
* **Data profiling** — upload a file and get summary statistics, quality issues, and suggested validation rules
* **Header validation** — AI-powered fuzzy matching for CSV/delimited file headers against the pipeline schema
* **Error explanation** — AI analyzes job failures and explains the root cause in plain English
* **Vector search & embeddings** — required when pushing data to vector destinations (Qdrant, Weaviate, Milvus, Chroma, pgvector) or running semantic search

## Independent Secrets

AI configuration is split into independent, self-describing Vault secrets, each pointing at its own endpoint/model/key. Each is a separate slot in `application.yaml`:

```yaml theme={null}
ai:
  enabled: "true"
  aiPrimary:
    secretName: "oss/ai-primary"
  codegen:
    secretName: "oss/codegen"
  embedding:
    secretName: "oss/embedding"
  webSearch:
    secretName: "oss/web-search"
```

| Slot | Purpose | Used by |
| - | - | - |
| `aiPrimary` | General AI tasks | Error explanation, brainstorm chat, profiling, header validation, CRON generation, AI Q\&A |
| `codegen` | Code-generation tasks | Tap scripts, `aiRule`, `aiTransformation`, JSON Schema / XSD generation, natural-language → SQL |
| `embedding` | Vector embeddings | Vector destinations (Qdrant, Weaviate, Milvus, Chroma, pgvector), semantic search |
| `webSearch` | Live web lookup (optional) | Tap script generation and tap diagnosis when the model needs current API documentation |

Each Vault secret carries its full configuration inline:

| Field | Required | Notes |
| - | - | - |
| `provider` | Yes | `anthropic`, `openai`, `azure`, `bedrock`, `grok`, or `ollama` |
| `endpoint` | Yes (blank OK for bedrock) | API endpoint URL. Azure has no default — use your resource URL, e.g. `https://YOUR-RESOURCE.openai.azure.com/openai/v1/chat/completions`. Bedrock derives its endpoint from the AWS region when blank |
| `model` | Yes | Model identifier. For Azure this is your **deployment name**; for Bedrock an invokable model id (e.g. `anthropic.claude-sonnet-5`) |
| `apiKey` | No (not used by ollama or bedrock) | Inline provider API key. The [shared API key store](#shared-api-key-store) is checked first and takes precedence when set. Bedrock authenticates with AWS credentials instead — see the [Bedrock path](#amazon-bedrock-path) |
| `version` | Optional | API version (Anthropic uses `2023-06-01`) |

The resolver reads each secret at the configured path and uses whatever it finds inside — there is no path derivation from a YAML `provider` field. This means the four slots can independently use different providers (e.g. Anthropic for `aiPrimary`, Ollama for `codegen`, OpenAI for `embedding`).

**All three seeded slots (`aiPrimary`, `codegen`, `embedding`) support Ollama** — you can run the entire platform on local models with no API keys. The Configuration UI offers Ollama as a provider choice for every section, with a free-text model input since Ollama supports hundreds of community models.

### Shared API Key Store

Provider API keys live in a shared per-provider secret, `{env}/ai-keys` (`oss/ai-keys` in single-tenant deployments), with one field per provider: `anthropicApiKey`, `openaiApiKey`, `azureApiKey`, and `grokApiKey`. When a slot needs a key, resolution order is:

1. **Shared key store** (`{env}/ai-keys`) — checked first; takes precedence over any inline key
2. **Inline `apiKey`** on the slot secret — legacy deployments that stored the key on the slot itself
3. **`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `AZURE_OPENAI_API_KEY` / `XAI_API_KEY`** container env vars — single-tenant only

Keys are stored once per provider rather than once per slot, so switching a slot between providers never loses the other provider's key. This is the store behind the Configuration UI's shared **API Keys** panel — entering a key there writes it to `{env}/ai-keys` for all slots to use.

The same store holds the Bedrock provider's AWS credentials, in four fields: `awsAccessKeyId`, `awsSecretAccessKey`, `awsSessionToken` (optional, for temporary credentials), and `awsRegion`. Bedrock resolves them in its own order: the store, then the standard `AWS_*` container env vars (single-tenant only), then the server's IAM role / AWS default credential chain — so an instance-role deployment can run Bedrock with no stored secrets at all.

It also holds the Azure Entra service-principal credentials for
[keyless Azure OpenAI auth](#keyless-auth-microsoft-entra-id), in three fields:
`azureTenantId`, `azureClientId`, and `azureClientSecret`. Azure resolves auth per
request: a stored API key wins; otherwise the SP trio; otherwise (single-tenant
only) the Azure default credential chain — `AZURE_*` env vars, managed identity on
Azure compute, workload identity, or `az login`.

## Quick Start with `.env`

On first boot, `docker/vault-init.sh` seeds the `ai-primary`, `codegen`, and `embedding` Vault secrets automatically (the optional `web-search` slot is not seeded — configure it from the Configuration tab) based on which API key you put in `.env` (see [How Configuration Persists](#how-configuration-persists) — after the first boot, the Configuration tab is the source of truth). Set `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, the `AZURE_OPENAI_*` trio, `XAI_API_KEY` (see the [Grok path](#grok-xai-path)), or `AI_PROVIDER=bedrock` (see the [Bedrock path](#amazon-bedrock-path)):

```bash theme={null}
# .env
ANTHROPIC_API_KEY=sk-ant-...
# OPENAI_API_KEY=
```

### OpenAI path

When `OPENAI_API_KEY` is set, all three slots are seeded against OpenAI:

| Slot | Provider | Endpoint | Model |
| - | - | - | - |
| `oss/ai-primary` | openai | `https://api.openai.com/v1/chat/completions` | `gpt-5.6-sol` |
| `oss/codegen` | openai | `https://api.openai.com/v1/chat/completions` | `gpt-5.6-sol` |
| `oss/embedding` | openai | `https://api.openai.com/v1/embeddings` | `text-embedding-3-small` |

GPT-6 models (`gpt-6-astra`) are supported for the chat and codegen slots. Requests to the GPT-6 family are routed to the Responses API automatically, whatever endpoint the slot is configured with, and no temperature is sent. Reasoning effort stays at the platform default (`medium`); the `xhigh` and `max` levels are not exposed.

### Azure OpenAI path

Use your Azure OpenAI resource instead of api.openai.com. All three values are required together — Azure endpoints embed your resource name and models are addressed by **deployment name**, so there are no universal defaults:

```bash theme={null}
# .env
AI_PROVIDER=azure
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://YOUR-RESOURCE.openai.azure.com
AZURE_OPENAI_MODEL=gpt-5-2          # your chat deployment's name
```

| Slot | Provider | Endpoint | Model |
| - | - | - | - |
| `oss/ai-primary` | azure | `{AZURE_OPENAI_ENDPOINT}/openai/v1/chat/completions` | your deployment name |
| `oss/codegen` | azure | `{AZURE_OPENAI_ENDPOINT}/openai/v1/chat/completions` | `CODEGEN_MODEL` or same deployment |
| `oss/embedding` | tei | `http://tei:80/v1/embeddings` | `BAAI/bge-m3` |

Embeddings default to the bundled service; set `EMBEDDING_PROVIDER=azure` to use an embedding deployment on your resource instead (see [mix-and-match](#mix-and-match-anthropic-chat--openai-embeddings)).

<Tip>
  **Name deployments after their model** (e.g. `gpt-5-2`, `o4-mini` — the Azure portal's default). The platform detects model-specific behavior, like omitting sampling parameters that reasoning models reject, from the model name — which for Azure is the deployment name.
</Tip>

Datris targets Azure's **v1 API** (`/openai/v1/...`), which needs no `api-version` parameter. Legacy deployment-scoped URLs also work — put the full URL including `?api-version=...` in the section's Endpoint field in the Configuration UI.

#### Keyless auth (Microsoft Entra ID)

Azure OpenAI also works **without an API key** — required when your resource has
`disableLocalAuth` set (API keys disabled, often tenant-wide by policy), and useful
anywhere you'd rather rotate credentials centrally. Pick the authentication mode in
the Configuration tab's Azure OpenAI credentials section:

* **Service principal** — works for any Datris deployment. Enter the Entra
  **Tenant ID**, **Client ID**, and **Client Secret** of an app registration.
  Alternatively set the `AZURE_TENANT_ID` / `AZURE_CLIENT_ID` / `AZURE_CLIENT_SECRET`
  container env vars (single-tenant only).
* **Managed identity** — zero stored secrets. Available when the Datris server runs
  on Azure compute (VM, AKS, App Service, Container Apps); nothing to enter at all.

Either way, the identity needs the **Cognitive Services OpenAI User** role on the
Azure OpenAI resource:

```bash theme={null}
az role assignment create \
  --assignee <client-id-or-principal-id> \
  --role "Cognitive Services OpenAI User" \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<resource>
```

Precedence: a stored API key always wins over Entra configuration. If you're moving
a resource to `disableLocalAuth`, switch the auth mode in the Configuration tab —
saving the switch clears the stored key so it can't shadow Entra auth. Chat,
CodeGen, embeddings, and the Assistant all honor the same resolution; tokens are
acquired and refreshed automatically.

### Amazon Bedrock path

Run Claude through your AWS account — IAM auth, AWS billing, no Anthropic API key. Bedrock is **explicit-only**: it activates with `AI_PROVIDER=bedrock`, never from the mere presence of AWS keys (those are routinely set for S3 destinations).

```bash theme={null}
# .env
AI_PROVIDER=bedrock
AWS_REGION=us-east-1
# Optional — leave both unset to use the host's IAM role / default credential chain:
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
# Optional model override (default shown):
# BEDROCK_MODEL=anthropic.claude-opus-5-5
```

| Slot | Provider | Endpoint | Model |
| - | - | - | - |
| `oss/ai-primary` | bedrock | *(blank — derived from region)* | `BEDROCK_MODEL` or `anthropic.claude-opus-5-5` |
| `oss/codegen` | bedrock | *(blank — derived from region)* | `CODEGEN_MODEL` or `anthropic.claude-opus-5-5` |
| `oss/embedding` | tei | `http://tei:80/v1/embeddings` | `BAAI/bge-m3` |

Notes:

* **Model ids** are Bedrock's `anthropic.`-prefixed forms (e.g. `anthropic.claude-opus-5-5`). Models that aren't on-demand invokable in your region are addressed by their **cross-region inference profile** id (e.g. `us.anthropic.claude-...`). The Configuration UI discovers the invokable list live from your account when its IAM policy allows it.
* **Enable model access first.** Models must be enabled for your account in the AWS Bedrock console — a model can appear in listings and still fail with `AccessDeniedException` on the first call until access is granted.
* **Claude Fable 5 / 5.1 need a one-time data-retention opt-in.** Fable 5 and 5.1 (and Mythos 5 / 5.1) require Bedrock's `provider_data_share` retention mode — prompts/completions are shared with Anthropic and retained up to 30 days for trust and safety (not used for training; other Claude models are unaffected by the setting). Without it, calls fail with "data retention mode 'default' is not available for this model". Opt in once per account: `PUT /data-retention` with `{"mode": "provider_data_share"}` on the Bedrock control plane (IAM action `bedrock:PutAccountDataRetention`) — see AWS's Bedrock "Data retention" documentation. There is no console UI for this setting at the time of writing.
* **Minimal IAM policy:** `bedrock:InvokeModel` on the models you use, plus `bedrock:ListFoundationModels` and `bedrock:ListInferenceProfiles` for the UI's live model dropdown (optional — without them the UI shows a standard list).
* **Endpoint stays blank** in the slot secrets; the invoke URL is derived from the region and model. Set an explicit endpoint only for VPC endpoints or GovCloud.
* **Embeddings and web search are not available on Bedrock** — embeddings default to the bundled bge-m3 service (or pair with OpenAI/Azure), and web search stays on Anthropic/OpenAI.
* Extended thinking works the same as on the direct Anthropic provider, including the always-on thinking of Claude Fable 5 and 5.1.

### Grok (xAI) path

Run chat and CodeGen on Grok through xAI's OpenAI-compatible API. One key from [console.x.ai](https://console.x.ai) covers both slots:

```bash theme={null}
# .env
XAI_API_KEY=...
# Optional model override (default shown):
# GROK_MODEL=grok-4.7
```

| Slot | Provider | Endpoint | Model |
| - | - | - | - |
| `oss/ai-primary` | grok | `https://api.x.ai/v1/chat/completions` | `GROK_MODEL` or `grok-4.7` |
| `oss/codegen` | grok | `https://api.x.ai/v1/chat/completions` | `CODEGEN_MODEL` or `grok-4.7` |
| `oss/embedding` | tei | `http://tei:80/v1/embeddings` | `BAAI/bge-m3` |

Notes:

* xAI has **no embeddings API**, so embeddings default to the bundled bge-m3 service (or pair with OpenAI/Azure via `EMBEDDING_PROVIDER` — see [mix-and-match](#mix-and-match-anthropic-chat--openai-embeddings)).
* **Web search is not available on Grok** — pair it with Anthropic or OpenAI web search if you need it.
* Grok's reasoning happens server-side; unlike Claude's extended thinking it isn't surfaced in the Assistant's thinking panel.
* With multiple keys in `.env`, pin the choice explicitly with `AI_PROVIDER=grok` (auto-detection only picks Grok when `XAI_API_KEY` is the sole key).

### Anthropic path (with bundled embedding service)

When `ANTHROPIC_API_KEY` is set, `aiPrimary` and `codegen` use Claude. Anthropic has no embeddings API, so `embedding` falls back to a **bundled embedding service** running `bge-m3` (a strong open-source 1024-dimension embedding model). No OpenAI key required, vector destinations work out of the box.

| Slot | Provider | Endpoint | Model |
| - | - | - | - |
| `oss/ai-primary` | anthropic | `https://api.anthropic.com/v1/messages` | `claude-opus-5-5` |
| `oss/codegen` | anthropic | `https://api.anthropic.com/v1/messages` | `claude-opus-5-5` |
| `oss/embedding` | tei | `http://tei:80/v1/embeddings` | `BAAI/bge-m3` |

The `tei` service (HuggingFace Text Embeddings Inference) is part of `docker-compose.yml`. On first start it downloads `bge-m3` (\~2.2 GB ONNX weights) into a persistent volume; subsequent restarts use the cached model.

### Fully local with Ollama

You can run all three slots on a local Ollama instance — no cloud API keys needed. Ollama is opt-in: uncomment the optional `ollama` service block in `docker-compose.yml` (and the `ollama-data` volume), then `docker compose up -d ollama` and pull the models you want (e.g. `docker exec ollama ollama pull qwen3:14b`). Alternatively, point at an Ollama instance on your host machine using `host.docker.internal`:

| Slot | Provider | Endpoint | Model (example) |
| - | - | - | - |
| `oss/ai-primary` | ollama | `http://ollama:11434/v1/chat/completions` (bundled) or `http://host.docker.internal:11434/v1/chat/completions` (host) | `qwen3:14b` |
| `oss/codegen` | ollama | same as above | `qwen2.5-coder:7b-instruct` |
| `oss/embedding` | ollama | same as above | `bge-m3` |

With every slot on Ollama no data goes to a cloud model, but the chat assistants (the Assistant, Search chat and the other chat panels) cannot run on Ollama and need a cloud provider; [Data Flows](/production/data-flows) lists where each kind of data goes for each provider choice.

If you only need local AI Primary and CodeGen but want to keep the bundled embedding service (TEI), leave `oss/embedding` pointed at `http://tei:80/v1/embeddings` with `model="BAAI/bge-m3"` and only switch the AI Primary and CodeGen slots to Ollama.

The Configuration UI makes this easy — select **Ollama (if optional service enabled)** as the provider and type your model name. For the AI Primary and CodeGen slots the endpoint defaults to `http://host.docker.internal:11434/v1/chat/completions` (edit it to `http://ollama:11434/v1/chat/completions` for the bundled service); for the Embedding slot it defaults to `http://ollama:11434/v1/embeddings`. No API key is required.

### Optional model overrides

Override the seeded models by setting these in `.env`:

```bash theme={null}
ANTHROPIC_MODEL=claude-opus-5-5
OPENAI_MODEL=gpt-5.6-sol
GROK_MODEL=grok-4.7
CODEGEN_MODEL=claude-opus-5-5   # or gpt-5.6-sol if using OpenAI, grok-4.7 for Grok
ANTHROPIC_OVERLOAD_FALLBACK_MODEL=claude-sonnet-5   # used for the current request when an Anthropic Opus call stays overloaded after retries
```

### Mix-and-match: Anthropic chat + OpenAI embeddings

The embedding slot is **decoupled from the chat slots** — set `EMBEDDING_PROVIDER` in `.env` to override what `vault-init.sh` seeds for `oss/embedding`, regardless of `AI_PROVIDER`. This is the recommended path when you want Claude for chat/codegen but don't want to run the bundled TEI sidecar (for example on a small host where TEI gets OOM-killed).

```bash theme={null}
# .env — Anthropic for chat/codegen, OpenAI for embeddings
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
EMBEDDING_PROVIDER=openai
```

Valid values for `EMBEDDING_PROVIDER`:

| Value | Endpoint | Default model | Requirements |
| - | - | - | - |
| `openai` | `https://api.openai.com/v1/embeddings` | `text-embedding-3-small` (1536-dim) | `OPENAI_API_KEY` set |
| `azure` | `{AZURE_OPENAI_ENDPOINT}/openai/v1/embeddings` | `text-embedding-3-small` (deployment name) | `AZURE_OPENAI_API_KEY` set, embedding deployment created on your resource |
| `tei` | `http://tei:80/v1/embeddings` | `BAAI/bge-m3` (1024-dim) | bundled TEI sidecar running |
| `ollama` | `http://ollama:11434/v1/embeddings` | `bge-m3` (1024-dim) | optional Ollama service enabled, model pulled |

`EMBEDDING_MODEL` overrides the default model for the chosen provider (e.g. `text-embedding-3-large` for higher quality at 3072-dim). `EMBEDDING_ENDPOINT` overrides the URL for the `tei` or `ollama` providers — useful when running TEI or Ollama on a separate host and pointing at it from inside Docker.

When `EMBEDDING_PROVIDER` is unset, embedding follows `AI_PROVIDER`: `anthropic` seeds the bundled TEI; `openai` seeds OpenAI embeddings.

<Warning>
  **Switching providers changes vector dimensions.** `bge-m3` is 1024-dim, `text-embedding-3-small` is 1536-dim, `text-embedding-3-large` is 3072-dim. Existing pgvector / Qdrant / Weaviate / Milvus / Chroma collections built on the previous embedder will fail-fast on the next run with a dimension-mismatch message — drop the destination table or collection and re-ingest.
</Warning>

## Hot Reload

When you save AI configuration from the Configuration UI, the server reloads the `ai-primary` and `codegen` configs from Vault immediately — **no restart required**. The next AI call uses the updated provider, model, and endpoint. The `embedding` config is read from Vault on each request and also requires no restart.

## How Configuration Persists

Vault stores its data on a durable Docker volume (`vault-data`), so secrets, AI provider and model choices, and other Configuration-tab changes **survive restarts and rebuilds** — `docker compose up --build`, `down && up`, and host reboots all keep your settings.

`.env` is a **first-boot seed only**. The first time Vault initializes, `vault-init.sh` seeds the secrets from `.env`. After that, Vault persists on disk and the **Configuration tab is the source of truth** — editing a value in `.env` later has no effect. To change AI configuration after the first boot, use the Configuration tab (saves go to Vault and stick).

This applies to everything stored in Vault (AI config and connection secrets). Deployment-level settings like `JAVA_OPTS` and `USE_USER_AUTH` work the opposite way — they're plain container env vars read on every boot, so `.env` edits *do* take effect on a container recreate. See [Changing Settings After Install](/configuration-reference#changing-settings-after-install) for the full two-bucket breakdown.

<Note>
  **Upgrading from an older release — one-time step.** Earlier versions kept Vault data in memory only, so any secret you had set **only** in the UI — **including tap connection secrets** — is not carried into the new on-disk store and must be re-entered once after upgrading. Secrets supplied through `.env` are re-seeded automatically. After this one-time step, everything persists across future rebuilds.
</Note>

### Re-seeding from `.env`

To discard the persisted Vault data and seed fresh from `.env` again (a clean reset), remove the volume:

```bash theme={null}
docker compose down
docker volume rm datris-platform-oss_vault-data   # the project-name prefix may differ
docker compose up -d
```

The next boot re-initializes Vault and seeds from `.env`.

## Setting Secrets Manually

To write a secret directly to Vault (note: manual Vault writes do **not** trigger hot reload — restart the server or re-save from the UI):

```bash theme={null}
docker compose exec -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN="$(docker compose exec -T datris cat /vault-token/token)" vault \
  vault kv put secret/oss/ai-primary \
  provider="anthropic" \
  endpoint="https://api.anthropic.com/v1/messages" \
  model="claude-opus-5-5" \
  apiKey="sk-ant-..." \
  version="2023-06-01"
```

Same shape for `oss/codegen` and `oss/embedding`. Verify with:

```bash theme={null}
docker compose exec -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN="$(docker compose exec -T datris cat /vault-token/token)" vault \
  vault kv get secret/oss/ai-primary
```

## Embedding Provider Notes

The embedding endpoint must speak the **OpenAI embeddings API** (`POST /v1/embeddings` with `{model, input}` body returning `{data: [{embedding: [...]}]}`). That contract is implemented by:

* **OpenAI** — `https://api.openai.com/v1/embeddings`
* **Azure OpenAI** — `https://YOUR-RESOURCE.openai.azure.com/openai/v1/embeddings` (model = your embedding deployment's name)
* **TEI (bundled)** — `http://tei:80/v1/embeddings` (the bundled HuggingFace Text Embeddings Inference sidecar serves `BAAI/bge-m3` out of the box)
* **Ollama** — `http://ollama:11434/v1/embeddings` (only when the optional Ollama service is enabled and `bge-m3` has been pulled there)
* **Ollama (external)** — `http://host.docker.internal:11434/v1/embeddings` (your own Ollama with any embedding model)
* **Self-hosted vLLM, LM Studio, etc.** — anything implementing the same contract

Anthropic has no embeddings API, so `provider: "anthropic"` is not valid for the `embedding` slot.

## Chat Provider Notes

The `aiPrimary` and `codegen` slots support Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, Grok (xAI), and Ollama. Ollama must expose the **OpenAI chat completions API** (`POST /v1/chat/completions`), which it does by default. When running Ollama on your host machine, the endpoint from inside Docker is `http://host.docker.internal:11434/v1/chat/completions`.

Azure OpenAI always speaks chat/completions against your configured endpoint (never the OpenAI Responses API, whose Azure availability varies by region and model). Web search is not available on the Azure provider — pair Azure chat with Anthropic or OpenAI web search if you need it.

Amazon Bedrock serves Claude models over the same Anthropic request/response shape as the direct provider, with AWS SigV4 authentication. Web search is not available on Bedrock either — pair it with Anthropic or OpenAI web search if needed.

Grok speaks the OpenAI chat completions API against `https://api.x.ai/v1/chat/completions` with Bearer authentication. Like Azure, it never routes to the OpenAI Responses API, and web search is not available on it.

## Web Search (Optional)

When enabled, four surfaces can consult the live web — official API documentation, deprecation notices, current package names — instead of relying only on the model's training data. Off by default; flip it on from the **Configuration → AI Providers** tab.

Web search is its own independent service — same model as Embedding. Pick a provider regardless of which provider runs the main AI work. The platform attaches the tool natively to the main AI call when the providers match (one round trip, the model decides when to search) or makes a separate out-of-band search call and injects the results as context (when providers differ — e.g. AI Primary=Anthropic, Web Search=OpenAI). Either way the model sees the current information.

| Field | Required | Notes |
| - | - | - |
| `enabled` | Yes | `true` or `false` |
| `provider` | Yes | `anthropic` or `openai` (Ollama has no native web search) |
| `endpoint` | Yes | Provider endpoint — Anthropic Messages or OpenAI Responses |
| `model` | Yes | Model that runs the search call |
| `apiKey` | Yes | Own key, copied from the right-hand panel based on the chosen provider (mirrors Embedding) |
| `version` | Optional | API version (Anthropic uses `2023-06-01`) |
| `maxUses` | Optional | Maximum searches per AI call (default 3) |

| Surface | Web search opt-in |
| - | - |
| Tap brainstorm (chat) | Yes — verifies recommended data sources still exist and surfaces current free-tier / auth state |
| Tap diagnosis (analyzes run logs after a failed test) | Yes — looks up unfamiliar exception messages, library deprecation notices |
| Tap fix path (Auto-Fix after a tap fails) | Yes — looks up current PyPI install names, current method signatures (replaces the legacy PyPI scraper when on) |
| First-pass tap script generation | No — happy path, model uses training data so generation stays fast |
| NL → SQL, schema gen, profiling, DQ rule gen, etc. | No — self-contained, they don't need the web |

OpenAI web search requires the **Responses API**. When the configured endpoint is `/v1/chat/completions`, the call is automatically routed to `/v1/responses`. Ollama has no native web search; selecting it as your AI Primary doesn't prevent you from using Anthropic or OpenAI for web search.

## Stronger Model for Code Generation

<Tip>
  **Highly recommended: a provider's strongest coding model for CodeGen.** Tap script generation, AI data quality rules, AI transformations, and NL→SQL all depend heavily on the model's coding ability — the top coding models from Anthropic and OpenAI produce dramatically better results than smaller or cheaper models. The same recommendation appears in `.env.example`.
</Tip>

CodeGen tasks (tap scripts, `aiRule`, `aiTransformation`, JSON Schema / XSD generation, natural-language → SQL) have their own slot so they can run a different model than the general default. On a fresh install the slot seeds the strongest code-generation model — `gpt-5.6-sol` with OpenAI, `claude-opus-5-5` with Anthropic — so code generation is at full quality out of the box, while `aiPrimary` seeds the recommended chat model (`claude-opus-5-5` / `gpt-5.6-sol`). The separate slot is your cost-tuning lever: point `aiPrimary` at a cheaper model (e.g. `claude-sonnet-5`) for the chatty paths and keep the stronger model only on the calls that benefit from it.

| Feature | Uses codegen slot? |
| - | - |
| Tap script generation, auto-fix | Yes |
| `aiRule` data quality rules | Yes |
| `aiTransformation` | Yes |
| CSV / JSON Schema / XSD generation | Yes |
| Natural language → SQL (`/query/ai`) | Yes |
| Error explanation, brainstorm chat, CRON generation | No (general slot) |
| Data profiling, AI Q\&A | No (general slot) |
| AI header validation | No (general slot) |

To use the same model for codegen as for everything else, point `oss/codegen` at the same secret values as `oss/ai-primary`.

## Multi-Tenant: Per-Tenant Overrides

In multi-tenant deployments (`multiTenant: "true"` in `application.yaml`) each tenant can override any of the three slots independently. The resolver checks per-request:

| Slot | Per-tenant Vault path | Falls back to |
| - | - | - |
| AI Primary | `{env}/ai-primary` | `oss/ai-primary` |
| CodeGen | `{env}/codegen` | `oss/codegen` |
| Embedding | `{env}/embedding` | `oss/embedding` |
| Web Search | `{env}/web-search` | `oss/web-search` |

(`{env}` is the tenant's environment name, e.g. `acme`.)

A per-tenant secret takes effect on the next AI call — no restart needed. The override is independent for each slot: a tenant can override just AI Primary, just CodeGen, just Embedding, or any combination.

### Setting your key from the UI

Any tenant can manage all three slots from the platform UI. The Configuration page is organized into sub-tabs:

* **AI Providers** — AI Primary, CodeGen, and Embedding provider + model selectors, plus a shared API Keys panel on the right (backed by the `{env}/ai-keys` secret — see [Shared API Key Store](#shared-api-key-store)). This is where you enter your provider API keys and pick models.
* **Secrets** — operator-facing platform credentials (Vault-backed). Admin-only when user-auth is on.
* **Users** — user accounts and roles. Visible when `USE_USER_AUTH=true`.
* **API-Keys** — issue and revoke scoped credentials for the CLI, MCP agents, and other programmatic clients. Visible when `USE_API_KEYS=true`. See [API Keys](/api-keys).
* **Data Sources**, **Code Repository**, **Audit Log**, **Agent Policy** — additional operator sub-tabs (see their respective pages).
* **Configuration assistant** (chat rail at the right edge) — see [Configuration Assistant](/configuration-assistant).

Some sub-tabs are role- and flag-gated — see the [API Keys matrix](/api-keys#supported-configurations) for the full visibility rules.

To set your AI keys:

1. Open the **Configuration** tab, then the **AI Providers** sub-tab.
2. Each section (AI Provider, CodeGen Provider, Embedding Provider) has its own provider and model selector. Providers include Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, Grok (xAI), and Ollama (local).
3. For Anthropic/OpenAI/Azure/Grok, enter your API key in the right-hand panel and pick a model from the dropdown. For Azure, also enter your resource endpoint and type your deployment name as the model. For Amazon Bedrock, enter your AWS credentials and region in the right-hand panel (or leave the keys blank to use the server's IAM role) — the model dropdown then lists what your account can actually invoke. For Ollama, type the model name directly — no API key needed.
4. Click **Save Configuration**. Changes take effect immediately — no restart required.

To fall a tenant back to the global `oss/*` defaults, delete its per-tenant secrets in Vault (see below).

### Setting tenant secrets directly in Vault

For operators provisioning tenants programmatically:

```bash theme={null}
docker compose exec -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN="$(docker compose exec -T datris cat /vault-token/token)" vault \
  vault kv put secret/acme/ai-primary \
  provider="anthropic" \
  endpoint="https://api.anthropic.com/v1/messages" \
  model="claude-opus-5-5" \
  apiKey="sk-ant-..." \
  version="2023-06-01"
```

Same shape for `{env}/codegen` and `{env}/embedding`.

## Misconfiguration

The platform fails fast at startup if the `aiPrimary` slot is missing or malformed — it is always required when `ai.enabled: true`. The `codegen` slot is optional: when its secret is absent, code-generation tasks fall back to `aiPrimary`. The `embedding` slot is read per request and is only required by features that depend on it (vector destinations and semantic search).


# AI Data Profiling
Source: https://docs.datris.ai/ai-data-profiling

AI-powered data profiling

Upload any data file and receive an AI-generated profile — summary statistics, data quality issues, and recommendations for validation rules and transformations. Use profiling to understand your data before setting up a pipeline configuration.

## Endpoint

`POST /api/v1/pipeline/profile`

## Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `file` | multipart file | (required) | The data file to profile (CSV, JSON, or XML) |
| `delimiter` | string | `,` | CSV delimiter character |
| `header` | boolean | `true` | Whether the CSV file has a header row |
| `sampleSize` | int | `200` | Number of rows to sample for large files |

## Example

```bash theme={null}
curl -s -X POST http://localhost:8080/api/v1/pipeline/profile \
  -F "file=@stock_prices.csv" \
  -F "delimiter=," \
  -F "header=true" \
  -F "sampleSize=200" | python3 -m json.tool
```

## Response

The endpoint returns a JSON object with three sections:

```json theme={null}
{
  "summary": {
    "rowCount": 200,
    "columnCount": 8,
    "columns": [
      {
        "name": "symbol",
        "inferredType": "string",
        "nullCount": 0,
        "uniqueCount": 195,
        "sampleValues": ["FAX", "IAF", "FCO"]
      },
      {
        "name": "date",
        "inferredType": "date",
        "nullCount": 0,
        "uniqueCount": 1,
        "sampleValues": ["2016-12-30"]
      },
      {
        "name": "open",
        "inferredType": "float",
        "nullCount": 1,
        "uniqueCount": 198,
        "sampleValues": ["4.65", "5.44", "7.91"]
      }
    ]
  },
  "qualityIssues": [
    "Column 'open' has 1 null/empty value",
    "Column 'volume' contains a negative value (-500)",
    "Column 'close' has a value exceeding $1,000,000"
  ],
  "recommendations": [
    "Add an aiRule to validate symbol format, price ranges, and cross-column relationships",
    "Consider sampling for large files to keep processing fast"
  ],
  "suggestedDataQuality": {
    "aiRule": {
      "instruction": "symbol must be 1-5 uppercase letters, all price columns (open, high, low, close, adj_close) must be positive and not exceed $1,000,000, volume must be positive, and high must be greater than or equal to low",
      "onFailureIsError": false
    }
  }
}
```

### Response fields

| Section | Description |
| - | - |
| `summary` | Row count, column count, and per-column statistics (inferred type, null count, unique count, sample values) |
| `qualityIssues` | Data quality problems detected in the sample — missing values, outliers, inconsistent formats, suspicious patterns |
| `recommendations` | Human-readable suggestions for validation rules and transformations |
| `suggestedDataQuality` | Ready-to-use `dataQuality` JSON block that can be copied directly into a pipeline configuration. Contains an `aiRule` with a comprehensive plain-English instruction covering all validation checks |

### Suggested data quality rules

The `suggestedDataQuality` section provides a complete, copy-paste-ready `dataQuality` configuration based on what the AI observed in the data:

* **`aiRule`** — a single comprehensive plain-English instruction covering all validation checks: format patterns (emails, phone numbers, dates), value ranges, cross-column relationships (e.g., high >= low), and business logic. If no validation rule is appropriate, this field is omitted.

See [AI Rule](/data-quality/ai-rules) for full documentation.

## How it works

1. You upload a file — no pipeline registration needed
2. For large CSV files, the pipeline randomly samples `sampleSize` rows (keeping the header)
3. The sampled content is sent to the AI model with a profiling prompt
4. The AI analyzes the data and returns a structured JSON profile

Profiling is a standalone operation — it does not require a registered pipeline, data quality rules, or any pipeline configuration. It is designed to be the first step when working with a new data source, before setting up validation or ingestion.

## Use cases

* **Explore new data** — understand the structure, types, and quality of an unfamiliar file before writing a pipeline configuration
* **Discover quality issues** — find missing values, outliers, format inconsistencies, and suspicious patterns
* **Generate rule ideas** — the AI suggests aiRule instructions and transformations based on what it observes
* **Validate assumptions** — confirm that a file matches expected schema and data quality before loading

## Sampling

For CSV files larger than `sampleSize` rows, the profiling endpoint automatically samples a random subset. The header row is always included. For JSON and XML files, the content is truncated if it exceeds the AI model's context window.

The default sample of 200 rows is typically sufficient to detect patterns, types, and quality issues. Increase `sampleSize` for more thorough profiling at the cost of slower response times.

## When the model declines

An AI provider's safety classifier can decline to look at a file, most often one that looks like personal data. Profiling then still returns `200` with the server's own statistics only: `summary.rowCount`, `summary.columnCount` and `summary.columns` computed over every row of a CSV file (inferred type, nulls, distinct count, value lengths; `sampleValues` empty; empty for JSON and XML files), empty `qualityIssues` and `recommendations`, no `suggestedDataQuality`, a top-level `"aiDeclined": true` and a fixed `note`: "the model declined this request; statistics only". The provider's own text is never included. Read an empty `qualityIssues` list with `aiDeclined` as "no AI review", not as a clean file.

| Provider family | Recognised as a decline |
| - | - |
| Anthropic (direct or AWS Bedrock) | `stop_reason: "refusal"` |
| OpenAI, Azure OpenAI, Grok and Ollama (chat completions, streamed or not) | `finish_reason: "content_filter"`, or a non-empty `message.refusal` (`delta.refusal` when streamed) |
| OpenAI Responses API models | an output item or message content part of type `refusal`, or `incomplete_details.reason: "content_filter"` |

Only response-side declines are recognised. Azure OpenAI's prompt-side content filter answers with HTTP 400 (`code: content_filter`) and is still returned as an error. Nothing else counts as a decline: a missing or wrong provider key, a timeout, a rate limit or any other provider error is still returned as an error, so a misconfiguration is not hidden.

Other AI features (search answers, CodeGen, tap generation, fix suggestions and the rest) have no local fallback: a decline there, including a `content_filter` finish that used to return the cut-off text, is reported as an error.

## Values withheld

When the server sets `DATRIS_AI_SAMPLE_VALUES=false`, no row value is sent to the model. For a CSV file the model gets per-column statistics computed over every row (inferred type, count, nulls, distinct count, minimum and maximum value length); for JSON, NDJSON and XML (by the `.json`, `.ndjson`, `.jsonl` or `.xml` extension; for an unrecognised extension, by content that starts with `{`, `[` or `<`) it gets a value-free skeleton of the document. `.csv`, `.tsv`, `.psv` and `.txt` files are always profiled as delimited. A header cell that does not look like a plain column name is sent as `column_N`, and a first line that looks like data (a dash next to a digit, an `@`, or a long run of digits in any cell) is counted as a row and every column is numbered. A cell with more than two digits or no letter at all also marks line 1 as data. A headerless first row made only of words still passes as names and is sent: single words in any case (`John,Smith,Boston,MA`; `jane,doe,asthma,f`; `Diabetes,Type,Two`), all-caps or lower-case multi-word cells (`JOHN SMITH`, `jane doe`), and names the capitalised-words test misses (`John A Smith`, `McDonald Smith`, `O Brien`). `header` defaults to `true`: profile a headerless file with `header=false` so no cell of its first line is sent. `sampleValues` are empty for every column and the response carries a top-level `"valuesWithheld": true`. Quality issues and the suggested rule are then based on names, types and statistics only. If the model's reply cannot be read as JSON (for example a very wide file whose reply is cut off), or the model call fails for any reason (the model declines the request, a provider error, a timeout), the response holds the server's statistics only: `summary.columns` (empty for JSON and XML files), empty `qualityIssues` and `recommendations`, `"valuesWithheld": true` and a `note` giving the cause in the server's own words; the provider's error text is never included. A missing or misconfigured provider key or model also produces this note when values are withheld; check it with [`datris doctor`](/doctor). With values sent (the default), a failed model call still returns an error; only a model decline falls back, as described in [When the model declines](#when-the-model-declines) (with `aiDeclined`, not `valuesWithheld`). See [Field Protection](/transformation/field-protection#what-still-sees-raw-data).

## Requirements

* `ai.enabled: true` must be set in `application.yaml`
* A configured AI provider (see [AI Configuration](/ai-configuration))
* Cloud providers (Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, Grok) recommended for best results


# AI Error Explanation
Source: https://docs.datris.ai/ai-error-explanation

AI-powered error explanations, surfaced as an AI Suggested Fix

When a pipeline job fails, the AI automatically analyzes the error and provides a plain-English explanation of the root cause and suggested fix. The result — labelled **AI Suggested Fix** in the job status — appears as a separate status entry after the error, making it easy to diagnose issues without reading stack traces.

## How it works

When any pipeline stage (data quality, transformation, destination loading) throws an exception:

1. The error is logged to status as usual
2. The error message and pipeline configuration are sent to the AI model
3. The AI returns a concise explanation of what went wrong and how to fix it
4. The explanation is written as a separate status entry (`AI Suggested Fix: ...`, with `Diagnosis:` and `Suggested fix:` lines) and logged to the pipeline logs

If the AI call itself fails (rate limit, timeout, etc.), the explanation is skipped with a server log warning — it never causes a secondary failure.

## Example

**Error status:**

```
Process completed, error: Aborting processing this pipeline, 3 error(s) were found
while performing data quality rules:
Data quality CodeGen failure, row: 2, reason: high (7.50) is less than low (8.01)
Data quality CodeGen failure, row: 3, reason: volume is negative (-500)
Data quality CodeGen failure, row: 4, reason: close and adj_close exceed $1,000,000
```

**AI Suggested Fix status (separate entry):**

```
AI Suggested Fix: Three source rows violate the price/volume rules; fix the upstream data.
Diagnosis: Row 2 has a high value lower than low (likely a column swap or data entry
error), row 3 contains a negative volume, and row 4 has close and adj_close values
exceeding $1,000,000.
Suggested fix: Correct the upstream source data or add a pre-processing transformation
step to sanitize these issues before the data quality check runs.
```

## Configuration

No additional configuration is needed. AI error explanation is automatically enabled since an AI provider is required to run the platform.

## Requirements

* A configured AI provider (see [AI Configuration](/ai-configuration))


# API Keys
Source: https://docs.datris.ai/api-keys

Enable API-key authentication for the UI, CLI, MCP clients, and any other programmatic access to Datris.

By default, the Datris OSS install accepts unauthenticated requests on the REST and MCP APIs — fine for laptop development, wrong for any shared or hosted setup. Turn on API-key authentication and every caller (the UI, the CLI, agent MCP clients) must present a key that the platform recognizes.

This page covers single-tenant deployments. Multi-tenant setups (`MULTI_TENANT=true`) use the same flag and a different mapping mechanism — see [Multi-tenant](#multi-tenant) below.

## What it does

When `USE_API_KEYS=true`:

* Programmatic clients (CLI, MCP-connected agents, external scripts) must include `x-api-key: <value>` on every request.
* The platform validates the value against a known set of issued keys.
* The capability framework enforces what each scoped key is allowed to do (see [the capability model](#the-capability-model)). Calls outside the key's scope return HTTP 403.
* The Datris UI behavior depends on whether [user authentication](/user-auth) is also enabled. With `USE_USER_AUTH=true` (recommended), the UI authenticates via session cookie alone — no API key is sent or required. With user-auth off, the UI falls back to the legacy "paste a key" prompt on first load.

The flag is independent of `USE_USER_AUTH` but they're paired conceptually — see [the supported matrix](#supported-configurations) below.

## Enabling it

The recommended setup pairs `USE_API_KEYS=true` with `USE_USER_AUTH=true` (see [the matrix below](#supported-configurations) for why).

### 1. Flip the flags

In `.env`:

```bash theme={null}
USE_USER_AUTH=true
USE_API_KEYS=true
```

### 2. Restart

```bash theme={null}
docker compose up -d --force-recreate datris
```

### 3. Use the platform

* **The UI**: log in with your admin account. The session cookie authenticates browser flows — no `x-api-key` is sent, no Connect prompt appears. The Assistant inherits your session for capability checks; its tool calls, version-history entries, and [audit log](/audit-log) records are attributed to you (`session:<your username>`), not to the platform's internal `ui` key.
* **Programmatic clients** (CLI, MCP agents, scripts): each one needs its own key, issued from **Configuration → API-Keys**. See [the issuance flow](#issuing-keys-for-cli-mcp-and-other-clients) below.
* **Tap scripts** reading platform data need nothing: each run gets a short-lived, read-only token that the tap wrapper attaches automatically (see [Taps → Querying Datris Data](/taps#querying-datris-data)).

That's it. No env var to track, no value to paste anywhere.

### Legacy mode (`USE_USER_AUTH=false`, `USE_API_KEYS=true`)

If you can't or don't want to enable user-auth, the UI falls back to the older "paste a key" flow:

* The Connect prompt appears on first load
* The default seeded value is `default-ui-key` (defined in `docker/vault-init.sh`); paste it once
* Browser stores it in localStorage; subsequent loads skip the prompt
* Rotate it any time from **Configuration → API-Keys → ui → Rotate** (the new value is mirrored automatically into the validation map, and the operator distributes it out of band)

This mode is fine for laptop development but **not** appropriate for shared installs — anyone with the URL can reach the Connect prompt, and the only gate is knowledge of the pasted value.

## Issuing keys for CLI, MCP, and other clients

Open **Configuration → API-Keys → Issue new key**.

1. Pick a label (`cli`, `claude-desktop`, `cursor`, `prod-loader`, etc.) — lowercase, hyphens/underscores ok.
2. Pick a template or build a capability list manually:
   * **read-only** — observer access to pipelines/taps/jobs/metadata + query/search
   * **rag-builder** — create pipelines/taps in a catalog, upload documents, run vector search
   * **reporting** — analyst-shaped read access
   * **ops** — run any tap or pipeline, kill stuck jobs, no edits
   * **full-access** — legacy `*:*` shape; avoid for agents
3. Review and create.
4. **Copy the value immediately** — it's shown once. The platform stores a hashed copy; the plaintext is not retrievable after this modal closes.
5. Paste the value into the agent's MCP client config (or CLI env var, etc.).

Each label is a distinct identity in the [audit log](/audit-log) and can be revoked independently of the others. Every issued key also gets a stable id (shown in the audit log) so a label that is revoked and later re-issued is not confused with its predecessor.

<Note>
  The Vault label `ui` is reserved for the seeded fallback UI key (used in legacy `USE_USER_AUTH=false` mode and for the Assistant's internal MCP-to-REST hop). You don't issue this one from the UI — it's seeded by `vault-init.sh` on first boot and rotated from the API-Keys tab like any other key.
</Note>

## Supported configurations

`USE_API_KEYS` and `USE_USER_AUTH` compose into four states:

| `USE_USER_AUTH` | `USE_API_KEYS` | Behavior |
| - | - | - |
| `false` | `false` | Full anonymous. No prompts, no auth, no gating. **OSS default; fine for laptop dev.** |
| `true` | `true` | UI gated by login; programmatic clients gated by key. **Recommended for shared installs.** |
| `true` | `false` | UI gated by login; programmatic clients are anonymous. Workable, slightly odd. |
| `false` | `true` | **Not recommended.** No coherent auth mechanism for the browser. Pick a different combination. |

The fourth row is the trap. When you require strict auth for programmatic clients but no login for the UI, the browser has nothing to present. API keys are intended to be used alongside user auth — set both flags together, or neither.

## Rotating a key

From the **API-Keys** tab in Configuration, click **Rotate** on any row:

1. A new value is generated server-side
2. The validation map (`oss/api-keys`) and operator-facing record are updated in sync
3. The new value is shown once in a copy-to-clipboard modal
4. Distribute the new value out of band to whoever was using it; they'll get 401s on their next request and need the new value

The previous value stops working the moment Rotate returns.

## Revoking a key

From the **API-Keys** tab, click **Revoke**. The key's metadata is marked revoked; the underlying value stays in `oss/api-keys` so failed auth attempts log as "revoked key" rather than "unknown key" — useful for tracking compromised credential reuse.

A revoked key is rejected with `401 {"error":"API key is revoked or invalid"}` on every authenticated route, and the denial is recorded in the audit log as a security event whose actor is the revoked key's label (an unknown key value is recorded as `unknown-key`, never the value itself). If the secret store itself cannot be read, keys are not resolved at all and the response is `503` rather than falling back to full access.

Revoked keys cannot be un-revoked. Issue a new key instead.

## How requests get logged

Each authenticated request emits a structured line in the datris container log:

```
capability check: route=POST /api/v1/pipeline required=pipeline:create key=claude-desktop legacy=false outcome=grant
```

The fields:

* `route` — HTTP method + path
* `required` — the capability the route needs (see [the capability model](#the-capability-model))
* `key` — the label of the API key (or `session:<username>` for cookie-authed UI flows)
* `legacy=true` — the key has full access via the backward-compatibility backstop; `false` means a scoped key
* `outcome` — `grant`, `would-deny`, `deny`, or `unmapped`

Enforcement is **on by default**: a scoped key calling a route it lacks the capability for gets `outcome=deny` and an HTTP 403. To trial a new scope policy without rejecting traffic — for example, you've issued a new scoped key and want to see what it would have been denied under real workload before pointing a production agent at it — set `CAPABILITY_ENFORCEMENT=log-only` in `.env`. The same calls then succeed and emit `outcome=would-deny` log lines instead of 403s. Flip back to `enforce` (or remove the line) once you've validated the policy.

## The capability model

Every API key carries a list of capability strings:

```
pipeline:read:catalog=support
document:upload:collection=support_docs
search:vector:collection=support_docs
job:read
```

Each capability is `resource:action[:scope]`. Scope segments restrict by container (catalog, database, collection), type, or ownership — never by leaf resource name, since agents pick those at runtime.

Tap secrets (`_type=tap`) are the only secrets a tap can read: a tap that names a platform secret is refused on save and at run time, so a key holding `tap:create` and `tap:run` cannot use a tap to read a platform secret that its `secret:read:_type=tap` scope keeps it from reading directly (unless an operator sets `DATRIS_TAP_SECRET_SCOPE=any`; see [Taps](/taps#taps-can-only-use-tap-secrets)).

`protect:reveal` (read back the original of an [`encrypt`-protected field](/transformation/field-protection#revealing-a-value)) and `protect:admin` (rotate its key, and edits to the field-protection secret) are never in a scoped template or in the editor and viewer role bundles: grant them explicitly to the key that needs them. Admins, the `full-access` template, and legacy unscoped keys hold them through `*:*`.

When user-auth is on, logged-in browser sessions are first-class identities too: `session:admin`, `session:editor1`, etc. The capability bundle is derived from the user's role (admin → full access, editor → create/update/run, viewer → read-only). The API-Keys tab in Configuration shows scoped *keys*; per-role capabilities are managed via the **Users** tab.

## Multi-tenant

`MULTI_TENANT=true` switches the model: a single `api-key-mappings` secret maps `keyValue → tenantEnv`, and every tenant carries its own per-environment data. The UI key flow above is single-tenant only; in multi-tenant mode the mappings are managed directly in the `api-key-mappings` Vault secret.

## See also

* [User Authentication](/user-auth) — login + roles for the UI
* [Configuration Reference](/configuration-reference) — full env var list
* [MCP Server](/mcp-server) — how external agents connect

## Capabilities vs. the agent policy

A capability says what a key *may* do. The [agent policy](/agent-policy) says what an agent may do *unattended*: for each action, whether it runs on its own, waits for a person to approve it, or is refused. The policy runs after the capability check and can only pause or refuse what the key already holds — it never grants anything. Two capabilities support it: `policy:read` (read the policy; any key) and `approval:read` (list and poll the approvals the key queued; use `approval:read:owner=self`). Deciding approvals is a person's job — an agent-initiated request is refused even with `approval:decide`.


# Approve a queued action (admin / editor)
Source: https://docs.datris.ai/api-reference/agent-policy/approve-a-queued-action-admin-editor

/openapi.yaml post /api/v1/approvals/{id}/approve
Performs the original request on the approver's behalf by replaying it through the normal request chain. Never permitted from an agent-initiated request. Refused as stale (409) if the target pipeline or tap changed since the action was queued.



# List queued agent actions
Source: https://docs.datris.ai/api-reference/agent-policy/list-queued-agent-actions

/openapi.yaml get /api/v1/approvals
Newest first. An agent-initiated caller sees only the actions it queued; a person sees everything.



# Read one queued action
Source: https://docs.datris.ai/api-reference/agent-policy/read-one-queued-action

/openapi.yaml get /api/v1/approvals/{id}



# Read the agent policy
Source: https://docs.datris.ai/api-reference/agent-policy/read-the-agent-policy

/openapi.yaml get /api/v1/policy
Per-action auto / approve / deny for agent-initiated changes. Anyone may read it (agents consult it before acting). `enabled` is false when USE_AGENT_POLICY is off, in which case every action is auto.



# Reject a queued action (admin / editor)
Source: https://docs.datris.ai/api-reference/agent-policy/reject-a-queued-action-admin-editor

/openapi.yaml post /api/v1/approvals/{id}/reject



# Replace the agent policy (admin)
Source: https://docs.datris.ai/api-reference/agent-policy/replace-the-agent-policy-admin

/openapi.yaml put /api/v1/policy
Admin-only, and never permitted from an agent-initiated request regardless of key capabilities. Unknown action keys, bad modes or malformed overrides are rejected with 400 naming the problem.



# AI data profiling
Source: https://docs.datris.ai/api-reference/ai/ai-data-profiling

/openapi.yaml post /api/v1/pipeline/profile
Upload a file and receive AI-generated summary statistics, quality issues, and suggested validation rules.



# AI-powered question answering
Source: https://docs.datris.ai/api-reference/ai/ai-powered-question-answering

/openapi.yaml post /api/v1/ai/answer
Use AI to answer a question based on provided context. Ideal for RAG workflows — retrieve relevant chunks via vector search, then pass them as context along with the user's question.



# AI schema generation
Source: https://docs.datris.ai/api-reference/ai/ai-schema-generation

/openapi.yaml post /api/v1/pipeline/generate
Upload a file and receive an AI-generated pipeline configuration with inferred field names and types.



# Discover invokable models for a provider
Source: https://docs.datris.ai/api-reference/ai/discover-invokable-models-for-a-provider

/openapi.yaml get /api/v1/ai/models
Live per-provider model discovery, reflecting what the configured account can actually invoke (unlike the static catalog). Currently supports provider=bedrock: the server calls the AWS Bedrock discovery APIs (ListFoundationModels + ListInferenceProfiles) with the configured AWS credentials and merges them into invokable ids — a model without on-demand support is returned as its cross-region inference-profile id (e.g. us.anthropic....). Results are cached for 5 minutes. The Configuration UI overlays catalog labels on this list and falls back to the static catalog when discovery is unavailable.



# Get the AI model catalog
Source: https://docs.datris.ai/api-reference/ai/get-the-ai-model-catalog

/openapi.yaml get /api/v1/ai/model-catalog
Same-origin proxy for the remote model catalog (datris.ai/models.json) used by the Configuration UI's model pickers. Successful fetches are cached for 5 minutes; on fetch failure the stale cache is served if one exists.



# Capability catalog (admin)
Source: https://docs.datris.ai/api-reference/api-keys/capability-catalog-admin

/openapi.yaml get /api/v1/keys/capabilities/catalog
Admin-only. Every capability resource, its actions, and the scope keys it accepts — drives the Keys UI capability editor.



# Capability templates (admin)
Source: https://docs.datris.ai/api-reference/api-keys/capability-templates-admin

/openapi.yaml get /api/v1/keys/templates
Admin-only. Common capability-set starting points (read-only, rag-builder, reporting, ops, full-access) offered by the Keys UI wizard.



# Issue a new API key (admin)
Source: https://docs.datris.ai/api-reference/api-keys/issue-a-new-api-key-admin

/openapi.yaml post /api/v1/keys
Admin-only. The returned `value` is shown ONCE and cannot be retrieved again — store it immediately.



# List API keys (admin)
Source: https://docs.datris.ai/api-reference/api-keys/list-api-keys-admin

/openapi.yaml get /api/v1/keys
Admin-only. Lists every issued API key with its capabilities and audit metadata. Key VALUES are never returned here — only on issue and rotate, one time. Keys issued before capability metadata existed are flagged `isLegacyFullAccess`.



# Revoke an API key (admin)
Source: https://docs.datris.ai/api-reference/api-keys/revoke-an-api-key-admin

/openapi.yaml delete /api/v1/keys/{label}
Admin-only. Marks the key revoked; it immediately stops authenticating, and subsequent attempts are logged as "revoked key".



# Rotate an API key (admin)
Source: https://docs.datris.ai/api-reference/api-keys/rotate-an-api-key-admin

/openapi.yaml post /api/v1/keys/{label}/rotate
Admin-only. Generates a new value for the key; the previous value stops working immediately. Capabilities are preserved. The returned value is shown once — update every caller that held the old value.



# Assistant chat (SSE)
Source: https://docs.datris.ai/api-reference/assistant/assistant-chat-sse

/openapi.yaml post /api/v1/assistant/chat
Runs the build-mode Assistant agent loop and streams agent-loop events (text deltas, tool calls, tool results, errors) as Server-Sent Events. The client cancels by disconnecting. Attachments reference files previously staged via `/assistant/attachment`.



# Initialize the Assistant
Source: https://docs.datris.ai/api-reference/assistant/initialize-the-assistant

/openapi.yaml get /api/v1/assistant/init
UI mount call — warms the MCP client caches and returns the tool catalog, the workflow reference resource, and the active chat model.



# Stage a chat attachment
Source: https://docs.datris.ai/api-reference/assistant/stage-a-chat-attachment

/openapi.yaml post /api/v1/assistant/attachment
Stages a file dropped into the Assistant chat. The bytes are cached server-side (tenant-scoped, TTL'd) and only the returned handle plus a small content sample travel through the chat; the platform substitutes the real bytes when the agent calls a file tool with the `attachmentId`. Limit 25 MB per file.



# Change the current user's password
Source: https://docs.datris.ai/api-reference/auth/change-the-current-users-password

/openapi.yaml post /api/v1/auth/change-password
Requires an authenticated session. The current-password check is skipped on a first-time set (`mustSetPassword`).



# Create a user (admin)
Source: https://docs.datris.ai/api-reference/auth/create-a-user-admin

/openapi.yaml post /api/v1/auth/users
Admin-only. Omit `password` to create the user in first-login mode — they set their own password on first sign-in.



# Delete a user (admin)
Source: https://docs.datris.ai/api-reference/auth/delete-a-user-admin

/openapi.yaml delete /api/v1/auth/users/{username}
Admin-only. The built-in `admin` recovery account can never be deleted, and the last remaining admin cannot be deleted.



# Get the current user
Source: https://docs.datris.ai/api-reference/auth/get-the-current-user

/openapi.yaml get /api/v1/auth/me
UI bootstrap probe — returns the session's user, or 401 if there is no session.



# List users (admin)
Source: https://docs.datris.ai/api-reference/auth/list-users-admin

/openapi.yaml get /api/v1/auth/users
Admin-only. Timestamps are formatted using the server's configured date format and timezone.



# Log in
Source: https://docs.datris.ai/api-reference/auth/log-in

/openapi.yaml post /api/v1/auth/login
Authenticates a user and sets an HTTP-only session cookie. Only available when user auth is enabled on the server. For a user whose password has not been set yet (`mustSetPassword`), any password is accepted and the client must immediately call `/auth/change-password`.



# Log out
Source: https://docs.datris.ai/api-reference/auth/log-out

/openapi.yaml post /api/v1/auth/logout
Deletes the server-side session and clears the session cookie.



# Single sign-on (OIDC) callback
Source: https://docs.datris.ai/api-reference/auth/single-sign-on-oidc-callback

/openapi.yaml get /api/v1/auth/oidc/callback
The redirect URI registered at the identity provider. Checks `state` against the `datris-oidc-tx` cookie, exchanges the code (client secret from Vault, PKCE verifier), validates the ID token, resolves the Datris user and sets the session cookie. Every outcome is a redirect: `/` on success, otherwise `/login?ssoError=<code>` with one of `sso_denied`, `sso_failed`, `sso_no_account`, `sso_unverified_email`, `sso_identity_changed`. Provider error text is logged on the server, never returned. Recorded in the audit log as `auth` / `login`.



# Start single sign-on (OIDC)
Source: https://docs.datris.ai/api-reference/auth/start-single-sign-on-oidc

/openapi.yaml get /api/v1/auth/oidc/login
Browser navigation, not an API call. Sets the short-lived HttpOnly `datris-oidc-tx` cookie (state, nonce and PKCE verifier; ten minutes, `SameSite=Lax`, path `/api/v1/auth/oidc`) and redirects to the identity provider's authorization endpoint with `response_type=code`, `code_challenge_method=S256`, `state` and `nonce`. Only available when user auth and OIDC are both on.



# Update a user (admin)
Source: https://docs.datris.ai/api-reference/auth/update-a-user-admin

/openapi.yaml patch /api/v1/auth/users/{username}
Admin-only. Change a user's role and/or reset their password (back to first-login mode). The built-in `admin` user is locked to the admin role, and the last remaining admin cannot be demoted.



# Catalog chat (SSE)
Source: https://docs.datris.ai/api-reference/chat/catalog-chat-sse

/openapi.yaml post /api/v1/catalog-chat/chat
Agent chat for organizing the inventory — group, move, rename, and describe taps and pipelines. Full tool catalog with catalog tools prioritized. The optional `context` object is a catalog inventory snapshot that grounds answers in what the user is looking at. Streams the same SSE event format as `/assistant/chat`.



# Configuration chat (SSE)
Source: https://docs.datris.ai/api-reference/chat/configuration-chat-sse

/openapi.yaml post /api/v1/config-chat/chat
Admin-only agent chat over the Configuration sub-tabs (AI providers, secrets, data sources, code repository, users, API keys, agent policy, audit log, doctor). Uses Datris-defined configuration tools, not the MCP tool catalog; user and API-key tools are present only when the matching sub-tab is shown. Mutating tools perform nothing on the first call: they answer `needs_confirmation` and the stream emits a `confirm_request` event; the change runs only after the user confirms it in the panel. Secret values are never passed through the model; they are collected through a `secret_request` form event. The optional `context` object is `{tab}`, the active sub-tab. Streams the same SSE event format as `/assistant/chat`.



# Ops chat (SSE)
Source: https://docs.datris.ai/api-reference/chat/ops-chat-sse

/openapi.yaml post /api/v1/ops-chat/chat
Agent chat for the Ops dashboard side panel — explains failures and acts on runs by name, with operational tools (run, status, kill) prioritized. The optional `context` object is a dashboard snapshot that grounds answers in the failures and volumes currently on screen. Streams the same SSE event format as `/assistant/chat`.



# Search chat (SSE)
Source: https://docs.datris.ai/api-reference/chat/search-chat-sse

/openapi.yaml post /api/v1/search-chat/chat
Conversational data discovery for the Search tab — finds the data that answers a question, queries/searches it, and answers with citations. The tool catalog is filtered server-side to READ-ONLY discovery/query/search tools; the agent never sees mutating tools. The optional `context` object may carry a catalog scope, e.g. `{"catalog": "finance"}` (or "Uncataloged"; "All"/empty means unscoped). Streams the same SSE event format as `/assistant/chat`.



# Get the tenant code repository configuration
Source: https://docs.datris.ai/api-reference/code-repository/get-the-tenant-code-repository-configuration

/openapi.yaml get /api/v1/code-repo
Returns the code repository connection used for tap script storage. The auth token itself lives in Vault and is never returned.



# Save the tenant code repository configuration
Source: https://docs.datris.ai/api-reference/code-repository/save-the-tenant-code-repository-configuration

/openapi.yaml put /api/v1/code-repo
Validates the connection when `enabled` (token can push, branch exists — verified via the repository `permissions` block, no probe commits) and persists the configuration.



# Test a code repository connection
Source: https://docs.datris.ai/api-reference/code-repository/test-a-code-repository-connection

/openapi.yaml post /api/v1/code-repo/test
Round-trip check for the posted configuration — repository reachable, token has push access, branch exists. Always returns 200 with an `ok` flag.



# Configuration API
Source: https://docs.datris.ai/api-reference/config-api

Upload configuration files and generate schemas

The Configuration API lets you upload validation schemas and generate JSON Schema or XSD from sample data using AI.

## Upload Config File

Upload a validation schema to the platform's configuration store (MinIO).

```
POST /api/v1/config/upload
Content-Type: multipart/form-data
```

| Parameter | Type | Required | Description |
| - | - | - | - |
| `file` | form-data | Yes | The configuration file to upload |
| `type` | form-data | Yes | File type: `validation-schema` or `javascript` |

The uploaded file is stored at `{env}-config/{type}/{filename}` in MinIO and can be referenced in pipeline configurations.

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/config/upload \
  -F "file=@my-schema.json" \
  -F "type=validation-schema"
```

### Response

```json theme={null}
{
  "filename": "my-schema.json",
  "path": "s3://oss-config/validation-schema/my-schema.json"
}
```

***

## Generate Schema from Sample Data

Use AI to generate a JSON Schema or XSD from sample data. The generated schema is stored in MinIO and can be referenced in pipeline configurations for data quality validation.

```
POST /api/v1/config/generate-schema
Content-Type: application/json
```

| Field | Type | Required | Description |
| - | - | - | - |
| `type` | string | Yes | Schema type: `json-schema` or `xsd` |
| `name` | string | Yes | Name for the generated schema file |
| `sampleData` | string | Yes | Sample data content for the AI to analyze |

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/config/generate-schema \
  -H "Content-Type: application/json" \
  -d '{
    "type": "json-schema",
    "name": "orders",
    "sampleData": "[{\"id\": 1, \"email\": \"jane@example.com\", \"total\": 249.99}]"
  }'
```

### Response

```json theme={null}
{
  "filename": "orders.json",
  "path": "s3://oss-config/validation-schema/orders.json",
  "schema": "{ ... generated JSON Schema ... }"
}
```

For `xsd` type, the filename uses `.xsd` extension and the response contains a generated W3C XML Schema.


# AI-generate a validation schema
Source: https://docs.datris.ai/api-reference/configuration/ai-generate-a-validation-schema

/openapi.yaml post /api/v1/config/generate-schema
Uses AI to generate a JSON Schema (Draft 4) or W3C XSD from sample data. The schema is stored in MinIO.



# Upload a configuration file
Source: https://docs.datris.ai/api-reference/configuration/upload-a-configuration-file

/openapi.yaml post /api/v1/config/upload
Upload a JSON Schema for pipeline dataQuality schema validation to storage. The uploaded file can then be referenced by pipeline configurations.



# Health Check API
Source: https://docs.datris.ai/api-reference/health-api

Health check endpoints

The Health Check API provides endpoints to check the connectivity status of all backend services, and to list which destinations and vector stores are actually available in this deployment.

## Check Service Health

```
GET /api/v1/health/services
```

Returns the health status of all configured backend services.

### Response

```json theme={null}
{
  "postgres": {"status": "up", "message": "Connected"},
  "mongodb": {"status": "up", "message": "Connected"},
  "minio": {"status": "up", "message": "Connected"},
  "activemq": {"status": "up", "message": "Connected"},
  "kafka": {"status": "up", "message": "Connected"},
  "qdrant": {"status": "not_configured"},
  "weaviate": {"status": "not_configured"},
  "milvus": {"status": "not_configured"},
  "chroma": {"status": "not_configured"},
  "pgvector": {"status": "not_configured"}
}
```

### Status Values

| Status | Description |
| - | - |
| `up` | Service is reachable and responding |
| `down` | Service is configured but not reachable (includes error message) |
| `not_configured` | No Vault secret configured for this service |

### Services Checked

**Core infrastructure** (always checked):

* `postgres` — PostgreSQL database
* `mongodb` — MongoDB database
* `minio` — MinIO object store
* `activemq` — ActiveMQ message queue
* `kafka` — Apache Kafka

**Vector databases** (checked only if configured via Vault secrets):

* `qdrant` — Qdrant vector database
* `weaviate` — Weaviate vector database
* `milvus` — Milvus vector database
* `chroma` — Chroma vector database
* `pgvector` — PostgreSQL with pgvector extension

### Example

```bash theme={null}
curl http://localhost:8080/api/v1/health/services
```

***

## List Available Destinations

```
GET /api/v1/destinations/available
```

Returns the structured destinations that are usable in this deployment, as a JSON array of destination names. This drives the pipeline wizard's and Assistant's destination pickers.

Two availability semantics are used on purpose:

* **Self-hosted destinations** (`mongodb`, `postgres`, `objectstore`) are **live-probed** — a destination is listed only if the service is actually reachable. The presence of a Vault secret alone is not enough, because the compose stack seeds secrets even when a service is disabled.
* **External SaaS destinations** (`snowflake`, `databricks`) have no local service to probe, so they are listed when any Platform secret contains a complete set of credentials for that destination. Credential secret names are user-chosen per pipeline, so every Platform secret is scanned rather than a fixed name looked up.

### Response

```json theme={null}
["mongodb", "postgres", "objectstore", "snowflake"]
```

### Example

```bash theme={null}
curl http://localhost:8080/api/v1/destinations/available
```

***

## List Available Vector Stores

```
GET /api/v1/vector-stores/available
```

Returns the vector stores that are live and reachable, as a JSON array of store keys. This drives the document-tap pipeline wizard's store picker and the Search tab's store dropdown, and is the recommended way to discover which stores can be searched via [`POST /api/v1/search/{store}`](/api-reference/search-api).

Each candidate store (`qdrant`, `weaviate`, `pgvector`, `milvus`, `chroma`) is actively probed using the same connectivity checks as `/health/services` — only stores whose status is `up` are returned. A store with a configured Vault secret but an unreachable service is excluded.

### Response

```json theme={null}
["pgvector", "qdrant"]
```

### Example

```bash theme={null}
curl http://localhost:8080/api/v1/vector-stores/available
```


# Check backend service health
Source: https://docs.datris.ai/api-reference/health/check-backend-service-health

/openapi.yaml get /api/v1/health/services
Returns the health status of all configured backend services including PostgreSQL, MongoDB, MinIO, ActiveMQ, Kafka, and vector databases (Qdrant, Weaviate, Milvus, Chroma, pgvector). Each service reports "up", "down", or "not_configured".



# List live, reachable vector stores
Source: https://docs.datris.ai/api-reference/health/list-live-reachable-vector-stores

/openapi.yaml get /api/v1/vector-stores/available
Returns the subset of `[qdrant, weaviate, pgvector, milvus, chroma]` whose Vault secret is present AND whose service responds to a probe. Drives the document-tap pipeline wizard's store picker. Secret presence alone isn't sufficient — the dev stack seeds placeholder secrets for every store, so this endpoint must actually probe each one (2s timeout per store, ~10s worst case if everything is unreachable).



# List usable structured destinations
Source: https://docs.datris.ai/api-reference/health/list-usable-structured-destinations

/openapi.yaml get /api/v1/destinations/available
Returns the subset of `[mongodb, postgres, objectstore, snowflake, databricks]` usable in this deployment. Drives the pipeline wizard's and Assistant's destination picker. Two availability semantics on purpose: the self-hosted destinations (mongodb, postgres, objectstore) are live-probed — secret presence alone isn't a reliable signal because the compose stack seeds their secrets even when a service is disabled; the external SaaS destinations (snowflake, databricks) have no local service to probe, so they are included when any Platform secret carries a complete credential set for them.



# Data Ingestion API
Source: https://docs.datris.ai/api-reference/ingestion-api

Data ingestion endpoints

## Upload a File

Upload a data file for processing by a configured pipeline.

```
POST /api/v1/pipeline/upload
Content-Type: multipart/form-data
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `file` | form-data | Yes | The file to upload |
| `pipeline` | form-data | Yes | Target pipeline name |
| `publishertoken` | form-data | No | Publisher identifier for tracking |

**Behavior:**

* **Compressed files** (`.zip`, `.gz`, `.tar`, `.jar`): Extracted in-process. For a CSV pipeline with several files in the archive, the files are concatenated (headers on files 2+ skipped) and submitted as one batch job; otherwise each extracted file is submitted as its own job
* **Uncompressed files**: Processed immediately in-memory
* **Schema evolution**: If a CSV file contains new columns not in the pipeline schema, they are automatically added (as `string` type) and the destination table is altered. See [Schema Evolution](/schemas#schema-evolution).

**Example:**

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/upload \
  -F "file=@/path/to/data.csv" \
  -F "pipeline=sales_data" \
  -F "publishertoken=batch-001"
```

**Response:** `200 OK` with the pipeline token (for uncompressed files):

```
pt-abc12345-6789-...
```

For compressed files, the response is `200 OK` with the single pipeline token of the batch job (CSV batch mode, or a single-file archive), or the plain-text summary `N file(s) submitted` when each extracted file was submitted as its own job.

***

## Generate Pipeline Schema

Upload a CSV file to automatically infer the schema and generate a partial pipeline configuration.

```
POST /api/v1/pipeline/generate
Content-Type: multipart/form-data
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `file` | form-data | Yes | Data file to analyze (CSV, JSON, or XML) |
| `pipeline` | form-data | No | Pipeline name (auto-derived from filename if omitted) |
| `delimiter` | form-data | No | CSV delimiter (default: `,`) |
| `header` | form-data | No | Whether file has header row |
| `allStrings` | form-data | No | If `true`, all fields are typed as `string` (default: `false`) |

**Example:**

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/generate \
  -F "file=@/path/to/sample.csv" \
  -F "pipeline=my_pipeline"
```

**Response:** `200 OK` with a partial PipelineConfig JSON:

```json theme={null}
{
  "name": "my_pipeline",
  "source": {
    "fileAttributes": {
      "csvAttributes": {
        "delimiter": ",",
        "header": true
      }
    },
    "schemaProperties": {
      "fields": [
        {"name": "id", "type": "int"},
        {"name": "name", "type": "string"},
        {"name": "amount", "type": "double"},
        {"name": "created_at", "type": "string"}
      ]
    }
  },
  "destination": {
    "database": {
      "dbName": "datris",
      "schema": "public",
      "table": "my_pipeline",
      "usePostgres": true
    }
  }
}
```

**Inferred types:** `boolean`, `int`, `bigint`, `float`, `double`, `string`, `date`, `timestamp`

**Note:** For CSV files, the AI analyzes the content and infers types. For JSON and XML files, a default config is generated with a single `_json` or `_xml` string field. When `allStrings` is `true`, all fields are set to `string` (used by the MCP server for reliable ingestion). Edit the generated JSON to add your destination configuration before registering it with `POST /pipeline`.

***

## Profile Data

Upload a data file for AI-powered profiling — get summary statistics, quality issues, and suggested validation rules.

```
POST /api/v1/pipeline/profile
Content-Type: multipart/form-data
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `file` | form-data | Yes | Data file to profile |
| `delimiter` | form-data | No | CSV delimiter (default: `,`) |
| `header` | form-data | No | Whether file has header row (default: `true`) |
| `sampleSize` | form-data | No | Number of rows to sample (default: `200`) |

**Example:**

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/profile \
  -F "file=@/path/to/data.csv" \
  -F "sampleSize=200"
```

**Response:** `200 OK` with a JSON profile including summary statistics, quality issues, recommendations, and suggested data quality rules. See [AI Data Profiling](/ai-data-profiling) for the full response format.


# Upload a file for processing
Source: https://docs.datris.ai/api-reference/ingestion/upload-a-file-for-processing

/openapi.yaml post /api/v1/pipeline/upload
Upload a data file to a registered pipeline. Returns a pipeline token for tracking job status.



# Clear MCP activity history
Source: https://docs.datris.ai/api-reference/mcp/clear-mcp-activity-history

/openapi.yaml delete /api/v1/mcp/activity
Clears the MCP server's activity buffer so the cleared state survives page reloads. Live sessions are untouched.



# Get MCP server activity
Source: https://docs.datris.ai/api-reference/mcp/get-mcp-server-activity

/openapi.yaml get /api/v1/mcp/activity
Proxies the MCP server's activity feed (agent sessions and tool calls) with server-side enrichment: timestamps are formatted with the platform's date settings and each session/call is labeled with the tenant or key name resolved from its API-key hint. Always returns 200 — when the MCP server is unreachable or auth fails, the payload carries an `error` message and an `errorKind` of "unreachable" or "auth" with empty session/call arrays.



# Metadata API
Source: https://docs.datris.ai/api-reference/metadata-api

Database and vector store metadata discovery

The Metadata API lets you explore the structure of PostgreSQL, MongoDB, and vector databases managed by the platform. Use these endpoints to discover what data is available before writing queries or running searches.

## PostgreSQL

### List Databases

```
GET /api/v1/metadata/postgres/databases
```

Returns a JSON array of PostgreSQL database names.

***

### List Schemas

```
GET /api/v1/metadata/postgres/schemas
```

| Parameter | Type | Default | Description |
| - | - | - | - |
| `database` | query | configured Postgres database | Database name. Defaults to the configured Postgres database (`postgres.database`, which is `datris` on a default install) |

Returns a JSON array of schema names. System schemas (`information_schema`, `pg_catalog`, `pg_toast`) are excluded.

***

### List Tables

```
GET /api/v1/metadata/postgres/tables
```

| Parameter | Type | Default | Description |
| - | - | - | - |
| `database` | query | configured Postgres database | Database name. Defaults to the configured Postgres database (`postgres.database`, which is `datris` on a default install) |
| `schema` | query | `public` | Schema name |
| `vectorOnly` | query | `false` | If `true`, return only pgvector tables (tables with an `embedding` column) |

Returns a JSON array of table names. When `vectorOnly` is `false`, pgvector tables are excluded from the results.

***

### List Columns

```
GET /api/v1/metadata/postgres/columns
```

| Parameter | Type | Default | Description |
| - | - | - | - |
| `database` | query | configured Postgres database | Database name. Defaults to the configured Postgres database (`postgres.database`, which is `datris` on a default install) |
| `schema` | query | `public` | Schema name |
| `table` | query | (required) | Table name |

Returns a JSON array of column definitions:

```json theme={null}
[
  {"name": "id", "type": "bigint"},
  {"name": "email", "type": "text"},
  {"name": "created_at", "type": "timestamp"}
]
```

***

## MongoDB

### List Databases

```
GET /api/v1/metadata/mongodb/databases
```

Returns a JSON array of MongoDB database names. System databases (`admin`, `config`, `local`) and the platform's internal database are excluded.

***

### List Collections

```
GET /api/v1/metadata/mongodb/collections
```

| Parameter | Type | Default | Description |
| - | - | - | - |
| `database` | query | (all databases) | Filter to a specific database |

When `database` is provided, returns collection names from that database. When omitted, returns collections from all databases with a `database.collection` namespace prefix.

***

## Vector Stores

### List Qdrant Collections

```
GET /api/v1/metadata/qdrant/collections
```

Returns a JSON array of Qdrant collection names. Returns `[]` if Qdrant is not configured.

***

### List Weaviate Classes

```
GET /api/v1/metadata/weaviate/classes
```

Returns a JSON array of Weaviate class names. Returns `[]` if Weaviate is not configured.

***

### List Milvus Collections

```
GET /api/v1/metadata/milvus/collections
```

Returns a JSON array of Milvus collection names. Returns `[]` if Milvus is not configured.

***

### List Chroma Collections

```
GET /api/v1/metadata/chroma/collections
```

Returns a JSON array of Chroma collection names. Returns `[]` if Chroma is not configured.

***

## Example

```bash theme={null}
# Discover databases
curl -s http://localhost:8080/api/v1/metadata/postgres/databases | jq .

# Explore a schema
curl -s "http://localhost:8080/api/v1/metadata/postgres/tables?database=datris&schema=public" | jq .

# Inspect a table
curl -s "http://localhost:8080/api/v1/metadata/postgres/columns?table=stock_prices" | jq .

# List vector collections
curl -s http://localhost:8080/api/v1/metadata/qdrant/collections | jq .
```


# List Chroma collections
Source: https://docs.datris.ai/api-reference/metadata/list-chroma-collections

/openapi.yaml get /api/v1/metadata/chroma/collections
List all collections in the Chroma vector database. Returns an empty array if Chroma is not configured.



# List Milvus collections
Source: https://docs.datris.ai/api-reference/metadata/list-milvus-collections

/openapi.yaml get /api/v1/metadata/milvus/collections
List all collections in the Milvus vector database. Returns an empty array if Milvus is not configured.



# List MongoDB collections
Source: https://docs.datris.ai/api-reference/metadata/list-mongodb-collections

/openapi.yaml get /api/v1/metadata/mongodb/collections
List MongoDB collections. If database is specified, lists collections in that database. Otherwise lists all collections across all databases in "db.collection" format.



# List MongoDB databases
Source: https://docs.datris.ai/api-reference/metadata/list-mongodb-databases

/openapi.yaml get /api/v1/metadata/mongodb/databases
List all MongoDB databases available in the platform. Excludes system databases (admin, config, local).



# List PostgreSQL databases
Source: https://docs.datris.ai/api-reference/metadata/list-postgresql-databases

/openapi.yaml get /api/v1/metadata/postgres/databases
List all PostgreSQL databases available in the platform.



# List PostgreSQL schemas
Source: https://docs.datris.ai/api-reference/metadata/list-postgresql-schemas

/openapi.yaml get /api/v1/metadata/postgres/schemas
List all schemas in a PostgreSQL database.



# List PostgreSQL tables
Source: https://docs.datris.ai/api-reference/metadata/list-postgresql-tables

/openapi.yaml get /api/v1/metadata/postgres/tables
List all tables in a PostgreSQL schema. Set vectorOnly to true to filter to tables with an embedding column (pgvector tables).



# List Qdrant collections
Source: https://docs.datris.ai/api-reference/metadata/list-qdrant-collections

/openapi.yaml get /api/v1/metadata/qdrant/collections
List all collections in the Qdrant vector database. Returns an empty array if Qdrant is not configured.



# List table columns
Source: https://docs.datris.ai/api-reference/metadata/list-table-columns

/openapi.yaml get /api/v1/metadata/postgres/columns
List all columns and their data types for a specific PostgreSQL table.



# List Weaviate classes
Source: https://docs.datris.ai/api-reference/metadata/list-weaviate-classes

/openapi.yaml get /api/v1/metadata/weaviate/classes
List all classes in the Weaviate vector database. Returns an empty array if Weaviate is not configured.



# Pipeline Configuration API
Source: https://docs.datris.ai/api-reference/pipeline-api

Pipeline configuration endpoints

Manage pipeline configurations through CRUD operations.

## Get a Pipeline

```
GET /api/v1/pipeline?pipeline={name}
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `pipeline` | query | Yes | Pipeline name |

**Example:**

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline?pipeline=sales_data"
```

**Response:** `200 OK` with the full PipelineConfig JSON.

***

## Unity Catalog sync state

```
GET /api/v1/pipelines/{name}/unity-catalog
```

The last Unity Catalog sync for a pipeline that sets `unityCatalog.enabled` (or that the install default `DATRIS_UNITY_CATALOG_DEFAULT=enabled` turned on): the metadata push and lineage for a Databricks pipeline (see [Databricks](/destinations/databricks#unity-catalog-metadata)), or the table registration for an object store Iceberg pipeline (see [Register Iceberg tables](/destinations/unity-catalog#register-iceberg-tables)).

**Example:**

```bash theme={null}
curl http://localhost:8080/api/v1/pipelines/orders_pipeline/unity-catalog
```

**Response:** `200 OK`. `enabled` says whether Unity Catalog is on for the pipeline and `enabledBy` says who decided: `pipeline` (the pipeline's own `unityCatalog` block, including an `"enabled": false` opt-out), `default` (the install default turned on a Databricks pipeline with no block), or `null` (off, no block). `lineageEnabled` follows the same resolution. `syncSwitchedOff` is `true` while `DATRIS_UNITY_CATALOG_SYNC=false`: nothing is written to Unity Catalog even when `enabled` is `true`. `coordinates` gives the Unity Catalog table (Databricks destinations only). `state` is `never` (no sync yet), `synced`, or `error` (with `lastError` holding the warehouse's message, or when the last `catalogMode: rest` run was refused; see `restRefusedReason`). `lastSyncAt` and `lastRunId` identify the last sync. `lineage` is the lineage publish status: `off` (`unityCatalog.lineage` is `false`, or the push is off), `never`, `published`, or `error` (the failure is a `lastError` line starting with `uc-lineage:`); `lastLineageAt` is when lineage was last published and `lineageHash` identifies the published definition (see [Lineage in Unity Catalog](/destinations/unity-catalog#lineage-in-unity-catalog)). For an object store Iceberg pipeline, `coordinates` names the registered table (`catalog.schema.table`, with `kind: "iceberg"` and the table's `location`), `registerEnabled` says whether registration is on, and `register` is `off`, `never`, `registered`, `stale` (Unity Catalog still points at an older metadata file) or `error` (the failure is a `lastError` line starting with `uc-register:`); `registeredMetadataLocation` and `lastRegisterAt` record the last registration. `catalogMode` is `register`, `rest` or `managed` (see [Keep the catalog current](/destinations/unity-catalog#keep-the-catalog-current-catalogmode-rest) and [Databricks: catalogMode managed](/destinations/unity-catalog#databricks-catalogmode-managed)); in `managed` mode `register` is `managed` when the last run committed through the catalog, `restMetadataLocation` is under the location the catalog chose, and `coordinates.location` is that location once known; in `rest` mode `register` is `rest` when the last run committed through the catalog and `refused` when it wrote by path instead, `restMetadataLocation` and `lastRestCommitAt` record the last catalog commit, and `restRefusedReason` says why a run was refused. `restCreatedTable` names a table the catalog created for the pipeline at a location Datris refused (a Databricks managed table): it was never written to, and an admin has to drop it. `404` if the pipeline does not exist.

```json theme={null}
{
  "pipeline": "orders_pipeline",
  "enabled": true,
  "enabledBy": "pipeline",
  "syncSwitchedOff": false,
  "coordinates": { "catalog": "datris", "schema": "default", "table": "orders", "qualified": "datris.default.orders" },
  "state": "synced",
  "lastSyncAt": "2026-09-29T12:00:00Z",
  "lastRunId": "...",
  "lineageEnabled": true,
  "lineage": "published",
  "lineageHash": "...",
  "lastLineageAt": "2026-09-29T12:00:01Z",
  "registerEnabled": false,
  "register": "off"
}
```

For an object store Iceberg pipeline:

```json theme={null}
{
  "pipeline": "orders_lake",
  "enabled": true,
  "coordinates": { "catalog": "analytics", "schema": "default", "table": "orders_lake", "qualified": "analytics.default.orders_lake", "kind": "iceberg", "location": "s3a://my-lake/orders" },
  "registerEnabled": true,
  "register": "registered",
  "registeredMetadataLocation": "s3a://my-lake/orders/metadata/00001-....metadata.json",
  "lastRegisterAt": "2026-09-29T12:00:02Z"
}
```

***

## List All Pipelines

```
GET /api/v1/pipelines
```

**Example:**

```bash theme={null}
curl http://localhost:8080/api/v1/pipelines
```

**Response:** `200 OK` with an array of all PipelineConfig objects.

***

## Create or Update a Pipeline

```
POST /api/v1/pipeline
Content-Type: application/json
```

**Body:** Full PipelineConfig JSON (see [Pipeline Configuration](/pipeline-configuration)).

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `changeNote` | query | No | Note recorded on the saved pipeline version |

**Example:**

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "name": "sales_data",
    "source": {
      "schemaProperties": {
        "fields": [
          {"name": "id", "type": "int"},
          {"name": "amount", "type": "double"}
        ]
      },
      "fileAttributes": {
        "csvAttributes": {"delimiter": ",", "header": true, "encoding": "UTF-8"}
      }
    },
    "destination": {
      "database": {
        "dbName": "mydb",
        "schema": "public",
        "table": "sales",
        "usePostgres": true
      }
    }
  }'
```

**Response:** `200 OK`

```json theme={null}
{"warnings": []}
```

`warnings` is advisory: the pipeline is saved either way. It is empty unless something about the configuration will not work as written, for example an object store Iceberg pipeline whose Unity Catalog secret points at a Databricks workspace with `catalogMode` `register` or `rest` (Databricks cannot serve either; the warning says to set `catalogMode` `managed` or use the databricks destination). Each warning is also written to the server log.

**Validation:** The configuration is validated before saving. Invalid configurations return `400` with `{"error": "<message>"}` and nothing is saved; `500` means the server itself failed. See [Pipeline Configuration](/pipeline-configuration) for all constraints.

***

## Delete a Pipeline

```
DELETE /api/v1/pipeline?pipeline={name}
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `pipeline` | query | Yes | Pipeline name |
| `deleteData` | query | No | Delete destination data (default: `true`). Only takes effect together with `deleteConfig=false`: deleting the configuration always deletes the data too, so `deleteData=false` on its own still removes the data. |
| `deleteConfig` | query | No | Delete pipeline configuration (default: `true`). Set to `false` to delete data only while keeping the pipeline config. |

There is no way to delete a pipeline's configuration and keep its destination data; a pipeline with no configuration would leave tables, collections, and files behind that nothing owns.

**Examples:**

```bash theme={null}
# Delete pipeline and all data
curl -X DELETE "http://localhost:8080/api/v1/pipeline?pipeline=sales_data"

# Delete data only, keep pipeline configuration
curl -X DELETE "http://localhost:8080/api/v1/pipeline?pipeline=sales_data&deleteData=true&deleteConfig=false"
```

**Response:** `200 OK` with a JSON body:

```json theme={null}
{ "warnings": [] }
```

`warnings` lists anything left for you to handle. For an object-store Iceberg pipeline whose table was committed through Unity Catalog (`catalogMode: rest`), it says `Unity Catalog still holds <catalog>.<schema>.<table>; have an admin drop it`, because Datris removes the files but never drops the catalog table (see [Unity Catalog](/destinations/unity-catalog)). For a `catalogMode: managed` table it says `Unity Catalog owns <catalog>.<schema>.<table> and its files (catalogMode managed); Datris deleted nothing there; drop <catalog>.<schema>.<table> in Unity Catalog to remove the data`.

***

## AI Schema Generation

To generate a pipeline configuration automatically from an uploaded file, see the [AI Schema Generation API](/api-reference/schema-generation-api).

***

## Authentication

If `useApiKeys` is enabled, all requests must include the `x-api-key` header:

```bash theme={null}
curl -H "x-api-key: your-api-key" http://localhost:8080/api/v1/pipelines
```

API keys are stored in Vault under `{environment}/api-keys` (`oss/api-keys` on a default install). `secrets.apiKeysSecretName` in `application.yaml` tells the validator where to read them and should point at that same path — the Keys UI and multi-tenant resolution always write to `{environment}/api-keys`.


# Adopt a CodeGen script edited in the code repository
Source: https://docs.datris.ai/api-reference/pipelines/adopt-a-codegen-script-edited-in-the-code-repository

/openapi.yaml post /api/v1/pipelines/{name}/codegen-scripts/{kind}/pull
For a script stored in the code repository: records the commit at the head of the configured branch (origin `repository`), so later runs execute the file as it is in the repository now. No commit is made. Later saves keep the adopted script while the instruction and source schema are unchanged. Requires `pipeline:create`. Audited.



# Apply destination column types
Source: https://docs.datris.ai/api-reference/pipelines/apply-destination-column-types

/openapi.yaml post /api/v1/pipeline/dest-types
Applies column types to an all-string destination. Destination-first: if data has already landed the table is migrated before the config changes — postgres retypes in place, snowflake/databricks validate every landed value with TRY_CAST and then swap the table. Any landed value that will not cast fails the whole apply with the column named and nothing changed. On success the pipeline definition is updated as a new version. The migration locks or replaces the table — do not apply while a run is in flight.



# Cancel a catalog's ordered enable
Source: https://docs.datris.ai/api-reference/pipelines/cancel-a-catalogs-ordered-enable

/openapi.yaml delete /api/v1/catalog/{name}/enable-ordered
The test, run or load wait in progress finishes (a running tap script is not killed); every tap not yet started is marked `cancelled` (a tap already enabled whose run had not started reads `enabled; run cancelled`). Requires `tap:run`, checked against the key's catalog scope and the job starter as owner; 403 otherwise.




# Create or update a pipeline configuration
Source: https://docs.datris.ai/api-reference/pipelines/create-or-update-a-pipeline-configuration

/openapi.yaml post /api/v1/pipeline
Saves (upserts) a pipeline configuration. Invalid field protection is rejected at save time (unknown or not-yet-supported `protect.method`, `preserve` outside `mask`, a string-producing method on a non-string field, any method but `hmac` on a `keyFields` column, or a source that is not delimited or JSON). Like every other validator error, the rejection returns HTTP 400 with `{"error": "..."}` naming the field.



# Delete a catalog (detach or cascade)
Source: https://docs.datris.ai/api-reference/pipelines/delete-a-catalog-detach-or-cascade

/openapi.yaml delete /api/v1/catalog/{name}
`mode=detach` (default) clears `catalog` on every member so it appears under Uncataloged, through a versioned write per member. `mode=cascade` deletes every member tap and pipeline, including each pipeline's destination data, and requires `confirm` equal to `name`. Both remove every placeholder of the catalog (matched by its `catalog` field, or by name suffix when that field is empty). A member already moved out of `name` by a concurrent change is listed under `failed`. Not transactional: failed members are listed under `failed` (207). Catalog operations (rename, detach, cascade) are serialized server-side, so concurrent requests run one after another and the later one sees the earlier one's result. Requires `pipeline:delete`; each member is also checked against the key's owner and catalog scopes, and the placeholder is left untouched if any member is refused. In cascade mode, if any tap delete fails, the catalog's pipelines are not deleted and are reported under `failed`. A cascade response also carries `warnings` (always present, possibly empty): one entry per deleted pipeline that needs a follow-up, such as a Unity Catalog table Datris could not drop; warnings never change the status code. Refused with 409 while an ordered enable of the catalog (`POST /api/v1/catalog/{name}/enable-ordered`) is running.




# Delete a pipeline configuration
Source: https://docs.datris.ai/api-reference/pipelines/delete-a-pipeline-configuration

/openapi.yaml delete /api/v1/pipeline



# Enable a catalog's taps in dependency order
Source: https://docs.datris.ai/api-reference/pipelines/enable-a-catalogs-taps-in-dependency-order

/openapi.yaml post /api/v1/catalog/{name}/enable-ordered
Starts one server-side job that walks the catalog's enable order (see `GET /api/v1/catalog/{name}/enable-order`) level by level, one tap at a time. Per tap: if any of its dependencies failed, was skipped or cancelled, or has not landed data, the tap is `skipped` with the reason; otherwise it is tested when it has no passing test (a failed test marks it `failed`), enabled (the test-before-cron gate applies; a refusal marks it `failed` with that text), given one real run (an error or zero records marks it `failed`) and the job waits until the run's pipeline loads have finished (`success`/`warning` = `landed`, `error` = `failed`, no finish within `datris.catalog.enableWaitMs`, default 30 minutes = `failed`). A tap that was already enabled with landed data is not re-run. No request body. Requires `tap:run`; the key must also hold `tap:update` and `tap:run` for every member tap (owner and catalog scope), otherwise 403 and nothing starts. Agent policy action `tap:run:enable-ordered` (inherits `tap:run`): an approval covers the whole job.




# Enable order of a catalog's taps
Source: https://docs.datris.ai/api-reference/pipelines/enable-order-of-a-catalogs-taps

/openapi.yaml get /api/v1/catalog/{name}/enable-order
Groups the catalog's taps into dependency levels: level 1 holds taps that read no other tap's tables, level 2 the taps that read level 1, and so on. A tap's dependencies are its `dependsOn` when set (even `[]`), else its stored `inferredDependsOn`. Per tap: `hasPassingTest` (the last test run succeeded on the tap's current script; a test stamped for an older script, e.g. after an overwrite restore, does not count), `lastRealRunLanded` (`lastRunStatus` is `success` with `lastRunRecordCount` > 0), `blockedBy` (dependencies that have not landed data) and `ready` (tested and not blocked). Taps on a dependency cycle are listed under `cycles`, not in a level. Read-only and not audited. Requires `tap:read`; taps the key may not read (owner or catalog scope) are omitted and named in `warnings`.




# Export a catalog as a zip backup
Source: https://docs.datris.ai/api-reference/pipelines/export-a-catalog-as-a-zip-backup

/openapi.yaml get /api/v1/catalog/{name}/export
Downloads one zip archive of the catalog: `manifest.json`, `taps/<name>.json` (run status, ownership, timestamps and script references cleared), `taps/<name>.py` (each Python tap's script), `taps/<name>.state.json` (incremental state, when the tap has any), `pipelines/<name>.json`, `pipelines/<name>/validation-schema/<file>` (a referenced validation schema) and `pipelines/<name>/codegen/<kind>.py` (CodeGen scripts, reference copies only). Placeholders are never exported. Secret values are never in the archive: `manifest.requiredSecrets` lists each secret name, the items using it and its type. `manifest.warnings` names members the key may not read (omitted from the archive, never a 403 for the whole export), scripts or schemas that could not be read, taps whose state was included, and install-specific values copied verbatim (object-store endpoint, bucket override, warehouse, absolute `s3://` validation schema), and every inline credential removed from a pipeline (preprocessor and REST destination `bearerToken`/`apiKey`, `restEndpoint` row-function token parameters). The manifest carries `formatVersion: 1`. Requires `pipeline:read`; each member is also checked against the key's owner and catalog scopes as `tap:read` / `pipeline:read`.




# Force a new CodeGen script
Source: https://docs.datris.ai/api-reference/pipelines/force-a-new-codegen-script

/openapi.yaml post /api/v1/pipelines/{name}/codegen-scripts/{kind}/regenerate
Generates a new script for `kind` now, from the source schema, with the current CodeGen model, and replaces the stored one. If generation fails the error is returned (502) and the current script stays in place. For a JSON or XML pipeline the script is marked pending and the next run generates it from its data. The script stays in its current store unless `storage` asks for another; the old copy is left in place. A repository-backed script whose file was edited in the repository since the recorded commit is not overwritten (409) unless `overwrite=true`. Requires `pipeline:create`. Audited.



# Get a pipeline configuration
Source: https://docs.datris.ai/api-reference/pipelines/get-a-pipeline-configuration

/openapi.yaml get /api/v1/pipeline



# List all pipeline configurations
Source: https://docs.datris.ai/api-reference/pipelines/list-all-pipeline-configurations

/openapi.yaml get /api/v1/pipelines



# Progress of a catalog's ordered enable
Source: https://docs.datris.ai/api-reference/pipelines/progress-of-a-catalogs-ordered-enable

/openapi.yaml get /api/v1/catalog/{name}/enable-ordered
Exactly `{"active": false}` when the catalog has no job. A finished job stays readable for one hour with `active` false and `finishedAt` set. The same body is the `job` block of `GET .../enable-order`. Requires `tap:read`, checked against the key's catalog scope (the catalog) and owner scope (who started the job); 403 otherwise. Not audited.




# Propose destination column types from landed data
Source: https://docs.datris.ai/api-reference/pipelines/propose-destination-column-types-from-landed-data

/openapi.yaml get /api/v1/pipeline/dest-types
Agent-created pipelines store every destination column as text. This stateless call samples up to 1000 rows that already landed in the destination (postgres, snowflake, or databricks), infers real column types deterministically, and returns them with per-column evidence: a few distinct sample values, and — for columns kept as string by a dirty value — the offending value and the type it blocked. Nothing is stored or changed.



# Propose field protection from a preset (HIPAA Safe Harbor)
Source: https://docs.datris.ai/api-reference/pipelines/propose-field-protection-from-a-preset-hipaa-safe-harbor

/openapi.yaml post /api/v1/pipeline/protect/preset
Classifies source fields against the preset's fixed name table (the 18 HIPAA Safe Harbor identifier classes) and proposes each recognised field's default method, clamped by the same rules as `/api/v1/pipeline/protect/suggest` (keyFields columns only hmac or none; non-string source or destination types only drop or none). Deterministic: no model call. Stateless: nothing is saved and nothing is audited. Pass `preset` with either `pipeline` (its stored source schema and constraints are used) or `fields`. At most 150 fields per call; `_json` and `_xml` are skipped. A stored JSON, XML or unstructured pipeline is refused with 400. Needs the same capability as reading a pipeline. An aid to applying Safe Harbor, not a certification.



# Rename a catalog
Source: https://docs.datris.ai/api-reference/pipelines/rename-a-catalog

/openapi.yaml put /api/v1/catalog/{name}
Relabels every tap and pipeline whose `catalog` equals `name` to `newName`, each through a versioned write (one new definition version per member, with a change note naming the rename and the actor). Every hidden placeholder of the catalog (a `__catalog__*` tap whose `catalog` field is `name`, or whose name suffix is `name` when that field is empty) is removed, and `__catalog__<newName>` is created with `catalog=newName`. A member already moved out of `name` by a concurrent change is listed under `failed` ("no longer in catalog"), and placeholders are left untouched when no member moved. A rename never merges: if `newName` already exists (any tap or pipeline carries it, or it has a placeholder; matched case-sensitively) the request is refused with 409 and nothing changes. To combine catalogs, move their items. Not transactional: members that fail are listed under `failed` and the rest still move (207). API keys whose capabilities are scoped `catalog=<name>` are listed under `affectedKeys`; their scopes are not rewritten. Catalog operations (rename, detach, cascade) are serialized server-side, so concurrent requests run one after another and the later one sees the earlier one's result. Requires `pipeline:update`; each member is also checked against the key's owner and catalog scopes. If any member is refused by scope, placeholders are left untouched (`placeholder: skipped`); for a catalog with only a placeholder, the placeholder is scope-checked (403 on denial). Refused with 409 while an ordered enable of the catalog (`POST /api/v1/catalog/{name}/enable-ordered`) is running.




# Restore a catalog from a zip backup
Source: https://docs.datris.ai/api-reference/pipelines/restore-a-catalog-from-a-zip-backup

/openapi.yaml post /api/v1/catalog/{name}/restore
Restores an archive produced by `GET /api/v1/catalog/{name}/export` as catalog `name` (the same or a new name, on the same or another install). A restore never merges: with `mode=new` (default) it is refused with 409 when catalog `name` already exists or any archived tap or pipeline name exists anywhere on the install (any catalog, either type), and nothing is written. With `mode=overwrite` (requires `confirm` equal to `name`) items already in `name` may be replaced; a name in another catalog is still a 409. When tap scripts are stored in a code repository, a script file already at branch head with different content is also a clash; identical content is reused without a new commit. Pipelines are saved exactly as `POST /api/v1/pipeline` saves them; tap scripts go to the install's default script storage; taps are saved with `enabled=false` and their run status cleared, so nothing runs until a user tests and enables them. Secrets are never written: `missingSecrets` lists the secrets the archived items reference that are absent on this install. Secrets are only checked where the key may read them: a key without secret read access gets a `<name>: not checked, this key may not read secrets` warning instead, whether or not the secret exists. Each tap's incremental state is restored for taps that did not exist before, unless `restoreState=false`. An archive with no members creates the empty catalog. Not transactional: failed members are listed under `failed` (207). Requires `pipeline:create`. Every member is scope-checked before anything is written: `tap:create` / `pipeline:create` in the target catalog as the caller, and for an item that already exists (overwrite) also `tap:update` / `pipeline:update` against its current catalog and owner. A refused member is listed under `failed` and nothing of it, including its validation schema, is written. Inline credentials (preprocessor and REST destination `bearerToken`/`apiKey`, `restEndpoint` row-function token parameters) are removed at export; the restore response's `warnings` names each one to re-enter. Refused with 409 while an ordered enable of the catalog (`POST /api/v1/catalog/{name}/enable-ordered`) is running.




# Reveal values of an encrypt-protected field
Source: https://docs.datris.ai/api-reference/pipelines/reveal-values-of-an-encrypt-protected-field

/openapi.yaml post /api/v1/protect/reveal
Decrypts `enc:v<n>:` ciphertexts that a pipeline's `encrypt` field protection wrote to a destination. Stateless: send the ciphertexts you read from the destination; Datris never queries a destination. A ciphertext is bound to its pipeline and field, so one copied to another pipeline or column fails. Needs the `protect:reveal` capability, which no key template or editor/viewer role carries; admins, the `full-access` template and legacy unscoped keys hold it through `*:*`. Not available as an MCP tool. Every call is recorded in the audit log as `protect / reveal` (field name and counts only, never a value), with outcome `warning` when any value failed; denied calls are recorded as `security` events.



# Rotate the field protection encryption key
Source: https://docs.datris.ai/api-reference/pipelines/rotate-the-field-protection-encryption-key

/openapi.yaml post /api/v1/protect/keys/rotate
Issues a new version of the environment's `encrypt` key and makes it current; runs after this encrypt with the new version (`enc:v<n>:`). Older versions stay in the secret, so values encrypted with them still reveal. The `hmac` key is never rotated. Needs `protect:admin`. Recorded in the audit log as `key / rotate` with resource `field-protection`.



# Stored CodeGen scripts for a pipeline
Source: https://docs.datris.ai/api-reference/pipelines/stored-codegen-scripts-for-a-pipeline

/openapi.yaml get /api/v1/pipelines/{name}/codegen-scripts
The Python scripts the pipeline's AI data-quality rule and AI transformation run. A script is generated when the pipeline is saved (from the source schema, no row values) or, when none can be used, by the first run that needs it, and every later run executes the stored script without a model call. A changed instruction or source schema regenerates it; a changed CodeGen model does not (`modelIsCurrent` is false then; use the regenerate endpoint to adopt the new model). With a code repository configured, new scripts are committed to it and runs execute the recorded commit; `drift` reports an edit made in the repository since then. Requires `pipeline:read`.



# Suggest field protection from field names and types
Source: https://docs.datris.ai/api-reference/pipelines/suggest-field-protection-from-field-names-and-types

/openapi.yaml post /api/v1/pipeline/protect/suggest
Asks the configured CodeGen model which source fields should carry `protect` (hmac, mask with an optional preserve, redact, drop, or none), with a one-line reason per field. Only field names and types are sent to the model; no row value leaves the server. Stateless: nothing is saved. Every suggestion passes the pipeline validator, so confirm the ones you want and save them as `protect` on the source fields. Pass either `pipeline` (its stored source schema is used) or `fields`. At most 150 fields per call (more returns 400 before the model is called); with `pipeline`, keyFields columns are only suggested hmac or none and non-string destination fields only drop or none. Needs the same capability as reading a pipeline.



# Unity Catalog sync state for a pipeline
Source: https://docs.datris.ai/api-reference/pipelines/unity-catalog-sync-state-for-a-pipeline

/openapi.yaml get /api/v1/pipelines/{name}/unity-catalog
Whether Unity Catalog sync is on for the pipeline (`enabledBy`: `pipeline` for its own block, `default` when DATRIS_UNITY_CATALOG_DEFAULT turned on a Databricks pipeline with no block, null when off with no block), its three-level table coordinates (Databricks destinations, or the registered table of an object-store Iceberg destination with `unityCatalog.catalog` set), the last sync (`state` is `never`, `synced` or `error`), the lineage publish status (`lineage` is `off`, `never`, `published` or `error`; lineage failures appear in `lastError` as lines prefixed `uc-lineage:`), and the Iceberg register status (`register` is `off`, `never`, `registered`, `stale`, `error`, `rest` or `refused`; register warnings appear in `lastError` as lines prefixed `uc-register:`).



# Browse Unity Catalog with a Databricks Platform secret
Source: https://docs.datris.ai/api-reference/provenance/browse-unity-catalog-with-a-databricks-platform-secret

/openapi.yaml get /api/v1/unity-catalog/browse
Read-only walk of what a Databricks Platform secret can see, one level per call: no `catalog` lists catalogs; `catalog` lists schemas; `catalog` + `schema` lists tables (tables tagged `datris_pipeline` carry `datrisPipeline`); adding `table` returns columns (name, type, comment) and tags. Catalog and schema listings are cached for 5 minutes per secret. Requires `metadata:read` plus read access to the named secret. Every call is audit-logged as `unity-catalog` / `browse`. The SQL warehouse is the `warehouse` param, else the secret's optional `warehouse` field (aliases `httpPath`, `http_path`, `DATABRICKS_WAREHOUSE`), else the warehouse of the first Databricks pipeline using the secret.



# Column-level lineage for one pipeline definition
Source: https://docs.datris.ai/api-reference/provenance/column-level-lineage-for-one-pipeline-definition

/openapi.yaml get /api/v1/lineage/columns/{pipeline}
Which destination columns come from which source fields. The deterministic tier (schema comparison: `passthrough` / `drop` edges with confidence `exact`, stamped provenance columns as `system`) is always returned. For pipelines with an AI transformation, `infer=true` runs the opt-in extraction — the codegen model reads the instruction, the field lists and the stored CodeGen script, and returns only mappings it can evidence (`rename` / `derive` / `drop`, confidence `inferred`). Inferred results are cached per definition version and never recomputed unless the definition changes.



# Find datasets by meaning (discovery, not execution)
Source: https://docs.datris.ai/api-reference/provenance/find-datasets-by-meaning-discovery-not-execution

/openapi.yaml get /api/v1/catalog/find
Ranks the pipelines the calling key may read against a natural-language query (name, description, tags, catalog, destination field names, source host). Each hit carries location, freshness, provenance handles, lineage and a pre-filled `howToQuery` hint naming an existing query/search tool — the caller still makes that call itself, under its own capabilities.



# One node's lineage neighborhood
Source: https://docs.datris.ai/api-reference/provenance/one-nodes-lineage-neighborhood

/openapi.yaml get /api/v1/lineage/{type}/{name}
The node, everything transitively upstream and/or downstream, the edges between them, freshness for the pipeline the node is (or feeds), and optionally the most recent recorded runs touching the node — what each read and wrote, per destination.



# Resolve a stamped run id to its origin
Source: https://docs.datris.ai/api-reference/provenance/resolve-a-stamped-run-id-to-its-origin

/openapi.yaml get /api/v1/provenance
Walks a `_datris_run_id` back through the chain run → job status → tap run → script commit → pipeline config version → declared source and returns one document. Pass the row's `_datris_tap_run` and `_datris_config_version` values when available for exact resolution.



# The whole lineage graph
Source: https://docs.datris.ai/api-reference/provenance/the-whole-lineage-graph

/openapi.yaml get /api/v1/lineage
The graph Source → Tap → Pipeline → Dataset → Catalog, derived from configuration plus recorded runs (cached ~1 minute). Node ids are `type:name`. Datasets a pipeline landed into under an earlier configuration appear with `historical: true` (and a dashed edge).



# Query API
Source: https://docs.datris.ai/api-reference/query-api

Query data endpoints

The Query API provides REST endpoints for querying every structured destination — PostgreSQL, MongoDB, object store, Snowflake, and Databricks — plus natural language queries and AI-powered answers. These endpoints mirror the MCP tools, making the same retrieval capabilities available via HTTP.

In the web UI, the Search tab's Traditional mode can also query Databricks and Snowflake pipelines: pick the pipeline and run read-only SQL against its destination table through the pipeline's own warehouse and credentials.

## Query PostgreSQL

Execute a read-only SQL SELECT query against PostgreSQL.

```
POST /api/v1/query/postgres
```

### Request

| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `sql` | string | Yes | | SQL SELECT query to execute |
| `database` | string | No | `datris` | PostgreSQL database name to connect to |
| `limit` | integer | No | 100 | Maximum rows to return. Pass `-1` for unlimited (tap scripts use this to read every row). Omit for the preview default of 100; any positive value is honored with no cap. |

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/postgres \
  -H "Content-Type: application/json" \
  -d '{
    "sql": "SELECT symbol, date, close FROM test.stock_price WHERE volume > 1000000",
    "database": "datris",
    "limit": 10
  }'
```

### Response

```json theme={null}
{
  "results": [
    {
      "symbol": "AAPL",
      "date": "2024-01-15",
      "close": 185.92
    }
  ],
  "count": 1
}
```

### Safety

The endpoint enforces multiple layers of protection:

* **SELECT-only** — only queries starting with `SELECT` are allowed
* **Read-only connection** — the JDBC connection is set to read-only mode
* **Statement blocklist** — queries containing `INSERT`, `UPDATE`, `DELETE`, `DROP`, `ALTER`, `CREATE`, `TRUNCATE`, `GRANT`, `REVOKE`, `COPY`, `CALL`, `EXECUTE`, or `EXEC` are rejected
* **No stacked queries** — semicolons are not allowed
* **No SQL comments** — `--` and `/* */` are rejected to prevent obfuscation
* **LIMIT enforcement** — if no `LIMIT` is present in the SQL, one is appended automatically using the request's `limit` value (or the default 100). Pass `limit: -1` to bypass the automatic append entirely — the caller's SQL runs as-is.

***

## Query MongoDB

Query a MongoDB collection with optional filter and projection.

```
POST /api/v1/query/mongodb
```

### Request

| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `collection` | string | Yes | | MongoDB collection name |
| `database` | string | No | configured MongoDB database | MongoDB database name to query |
| `filter` | object | No | `{}` | MongoDB query filter |
| `projection` | object | No | all fields | Fields to include/exclude |
| `limit` | integer | No | 20 | Maximum documents to return. Pass `-1` for unlimited (tap scripts use this to read every document). Omit for the preview default of 20; any positive value is honored with no cap. |

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/mongodb \
  -H "Content-Type: application/json" \
  -d '{
    "collection": "orders",
    "filter": {"status": "active"},
    "projection": {"name": 1, "_id": 0},
    "limit": 10
  }'
```

### Response

```json theme={null}
{
  "results": [
    {
      "name": "Order-001"
    }
  ],
  "count": 1
}
```

### Safety

* **Blocked operators** — filters containing `$where`, `$function`, or `$accumulator` are rejected, as these allow arbitrary JavaScript execution
* **Standard MongoDB query operators** (`$eq`, `$gt`, `$in`, `$regex`, etc.) are safe and fully supported

***

## Query Object Store

Read rows from a pipeline's object store destination (MinIO or S3). The pipeline name selects the bucket, prefix, and credentials — the caller never supplies connection details.

```
POST /api/v1/query/objectstore
```

### Request

| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `pipeline` | string | Yes | | Pipeline name. Must be a pipeline with an object store destination. |
| `limit` | integer | No | 100 | Maximum rows to return. Values of 0 or less fall back to 100; capped at 10,000. |

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/objectstore \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "sales_data",
    "limit": 10
  }'
```

### Response

```json theme={null}
{
  "pipeline": "sales_data",
  "path": "s3a://datris-bucket/sales_data",
  "format": "parquet",
  "columns": ["symbol", "date", "close"],
  "results": [
    {
      "symbol": "AAPL",
      "date": "2024-01-15",
      "close": 185.92
    }
  ],
  "count": 1
}
```

If the pipeline exists but no runs have written data yet, the endpoint returns an empty result (`"count": 0`) rather than an error.

### Error Cases

Returns `400 Bad Request` with `{"error": "..."}` when:

* The pipeline is not found
* The pipeline does not have an object store destination
* The pipeline's object store configuration is missing a prefix key
* The read times out (default 90 seconds, tunable via the `DATRIS_OBJECTSTORE_QUERY_TIMEOUT_SEC` environment variable)

***

## Query Snowflake

Execute a read-only query against the Snowflake account a pipeline loads into. The pipeline name selects the credentials, warehouse, database, and schema — credentials never leave the server.

```
POST /api/v1/query/snowflake
```

### Request

| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `pipeline` | string | Yes | | Pipeline name. Must be a pipeline with a Snowflake destination. |
| `sql` | string | No | `SELECT * FROM {destination table}` | SQL to execute. When omitted, previews the pipeline's destination table. |
| `limit` | integer | No | 100 | Maximum rows to return. Pass `-1` for unlimited (also disables the query timeout). |

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/snowflake \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "sales_data",
    "sql": "SELECT symbol, close FROM SALES_DB.PUBLIC.STOCK_PRICE WHERE volume > 1000000",
    "limit": 10
  }'
```

### Response

```json theme={null}
{
  "pipeline": "sales_data",
  "sql": "SELECT symbol, close FROM SALES_DB.PUBLIC.STOCK_PRICE WHERE volume > 1000000 LIMIT 10",
  "results": [
    {
      "SYMBOL": "AAPL",
      "CLOSE": 185.92
    }
  ],
  "count": 1
}
```

The `sql` field echoes the query that actually ran, including any automatically appended `LIMIT`.

### Safety

* **Read-only allowlist** — queries must start with `SELECT`, `WITH`, `SHOW`, `DESCRIBE`, or `DESC`. `SHOW`/`DESCRIBE` are allowed so agents can discover databases, schemas, tables, and columns.
* **Statement blocklist** — queries containing `INSERT`, `UPDATE`, `DELETE`, `DROP`, `ALTER`, `CREATE`, `TRUNCATE`, `GRANT`, `REVOKE`, `CALL`, `EXECUTE`, `EXEC`, `PUT`, `REMOVE`, `COPY`, `MERGE`, or `USE` are rejected (`USE` is blocked so a query cannot re-point the session at another database or role)
* **No stacked queries** — semicolons are not allowed
* **No SQL comments** — `--` and `/* */` are rejected to prevent obfuscation
* **LIMIT enforcement** — if a `SELECT`/`WITH` query has no `LIMIT`, one is appended using the request's `limit` value. `SHOW`/`DESCRIBE` output is row-capped at the same value.
* **Query timeout** — 30 seconds, bounding both hung requests and warehouse spend. Not applied when `limit: -1`.

### Error Cases

Returns `400 Bad Request` with `{"error": "..."}` when:

* The pipeline is not found
* The pipeline does not have a Snowflake destination
* The SQL fails validation (allowlist, blocklist, semicolons, comments)

***

## Query Databricks

Execute a read-only query against the Databricks workspace a pipeline loads into. The pipeline name selects the credentials, SQL warehouse, catalog, and schema — credentials never leave the server.

```
POST /api/v1/query/databricks
```

### Request

| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `pipeline` | string | Yes | | Pipeline name. Must be a pipeline with a Databricks destination. |
| `sql` | string | No | `SELECT * FROM {destination table}` | SQL to execute. When omitted, previews the pipeline's destination table. |
| `limit` | integer | No | 100 | Maximum rows to return. Pass `-1` for unlimited (also disables the query timeout). |

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/databricks \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "sales_data",
    "sql": "SELECT symbol, close FROM main.default.stock_price WHERE volume > 1000000",
    "limit": 10
  }'
```

### Response

```json theme={null}
{
  "pipeline": "sales_data",
  "sql": "SELECT symbol, close FROM main.default.stock_price WHERE volume > 1000000 LIMIT 10",
  "results": [
    {
      "symbol": "AAPL",
      "close": 185.92
    }
  ],
  "count": 1
}
```

The `sql` field echoes the query that actually ran, including any automatically appended `LIMIT`.

### Safety

Same protections as [Query Snowflake](#query-snowflake), with the blocklist extended for Databricks-specific data-movement and maintenance verbs: `OPTIMIZE`, `VACUUM`, `RESTORE`, `CLONE`, `REFRESH`, `MSCK`, and `DECLARE` are also rejected.

<Note>
  If the SQL warehouse is stopped, opening the connection blocks while it auto-starts — this happens before the 30-second query timeout applies, so the first query after idle time can take longer.
</Note>

### Error Cases

Returns `400 Bad Request` with `{"error": "..."}` when:

* The pipeline is not found
* The pipeline does not have a Databricks destination
* The SQL fails validation (allowlist, blocklist, semicolons, comments)

***

## Kill Job

Kill a running pipeline job by its pipeline token. The job thread is interrupted and the job is marked as cancelled.

```
POST /api/v1/job/kill
```

### Request

| Field | Type | Required | Description |
| - | - | - | - |
| `pipelineToken` | string | Yes | Pipeline token of the running job to kill |

### Example

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/job/kill \
  -H "Content-Type: application/json" \
  -d '{
    "pipelineToken": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  }'
```

### Response

```json theme={null}
{
  "status": "cancelled",
  "pipelineToken": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}
```

### Error Cases

* Job not found for the given pipeline token
* Job is not in `PROCESSING` state (already completed or cancelled)
* Job thread is no longer alive


# Natural-language query (AI)
Source: https://docs.datris.ai/api-reference/query/natural-language-query-ai

/openapi.yaml post /api/v1/query/natural
Ask a question in plain English against a PostgreSQL table. The server reads the table's columns, uses AI to generate a SQL SELECT, executes it, and returns both the SQL and the results.



# Query Databricks
Source: https://docs.datris.ai/api-reference/query/query-databricks

/openapi.yaml post /api/v1/query/databricks
Execute a read-only SQL query against the Databricks destination of a pipeline (a Unity Catalog managed Delta table). Connection credentials come from the pipeline's configured `credentialsSecret`.



# Query MongoDB
Source: https://docs.datris.ai/api-reference/query/query-mongodb

/openapi.yaml post /api/v1/query/mongodb
Query a MongoDB collection with optional filter and projection. Blocks $where and $function operators.



# Query object store
Source: https://docs.datris.ai/api-reference/query/query-object-store

/openapi.yaml post /api/v1/query/objectstore
Read rows back from the files an object-store-destination pipeline has written (Parquet, ORC, CSV, or JSON), without downloading them.



# Query PostgreSQL
Source: https://docs.datris.ai/api-reference/query/query-postgresql

/openapi.yaml post /api/v1/query/postgres
Execute a read-only SQL SELECT query against PostgreSQL. Enforces SELECT-only, blocks dangerous keywords, and auto-appends LIMIT.



# Query Snowflake
Source: https://docs.datris.ai/api-reference/query/query-snowflake

/openapi.yaml post /api/v1/query/snowflake
Execute a read-only SQL query against the Snowflake destination of a pipeline. Connection credentials come from the pipeline's configured `credentialsSecret`.



# Abandon an open incident (admin / editor)
Source: https://docs.datris.ai/api-reference/recovery-agent/abandon-an-open-incident-admin-editor

/openapi.yaml post /api/v1/incidents/{id}/abandon
A person closing an incident by hand. Never permitted from an agent-initiated request.



# List recovery-agent incidents
Source: https://docs.datris.ai/api-reference/recovery-agent/list-recovery-agent-incidents

/openapi.yaml get /api/v1/incidents
Incidents the platform opened for failed, stale or anomalous data flows, newest first. Read-only — only the platform opens incidents. `enabled` is false while RECOVERY_AGENT_ENABLED is off.



# Read one incident
Source: https://docs.datris.ai/api-reference/recovery-agent/read-one-incident

/openapi.yaml get /api/v1/incidents/{id}



# Server-side Activity signals
Source: https://docs.datris.ai/api-reference/recovery-agent/server-side-activity-signals

/openapi.yaml get /api/v1/activity/signals
One definition of operational signals — failures (with the recovered flag), stale scheduled taps, and pipeline volume anomalies — shared by the dashboard, the Ops chat context and the recovery agent.



# AI Schema Generation API
Source: https://docs.datris.ai/api-reference/schema-generation-api

AI schema generation endpoints

The pipeline can automatically generate a ready-to-use pipeline configuration from an uploaded file using an AI model. The generated config can be pasted directly into `POST /api/v1/pipeline` to register the pipeline without writing any JSON by hand.

## How It Works

```
Uploaded file
  |
  v
File type detection (CSV / JSON / XML)
  |
  +-- JSON / XML --> Fixed schema (_json or _xml field)
  |                  No AI call needed
  |
  +-- CSV / other --> First 100 lines sent to AI model
                      AI infers column names and data types
                      Returns JSON array of field definitions
  |
  v
Config builder assembles full PipelineConfig JSON
  - source.schemaProperties.fields  (AI-inferred or fixed)
  - source.fileAttributes            (csvAttributes / jsonAttributes / xmlAttributes)
  - destination                      (Postgres for CSV, MongoDB for JSON/XML)
  |
  v
Response: complete PipelineConfig JSON ready to register
```

The response is **not** registered automatically — it is returned to the caller so you can review it, fill in the placeholder values, and then POST it to `/api/v1/pipeline`.

***

## Endpoint

```
POST /api/v1/pipeline/generate
Content-Type: multipart/form-data
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `file` | form-data (file) | Yes | The file to analyze |
| `pipeline` | query | No | Pipeline name. If omitted, derived from the filename (lowercased, non-alphanumeric characters replaced with `_`) |
| `delimiter` | query | No | Column delimiter for delimited files. Defaults to `,` |
| `header` | query | No | Whether the file has a header row (default `false`; pass `true` for a file with a header row) |
| `allStrings` | query | No | If `true`, all fields are typed as `string` (default: `false`) |
| `x-api-key` | header | No | API key (required if `useApiKeys: true`) |

***

## Schema Rules by File Type

| File type | Schema | Default destination |
| - | - | - |
| CSV / delimited | AI infers field names and types from file content | PostgreSQL (`usePostgres: true`) |
| JSON (`.json`) | Single field: `_json` (type `string`) | MongoDB (`useMongoDB: true`) |
| XML (`.xml`) | Single field: `_xml` (type `string`) | MongoDB (`useMongoDB: true`) |

JSON and XML files use a fixed schema because the pipeline stores them as raw documents — no column inference is needed.

**Valid AI-inferred types:** `boolean`, `int`, `bigint`, `float`, `double`, `string`, `date`, `timestamp`

### When the model declines

An AI provider's safety classifier can decline to look at a file, most often one that looks like personal data. For a CSV or other delimited file the endpoint then still returns `200` with every field typed `string`. With `header=true` and a first line that reads as column names, the fields keep the header's names exactly as `allStrings=true` writes them, except that an empty header cell (empty, plain spaces or a byte-order mark only; an empty or newline-only file included), a cell that only renders blank (such as a Hangul filler), or a name repeated ignoring case and surrounding spaces, is named `column_N` for its position, so no field name is blank or duplicated. A cell holding only other characters that are not letters (a non-breaking or zero-width space, for example) makes line 1 read as data: every column is then numbered and `csvAttributes.header` is `false`. When `header` is false (the default when the parameter is omitted) or line 1 looks like data, the columns are named `column_1`, `column_2`, … and the config sets `csvAttributes.header` to `false`. The response carries a top-level `"aiDeclined": true`, and the pipeline wizard shows a one-line note; set the types by hand. `aiDeclined` is a response-only flag: it is never saved with a pipeline. The provider's own text is not included.

| Provider family | Recognised as a decline |
| - | - |
| Anthropic (direct or AWS Bedrock) | `stop_reason: "refusal"` |
| OpenAI, Azure OpenAI, Grok and Ollama (chat completions, streamed or not) | `finish_reason: "content_filter"`, or a non-empty `message.refusal` (`delta.refusal` when streamed) |
| OpenAI Responses API models | an output item or message content part of type `refusal`, or `incomplete_details.reason: "content_filter"` |

Only response-side declines are recognised. Azure OpenAI's prompt-side content filter answers with HTTP 400 (`code: content_filter`) and is still returned as an error. Nothing else counts as a decline: a missing or wrong provider key, a timeout, a rate limit or any other provider error is still returned as an error, so a misconfiguration is not hidden.

Other AI features (search answers, CodeGen, tap generation, fix suggestions and the rest) have no local fallback: a decline there, including a `content_filter` finish that used to return the cut-off text, is reported as an error.

JSON Schema and XSD generation (`POST /api/v1/config/generate-schema`) have no local fallback: a decline there is still returned as an error, with the message "The model declined this request (…)".

***

## Example: CSV File

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/generate \
  -H "x-api-key: your-api-key" \
  -F "file=@./stock_price.csv" \
  -F "pipeline=stock_price" \
  -F "header=true"
```

Response:

```json theme={null}
{
  "name": "stock_price",
  "source": {
    "schemaProperties": {
      "fields": [
        { "name": "symbol",    "type": "string" },
        { "name": "date",      "type": "string" },
        { "name": "open",      "type": "double" },
        { "name": "high",      "type": "double" },
        { "name": "low",       "type": "double" },
        { "name": "close",     "type": "double" },
        { "name": "volume",    "type": "int"    },
        { "name": "adj_close", "type": "double" }
      ]
    },
    "fileAttributes": {
      "csvAttributes": { "delimiter": ",", "header": true, "encoding": "UTF-8" }
    }
  },
  "destination": {
    "database": {
      "dbName": "DATABASE_NAME",
      "schema": "SCHEMA_NAME",
      "table": "TABLE_NAME",
      "usePostgres": true
    }
  }
}
```

Replace `DATABASE_NAME`, `SCHEMA_NAME`, and `TABLE_NAME` with real values, then register the pipeline:

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline \
  -H "Content-Type: application/json" \
  -d '<paste response here>'
```

***

## Example: JSON File

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/generate \
  -H "x-api-key: your-api-key" \
  -F "file=@./events.json"
```

Response:

```json theme={null}
{
  "name": "events",
  "source": {
    "schemaProperties": {
      "fields": [
        { "name": "_json", "type": "string" }
      ]
    },
    "fileAttributes": {
      "jsonAttributes": { "everyRowContainsObject": false, "encoding": "UTF-8" }
    }
  },
  "destination": {
    "database": {
      "dbName": "DATABASE_NAME",
      "table": "TABLE_NAME",
      "useMongoDB": true
    }
  }
}
```

***

## Configuration

AI schema generation is disabled by default. To enable it:

Schema generation uses the **codegen** AI slot. As of v1.5.6, AI configuration is split into three independent self-describing Vault secrets — see [AI Configuration](/ai-configuration) for the full picture.

**`application.yaml`** (or `docker/config/application.yaml` for Docker deployments):

```yaml theme={null}
ai:
  enabled: "true"
  aiPrimary:
    secretName: "oss/ai-primary"
  codegen:
    secretName: "oss/codegen"
  embedding:
    secretName: "oss/embedding"
```

**Vault secret** (each secret is self-describing — provider, endpoint, model, apiKey, and optionally version, all inline):

```bash theme={null}
vault kv put secret/oss/codegen \
  provider="anthropic" \
  endpoint="https://api.anthropic.com/v1/messages" \
  model="claude-opus-4-8" \
  apiKey="sk-ant-..." \
  version="2023-06-01"
```

For OpenAI:

```bash theme={null}
vault kv put secret/oss/codegen \
  provider="openai" \
  endpoint="https://api.openai.com/v1/chat/completions" \
  model="gpt-5.4" \
  apiKey="sk-..."
```

The Vault secret keys:

| Key | Description |
| - | - |
| `provider` | `anthropic`, `openai`, `azure`, `bedrock`, `grok`, or `ollama` |
| `endpoint` | The AI provider API URL |
| `model` | The model name to use |
| `apiKey` | The API key for authentication (omit for local Ollama) |
| `version` | Optional API version header (e.g. Anthropic's `anthropic-version`) |

The pipeline reads the secret at startup. If `ai.enabled: true` and the codegen secret is missing or malformed, startup will fail with a descriptive error. `docker/vault-init.sh` seeds this automatically from `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` in `.env`.

***

## Supported Providers

| Provider | `provider` value | Auth header |
| - | - | - |
| Anthropic Claude | `anthropic` | `x-api-key` + `anthropic-version: 2023-06-01` |
| OpenAI | `openai` | `Authorization: Bearer` |
| Azure OpenAI | `azure` | API key or Entra ID (keyless) |
| Amazon Bedrock | `bedrock` | AWS credentials |
| Grok (xAI) | `grok` | `Authorization: Bearer` |
| Ollama (local) | `ollama` | none |

Any other value for `provider` will cause startup to fail with an unsupported provider error.


# Search API
Source: https://docs.datris.ai/api-reference/search-api

Search endpoints

The Search API provides a single REST endpoint for semantic search across vector databases. The endpoint converts a natural language query into an embedding vector, searches the vector store named in the URL, and returns the most similar results with relevance scores.

Embedding and vector database connection details are stored in HashiCorp Vault and resolved **server-side**. Clients never send secret names — the server always uses its own configured secret names (in multi-tenant mode this enforces tenant isolation), so any secret-name fields in a request body are ignored.

## Search a Vector Store

```
POST /api/v1/search/{store}
```

One endpoint serves all vector stores. The `{store}` path parameter selects the store:

| Store | Container field | Container default | Store-specific fields |
| - | - | - | - |
| `qdrant` | `collection` | `documents` | |
| `weaviate` | `className` | `Documents` | |
| `milvus` | `collection` | `documents` | |
| `chroma` | `collection` | `documents` | |
| `pgvector` | `table` | `documents` | `schema` (default `public`) |

### Common Parameters

All stores share these request-body parameters:

| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `query` | string | Yes | | Natural language search query |
| `topK` | integer | No | 5 | Number of results to return |

Plus the store's container field from the table above (e.g. `collection` for Qdrant, `className` for Weaviate, `table`/`schema` for pgvector).

### Common Response Format

```json theme={null}
{
  "results": [
    {
      "text": "document chunk content",
      "chunk_index": 0,
      "source_pipeline": "pipeline_name",
      "filename": "document.pdf",
      "_score": 0.89
    }
  ],
  "count": 1
}
```

The `_score` field indicates relevance (higher is more similar, normalized to 0–1 where applicable).

### Error Responses

An unknown `{store}` value returns `400 Bad Request` with a message naming the valid stores:

```
Unknown vector store: 'faiss'. Valid stores: chroma, milvus, pgvector, qdrant, weaviate
```

Other user-actionable errors also return `400 Bad Request` with a plain-text message — most notably a vector dimension mismatch, which usually means the embedding provider was changed between ingest and query. Fix it by switching the embedding provider in Configuration to match the collection's dimension, or by re-ingesting under the current provider.

Only stores that are actually running and reachable can be searched successfully. Use [`GET /api/v1/vector-stores/available`](/api-reference/health-api#list-available-vector-stores) to discover which stores are live before searching — the UI's Search tab uses this to gate its store dropdown.

Unexpected server errors return `500 Internal Server Error`.

***

### Qdrant

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/search/qdrant \
  -H "Content-Type: application/json" \
  -d '{
    "query": "quarterly revenue projections",
    "collection": "documents",
    "topK": 5
  }'
```

### Weaviate

The container field is `className` (PascalCase, default `Documents`).

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/search/weaviate \
  -H "Content-Type: application/json" \
  -d '{
    "query": "quarterly revenue projections",
    "className": "Documents",
    "topK": 5
  }'
```

### Milvus

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/search/milvus \
  -H "Content-Type: application/json" \
  -d '{
    "query": "quarterly revenue projections",
    "collection": "documents",
    "topK": 5
  }'
```

### Chroma

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/search/chroma \
  -H "Content-Type: application/json" \
  -d '{
    "query": "quarterly revenue projections",
    "collection": "documents",
    "topK": 5
  }'
```

### pgvector

The container field is `table`, with an optional `schema` (default `public`).

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/search/pgvector \
  -H "Content-Type: application/json" \
  -d '{
    "query": "quarterly revenue projections",
    "table": "documents",
    "schema": "public",
    "topK": 5
  }'
```

***

## Natural Language Query

Ask a question in plain English — the AI generates and executes a SQL query against PostgreSQL.

```
POST /api/v1/query/natural
```

| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `question` | string | Yes | | Natural language question |
| `table` | string | Yes | | Target table name |
| `schema` | string | No | `public` | PostgreSQL schema |
| `database` | string | No | `datris` | PostgreSQL database |
| `limit` | integer | No | 100 | Maximum rows to return |

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/natural \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What are the top 5 stocks by volume?",
    "table": "stock_prices"
  }'
```

Returns the AI-generated SQL query and its results.

***

## AI Answer (RAG)

Answer a question using AI based on provided context — typically used after a vector search to generate a natural language answer from retrieved chunks.

```
POST /api/v1/ai/answer
```

| Field | Type | Required | Description |
| - | - | - | - |
| `query` | string | Yes | The question to answer |
| `context` | string | Yes | Context text (e.g., concatenated search results) |

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/ai/answer \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What was Apple'\''s revenue last quarter?",
    "context": "Apple reported revenue of $94.9 billion for Q1 2026..."
  }'
```

Returns the AI-generated answer based on the provided context.

***

## Vault Secret Structure

These secrets are configured on the server (seeded by `vault-init.sh` and managed via the Configuration tab) — they are never passed in search requests.

### Embedding Secret

Self-describing — the resolver reads `provider`, `endpoint`, `model`, `apiKey`, and (optionally) `version` from inside the secret. The server uses the secret named by `ai.embedding.secretName` (`oss/embedding`). See [AI Configuration](/ai-configuration).

```json theme={null}
{
  "provider": "openai",
  "endpoint": "https://api.openai.com/v1/embeddings",
  "model": "text-embedding-3-small",
  "apiKey": "sk-..."
}
```

For Anthropic-only deployments, the bundled TEI sidecar serves `bge-m3` and `vault-init.sh` seeds the embedding secret to point at it (`provider: "tei"`, `endpoint: "http://tei:80/v1/embeddings"`, `model: "BAAI/bge-m3"`).

### Vector Database Secrets

Each store has a server-configured secret resolved by store key:

**Qdrant:**

```json theme={null}
{ "host": "localhost", "port": "6334", "apiKey": "" }
```

**Weaviate:**

```json theme={null}
{ "host": "localhost", "port": "8079", "scheme": "http", "apiKey": "" }
```

**Milvus:**

```json theme={null}
{ "host": "localhost", "port": "19530", "apiKey": "" }
```

**Chroma:**

```json theme={null}
{ "host": "localhost", "port": "8000" }
```

**pgvector:**

```json theme={null}
{ "jdbcUrl": "jdbc:postgresql://localhost:5432/datris", "username": "postgres", "password": "postgres" }
```


# Semantic search (all vector stores)
Source: https://docs.datris.ai/api-reference/search/semantic-search-all-vector-stores

/openapi.yaml post /api/v1/search/{store}
Search a vector store using natural language. One endpoint serves every supported store — the `{store}` path segment selects it. Generates an embedding from the query and returns the most similar document chunks. Connection and embedding secrets are always resolved server-side from the server's configuration; any client-supplied secret names are ignored. Request fields vary slightly per store — qdrant, milvus, and chroma name their container with `collection`, weaviate with `className`, pgvector with `table` (plus an optional `schema`).



# Secrets API
Source: https://docs.datris.ai/api-reference/secrets-api

Manage secrets in HashiCorp Vault

The Secrets API lets you list, read, update, and delete secrets stored in HashiCorp Vault. Secrets are scoped to the current environment (e.g., `oss`).

## List Secrets

```
GET /api/v1/secrets
```

Returns a JSON array of secret names for the current environment.

| Parameter | Type | Required | Description |
| - | - | - | - |
| `type` | query | No | Filter by the secret's `_type` tag (e.g. `tap`). `platform` returns every secret *not* tagged `_type=tap`. |

### Example

```bash theme={null}
curl -s http://localhost:8080/api/v1/secrets | jq .
```

### Response

```json theme={null}
["activemq", "anthropic", "chroma", "embedding", "kafka-producer", "minio", "mongodb", "ollama", "openai", "pgvector", "postgres", "qdrant", "weaviate"]
```

***

## Get Secret

```
GET /api/v1/secrets/{name}
```

| Parameter | Type | Description |
| - | - | - |
| `name` | path | Secret name |

Sensitive fields are always masked as `••••••••`: any field whose name (case-insensitive, ignoring `_`/`-`) contains `password`, `passwd`, `pwd`, `secret`, `token`, `key`, `credential`, `signature`, `bearer`, or `private` — e.g. `apiKey`, `secretKey`, `accessKey`. Non-sensitive fields show their actual values.

### Example

```bash theme={null}
curl -s http://localhost:8080/api/v1/secrets/anthropic | jq .
```

### Response

```json theme={null}
{
  "name": "anthropic",
  "fields": {
    "endpoint": "https://api.anthropic.com/v1/messages",
    "model": "claude-opus-4-8",
    "apiKey": "••••••••"
  }
}
```

### Error

Returns `404` if the secret does not exist.

***

## Update Secret

Create or update a secret.

```
PUT /api/v1/secrets/{name}
Content-Type: application/json
```

| Parameter | Type | Description |
| - | - | - |
| `name` | path | Secret name |
| body | JSON | Key-value pairs to store |

Updating an existing secret replaces its fields with the body you send, with one safeguard: sending the `••••••••` mask (or any empty string) for a sensitive field keeps its current value, so you can add or correct one field without re-entering credentials. To remove a field, omit its key from the body. The shared `ai-keys` store is the exception: fields you omit are kept, and an empty string clears a field.

### Example

```bash theme={null}
curl -X PUT http://localhost:8080/api/v1/secrets/anthropic \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "https://api.anthropic.com/v1/messages",
    "model": "claude-opus-4-8",
    "apiKey": "sk-ant-..."
  }'
```

### Response

```json theme={null}
{"status": "ok"}
```

***

## Delete Secret

```
DELETE /api/v1/secrets/{name}
```

| Parameter | Type | Description |
| - | - | - |
| `name` | path | Secret name to delete |

### Example

```bash theme={null}
curl -X DELETE http://localhost:8080/api/v1/secrets/my-secret
```

### Response

```json theme={null}
{"status": "ok"}
```


# Create or update a secret
Source: https://docs.datris.ai/api-reference/secrets/create-or-update-a-secret

/openapi.yaml put /api/v1/secrets/{name}
Creates or updates a secret with the provided key-value fields.



# Delete a secret
Source: https://docs.datris.ai/api-reference/secrets/delete-a-secret

/openapi.yaml delete /api/v1/secrets/{name}



# Get secret details
Source: https://docs.datris.ai/api-reference/secrets/get-secret-details

/openapi.yaml get /api/v1/secrets/{name}
Returns the key-value fields of a secret. Sensitive fields (names containing password, secret, token, key, credential, signature, bearer, or private) are always masked.



# List all secrets
Source: https://docs.datris.ai/api-reference/secrets/list-all-secrets

/openapi.yaml get /api/v1/secrets
Lists all secret names under the current environment path in Vault.



# Pipeline Status API
Source: https://docs.datris.ai/api-reference/status-api

Pipeline status endpoints

Query job processing status by publisher token, pipeline token, or pipeline name, or clear all ingestion history.

## Get Status by Publisher Token

```
GET /api/v1/pipeline/status?publishertoken={token}
```

Use this to watch every ingestion job a single caller submitted. Tap runs set a `publisherToken` on every job they spawn, so one query covers:

* Structured taps → the single job that ingested the fetched records.
* Document taps → every per-document job (one per file in a run).

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `publishertoken` | query | Yes | Publisher token returned from `/tap/run` (`publisherToken` field) or `upload_data` |
| `withrollup` | query | No | When `true`, wraps the response in a rollup object that classifies each job — see [Rollup Response](#rollup-response) below. Recommended for agents that need a single boolean to poll on. |

**Example:**

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/status?publishertoken=pub-abc12345-..."
```

**Response:** `200 OK` — status entries from every job whose `publisherToken` matches, sorted oldest-first. Each job has its own `pipelineToken`; group rows by `pipelineToken` to see one job at a time.

If both `publishertoken` and `pipelinetoken` are supplied, `publishertoken` wins.

***

## Get Status by Pipeline Token

```
GET /api/v1/pipeline/status?pipelinetoken={token}
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `pipelinetoken` | query | Yes\* | Pipeline token from upload response |
| `withrollup` | query | No | When `true`, wraps the response in a rollup object — see [Rollup Response](#rollup-response). |

**Example:**

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/status?pipelinetoken=pt-abc12345-..."
```

**Response:** `200 OK` - an array of status entries, one per processing stage:

```json theme={null}
[
  {
    "id": 1,
    "dateTime": "2026-03-15T10:00:00Z",
    "pipeline": "sales_data",
    "processName": "StreamNotifier",
    "publisherToken": null,
    "pipelineToken": "pt-abc12345-...",
    "filename": "sales_data",
    "state": "begin",
    "code": "begin",
    "description": "Process started",
    "epoch": 1710500400000
  },
  {
    "id": 2,
    "dateTime": "2026-03-15T10:00:03Z",
    "pipeline": "sales_data",
    "processName": "PostgresLoader",
    "publisherToken": null,
    "pipelineToken": "pt-abc12345-...",
    "filename": "sales_data",
    "state": "end",
    "code": "end",
    "description": "Process completed",
    "epoch": 1710500403000
  }
]
```

***

## Get Status by Pipeline Name

```
GET /api/v1/pipeline/status?pipelinename={name}&page={page}
```

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `pipelinename` | query | Yes\* | Pipeline name |
| `page` | query | No | Page number (default: 1) |

**Example:**

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/status?pipelinename=sales_data&page=1"
```

**Response:** `200 OK` - an array of job summaries:

```json theme={null}
[
  {
    "createdAtTimestamp": "2026-03-15T10:00:00Z",
    "createdAt": 1710500400000,
    "updatedAt": 1710500403000,
    "pipeline": "sales_data",
    "pipelineToken": "pt-abc12345-...",
    "process": "PostgresLoader",
    "startTime": "2026-03-15T10:00:00Z",
    "endTime": "2026-03-15T10:00:03Z",
    "totalTime": "3s",
    "status": "end"
  }
]
```

***

## Rollup Response

Add `&withrollup=true` to a `publishertoken` or `pipelinetoken` query and the response is wrapped:

```json theme={null}
{
  "rollup": {
    "allDone": false,
    "status": "processing",
    "jobs": [
      {
        "pipelineToken": "pt-abc12345-...",
        "pipeline": "sales_data",
        "filename": "sales_data.csv",
        "status": "success",
        "startedAt": "2026-04-29 14:32:11.123",
        "lastEventAt": "2026-04-29 14:32:47.901",
        "elapsed": "36.78 sec",
        "lastError": null
      },
      {
        "pipelineToken": "pt-def67890-...",
        "pipeline": "sales_data",
        "filename": "orders.csv",
        "status": "error",
        "startedAt": "2026-04-29 14:32:11.140",
        "lastEventAt": "2026-04-29 14:33:02.512",
        "elapsed": "51.37 sec",
        "lastError": {
          "processName": "PostgresLoader",
          "description": "duplicate key value violates unique constraint \"customers_pkey\""
        }
      }
    ]
  },
  "events": [ /* the same raw status entries as without withrollup */ ]
}
```

`rollup.allDone` flips to `true` when every job has reached a terminal state. `rollup.status` is the aggregate outcome — `success`, `warning`, `error`, or `processing`. Per-job `status` values are `success`, `warning`, `error`, `processing`, or `timed_out` (no terminal event within 8 hours of the first event). When a job's `status` is `error`, `lastError` carries the failing process and its message.

**Live Read result fields.** A job whose pipeline is a [Live Read](/destinations/live-read) pipeline (the `scratch` destination) carries its result on the rollup once the run finishes. These fields are absent on every other job:

| Field | Type | Description |
| - | - | - |
| `resultUri` | string | Where the result is held. Present only when the run produced a result. |
| `resultRowCount` | integer | Total rows in the result. |
| `resultExpiresAt` | string | When the result is deleted; reads after this return `410`. |
| `resultPreview` | array | The first rows, inline — up to the configured inline cap (`scratchInlineRows`). |
| `resultTruncated` | boolean | `true` when the result has more rows than `resultPreview` holds — page the rest with [Get a Live Read Pipeline's Result](#get-a-live-read-pipelines-result). |

Polling pattern (agents and clients):

1. Call with `withrollup=true`.
2. Re-call every few seconds until `rollup.allDone` is `true`.
3. Read `rollup.status` for the outcome and `rollup.jobs[].lastError` for any failures.

Without `withrollup=true`, all three queries return their original array shapes (no wrapper) — backward compatible.

***

\*Use one of `publishertoken`, `pipelinetoken`, or `pipelinename`. When multiple are supplied, `publishertoken` wins over `pipelinetoken` wins over `pipelinename`.

***

## Get a Live Read Pipeline's Result

```
GET /api/v1/pipeline/result?pipelinetoken={token}&offset={offset}&limit={limit}
```

Page through the rows a [Live Read](/destinations/live-read) pipeline run produced. Paging reads the stored result — it never re-runs the source. Only Live Read pipelines have a result.

**Parameters:**

| Parameter | Type | Required | Description |
| - | - | - | - |
| `pipelinetoken` | query | Yes\* | Pipeline token of one ingestion job |
| `publishertoken` | query | Yes\* | Publisher token from `/tap/run` — resolves to the Live Read job that run submitted |
| `offset` | query | No | Row offset to start from (default `0`) |
| `limit` | query | No | Rows requested for this page. Clamped server-side to the inline cap (`scratchInlineRows`), so advance by the page's `returnedCount`, never by `limit`. |

\*Exactly one of `pipelinetoken` or `publishertoken`.

**Example:**

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/result?pipelinetoken=pt-abc12345-...&offset=200"
```

**Response:** `200 OK`

```json theme={null}
{
  "records": [
    { "id": "201", "city": "c201" },
    { "id": "202", "city": "c202" }
  ],
  "rowCount": 300,
  "returnedCount": 100,
  "offset": 200,
  "truncated": false,
  "resultUri": "...",
  "resultExpiresAt": "2026-09-19T10:00:05Z"
}
```

| Field | Description |
| - | - |
| `records` | This page's rows, one object per row |
| `rowCount` | Total rows in the whole result |
| `returnedCount` | Rows on this page |
| `offset` | The offset this page starts at |
| `truncated` | `true` when more rows follow — the next page starts at `offset + returnedCount` |
| `resultUri` | Where the result is held |
| `resultExpiresAt` | When the result is deleted |

**Errors:**

| Status | Meaning |
| - | - |
| `400` | `offset` or `limit` is not an integer |
| `404` | No run matched the token, or the pipeline is not a Live Read pipeline — only Live Read pipelines have a result |
| `410` | The result has expired; run the pipeline again |

***

## Clear All Status

```
DELETE /api/v1/pipeline/status
```

Deletes **all** ingestion history — every status entry (the detail events returned by the token queries) and every job summary (the entries returned by the pipeline-name query). There are no parameters and no filtering: this is an all-or-nothing reset, typically used to clear out accumulated history on a dev or demo instance.

**Example:**

```bash theme={null}
curl -X DELETE "http://localhost:8080/api/v1/pipeline/status"
```

**Response:** `200 OK` — the total number of entries removed (detail entries plus summary entries combined):

```json theme={null}
{
  "deleted": 1234
}
```

<Warning>
  This permanently removes the processing history for every pipeline, not just one. Pipeline configurations and loaded data are untouched — only the status/history records are deleted.
</Warning>

***

## Status Fields

Each status entry (`PipelineStatus`) contains:

| Field | Description |
| - | - |
| `id` | Entry index (internal) |
| `dateTime` | Human-readable timestamp |
| `pipeline` | Pipeline name |
| `processName` | Processing stage name (see below) |
| `publisherToken` | Publisher identifier (if provided on upload) |
| `pipelineToken` | Pipeline job token |
| `filename` | Source filename |
| `state` | Stage state: `begin`, `processing`, `end`, or `error` |
| `code` | Same as `state` |
| `description` | Detail message. On a failed job's `Process completed, error:` entry, that prefix followed by the error and its causes |
| `detail` | Optional. On a failed job's `Process completed, error:` entry, the full stack trace; absent otherwise and on entries written by earlier versions |
| `epoch` | Unix epoch milliseconds |

## State Values

| State | Description |
| - | - |
| `begin` | Processing stage started |
| `processing` | In progress with detail message |
| `end` | Processing stage completed |
| `error` | Processing stage failed |

## Process Names

| Process | Description |
| - | - |
| `FileNotifier` | File intake from MinIO bucket |
| `StreamNotifier` | Direct upload intake (also tap-fed and Kafka-fed jobs) |
| `FileUploadAPIController` | Upload request handling before the job is created |
| `DataQuality` | Data quality validation |
| `Transformation` | Data transformation |
| `JobRunner` | Destination orchestration. When a destination loader fails, the job's `lastError` is attributed to that loader (below), not to `JobRunner`; the `JobRunner` end event carries the full stack trace |
| `PostgresLoader` | PostgreSQL loading |
| `SnowflakeLoader` | Snowflake loading |
| `DatabricksLoader` | Databricks loading |
| `MongoDBLoader` | MongoDB loading |
| `SparkObjectStoreLoader` | Object store writing |
| `KafkaLoader` | Kafka producing |
| `ActiveMQLoader` | ActiveMQ queue writing |
| `RestEndpointRunner` | REST endpoint posting (preprocessor and REST destination) |
| `PGVectorLoader` | pgvector loading |
| `QdrantLoader` | Qdrant loading |
| `WeaviateLoader` | Weaviate loading |
| `MilvusLoader` | Milvus loading |
| `ChromaLoader` | Chroma loading |


# Clear all ingestion history
Source: https://docs.datris.ai/api-reference/status/clear-all-ingestion-history

/openapi.yaml delete /api/v1/pipeline/status
Deletes every job status entry — detail rows and per-pipeline summaries — for the environment.



# Get a Live Read (scratch) pipeline's result
Source: https://docs.datris.ai/api-reference/status/get-a-live-read-scratch-pipelines-result

/openapi.yaml get /api/v1/pipeline/result
Page through the rows a Live Read (scratch destination) pipeline run produced. Paging reads the stored result and never re-runs the source. Only Live Read pipelines have a result. `limit` is clamped server-side to the inline cap, so the next page starts at `offset + returnedCount`, never `offset + limit`. Results expire after the retention window.



# Get job status
Source: https://docs.datris.ai/api-reference/status/get-job-status

/openapi.yaml get /api/v1/pipeline/status
Query job status by pipeline token or pipeline name. Returns detailed status when queried by token, or paginated summary when queried by pipeline name.



# Get server version
Source: https://docs.datris.ai/api-reference/status/get-server-version

/openapi.yaml get /api/v1/version



# Kill a running job
Source: https://docs.datris.ai/api-reference/status/kill-a-running-job

/openapi.yaml post /api/v1/job/kill
Interrupt and cancel a running pipeline job by its pipeline token.



# Create or update a tap prompt fragment
Source: https://docs.datris.ai/api-reference/tap-prompts/create-or-update-a-tap-prompt-fragment

/openapi.yaml post /api/v1/tap-prompts
Creates a new fragment or updates the existing fragment with the same `key`. Timestamps are managed server-side.



# Delete a tap prompt fragment
Source: https://docs.datris.ai/api-reference/tap-prompts/delete-a-tap-prompt-fragment

/openapi.yaml delete /api/v1/tap-prompts/{key}



# Get a tap prompt fragment
Source: https://docs.datris.ai/api-reference/tap-prompts/get-a-tap-prompt-fragment

/openapi.yaml get /api/v1/tap-prompts/{key}



# List tap prompt fragments
Source: https://docs.datris.ai/api-reference/tap-prompts/list-tap-prompt-fragments

/openapi.yaml get /api/v1/tap-prompts
Returns every prompt fragment. Fragments are per-source snippets injected into the tap code-generation system prompt when a tap's description mentions the fragment's key or an alias.



# Suggest fragment content (AI)
Source: https://docs.datris.ai/api-reference/tap-prompts/suggest-fragment-content-ai

/openapi.yaml post /api/v1/tap-prompts/suggest
Uses AI to draft (or refine) the content of a prompt fragment from its key, aliases, and any existing content.



# Taps API
Source: https://docs.datris.ai/api-reference/taps-api

API endpoints for managing taps

<Note>
  **Beta.** The Taps API is stable but may add fields or refine response shapes in upcoming releases. Existing endpoints will remain backward-compatible where possible.
</Note>

## List Taps

```
GET /api/v1/taps
```

Returns all registered taps.

**Response:** JSON array of tap configurations.

## Get Tap

```
GET /api/v1/tap?name={tapName}
```

Returns a single tap configuration plus the Python script body from MinIO.

**Parameters:**

| Parameter | Type | Description |
| - | - | - |
| `name` | query | Tap name (required) |

**Response (success):**

```json theme={null}
{
  "name": "stock-prices-tap",
  "description": "Fetch daily stock prices from yfinance",
  "scriptPath": "tap-scripts/stock-prices-tap_a1b2c3d4.py",
  "targetPipeline": "stock-prices-pipeline",
  "script": "def fetch():\n    ...",
  "packages": ["yfinance"],
  "enabled": true,
  "tapType": "structured"
}
```

**Response when the stored script is missing from object storage:**

```json theme={null}
{
  "name": "stock-prices-tap",
  "scriptPath": "tap-scripts/stock-prices-tap_a1b2c3d4.py",
  "script": null,
  "scriptMissing": true,
  ...
}
```

`scriptMissing` only appears when `scriptPath` is set but the object is absent — typically after an interrupted edit or re-generation wiped the old script before the new one landed. Distinguish from "never generated" (`scriptPath` null, `script` null, no `scriptMissing` key). The Edit Tap UI uses this flag to render a recoverable-state banner instead of the generic "Generate a script first" validation.

## Create or Update Tap

```
POST /api/v1/tap
Content-Type: application/json
```

**Body:**

```json theme={null}
{
  "name": "stock-prices-tap",
  "description": "Fetch daily stock prices from yfinance",
  "scriptPath": "tap-scripts/stock-prices-tap_a1b2c3d4.py",
  "targetPipeline": "stock-prices-pipeline",
  "packages": ["yfinance"],
  "secretName": "my-api-creds",
  "cronExpression": "0 0 0 * * ?",
  "enabled": true,
  "tapType": "structured"
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `tapType` | string | `"structured"` | `"structured"` or `"document"`. Document taps return `{uri, filename, content}` dicts and require a `targetPipeline` with an `unstructuredAttributes` source and a vector-store destination. Cannot be changed once set. |
| `catalog` | string | `null` | Free-form label that groups related taps and pipelines in the [Data Catalog](/data-catalog). Empty or null = Uncataloged. |

**Document-tap compatibility rule.** When `tapType == "document"` and `targetPipeline` is non-empty, the server validates that the pipeline has:

* `source.fileAttributes.unstructuredAttributes != null`
* A vector-store destination (one of `qdrant`, `pgvector`, `weaviate`, `milvus`, `chroma`)

Mismatches return **HTTP 400** with a message naming the violation. Also enforced at run time in `TapRunner.feedDocumentPipeline` in case the pipeline is reshaped after the tap is saved.

**Test-before-cron rule (HTTP 409).** A tap may only carry a `cronExpression` once the exact script (or, for HTTP taps, the exact `endpointUrl`) that will run on that schedule has passed a test run (`POST /api/v1/tap/run` with `mode: "test"`). The save is refused with **HTTP 409** and nothing is persisted (no new version is minted) when either:

* the `cronExpression` is being set or changed and the stored tap has no successful test for the script being saved, or
* the `cronExpression` is unchanged but the script (new `scriptPath` / `scriptCommitSha` / `endpointUrl`) differs from the one that last passed a test.

```json theme={null}
{
  "error": "Tap 'stock-prices-tap' cannot be scheduled: its script has never passed a test run. Remedy: save the tap without `cronExpression`, call `test_tap`, then `update_tap` with the cron."
}
```

Remedy, in order: save the tap **without** `cronExpression`; run it with `mode: "test"` until it succeeds (the server stamps the tested script on the tap as `lastTestRunScriptId`); then save again with the `cronExpression`. Clearing a cron, and cadence-only or metadata edits (description, tags, `targetPipeline`, `enabled`) on a tap whose current script has passed a test, are always accepted. Test state is read from the stored tap only — `lastTestRun*` fields in the request body are ignored. Pre-existing scheduled taps keep running and stay editable; the first cron or script change on such a tap is accepted if the tap has ever tested or run successfully, otherwise it needs a test run first. The same rule applies to `POST /api/v1/tap/version/restore`. `POST /api/v1/tap/script` applies the rule up front: storing different bytes for a tap that currently has a `cronExpression` is refused with the same **HTTP 409** body before anything is written to the object store or code repository (so the repository never drifts ahead of the tap's pinned commit). Clear the cron first (save with `cronExpression: null`), then store, test, and set the cron. Storing identical bytes, or storing for an unscheduled tap, is always accepted. The AI script endpoints (`/tap/generate`, `/tap/fix`, `/tap/review`, `/tap/optimize`) still return the new script and its `scriptPath` with HTTP 200, but do not move a scheduled tap onto the new bytes; when that happens the body also carries the gate message under `error`.

## Delete Tap

```
DELETE /api/v1/tap?name={tapName}
```

Deletes the tap configuration and its script from MinIO. For document taps, also clears the ledger entries and deletes all staged MinIO objects for that tap.

## Brainstorm (AI Chat)

```
POST /api/v1/tap/brainstorm
Content-Type: application/json
```

Multi-turn conversational endpoint that helps the user refine a vague tap idea into a clear instruction. The UI calls this on every message in the brainstorm chat panel of the tap creation wizard. The AI:

* Asks one focused clarifying question at a time
* Suggests specific data sources (e.g., yfinance, Alpha Vantage, Open-Meteo)
* Recognizes Datris platform tables and uses the metadata/query endpoints
* Returns an updated **instruction draft** on every turn so the UI can keep the instruction box in sync
* Returns **suggestedEnvVars** when an external API requires authentication

**Body:**

```json theme={null}
{
  "messages": [
    {"role": "user", "content": "I'm looking for company earnings data"},
    {"role": "assistant", "content": "Which companies?..."},
    {"role": "user", "content": "Tickers from the consumer_discretionary_earnings table on Datris"}
  ],
  "currentDescription": "Optional: the current draft instruction so the AI has context",
  "tapType": "structured"
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `tapType` | string | `"structured"` | Selects the system prompt. `"document"` uses a document-tap-aware prompt that asks about document sources (SharePoint, S3, URL listings, etc.) and never asks about chunking, embedding, or vector-store details (those are pipeline concerns). |

**Response:**

```json theme={null}
{
  "reply": "Got it. Which earnings data source — yfinance (free), Alpha Vantage, or Polygon.io?",
  "description": "Query the consumer_discretionary_earnings table via /api/v1/query/postgres to get the ticker list. (Source TBD.)",
  "suggestedEnvVars": []
}
```

After the user picks Alpha Vantage, a subsequent call returns:

```json theme={null}
{
  "reply": "Good choice. The script will need an Alpha Vantage API key — please create a tap secret with ALPHA_VANTAGE_API_KEY in the credentials section below.",
  "description": "Query the consumer_discretionary_earnings table via /api/v1/query/postgres to get tickers, then for each ticker call the Alpha Vantage EARNINGS endpoint and return historical earnings records. Use os.environ.get('ALPHA_VANTAGE_API_KEY') for authentication.",
  "suggestedEnvVars": ["ALPHA_VANTAGE_API_KEY"]
}
```

The AI never suggests `DATRIS_POSTGRES_DATABASE`, `DATRIS_MONGODB_DATABASE`, `DATRIS_PLATFORM_HOST`, or `DATRIS_PLATFORM_PORT` in `suggestedEnvVars` — those are always injected by the platform.

## Generate Script (AI)

```
POST /api/v1/tap/generate
Content-Type: application/json
```

Uses AI to generate a Python `fetch()` script from a plain-English description. The system prompt branches on `tapType` — document taps get instructions to return `{uri, filename, content}` dicts and a different set of rules (no chunking, no embedding, no local-filesystem fallback).

**Body:**

```json theme={null}
{
  "description": "Discover all PDFs in the S3 bucket 'contracts' under prefix 2026/",
  "tapName": "contracts-tap",
  "oldScriptPath": "tap-scripts/old_script.py",
  "secretName": "aws-creds",
  "tapType": "document"
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `tapType` | string | `"structured"` | Selects the system prompt. Set to `"document"` when generating a document-tap script. |

**Response:**

```json theme={null}
{
  "script": "import requests\n\ndef fetch():\n    ...",
  "packages": ["yfinance"],
  "scriptPath": "tap-scripts/stock-prices-tap_a1b2c3d4.py",
  "injectedPrompts": ["rate-limits"]
}
```

`injectedPrompts` lists the names of the [tap prompt fragments](#tap-prompt-fragments) that were injected into the generation prompt (empty when none matched).

## Fix Script (AI Diagnosis)

```
POST /api/v1/tap/fix
Content-Type: application/json
```

Uses AI to fix a script based on a diagnosis of what went wrong.

**Body:**

```json theme={null}
{
  "tapName": "stock-prices-tap",
  "script": "current script content...",
  "diagnosis": "The API endpoint is wrong...",
  "logs": "script output logs...",
  "error": "error message...",
  "oldScriptPath": "tap-scripts/old_script.py"
}
```

**Response:** Same format as Generate Script.

## Test Tap

```
POST /api/v1/tap/test?testLimit={N}
Content-Type: application/json
```

Executes the tap script without sending data to a pipeline. Returns results, logs, and AI diagnosis if issues detected.

**Query parameters:**

| Name | Type | Description |
| - | - | - |
| `testLimit` | integer (optional) | When `> 0`, the runner injects `DATRIS_TAP_TEST_LIMIT=<N>` into the script process. Well-written tap scripts honor this env var to cap their source reads and iteration loops, making tests faster and lower-cost. Cron runs and `mode=run` runs never set this — production runs always read every row. (A `/tap/run` call with `mode=test` caps at 20 records by default; see [Run Tap](#run-tap).) The UI's "Limit test sample to 20 records" checkbox in Create Tap → step 2 sends `testLimit=20`. |

**Body:** A TapConfig JSON object (same as Create).

**Structured response** (`dataType: "csv"` shown):

```json theme={null}
{
  "records": [{"ticker": "AAPL", "close": 255.92}, ...],
  "recordCount": 503,
  "dataType": "csv",
  "columns": ["ticker", "date", "open", "high", "low", "close", "volume"],
  "logs": "Retrieved 503 tickers...\nBatch 1/11...",
  "error": null,
  "aiExplanation": null,
  "durationMs": 12400
}
```

**Document-tap response** (`dataType: "document"`):

```json theme={null}
{
  "records": [
    {
      "uri": "s3://contracts/2026/acme-msa.pdf",
      "filename": "acme-msa.pdf",
      "content": "JVBERi0xLjQKJe…",
      "content_hash": "2f1c…",
      "metadata": {"source": "s3"}
    }
  ],
  "recordCount": 1,
  "dataType": "document",
  "columns": null,
  "logs": "Listed 1 object in s3://contracts/2026/",
  "error": null,
  "durationMs": 3100
}
```

If errors or 0 records are detected, `aiExplanation` contains an AI-generated diagnosis. `durationMs` is the end-to-end test wall time in milliseconds — this is what the [Optimize Script](#optimize-script) endpoint uses as the baseline timing for its perf rewrite.

For `dataType: "csv"`, **column names in `columns` and the keys inside each record in `records` are normalized** by the platform: lowercase, `[a-z0-9_]` only, with `%` rewritten to `percent`. Source data with names like `EPS Estimate` or `Surprise(%)` will appear as `eps_estimate` and `surprise_percent`. JSON/XML results destined for MongoDB are not normalized. See [Schema Definition → Column Naming Rules](/schemas#column-naming-rules). Document-tap records pass through unchanged — the platform never rewrites URI, filename, content, or metadata keys.

## Review Script (Post-Run)

```
POST /api/v1/tap/review
Content-Type: application/json
```

After a successful test, asks the LLM to scan the script's captured stderr/stdout for signals that the script *itself* should change — not for performance. The reviewer looks for rate-limit / throttle / burst warnings, deprecation hints, pagination / partial-response cues, and schema-drift / auth warnings. When a signal is found, the script is regenerated with the appropriate fix (add `time.sleep`, switch to the recommended endpoint, add pagination, update parsing for renamed fields) and persisted to MinIO as a new version. When no signal is found, the script is returned unchanged.

The UI invokes this automatically **before** [Optimize Script](#optimize-script) — if the reviewer rewrites the script, the optimizer is skipped on that pass (correctness from output outranks speed). Callers integrating directly should follow the same order: call `/tap/review` first, re-test on `rewritten=true`, and only call `/tap/optimize` when `rewritten=false`.

**Body:**

```json theme={null}
{
  "tapName": "stock-prices-tap",
  "script": "current working script content...",
  "recordCount": 503,
  "durationMs": 298000,
  "logs": "Retrieved 503 tickers...\nSource warning: Burst pattern detected...",
  "oldScriptPath": "tap-scripts/stock-prices-tap_v3.py"
}
```

| Field | Type | Description |
| - | - | - |
| `tapName` | string | Tap identifier. Used for script path versioning. |
| `script` | string | The current working script that passed its test. |
| `recordCount` | integer | How many records the last test returned. |
| `durationMs` | integer | Wall time of the last test run in ms (from `/tap/test` response). |
| `logs` | string | Captured script stderr/stdout — the reviewer matches signals liberally against this text. |
| `oldScriptPath` | string | MinIO path of the current script. If rewritten, the new script is stored as a new version and the old path is retained for rollback. |

**Response:**

```json theme={null}
{
  "script": "regenerated Python 3 script...",
  "packages": ["extra-package"],
  "scriptPath": "tap-scripts/stock-prices-tap_v4.py",
  "changes": [
    "Added 0.25s sleep between API calls — source reported burst pattern (5 rps limit)",
    "Lowered max_workers from 10 to 3"
  ],
  "rewritten": true
}
```

When `rewritten` is `false`, `script` / `scriptPath` match the input and `changes` is empty — the reviewer found nothing in the output worth acting on. Prompt fragments configured via [Tap Prompt Fragments](#tap-prompt-fragments) are auto-injected into the reviewer's system prompt when their key or any alias appears in the script.

## Optimize Script

```
POST /api/v1/tap/optimize
Content-Type: application/json
```

After a successful test, asks the LLM to restructure a working script for performance — e.g. swap serial HTTP calls for a `ThreadPoolExecutor`, reuse a `requests.Session()`, or drop unnecessary `time.sleep()` — while preserving correctness (`fetch()` signature, `DATRIS_TAP_TEST_LIMIT` handling, Vault env-var reads, retry/backoff behavior, and `raise_for_status` calls).

The UI invokes this automatically after a green test in the Create Tap wizard, with a regression guard that auto-reverts if the rewrite runs ≥20% slower. Callers integrating this endpoint directly should implement the same re-test + revert pattern.

**Body:**

```json theme={null}
{
  "tapName": "stock-prices-tap",
  "script": "current working script content...",
  "recordCount": 503,
  "durationMs": 298000,
  "logs": "Retrieved 503 tickers...",
  "oldScriptPath": "tap-scripts/stock-prices-tap_v3.py"
}
```

| Field | Type | Description |
| - | - | - |
| `tapName` | string | Tap identifier. Used for script path versioning. |
| `script` | string | The current working script that passed its test. |
| `recordCount` | integer | How many records the last test returned. Used in the user prompt so the LLM can reason about per-record timing. |
| `durationMs` | integer | Wall time of the last test run in ms (from `/tap/test` response). |
| `logs` | string | Recent script stderr output (last \~40 lines used). |
| `oldScriptPath` | string | MinIO path of the current script. The optimized script is stored as a new version; the old path is retained so the caller can revert. |

**Response:**

```json theme={null}
{
  "script": "optimized Python 3 script...",
  "packages": ["extra-package"],
  "scriptPath": "tap-scripts/stock-prices-tap_v4.py",
  "changes": [
    "Parallelized ticker fetches with ThreadPoolExecutor(10)",
    "Removed per-item 0.25s sleep"
  ]
}
```

If the LLM concludes the script is already well-optimized, `changes` is an empty array and `script` / `scriptPath` match the input. In that case no re-test is needed.

## Run Tap

```
POST /api/v1/tap/run
Content-Type: application/json
```

Executes a saved tap. Optionally sends data to the configured pipeline.

**Body:**

```json theme={null}
{
  "name": "stock-prices-tap",
  "mode": "run",
  "params": {"since": "2026-05-01"}
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `name` | string | (required) | Tap name |
| `mode` | string | `"test"` | `"run"` feeds records into `targetPipeline`; `"test"` executes the script and returns sample results without persisting. A successful `mode=test` also records `lastTestRunStatus: "success"` and `lastTestRunScriptId` (the identity of the tested script or endpoint) on the tap, which is what allows a `cronExpression` to be saved — see the test-before-cron rule under Create or Update Tap |
| `testLimit` | integer (optional) | `20` when `mode=test` | Max records a test pulls before the runner stops the script. Applies to `mode=test` only: the runner injects `DATRIS_TAP_TEST_LIMIT=<N>` and stops pulling from a generator `fetch()` after `N` records. `0` (or a negative value) disables the cap and streams the whole source. Ignored for `mode=run`, which always reads every row |
| `params` | object | `{}` | Optional per-run parameters exposed to the script as environment variables (nested values JSON-encoded) |

**Response:**

```json theme={null}
{
  "tap": "stock-prices-tap",
  "description": "Fetch daily stock prices...",
  "status": "success",
  "mode": "run",
  "targetPipeline": "stock-prices",
  "persisted": true,
  "publisherToken": "5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e",
  "pipelineTokens": ["a1b2c3d4-..."],
  "recordCount": 503,
  "dataType": "csv",
  "columns": ["ticker", "date", "open", "high", "low", "close", "volume"],
  "logs": "...",
  "error": null
}
```

`columns` lists the column names the script produced (normalized for `dataType: "csv"`, see [Test Tap](#test-tap)); it is `null` when the run produced no records.

**`mode=run` does not return the records themselves** — the data is in transit to `targetPipeline`, and the agent / caller should verify completion via `get_pipeline_status`, not read from the response body. `recordCount` summarizes how many records were submitted. To preview what a script produces, call the same endpoint with `mode=test`, which pulls only the first 20 records by default (raise or remove the cap with `testLimit`) and returns up to 20 sample rows in a `records` array (with `recordsTruncated: true` set when the script produced more than that). Because of the cap, a test’s `recordCount` is the preview size, not what a `mode=run` call will produce.

`persisted: true` means the records were submitted to `targetPipeline`. When `persisted: false`, a `persistedReason` field names the cause — one of `test_mode`, `run_error`, `no_records`, `no_target_pipeline`, or `debounced`. On persisted runs, `publisherToken` groups every ingestion job this run submitted; `pipelineTokens` lists each. Document taps fan out to many `pipelineTokens` but share one `publisherToken`.

### Run debounce (mode=run only)

`/tap/run` debounces `mode=run` requests per tap on a 5-second window to suppress accidental duplicates — agents that emit parallel `tool_use` blocks, UI buttons that get double-clicked, or transport-level retries. If a second run hits within the window, the response is HTTP 200 with `status: "skipped"`, `persisted: false`, `persistedReason: "debounced"`, and an `error` string explaining how long ago the previous run started. The previous run keeps executing — pivot via `get_tap_logs` and `get_pipeline_status` to track its outcome. `mode=test` is read-only and is never debounced.

### Watching a tap run

A `/tap/run` response comes back as soon as records are handed off to the async ingestion pipeline — the actual load is still in progress. Use `publisherToken` to poll:

```
GET /api/v1/pipeline/status?publishertoken={publisherToken}&withrollup=true
```

Returns a `{rollup, events}` wrapper covering every ingestion job this tap run submitted (for structured taps that's one job; for document taps it's one per submitted document). Poll until `rollup.allDone` is `true`, then read `rollup.status` for the outcome and `rollup.jobs[].lastError` for any failure. See the [Pipeline Status API → Rollup Response](/api-reference/status-api#rollup-response) for the full shape.

## Generate CRON Expression (AI)

```
POST /api/v1/tap/cron
Content-Type: application/json
```

Converts a plain-English schedule description to a Quartz CRON expression.

**Body:**

```json theme={null}
{
  "description": "Every weekday at 4pm ET"
}
```

**Response:**

```json theme={null}
{
  "cronExpression": "0 0 16 ? * MON-FRI"
}
```

## Run History

```
GET /api/v1/tap/logs?name={tapName}
```

Returns the last 50 run log entries for a tap, sorted by most recent first.

**Response:**

```json theme={null}
[
  {
    "tapName": "stock-prices-tap",
    "runTime": "2026-04-04T20:23:22Z",
    "status": "success",
    "recordCount": 503,
    "dataType": "csv",
    "logs": "Retrieved 503 tickers...",
    "error": null,
    "mode": "test",
    "durationMs": 45000,
    "publisherToken": null,
    "aiSummary": null,
    "aiDiagnosis": null,
    "aiSuggestion": null,
    "scriptCommitSha": null,
    "pipelineRollup": null
  }
]
```

| Field | Description |
| - | - |
| `publisherToken` | Set on `mode=run` entries that fed a pipeline; groups the ingestion jobs the run submitted |
| `aiSummary`, `aiDiagnosis`, `aiSuggestion` | AI fix suggestion filled in after a scheduled run exhausts its automatic retries; `aiSummary` is the one-line headline, the other two carry the full explanation |
| `scriptCommitSha` | Repo-backed taps only — the commit the run's script was read at; `null` for MinIO-stored scripts |
| `pipelineRollup` | Downstream pipeline job rollup (same shape as `rollup` in the [Status API](/api-reference/status-api)) for runs that have a `publisherToken`; `null` otherwise |

## Document Ledger

Document taps track which files they've already processed in a **ledger** — a MongoDB collection keyed by `{tapName}|{uri}`. See [Document Taps → The Document Ledger](/taps#the-document-ledger) for the concept; these endpoints manage it.

### Read ledger

```
GET /api/v1/tap/ledger?name={tapName}
```

Returns every ledger entry owned by the tap.

**Response:**

```json theme={null}
[
  {
    "uri": "s3://contracts/2026/acme-msa.pdf",
    "tapName": "contracts-tap",
    "stagedPath": "tap-docs/contracts-tap/a1b2c3d4_acme-msa.pdf",
    "filename": "acme-msa.pdf",
    "contentHash": "2f1c…",
    "firstSeenAt": "2026-04-18 13:42:38 EDT",
    "lastSeenAt": "2026-04-19 07:00:11 EDT",
    "status": "processed",
    "metadata": {"source": "s3", "region": "EU"}
  }
]
```

### Delete one entry (force re-process)

```
DELETE /api/v1/tap/ledger?name={tapName}&uri={uri}
```

Removes a single entry and its staged MinIO object. The next tap run will re-ingest that specific document from source.

### Clear the entire ledger (force full re-scan)

```
DELETE /api/v1/tap/ledger?name={tapName}
```

Removes every entry for the tap plus all staged MinIO objects. The next run re-ingests every document the source exposes.

Also triggered automatically when:

* The tap is deleted (`DELETE /api/v1/tap`).
* The target pipeline's data is cleared (`DELETE /api/v1/pipeline?pipeline=X&deleteData=true`). Without this, the tap would skip docs it already "processed" and the pipeline would stay empty.

## Available Vector Stores

```
GET /api/v1/vector-stores/available
```

Returns the subset of `[qdrant, weaviate, pgvector, milvus, chroma]` whose Vault secret is present **and** whose service is currently reachable. Used by the document-tap pipeline wizard to drive the store picker. Secret presence alone isn't sufficient — the dev stack seeds placeholder secrets for every store, so the endpoint actually probes each service.

**Response:**

```json theme={null}
["pgvector"]
```

Probes use the same logic as `/api/v1/health/services`, with a 2-second timeout per store (\~10 s worst case if everything is unreachable).

## Tap Prompt Fragments

User-configured system prompt fragments that auto-inject into tap AI flows when the user's text matches the fragment's key or any alias. See [Tap Prompt Fragments](/tap-prompts) for the full concept; this section covers the REST endpoints.

Fragments are per-tenant, stored in MongoDB at `{env}-tap-prompt`, and matched case-insensitively with word boundaries.

### List Fragments

```
GET /api/v1/tap-prompts
```

**Response:** Array of `TapPromptFragment` objects.

```json theme={null}
[
  {
    "key": "AWS",
    "aliases": ["S3", "EC2", "Lambda", "boto3"],
    "content": "When fetching data from AWS services, prefer boto3 with IAM credentials...",
    "enabled": true,
    "createdAt": "2026-04-20 14:05:00 UTC",
    "updatedAt": "2026-04-20 14:05:00 UTC"
  }
]
```

### Get Fragment

```
GET /api/v1/tap-prompts/{key}
```

Returns a single fragment or HTTP 404 if `{key}` is not found. `{key}` should be URL-encoded.

### Create or Update Fragment

```
POST /api/v1/tap-prompts
Content-Type: application/json
```

Create-or-update semantics — if a fragment with the same `key` already exists, it is overwritten. `createdAt` is preserved across updates; `updatedAt` is refreshed on every write. Writing through this endpoint invalidates the in-process fragment cache immediately, so the next tap generation / fix / optimize / brainstorm call sees the change without restart.

**Body:**

```json theme={null}
{
  "key": "Polygon",
  "aliases": ["polygon.io"],
  "content": "Use the requests library with os.environ.get(\"POLYGON_API_KEY\") and pass auth via the Authorization: Bearer {key} header (query param ?apiKey= also works but prefer the header).",
  "enabled": true
}
```

| Field | Type | Required | Description |
| - | - | - | - |
| `key` | string | Yes | Primary match keyword. Unique per tenant. Matched case-insensitively with `\b` word boundaries. |
| `aliases` | string\[] | No | Additional match keywords. Same matching rules as `key`. |
| `content` | string | Yes | System-prompt text appended under a `## User-provided context` block when a match fires. |
| `enabled` | boolean | No (default `true`) | When `false`, the fragment stays in storage but is excluded from matching. |

**Response:** `{"status": "ok"}` on success.

### Delete Fragment

```
DELETE /api/v1/tap-prompts/{key}
```

Deletes the fragment and invalidates the cache. URL-encode `{key}`. Returns `{"status": "ok"}`.

### Suggest Fragment Content

```
POST /api/v1/tap-prompts/suggest
Content-Type: application/json
```

Asks the LLM to draft a content body from the fragment's key and aliases. Useful as a starting point when creating a fragment — the UI exposes this via the **Suggest** button next to the Content field. If `content` is non-empty in the request, the LLM is instructed to refine/expand it rather than replace.

**Body:**

```json theme={null}
{
  "key": "Stripe",
  "aliases": ["stripe.com"],
  "content": ""
}
```

**Response:**

```json theme={null}
{
  "content": "Use the stripe Python SDK (pip install stripe). Set stripe.api_key = os.environ.get(\"STRIPE_API_KEY\"). Use auto_paging_iter() for list endpoints to transparently handle pagination..."
}
```

The response is plain text — the UI drops it into the Content textarea for the user to review before saving.

## Authentication

All endpoints require the `x-api-key` header if API key authentication is enabled.


# Brainstorm chat (AI)
Source: https://docs.datris.ai/api-reference/taps/brainstorm-chat-ai

/openapi.yaml post /api/v1/tap/brainstorm
Multi-turn conversational endpoint that helps refine a vague tap idea into a clear instruction.
The AI asks one focused clarifying question at a time, suggests data sources, recognizes Datris
platform tables, and returns an updated instruction draft on every turn.




# Clear the ledger (force re-scan) or delete one entry (force re-process)
Source: https://docs.datris.ai/api-reference/taps/clear-the-ledger-force-re-scan-or-delete-one-entry-force-re-process

/openapi.yaml delete /api/v1/tap/ledger
With only `name`, clears the entire ledger for the tap and deletes every staged MinIO object — the next run re-ingests every document from source. With `name` and `uri`, deletes one entry and its staged object so that specific document is re-processed on the next run.



# Create or update a tap
Source: https://docs.datris.ai/api-reference/taps/create-or-update-a-tap

/openapi.yaml post /api/v1/tap
Creates a new tap or updates an existing tap with the same `name`.

Cron validation: `cronExpression` is a Quartz expression of 6 fields
(seconds minutes hours day-of-month month day-of-week) plus an optional
year. A malformed expression — including the 5-field Unix form — is
refused with 400 and nothing is persisted; for a 5-field input the error
carries the 6-field equivalent to resend. A blank cron means unscheduled.

Test-before-cron: a `cronExpression` may only be saved once the exact script
(or HTTP endpoint) that will run on it has passed a `mode=test` run. Setting or
changing a cron on an untested script, or changing the script under an existing
cron, is refused with 409 and nothing is persisted. Remedy: save without
`cronExpression`, run `POST /api/v1/tap/run` with `mode=test`, then save again
with the cron.




# Delete a tap
Source: https://docs.datris.ai/api-reference/taps/delete-a-tap

/openapi.yaml delete /api/v1/tap
Deletes the tap configuration and its script from MinIO.



# Fix tap script (AI diagnosis)
Source: https://docs.datris.ai/api-reference/taps/fix-tap-script-ai-diagnosis

/openapi.yaml post /api/v1/tap/fix
Uses AI to fix a script based on a diagnosis of what went wrong.



# Generate CRON expression (AI)
Source: https://docs.datris.ai/api-reference/taps/generate-cron-expression-ai

/openapi.yaml post /api/v1/tap/cron
Converts a plain-English schedule description to a Quartz CRON expression.



# Generate tap script (AI)
Source: https://docs.datris.ai/api-reference/taps/generate-tap-script-ai

/openapi.yaml post /api/v1/tap/generate
Uses AI to generate a Python `fetch()` script from a plain-English description. The system prompt branches on `tapType` — document taps get a prompt that instructs the LLM to return `{uri, filename, content}` dicts, never chunk or embed, and never fall back to scanning the local filesystem.



# Get a tap configuration
Source: https://docs.datris.ai/api-reference/taps/get-a-tap-configuration

/openapi.yaml get /api/v1/tap
Returns a single tap configuration, including the script content from MinIO.



# Get run history across all taps
Source: https://docs.datris.ai/api-reference/taps/get-run-history-across-all-taps

/openapi.yaml get /api/v1/tap/logs/all
Returns run log entries for every tap, newest window first. Feeds the Ops activity dashboard's aggregation. Defaults to the last 30 days, capped at 2000 rows.



# Get tap run history
Source: https://docs.datris.ai/api-reference/taps/get-tap-run-history

/openapi.yaml get /api/v1/tap/logs
Returns the last 50 run log entries for a tap, sorted by most recent first.



# List all taps
Source: https://docs.datris.ai/api-reference/taps/list-all-taps

/openapi.yaml get /api/v1/taps
Returns all registered tap configurations.



# Live progress of an in-flight script generation
Source: https://docs.datris.ai/api-reference/taps/live-progress-of-an-in-flight-script-generation

/openapi.yaml get /api/v1/tap/generate/status
Reports what a blocking `/tap/generate` call is currently doing so clients can show real progress during the 1–3 minute wait. Poll every few seconds while a generate request is in flight. `active: false` means no generation is running for this tap.



# Move a tap script between storage backends
Source: https://docs.datris.ai/api-reference/taps/move-a-tap-script-between-storage-backends

/openapi.yaml post /api/v1/tap/migrate-storage
Reads the script from its current backend, writes it to the target (built-in object storage or the code repository), and stamps the tap as a versioned edit. Idempotent when the tap is already on the target. Built-in objects are never deleted on migrate-out — version snapshots pin them.



# Optimize tap script (AI)
Source: https://docs.datris.ai/api-reference/taps/optimize-tap-script-ai

/openapi.yaml post /api/v1/tap/optimize
Uses AI to optimize a working script for speed and efficiency (based on the latest run's record count and duration). Always stores the returned script; if the tap already exists, its `scriptPath` is updated.



# Overwrite a tap's incremental-sync state
Source: https://docs.datris.ai/api-reference/taps/overwrite-a-taps-incremental-sync-state

/openapi.yaml post /api/v1/tap/state
Sets the state object the next run receives via DATRIS_TAP_STATE — e.g. rewind a cursor so a window is re-fetched. The state shape is defined by the tap's own script. Committed state is capped at 64 KB.



# Pull a repo-backed tap script from its branch head
Source: https://docs.datris.ai/api-reference/taps/pull-a-repo-backed-tap-script-from-its-branch-head

/openapi.yaml post /api/v1/tap/script/pull
Re-reads a repository-stored tap script at the configured branch head and reports drift against the tap's pinned commit. With `apply=true`, also advances the pin (a versioned edit).



# Read a document tap's ledger
Source: https://docs.datris.ai/api-reference/taps/read-a-document-taps-ledger

/openapi.yaml get /api/v1/tap/ledger
Returns every ledger entry owned by the tap. The ledger records which documents the tap has already staged so re-runs skip unchanged files.



# Read a tap's incremental-sync state
Source: https://docs.datris.ai/api-reference/taps/read-a-taps-incremental-sync-state

/openapi.yaml get /api/v1/tap/state
Returns the bookmark/cursor the tap's script saved after its last successful run — the object the next run receives via the DATRIS_TAP_STATE environment variable. `state` is null when the tap has never committed state (non-incremental tap, or no successful run yet).



# Reset a tap's incremental-sync state
Source: https://docs.datris.ai/api-reference/taps/reset-a-taps-incremental-sync-state

/openapi.yaml delete /api/v1/tap/state
Deletes the stored state entirely. The next run sees no DATRIS_TAP_STATE and performs a full first-run fetch.



# Review tap script (AI)
Source: https://docs.datris.ai/api-reference/taps/review-tap-script-ai

/openapi.yaml post /api/v1/tap/review
Uses AI to review a script after a successful test run — correctness, robustness, and adherence to tap conventions. May return the script rewritten. When it rewrites and the tap already exists, the tap's `scriptPath` is updated to the new object.



# Run a tap
Source: https://docs.datris.ai/api-reference/taps/run-a-tap

/openapi.yaml post /api/v1/tap/run
Executes a saved tap. Optionally sends data to the configured pipeline.



# Store a tap script
Source: https://docs.datris.ai/api-reference/taps/store-a-tap-script

/openapi.yaml post /api/v1/tap/script
Stores a (typically hand-edited) Python script in the resolved storage backend and, if the tap already exists, updates its script reference. Backend resolution: explicit `storage` in the body, else the existing tap's backend, else the tenant default (the code repository when one is enabled, built-in storage otherwise).



# Test a tap
Source: https://docs.datris.ai/api-reference/taps/test-a-tap

/openapi.yaml post /api/v1/tap/test
Executes the tap script without sending data to a pipeline. Returns a preview of the
records (the first 20 of a record list, with `recordsTruncated: true` when there were
more), logs, and an AI-generated diagnosis if errors or zero records are detected.




# Version API
Source: https://docs.datris.ai/api-reference/version-api

Version information endpoints

Returns the pipeline server version, the feature flags this install runs with, and the tap budgets in force (read those before sizing a large tap run).

## Get Version

```
GET /api/v1/version
```

**Example:**

```bash theme={null}
curl http://localhost:8080/api/v1/version
```

**Response:** `200 OK`

```json theme={null}
{
  "version": "1.28.0",
  "environment": "oss",
  "multiTenant": "false",
  "hosted": "false",
  "useUserAuth": "false",
  "useApiKeys": "false",
  "useAuditLog": "false",
  "useAgentPolicy": "false",
  "recoveryAgentEnabled": "false",
  "postgresDatabase": "datris",
  "mongodbDatabase": "datris",
  "useTapRunner": "false",
  "pipelineMaxPayloadMB": "4096",
  "pipelineMaxPayloadMBSource": "default",
  "tapScriptTimeoutSeconds": "300",
  "tapScriptTimeoutSecondsSource": "default",
  "tapRunTimeoutSeconds": "3600",
  "tapRunTimeoutSecondsSource": "default"
}
```

| Field | Description |
| - | - |
| `version` | Server version |
| `environment` | Environment name (e.g., `oss` for self-hosted, or tenant prefix for managed hosting) |
| `multiTenant` | Whether the server is running in multi-tenant mode |
| `hosted` | Whether this is a managed/hosted deployment |
| `useUserAuth` | Whether user authentication is enabled |
| `useApiKeys` | Whether API key authentication is enabled |
| `useAuditLog` | Whether the audit log is enabled |
| `useAgentPolicy` | Whether agent policy enforcement is enabled |
| `recoveryAgentEnabled` | Whether the recovery agent is enabled |
| `useTapRunner` | Whether tap scripts run in the separate tap-runner service |
| `postgresDatabase` | Canonical Postgres database name. User-facing; UI reads this as authoritative and doesn't let users edit it. |
| `mongodbDatabase` | Canonical MongoDB database name (user-facing). In multi-tenant mode this is the tenant environment name; in single-tenant it comes from `mongodb.database` in application.yaml. Distinct from the internal platform database (`mongodb.internalDatabase`, default `oss`) which holds pipeline/tap configs and run state and is never surfaced in the UI. |
| `pipelineMaxPayloadMB` | Per-run payload disk budget in force on this install, in MB (`0` = unlimited). Agents size a tap run against this value, not against the documented default of 4096. |
| `pipelineMaxPayloadMBSource` | Where the disk budget came from: `env` (`PIPELINE_MAX_PAYLOAD_MB`), `deprecated-alias` (`TAP_MAX_OUTPUT_MB`) or `default` |
| `tapScriptTimeoutSeconds` | Wall-clock ceiling in force for a tap test, in seconds |
| `tapScriptTimeoutSecondsSource` | Where the test ceiling came from: `env` (`TAP_SCRIPT_TIMEOUT_SECONDS`) or `default` |
| `tapRunTimeoutSeconds` | Wall-clock ceiling in force for a real or scheduled tap run, in seconds |
| `tapRunTimeoutSecondsSource` | Where the run ceiling came from: `env` (`TAP_RUN_TIMEOUT_SECONDS`), `derived` (unset, resolved to `max(3600, tapScriptTimeoutSeconds)`) or `default` |


# Diff two pipeline versions
Source: https://docs.datris.ai/api-reference/versions/diff-two-pipeline-versions

/openapi.yaml get /api/v1/pipeline/version/diff



# Diff two tap versions
Source: https://docs.datris.ai/api-reference/versions/diff-two-tap-versions

/openapi.yaml get /api/v1/tap/version/diff
Server-computed, ready-to-render diff of the tap configuration and script between two versions.



# Get a pipeline version snapshot
Source: https://docs.datris.ai/api-reference/versions/get-a-pipeline-version-snapshot

/openapi.yaml get /api/v1/pipeline/version



# Get a tap version snapshot
Source: https://docs.datris.ai/api-reference/versions/get-a-tap-version-snapshot

/openapi.yaml get /api/v1/tap/version



# List pipeline definition versions
Source: https://docs.datris.ai/api-reference/versions/list-pipeline-definition-versions

/openapi.yaml get /api/v1/pipeline/versions
Lists the saved definition versions of a pipeline, newest first.



# List tap definition versions
Source: https://docs.datris.ai/api-reference/versions/list-tap-definition-versions

/openapi.yaml get /api/v1/tap/versions
Lists the saved definition versions of a tap, newest first. Every versioned save (create, update, restore) appends a snapshot.



# Restore a pipeline version
Source: https://docs.datris.ai/api-reference/versions/restore-a-pipeline-version

/openapi.yaml post /api/v1/pipeline/version/restore
Restores the pipeline's definition from the given version snapshot, preserving ownership. The restore itself is saved as a new version.



# Restore a tap version
Source: https://docs.datris.ai/api-reference/versions/restore-a-tap-version

/openapi.yaml post /api/v1/tap/version/restore
Restores the tap's DEFINITION from the given version snapshot, preserving the live tap's creation metadata, ownership, and run-status fields. The restore itself is saved as a new version.



# Assistant
Source: https://docs.datris.ai/assistant

Chat with an in-product agent that finds data sources and builds taps and pipelines for you.

The **Assistant** tab is an in-product agent that wires up taps and pipelines
for you. You describe what data you want; the agent finds a source, generates
the tap script, creates the pipeline, links them, and shows you the result —
all in one chat.

It uses the same MCP tools that external agents (Claude Desktop, Cursor) see
when they connect to your Datris MCP server. There is one canonical workflow,
shared between in-product and external agents.

## What you can ask

The Assistant is open-ended. Some examples:

* *"Find a source of US public-company SEC filings and build a tap and pipeline."*
* *"Pull current weather data from a public API and create a pipeline for it."*
* *"Ingest the most recent NYC taxi trip data into a pipeline."*
* *"List the taps that are currently scheduled and tell me which haven't run in over a week."*
* *"Show me the rows in the `weather` pipeline."* (works for MongoDB, PostgreSQL, Snowflake, Databricks, **Object Store (Parquet / ORC)**, or vector destinations — the agent picks the right query tool for the destination)

The agent will reach for tools as it needs them: web search to find a source,
`list_taps` / `list_pipelines` to check for existing work, `create_tap_secret`
when credentials are needed, `create_tap`, `test_tap`, `create_pipeline`, and
so on.

## Drop a file to build a pipeline

You can drag a file straight into the chat — or attach one with the file
picker — and the Assistant will build a pipeline around it for you. Drop a CSV,
JSON, XML, or document file onto the chat window (or click the attach control
and pick it), then describe what you want done with it.

When you send a message with a file attached, the agent:

1. **Reads a sample** of the file — a text extract from the attached content —
   so it can see the columns and data shape.
2. **Proposes a destination**, defaulting sensibly for the data and naming the
   alternatives (Postgres, MongoDB, an object store, Snowflake, Databricks, or a
   vector store) so you
   can pick a different one.
3. **Confirms before building** — it tells you what it's about to create and
   waits for your go-ahead.
4. **Asks about an upsert key** when relevant, so repeated loads update rows in
   place instead of duplicating them.
5. **Creates the pipeline, loads the data, and reports the result** — including
   how many rows landed in the destination.

This is the fastest path from "here's a file" to a working pipeline: no wizard,
no manual schema entry — just drop it in and confirm.

## What you'll see in real time

Three live layers in every assistant turn:

1. **💭 Thinking** — the model's chain-of-thought, streaming character-by-character.
   Anthropic tenants see the full reasoning; OpenAI tenants see a high-level
   summary instead. Click the block to expand.
2. **Tool cards** — inline within the response. Each card shows a human-friendly
   label (*"🔍 Searching the web for SEC EDGAR API…"*, *"🛠 Creating tap sec-edgar…"*)
   with a spinner while running, then a ✓ or ✕. Click to expand the raw input
   and result.
3. **Streaming prose** — the response text appears as the model writes it, with
   tool calls inlined where the model invoked them.

When the agent finishes, **Open tap** and **Open pipeline** links appear at the
bottom of the turn so you can jump straight to anything it created.

## Safety

The agent has access to every MCP tool, including destructive ones
(`delete_tap`, `delete_pipeline`, `delete_tap_secret`, `update_secret`). It is
instructed to **never invoke a destructive tool without asking you first** —
if you say "delete the broken tap," the agent will name what it's about to
delete and wait for your confirmation in the chat before proceeding.

The Stop button at the bottom of the chat aborts the loop server-side at the
next checkpoint, so you can interrupt anytime. Stop halts the in-flight model
call within a fraction of a second — cancelled responses cost only what was
already streamed.

## Platform-tab secrets are visible but read-only

When a pipeline destination needs operator-owned credentials (AWS S3, an
external Postgres, a vector-store endpoint, etc.), the agent can **discover**
secrets you've created on Configuration → Secrets → Platform and verify their
field shape — but it cannot create, modify, or delete them. If nothing
suitable exists, the agent will name the field shape the destination needs and
ask you to add the secret in the Secrets tab, then give it the name to use.
Secret values are never exposed to the agent.

## Long-running runs are carried to completion

When the agent runs a tap (`run_tap` or `upload_data`), it owns that work all
the way to a terminal `success` / `warning` / `error` outcome. It will not
kick off the job and tell you to check back later — that defeats the point of
delegating.

The polling pattern it uses:

1. Poll `get_pipeline_status` immediately after the run is submitted (most
   small structured taps finish in 1–5 seconds; no point sleeping first).
2. If still in progress, sleep with `wait_seconds` using exponential backoff
   (5s → 10s → 20s → 30s → 60s → 60s …, capped at 60s normally, 120s only if
   things are genuinely glacial).
3. Reset to a short wait (5–15s) the moment a poll shows new jobs flipping to
   a terminal state — progress means it's about to finish.
4. Emit a one-line progress update each cycle ("12 of 28 jobs done, 16 still
   processing — checking again in 30s") so you can see the run advance.
5. After \~20 polls, if the run is still in progress, pause and ask whether to
   keep waiting, kill the run, or check later — never silently stop.

The same pattern applies to `upload_data` → `get_job_status`. If the chat
seems to be "hanging" mid-run, it's waiting between polls on purpose.

## Reasoning visibility by provider

The Assistant names the chat provider and model it is using (for example *Anthropic · claude-fable-5-1 · extended thinking on*) on the empty conversation and in the top bar once you start chatting, with a link to **Configuration → AI Providers** where you change it. The line refreshes when you return to the Assistant tab or start a new chat, so it follows a change you just saved there.

| Provider | Streaming text | Tool cards | Reasoning block |
| - | - | - | - |
| Anthropic Claude (Opus 5.5, Fable 5.1) | ✅ | ✅ | Full chain-of-thought (extended thinking) |
| OpenAI GPT-5.x / GPT-6 Astra | ✅ | ✅ | Reasoning summary |
| Azure OpenAI | ✅ | ✅ | None (reasoning happens server-side but isn't surfaced) |
| Claude via Amazon Bedrock | ✅ | ✅ | Full thinking blocks (same as direct Anthropic; responses arrive at the end of each turn rather than streaming) |
| Grok (xAI) | ✅ | ✅ | None (reasoning happens server-side but isn't surfaced) |
| Ollama / Llama | ✅ | If the model supports tool calling | None |

Set `ai.extendedThinking: "false"` in `application.yaml` to disable the
reasoning block platform-wide. Default is on. Extended thinking adds roughly
\$0.05/turn and 2–8s of latency on Opus — cheap insurance for transparency
on an agent that's writing production-bound config.

## Configuration

The Assistant uses your tenant's **AI primary** config (the chat slot). Tap
script generation, AI data quality, and AI transformations use the separate
**codegen** slot. On a fresh install the Anthropic default is Claude Opus 5.5
(`claude-opus-5-5`) for both slots; the OpenAI default is `gpt-5.5`.

There is no per-tenant Assistant configuration to manage beyond what you
already have for AI primary. See [AI Configuration](/ai-configuration) for the
underlying AI primary / codegen / web search / embedding settings.

### Authentication

The Assistant inherits the identity of the user running it. How that maps
to permissions depends on which auth mode the platform is running in:

* **`USE_USER_AUTH=true`** (recommended): the Assistant runs as your
  logged-in user. Audit log entries and capability checks both show
  `session:<your username>`. Permissions follow your role — admin gets
  full access, editor gets create/update/run on data resources, viewer
  gets read-only. See [User Authentication](/user-auth) for the role-to-
  capability mapping.
* **`USE_USER_AUTH=false` + `USE_API_KEYS=true`**: the Assistant uses
  whatever value you pasted into the Connect prompt (typically the seeded
  `ui` key). Same identity for everyone using the browser.
* **`USE_API_KEYS=false`**: no auth on tool calls; the Assistant runs as
  the platform's anonymous identity.

The Assistant doesn't have its own dedicated API key — its server-side
MCP-bound tool calls use the same identity the chat endpoint was
authenticated under. Want to give the Assistant *narrower* permissions
than the human user has? That's not exposed today; the workaround is to
log in as a different user (an editor or viewer) when running the
Assistant, so the role mapping applies. See [API Keys](/api-keys) for
the full capability model.

## Limits

* **50 iterations per turn.** The agent loop is hard-capped to bound cost. If
  the agent runs out of iterations, it stops and asks you to send a follow-up
  message to continue.
* **No server-side conversation history.** The transcript lives in the
  browser tab's session storage — it survives a refresh but is gone once the
  tab is closed. Durable persistence is on the roadmap.
* **32K output tokens per LLM call.** Sufficient for thinking + reasoning +
  tool selection on every public model we support.

## How it relates to other tabs

* **Chat panels** — four panels, one job each: the Assistant (build taps and
  pipelines), Ops chat (runs, failures, incidents — see [Incidents](/incidents)),
  Catalog chat (curation — see [Data Catalog](/data-catalog)) and the
  Configuration assistant (settings — see [Configuration Assistant](/configuration-assistant)).
* **MCP tab** — static catalog of the tools the Assistant uses. Read this if
  you want to understand what each tool does.
* **Agent Monitor (pop-out)** — read-only activity monitor of *external* MCP
  clients (Claude Desktop, Cursor, scripts) calling your server. Opens in its
  own browser window via the pop-out icon (open-in-new) in the Agent Monitor
  header under **MCP → Activity**, so you can keep an eye on Connections and
  the Activity Log while you work elsewhere. The
  Assistant is the in-product counterpart — same tools, different surface.
* **Catalog tab** — anything the Assistant creates (taps and pipelines) lands
  here, grouped by catalog. The "→ Open tap" / "→ Open pipeline" links at the
  end of each conversation route you there.


# Audit log for AI agent data access
Source: https://docs.datris.ai/audit-log

With the audit log on, a durable record of creates, changes, runs, and deletes: humans by login, agents by API key, the Assistant on the user's behalf. Best effort by default, strict mode on request. Filterable in the UI, exportable to CSV, mirrored to your SIEM.

The audit log answers questions like *who deleted that pipeline last Tuesday*, *which key rotated that secret*, *did anyone log in from a new address overnight*, and *what did the `claude-desktop` agent do against pipeline X*.

It is separate from the [Agent Monitor](/monitoring#agent-monitor), which is a live, in-memory view of the last 200 MCP tool calls. The audit log is durable, admin-only, and queryable. With the audit log on it covers every platform write (not just MCP). By default delivery is best effort: if the config store cannot keep up, entries can be dropped and are counted. [Strict mode](#strict-mode) refuses a request instead of losing its entry.

## Enabling it

Off by default — existing installs are unchanged until you opt in.

1. Add the flag to your `.env`:

   ```bash theme={null}
   USE_AUDIT_LOG=true
   # optional — days to keep entries (0 = keep forever)
   AUDIT_LOG_RETENTION_DAYS=90
   # optional — also record reads (queries, searches, listings)
   AUDIT_LOG_READS=false
   # optional — refuse audited requests rather than drop entries (see Strict mode)
   AUDIT_LOG_STRICT=false
   ```

2. Recreate the Datris container:

   ```bash theme={null}
   docker compose up -d --force-recreate datris
   ```

3. Open **Configuration → Audit Log**. With user authentication on, the sub-tab is visible to admins only.

A restart is required to turn the flag on or off — it is read once at startup, like the other auth flags.

## Who is recorded

Every entry names an actor. The type tells you which kind of identity it was:

| Actor type | How it is identified | Example |
| - | - | - |
| `user` | The logged-in session (requires [User Authentication](/user-auth)) | `tfearn`, role `admin` |
| `api-key` | The API key's **label** plus a stable **key id** | `claude-desktop`, `k_7f3a9c1b2d4e` |
| `assistant` | The in-platform Assistant (or Ops / Catalog chat, or the Configuration assistant) acting **on behalf of** the logged-in user | `tfearn via Assistant` |
| `tap` | A tap script reading platform data during a run, via its per-run token | `crypto-prices` |
| `system` | No request behind it: scheduled runs, Kafka / object-store triggers, server start | — |

Agents are identified by their API key, humans by their session. The key's **label** is the identity you already see in the API-Keys tab, version history, and the Agent Monitor. The key's **value** is a secret and is never written anywhere — not hashed, not truncated.

The **key id** exists because a label can be revoked and re-issued. Rotating a key keeps the identity (same label, new secret). Revoking `claude-desktop` and issuing a new `claude-desktop` next week is a *different* key wearing the same name; the id tells them apart. Keys issued before this feature have no id and are attributed by label only.

<Note>
  Actions the Assistant takes for you show up as `assistant` with your username — not as the platform's internal `ui` key. The same attribution applies to pipeline and tap version history, so "who changed this?" has the same answer in both places.
</Note>

In the legacy no-auth mode (`USE_USER_AUTH=false`, `USE_API_KEYS=false`) there is no identity to record; entries are attributed to `anonymous`.

## What is recorded

With the audit log on: writes, runs, deletes, logins, key and user management, and every denied request (best effort by default; see [Strict mode](#strict-mode) for the complete guarantee). Reads are not recorded by default — they are high-volume and rarely what an audit is about — with one exception: reading a **secret** is always logged.

| Category | Actions |
| - | - |
| `pipeline` | create, update, delete, version restore, destination type changes; `purge-source` (system event) when a [protected](/transformation/field-protection) run deletes its raw ingest object(s), with outcome `warning` if a delete failed |
| `tap` | create, update, delete, run (manual and scheduled), script and state changes |
| `secret` | read, write, delete |
| `config` | AI provider, embedding, and platform configuration changes |
| `job` | kill |
| `document` | upload (bursts are folded into one row with a count) |
| `code-repo` | repository connection changes |
| `auth` | login (success and failure), logout, password change |
| `user` | create (including the first-boot `admin` seed, with reason `bootstrap`), update (role change), reset-password (an admin cleared the password), rotate-password (startup rotation of an account with no password, with reason `no-password-hash`; recorded just before that boot's `system` / `start` entry), delete |
| `key` | issue, revoke, rotate; `issue` with resource `field-protection` (system event) when the server issues the environment's [field protection](/transformation/field-protection) hmac or encryption key on the first protected run; `rotate` with resource `field-protection` when an operator rotates the field protection encryption key or adds or changes an `enc.v<n>` through the secrets API; `retire` and `set-current` with resource `field-protection` when a `protect:admin` key removes an `enc.v<n>` or moves `encCurrent` through the secrets API (resource name lists the changed fields, never values) |
| `protect` | reveal: an [`encrypt`-protected value](/transformation/field-protection#revealing-a-value) read back, with the pipeline, the field name, and the counts requested, revealed, and failed (never a value); outcome `warning` when any value failed |
| `security` | any denied request — insufficient role, missing capability, expired session, and a non-`ui` key attempting to act on behalf of a user |
| `system` | server start and stop |
| `audit` | CSV export of the audit log itself |
| `mcp` | Clearing the Agent Monitor activity log |

Each entry also carries the outcome (`success`, `failure`, `denied`, or `warning` for an action that succeeded but needs follow-up, such as a pipeline delete that leaves a Unity Catalog table behind), the HTTP status, how long the request took, the caller's IP address (first hop of `X-Forwarded-For` when the UI sits behind a proxy) and user agent, and the name of the resource touched.

To record reads as well — every query, search, and listing an agent runs — set `AUDIT_LOG_READS=true` in `.env` and recreate the container. Expect a lot more rows: each query is its own entry, so filter by category when you're looking for changes rather than activity.

### What is never recorded

Passwords, password hashes, API-key values, session cookies, secret values, and credentials embedded in connection strings. Request bodies are not stored. Where an entry does carry extra detail (a role change records `from` and `to`, for example), every field whose name looks like a credential is masked before it is written. With user authentication on, entries made through the [Configuration assistant](/configuration-assistant) carry `via: config-chat` in their detail and are attributed to the signed-in admin.

## Reading it

**Configuration → Audit Log** shows the newest entries first with filters for time range, category, action, actor type, actor, outcome, and resource name. Click a row for the full entry. **Export CSV** downloads the current filter (up to 50,000 rows); the export itself is recorded as an `audit / export` event.

The same data is available programmatically:

```bash theme={null}
curl "http://localhost:8080/api/v1/audit-log?category=pipeline&action=delete&since=2026-08-01T00:00:00Z" \
  -H "x-api-key: $DATRIS_API_KEY"
```

Parameters: `since`, `until` (ISO-8601 or epoch millis), `category`, `action`, `actor` (label), `actorType`, `outcome`, `resource` (substring match), `limit` (default 100, max 1000), and `cursor` from the previous page's `nextCursor`. `/api/v1/audit-log/facets` lists the values seen in the last 30 days; `/api/v1/audit-log/export` returns CSV with the same filters.

Reading requires the `admin` role for a user session, or the `audit:read` capability for an API key. Full-access keys hold it implicitly; a scoped key must be issued with it. Entries cannot be edited or deleted through the API.

## Retention and long-term storage

Entries live in the platform's MongoDB config store (collection `{environment}-audit-log`) and expire after `AUDIT_LOG_RETENTION_DAYS` (default 90). Set it to `0` to keep them until you remove them yourself.

Every entry is **also written to the server log** on the logger `ai.datris.audit`, one line per event. Under the `production` Spring profile that line is a JSON object, so any log aggregator or SIEM already reading the container's output gets the full audit trail with no extra integration. That stream is the long-term record; Mongo is the queryable window. Turn the mirror off with `auditLog.emitLogLine: false` if you only want the database copy.

Two Prometheus metrics track health on the existing `/actuator/prometheus` endpoint: `datris_audit_events_total{category,outcome}` and `datris_audit_dropped_total`. By default (best effort) entries are written asynchronously and never slow a request; if the config store cannot keep up, the oldest pending entries are dropped and counted in `datris_audit_dropped_total`, and the Audit Log tab shows a warning. If you need no entry to be lost, turn on [strict mode](#strict-mode).

## Strict mode

By default the audit log favours availability: a request is never blocked, and an entry that cannot be written is dropped and counted. If you need the opposite trade, a request that cannot be recorded does not happen, set:

```bash theme={null}
USE_AUDIT_LOG=true
AUDIT_LOG_STRICT=true
```

and recreate the Datris container. `AUDIT_LOG_STRICT` does nothing while `USE_AUDIT_LOG` is off (`datris doctor` warns about that combination under `audit.strict`).

With strict mode on:

* **Nothing is evicted.** A full queue never pushes an older entry out, and `datris_audit_dropped_total` stays at 0.
* **A failed write is retried, not skipped.** While the config store (MongoDB) is down, the writer retries the same entry with backoff and keeps the rest of the queue in order. When the store comes back, every queued entry is written once, in the order it was accepted.
* **Audited requests are refused while the queue is backed up.** Once the pending queue reaches its high-water mark (9,000 of 10,000 entries), every request that would be audited (writes; reads too with `AUDIT_LOG_READS=true`; secret reads always) gets HTTP **503** with a `Retry-After` header and a JSON message naming `AUDIT_LOG_STRICT`, before the request is handled. Tap callbacks and HTTP ingest triggers fail with the same 503. Triggers that do not come over the API wait instead: the MinIO bucket webhook answers 503 and MinIO redelivers the event later (this relies on MinIO's webhook event queue: the bundled `minio-init` sets `queue_dir` with a limit of 1,000 undelivered events, past which MinIO drops events, and a MinIO you configured yourself without `queue_dir` does not redeliver at all), the Kafka consumer stops reading (no offset is committed past an unprocessed message), and scheduled tap runs are skipped until the queue drains, then fire on the next tick. Unaudited reads keep working, and `GET /api/v1/audit-log/status` is never refused, so the UI banner and scripts can see the state (that status read is not recorded while the gate is closed). Requests are accepted again as soon as the queue drains.
* **An entry that still cannot be queued is counted.** A request refused by authentication or authorization while the gate is closed is not held: its denial entry is counted straight away. If a request already past the gate waits more than a few seconds for room, its entry is counted in `datris_audit_unrecorded_total` and written whole at ERROR level on the `ai.datris.audit` logger, so the log stream still has it.

`GET /api/v1/audit-log/status` reports `strict`, `unrecorded`, and `acceptingWrites` alongside `dropped` and `queueDepth`.

**Residual risk.** Accepted entries wait in an in-memory queue until the writer stores them. A hard kill of the server (out-of-memory kill, `docker kill`, host crash) loses entries that were accepted but not yet written; the `ai.datris.audit` log line is emitted when an entry is written, not when it is accepted. A normal stop (`docker stop`, `docker compose stop` or `down`) flushes the queue first. The flush is bounded to 10 seconds, even when the config store hangs rather than fails: entries it cannot store in that time are counted in `datris_audit_unrecorded_total` (strict mode), and, under strict mode or whenever an entry was left behind, the total is printed to the container's stderr as `audit-unrecorded total=N at shutdown`, so it survives the server's logging shutdown. The Kafka consumer stops before the flush, and an entry submitted after the flush (for example by a tap run still finishing) is refused and counted the same way. Compose gives the server 30 seconds to stop before killing it.

## Cross-referencing the Agent Monitor

Audit entries produced by an MCP agent carry the same key label the Agent Monitor displays, so you can match a tool call in the monitor to the platform-side record of what it did. The audit log is the durable record that something happened; the monitor is the ephemeral detail of the tool call, kept only while the MCP server is running.


# Changelog
Source: https://docs.datris.ai/changelog

Release history for Datris

<Update label="v1.50.0" description="October 15, 2026">
  **Busy hosts stay up, and Claude Opus 5.5 becomes the default chat model.**

  * **Busy hosts stay up.** Tap scripts and AI rule scripts now run a bounded number at a time, inside CPU, memory and process limits. When the runner is full, a run waits its turn instead of piling on; its run history shows how long it queued, and `datris doctor` warns when runs had to wait or were turned away. All limits are adjustable in `.env`.
  * **Claude Opus 5.5 is the default chat model for Anthropic.** New installs on the Anthropic and Bedrock paths seed Claude Opus 5.5 for the Assistant and other chat features (it was already the code-generation default), and the model picker recommends it. Existing installs keep the model they have; change it any time on Configuration → AI Providers.

  **Upgrading:** `docker compose pull && docker compose up -d`; the server, UI, MCP server and script runner images changed. The new compose file gives both script runners CPU, memory and process ceilings and each script a 1 GB memory limit (ceilings, not reservations). Raise them in `.env` for scripts that need more; see [Tap Execution & Isolation](/tap-execution-isolation#configuration). No other configuration changes are required.
</Update>

<Update label="v1.49.0" description="October 14, 2026">
  **Bring a restored catalog back to life in one click, and let assistants read whole scripts.**

  * **Enable a restored catalog in the right order.** Taps that read tables produced by other taps in the same catalog are recognised from their scripts (or declared by you), and each catalog can show the order to enable its taps in, with what still blocks each one. Disabled taps on the Catalog tab show their level or the sources they need, **Run now** lands a tested tap's data without enabling it, and the restore result says how many restored taps depend on others. Agents get the same enable order through a new read-only MCP tool.
  * **Enable a whole catalog in order with one click.** A new **Enable in order** action on a catalog's card tests, enables and runs each tap in its enable order, waiting for each tap's data to land before starting the taps that read it. Taps whose sources failed are skipped with the reason, progress shows on the card, and the job can be cancelled. Agents can start the same job, and an approval-required agent policy asks for one approval for the whole catalog.
  * **Updated object store image.** The bundled MinIO image moves to a newer build with current security patches. Run `docker compose pull && docker compose up -d --force-recreate minio minio-init`; buckets and data are kept.
  * **In-app assistants read whole tap scripts.** In-app assistants can now read whole tap scripts and larger tool results, so they can review and fix a long script without asking you to paste it. A tap version's script now also shows for taps stored in a code repository.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate minio minio-init`; the server, UI, MCP server and object store images changed. No configuration changes are required; the new endpoints, tools and settings are additive.
</Update>

<Update label="v1.48.0" description="October 13, 2026">
  **Back up a catalog as one file and restore it anywhere.**

  * **Download a catalog.** The download icon on a catalog's card in the Catalog tab saves the whole catalog as a zip archive: every tap and pipeline definition, each tap's script and incremental state, and the validation schemas its pipelines reference. Secret values, destination data, run history and version history are never included.
  * **Import a catalog.** The **Import catalog** button restores an archive on the same install or another one, under the original name or a new one. A restore never merges into an existing catalog: clashes are refused and listed with nothing changed. Choose **Overwrite catalog** and type its name to replace an existing catalog's contents, or skip the incremental tap state to make a clone that starts from scratch.
  * **Secrets stay yours.** The archive lists the secret names the catalog needs, and the restore result names any missing on the target. Inline REST bearer tokens and API keys on a pipeline are removed on export; re-enter them after the restore.
  * **Restored taps start disabled.** Taps come back with a **Disabled** badge in place of their schedule; click it to enable the tap or to test it first.
  * **CLI and agents.** `datris catalog export` and `datris catalog restore`, plus the `export_catalog` and `restore_catalog` MCP tools, with the same refuse-on-clash and confirm-to-overwrite rules.

  **Upgrading:** `docker compose pull && docker compose up -d`; the server, UI and MCP server images changed. No configuration changes are required; the new endpoints and tools are additive.
</Update>

<Update label="v1.47.2" description="October 12, 2026">
  **See which AI model is answering.**

  * **The Assistant names its model.** The top of the Assistant conversation shows the chat provider and model in use (for example "Anthropic · claude-fable-5-1 · extended thinking on"), and the line stays visible after the first message. Administrators get a link to change it on Configuration → AI Providers; the line updates as soon as the configuration is saved.
  * **Generated tap scripts say who wrote them.** When the Assistant or an agent generates a tap script, the result reports which CodeGen provider and model produced it. The tap generation status carries the provider and model from the first poll.
  * **Consistent provider names.** The Assistant, tap generation and the Configuration tab use the same display names for AI providers.

  **Upgrading:** `docker compose pull && docker compose up -d`; the server, UI and MCP server images changed. No configuration changes are required; the additions are new fields only.
</Update>

<Update label="v1.47.1" description="October 11, 2026">
  **Server framework update.**

  * **Current, supported server framework.** The server now runs on Spring Boot 4, Spring Framework 7 and Tomcat 11, a line that still receives security fixes. This closes two open vulnerability reports against the web framework; neither affected path is used by Datris. The API, responses, sign-in sessions, Assistant streaming, metrics and audit log behave exactly as before.

  **Upgrading:** `docker compose pull && docker compose up -d`; only the server image changed. No configuration changes are required.
</Update>

<Update label="v1.47.0" description="October 9, 2026">
  **Single sign-on through your identity provider, and security updates.**

  * **Sign in with SSO.** With user authentication on, people can sign in through your identity provider (Okta, Microsoft Entra ID, Google Workspace, Keycloak, Auth0, or any OpenID Connect provider) from a new **Sign in with SSO** button on the login screen. OpenID Connect authorization code flow with PKCE, so MFA, password rules and lockout come from your provider. SSO users are ordinary Datris users with one of the three roles; sessions, role checks, the Users screen and the audit log work as for password users. API keys, the CLI and MCP clients are unchanged. See [Single sign-on (OIDC)](/user-auth#single-sign-on-oidc).
  * **You decide who gets in.** Users already on **Configuration → Users** sign in with their role; anyone else is refused unless `OIDC_DEFAULT_ROLE` is `viewer` or `editor`. The built-in `admin` account is never reachable through SSO. The first sign-in binds the user to the provider's subject id; a different identity with the same username is refused.
  * **Never in the way of password login.** A wrong or incomplete setting switches SSO off with one ERROR line and the server starts as usual; an unreachable provider fails the SSO button with a clear message while password login keeps working, and SSO recovers on its own.
  * **Audited.** Every SSO sign-in and refusal is an `auth` / `login` entry with `method: oidc`.
  * **Security updates.** Refreshed server and UI build dependencies to clear the open vulnerability reports. No behaviour change.
  * **MinIO health check** no longer depends on a config file inside the container.

  **Upgrading:** refresh the compose file (`git pull` or re-download `docker-compose.standalone.yml`), then `docker compose pull && docker compose up -d`; the server and UI images changed. SSO is off by default. To turn it on, set `USE_USER_AUTH=true`, `OIDC_ENABLED=true`, `OIDC_ISSUER`, `OIDC_CLIENT_ID` and `OIDC_REDIRECT_URI`, put the client secret in Vault under `oss/oidc` (key `clientSecret`), and recreate the server. A secret named `oidc` is now platform-managed (taps and pipelines cannot reference it). The version endpoint gains `oidcEnabled`. SSO is single-tenant; no SAML or provider group-to-role mapping yet.
</Update>

<Update label="v1.46.0" description="October 8, 2026">
  **Strict evidence mode, a complete record of password changes, a data-flow matrix, and graceful shutdown.**

  * **Strict evidence mode.** `PROVENANCE_STRICT=true` makes a stamping pipeline fail its run instead of loading rows it could not stamp (XML sources are exempt and say so). `AUDIT_LOG_STRICT=true` refuses an audited request with a clear 503 while the audit log cannot record it, instead of dropping an older entry; bucket drops, Kafka messages and scheduled tap runs wait the same way, and everything replays in order when the store returns. Both default to off. `datris doctor` reports the mode; the audit-log status endpoint shows `acceptingWrites`. See [Audit Log](/audit-log#strict-mode) and [Provenance](/provenance#strict-mode).
  * **Password resets and rotations are audited.** `reset-password` names the account and the acting session or key; role changes carry the new and previous role; the startup bootstrap rotation is `rotate-password` with the `system` actor (the password never enters the audit log); the first-boot `admin` seed is `create` with reason `bootstrap`. See [Recovering admin access](/user-auth#recovering-admin-access).
  * **Data Flows.** A new [page](/production/data-flows) lists, for each kind of data, every place it can go and the switch that changes it, with a no-cloud-exit recipe. Evidence claims across the docs now say when they hold.
  * **Graceful shutdown.** `docker stop` now reaches the server: requests drain, the Kafka consumer closes cleanly, queued audit entries are written, exit code 143. Previously every stop was a hard kill after ten seconds.

  **Upgrading:** `docker compose pull && docker compose up -d`; only the server image changed. Refresh the compose file for the new 30 s stop grace period. Stops are now graceful (exit 143, usually 1-3 s; up to \~12 s with the audit log on and the store down). Strict mode is opt-in; with both switches off, behaviour is unchanged. Audit action names changed: password reset is `reset-password` (was `update`), first-boot seed is `create` + reason `bootstrap` (was `seed-admin`); existing rows unchanged. New: status fields `strict`/`unrecorded`/`acceptingWrites`, doctor checks `audit.strict`/`provenance.strict`, metric `datris_audit_unrecorded_total`.
</Update>

<Update label="v1.45.0" description="October 8, 2026">
  **AI rule and transformation scripts are generated once, stored with your pipeline, and run in an isolated container.**

  * **One script per instruction, written when you save.** Saving a pipeline with an AI data-quality rule or an AI transformation generates its Python script then and stores it. Every run executes that stored script: an unchanged instruction behaves the same on every run, runs make no model call, and a model outage no longer stops ingestion. A later save regenerates only when the instruction or the source schema changed; pipelines created before the upgrade generate and store on their first run. See [AI rules](/data-quality/ai-rules#when-the-script-is-generated) and [AI transformation](/transformation/ai-transformation#when-the-script-is-generated).
  * **Read, review and replace the script.** The save response, a new API endpoint and the Lineage panel show each script, when it was generated and by which model. Regenerate on demand; a failing script is never silently replaced, and the run error says when it was generated and how to replace it.
  * **Scripts in your code repository.** With a code repository configured, pipeline scripts are committed under `pipelines/<pipeline>/` next to your taps, and runs execute the recorded commit. Hand edits are adopted deliberately with one call; a save that would overwrite one is refused with a warning, and runs keep the recorded commit until you resolve it. See [Pipeline scripts](/tap-github-storage#pipeline-scripts).
  * **Generated scripts run outside the server.** On compose, they execute in a new hardened container, `datris-codegen-runner`: no platform secrets, no server filesystem, no route to the database, object store, secrets store, internet or the Datris API; a dedicated scratch volume emptied after every run. `datris doctor` reports the mode. See [Execution isolation](/tap-execution-isolation#generated-data-quality-and-transformation-scripts).
  * **Blank AI instructions are ignored** instead of calling the model.
  * **Doctor covers the new container** in the environment-drift check and a new `codegen.isolation` row.

  **Upgrading:** refresh `docker-compose.yml` (`git pull` or re-download `docker-compose.standalone.yml`), then `docker compose pull && docker compose up -d`. All four images changed; one new container, one new internal network (`codegen-net`) and one new named volume (`codegen-scratch`) appear; no `.env` change. Pulling images without refreshing the compose file keeps scripts in-process with a startup and doctor warning; refreshing the compose file without pulling fails loudly on every AI rule and transformation, never silently in-process. Scripts are now written from column names and types, not sample rows. A failing script stays until you change the instruction or regenerate. Scripts that reached the network or server files now fail; `USE_CODEGEN_RUNNER=false` restores in-process execution. Keep `codegen-net` `internal: true`.
</Update>

<Update label="v1.44.1" description="October 6, 2026">
  **`datris doctor` tells you when a newer version is available.**

  * **Know when you are behind.** A new `version.update` row reads "up to date" or names the newer version and the upgrade command. To find out, doctor tells datris.ai your Datris version, operating system and CPU architecture, and nothing else, each time you run it. Never from the server, on a timer, or during `--pre-upgrade`; nothing is stored on your machine. See [Doctor](/doctor#update-check).
  * **Off with one line:** `DATRIS_UPDATE_CHECK=0` in `.env` (run doctor from the install directory or pass `--project-dir`, or set it in the environment). A fresh install prints a short notice, and the installer writes the line when given `DATRIS_UPDATE_CHECK=0`.
  * **Never in the way.** If datris.ai cannot be reached within three seconds the row is skipped and doctor's result is unaffected.
  * **How to verify:** [Security Architecture](/production/security-architecture#no-telemetry) lists both outbound requests Datris can make and commands to confirm there are no others.
  * **Security updates.** Refreshed the server, UI and tap-runner dependencies and the tap-runner base image to clear the open vulnerability reports. No behaviour change.

  **Upgrading:** run `docker compose pull && docker compose up -d --force-recreate` (all four images changed). The update check is on after you upgrade the CLI (`pip install -U datris-mcp-server` / `brew upgrade datris`), so each `datris doctor` run now makes one request to datris.ai; add `DATRIS_UPDATE_CHECK=0` to `.env` to keep it off. On a deployment behind the latest release, `datris doctor` now exits 1 where it exited 0. `datris doctor --pre-upgrade` is unchanged.
</Update>

<Update label="v1.44.0" description="October 6, 2026">
  **One switch to turn on the governance controls, a doctor check that reports when they are off, and corrected first-login instructions.**

  * **Switch on the governance controls in one step.** A fresh install takes `DATRIS_GOVERNED=1` to turn on user login, API keys, the audit log and the agent policy; an interactive install asks once, defaulting to no. An existing install adds four documented lines to its `.env`. Nothing changes for installs that do not ask for it. See [Quick Start](/quick-start#2-switch-on-the-governance-controls).
  * **`datris doctor` reports the governance controls.** A new `governance.controls` row names any control that is off, with the lines to add. A warning, never an error. See [Doctor](/doctor).
  * **First-login instructions corrected.** `admin` is created with a bootstrap password printed once to the server log, not a blank password. The docs and login screen say where to read it and to save it before recreating the container. A new [Recovering admin access](/user-auth#recovering-admin-access) section covers the case where it is gone.
  * **Fixed: a headless install on Debian or Ubuntu stopped silently** right after fetching the runtime files. It now continues.
  * **Governed installs skip a check that needs a key.** The installer no longer waits on the post-boot store check when API keys are enforced; it prints the command to run once a key exists.
  * **Docs and website wording** now say the controls are switched on for production rather than "on by default".

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate`. The server, UI and MCP server images changed. No defaults change. `datris doctor` now exits 1 on an install with any of the four controls off, where it exited 0. Before turning the controls on, save the `admin` bootstrap password from the server log (`docker compose logs datris | grep "Bootstrap login"`); if nothing is printed, see [Recovering admin access](/user-auth#recovering-admin-access).
</Update>

<Update label="v1.43.0" description="October 5, 2026">
  **A HIPAA Safe Harbor preset, a switch that keeps row values away from AI helpers, clearer errors, and an installer agents can run.**

  * **HIPAA Safe Harbor preset for field protection.** Datris recognises the Safe Harbor identifier classes from field names with a fixed table and no AI call, proposes a method for each, and lists what needs a person's review. When enforced, a pipeline cannot be saved with an unprotected identifier and new columns are protected before they land. Available in the wizard, on the pipeline page and to agents. An aid, not a certification. See [Field Protection](/transformation/field-protection).
  * **One switch so helpers send no row values to a model.** With `DATRIS_AI_SAMPLE_VALUES=false`, schema generation, profiling, Assistant attachments and AI rule and transformation generation work from structure only. The default is unchanged, and `datris doctor` reports the setting.
  * **A new mask option** keeps the first three characters.
  * **Schema generation and profiling no longer fail when the model declines.** You get a usable result with a note.
  * **An invalid pipeline is reported as an invalid request**, with the same message, shown as plain text.
  * **Failed runs show the error, not a Java stack trace.** The trace is behind a toggle in the Ops activity view and in the server log.
  * **Fixed: a column the destination already declares is no longer added twice** when it later arrives as a new source column.
  * **Fixed: editing a pipeline in the wizard keeps its protection settings.**
  * **Install Datris from an agent.** See [Install for Agents](/install-for-agents). The installer also stops clearly when no AI provider is given and cleans up after a failed first run.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate`. The server, UI and MCP server images changed. A refused pipeline save now returns HTTP 400 where it returned 500, with the same message. A failed run's stack trace moved from the status description to a separate detail field. To use `DATRIS_AI_SAMPLE_VALUES` on an existing install, add it to your `.env`.
</Update>

<Update label="v1.42.0" description="October 3, 2026">
  **Field protection for PII and PHI, reveal for authorized users, and tighter rules for secrets.**

  * **Protect sensitive columns before they land anywhere.** Mark a field and its original value never reaches a destination, a preview, or an AI step. Choose a stable one-way pseudonym, a mask that keeps the part you pick, a fixed redaction, dropping the column, or encryption. Protection runs on your server before data quality and transformation, and the raw copies Datris held for the run are removed afterwards. See [Field Protection](/transformation/field-protection).
  * **Encrypt what an authorized person may need to see again.** Encrypted fields are stored as ciphertext. Someone with the reveal permission can get the original back; every reveal is audited and no user ever holds a key. The permission is in no built-in role or key template. Keys can be rotated without losing older rows.
  * **Reveal from the Search tab.** Encrypted values show as a lock in results for Databricks, Snowflake, object store, PostgreSQL and MongoDB, and an admin can reveal the current result with one click. Nothing is kept afterwards. Assistants and agents cannot reveal.
  * **Suggestions for what to protect.** Datris proposes a method per field from its name and type alone. No data is sent and nothing is applied until you accept it.
  * **Protection in the wizard and for agents.** The schema step has a Protect column with an explanation of each option. Agents can set protection and ask for suggestions, and confirm the fields with you first. Lineage shows protected and dropped columns.
  * **Security: stronger protection for stored secrets.** Taps can now only use tap secrets, the keys Datris manages for itself cannot be used by a tap or pipeline, and secret names are checked more strictly. We recommend upgrading.
  * **Databricks loads are no longer reported as failed when they succeeded.**
  * **MongoDB results read as a table in Search** when the documents are flat.
  * **`datris doctor` checks tap secrets.**

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate`. The server, UI and MCP server images changed. A tap whose secret is not a tap secret now fails its next run and cannot be saved; create a tap secret for it, or set `DATRIS_TAP_SECRET_SCOPE=any` to keep the previous behaviour. Secret names with spaces or URL-special characters are refused. Protected pipelines delete their source file after a run unless they opt out. See [Taps](/taps#taps-can-only-use-tap-secrets).
</Update>

<Update label="v1.41.1" description="October 2, 2026">
  **Security dependency updates and a skill that teaches your coding agents to use Datris.**

  * **Security: updated third-party libraries.** Dependency updates close several published vulnerabilities, including high-severity ones, in libraries the server and the UI build use. No Datris behaviour changes. Shared installs should upgrade soon.
  * **A skill for your project's coding agents.** The new `datris-platform` skill tells Claude Code, Codex, Hermes, OpenClaw and similar agents to use Datris for data work in your own projects, with MCP tools first and the REST API only for code written into the project. See [Agent Skill](/agent-skill) for install commands.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate`. The server, UI and MCP server images changed. No configuration changes are required.
</Update>

<Update label="v1.41.0" description="October 2, 2026">
  **Iceberg tables that work on Databricks, Unity Catalog metadata on by default, and a security fix for API keys.**

  * **Iceberg pipelines can now target Databricks Unity Catalog.** A new managed mode lets the catalog choose where the table lives while Datris writes to it through the catalog, so Iceberg output on Databricks is governed, queryable and kept current after every run. The pipeline status shows the table and its location. Deleting the pipeline leaves the catalog's data in place.
  * **Unity Catalog metadata is on by default for Databricks pipelines.** Descriptions, tags, properties and lineage are written without a per-pipeline opt-in. A pipeline can still turn it off explicitly, and a single environment setting restores the previous opt-in behaviour. The status endpoint tells you where the setting came from.
  * **Clearer guidance when a catalog cannot do what you asked.** Deleting a catalog reports any Unity Catalog cleanup it could not complete, and saving a pipeline that points an Iceberg register mode at Databricks warns right away and names the mode that works.
  * **Safer Iceberg history for adopted tables.** Tables Datris adopts from an existing catalog keep a longer metadata history and record where they came from.
  * **Security: revoked API keys are now rejected immediately.** Revoke a key and every request made with it is refused from that moment, and the refusal is recorded in the audit log under the key's name. We recommend upgrading.
  * **API errors are returned as JSON.** Server errors no longer come back as raw text; clients, the UI and the CLI get a short, readable message. Agents calling tools over MCP see failures flagged as errors instead of ordinary results, and the CLI explains when its key was rejected.

  **Upgrading**

  Run `docker compose pull && docker compose up -d --force-recreate`. The server, UI and MCP server images changed. Databricks pipelines that never configured Unity Catalog now get metadata written after each load; set `DATRIS_UNITY_CATALOG_DEFAULT=disabled` to keep the old opt-in behaviour, or add `unityCatalog: {enabled: false}` to an individual pipeline. See [Unity Catalog](/destinations/unity-catalog).
</Update>

<Update label="v1.40.0" description="October 1, 2026">
  **Unity Catalog integration for Databricks, warehouse queries in the Search tab, and less to configure for Databricks pipelines.**

  * **Datris annotates your Databricks tables in Unity Catalog.** Turn on Unity Catalog for a pipeline and, after every successful load, its table gets a description, stable tags (which pipeline loaded it, its data-quality status, managed by Datris) and run details in the table properties, all visible in Catalog Explorer. Nothing is written unless a pipeline opts in, and a metadata problem is reported as a warning while the load still succeeds.
  * **Lineage appears in Unity Catalog too.** The same opt-in publishes source → pipeline → table lineage, with column mappings, to the table's Lineage tab.
  * **Agents can look around Unity Catalog before building a pipeline.** A new tool lists the catalogs, schemas, tables and columns a connection can see, and data discovery can include warehouse tables in its results, marking the ones Datris loaded.
  * **Iceberg tables can be registered with, or committed through, an Iceberg REST catalog.** Object-store pipelines writing Iceberg can keep an external catalog current. Databricks Unity Catalog does not yet accept tables at a location you choose; Datris detects this, warns, and keeps writing safely. Use the Databricks destination for governed Databricks tables.
  * **Query Databricks and Snowflake from the Search tab.** Pick a pipeline, run read-only SQL and see the results in place. Error messages across every query type are now short and specific.
  * **Less to configure for Databricks.** A pipeline no longer needs the SQL warehouse in its configuration when the connection secret provides it, and assistants stop asking for it.
  * **Deleting a pipeline tells you when a catalog still references its table.** The CLI shows the same notice. Its keep-data option, which never kept data, now refuses instead of deleting silently.
  * **Editors and viewers can run read-only queries** against object-store, Snowflake and Databricks destinations.
  * **Fix: uploads to pipelines that leave out the CSV delimiter no longer fail.**
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. All four images changed. No configuration changes are required; Unity Catalog features stay off until a pipeline opts in. To write tags and lineage, the Databricks service principal needs two additional grants. See [Unity Catalog](/destinations/unity-catalog).
</Update>

<Update label="v1.39.1" description="September 29, 2026">
  **Security maintenance release.**

  * **Security update.** The JSON processing library used by the server is updated to close three recently disclosed advisories, one rated high. No functional changes.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. Only the server image changed. No configuration changes required.
</Update>

<Update label="v1.39.0" description="September 28, 2026">
  **Catalogs can now be renamed, and deleting one asks whether to keep or delete its items.**

  * **Rename a catalog in place.** A named catalog can be renamed from the Catalog tab; every tap and pipeline in it moves to the new name in one step. If API keys were limited to the old name, you are told which ones to update.
  * **Deleting a catalog keeps its items by default.** You choose between keeping the items (they move to Uncataloged) and deleting them together with their data, which requires typing the catalog name. Items that could not be changed are listed with the reason.
  * **Catalog names may now use capital letters, when created or renamed.**
  * **Renaming to an existing name is refused.** Renaming a catalog to a name that already exists is refused; use Move all contents to combine catalogs.
  * **No near-duplicate names.** Creating a catalog whose name differs from an existing one only by capitalisation is refused.
  * **Uncataloged is protected.** It can no longer be renamed or deleted as a group; its items are deleted one at a time.
  * **The assistant can rename catalogs.** Ask the Catalog assistant, or any connected agent, to rename or remove a catalog and it does so in one step. Agents can only remove a catalog while keeping its items. See [Data Catalog](/data-catalog).
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. The server and UI images must be updated together: an older server does not support catalog rename or delete from the new UI.
</Update>

<Update label="v1.38.3" description="September 27, 2026">
  **Clearer errors for files whose names cannot be parsed.**

  * **The real ingest error is reported.** When an uploaded file's name cannot be matched to a pipeline, the log now says so instead of reporting an unrelated internal error. Files that fail later in ingestion still record their status exactly as before.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. Only the server image changed. See [Object-store ingestion](/ingestion/object-store).
</Update>

<Update label="v1.38.2" description="September 26, 2026">
  **Object-store ingestion is more forgiving: a bad file no longer blocks its neighbours, and root-level uploads work.**

  * **One bad file no longer stops the other files uploaded at the same time.** When several files land in the raw bucket together and one of them cannot be read, the others still run in the same pass. The failure is logged with its bucket and file name.
  * **Uploads dropped in the root of the raw bucket are now ingested.** A file placed directly in the bucket, with no folder prefix, is picked up like any other instead of failing with a naming-convention error.
  * **Every file in a multi-file event is ingested.** An event carrying several files for the same bucket now processes each distinct file; previously only one of them was kept.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. Only the server image changed. See [Object-store ingestion](/ingestion/object-store).
</Update>

<Update label="v1.38.1" description="September 25, 2026">
  **Installs no longer depend on a third-party registry staying public.**

  * **MinIO ships from the same place as the rest of the platform.** The object store image is now published alongside the other Datris images, so a fresh install or `docker compose pull` no longer depends on a third-party registry staying public. Existing object-storage data is kept.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. The `minio` and `minio-init` containers are recreated because the image changed; object-storage data is kept.
</Update>

<Update label="v1.38.0" description="September 25, 2026">
  **Change your platform's settings by asking: every change waits for your Confirm, and secrets never touch the chat.**

  * **A Configuration assistant.** The Configuration tab has an Ask panel at its right edge that reads and changes what the sub-tabs cover: AI providers, secrets, data sources, the code repository, users, API keys and agent policy. It proposes one change at a time on a card, and nothing is written until you click Confirm; a confirmation works once and expires after ten minutes. Credentials are entered in a masked form on the card, never in the conversation. A new user's temporary password or a new API key is shown once on the card, with a copy button. The panel is for admins, needs a signed-in session when user auth is on, and is not shown on trial installs. See [Configuration Assistant](/configuration-assistant).
  * **Ask the audit log.** Ask the assistant who changed what and when in plain language, and it answers from the audit entries themselves.
  * **Doctor from chat.** Ask the assistant to run the health checks, optionally with a test request to each AI provider, and it summarizes what needs attention.
  * **Deep links to Code Repository.** A link that opens Configuration on the Code Repository sub-tab now lands there instead of on the default sub-tab.
  * **Changes made through the assistant are marked in the audit log.** With user auth on, changes made through the assistant are attributed to the signed-in admin and marked `via: config-chat` in the audit log. See [Audit Log](/audit-log).
  * **More chats at once.** Several chats can stream at the same time without one failing on a connection-pool timeout; the limits are configurable, see [Configuration Reference](/configuration-reference).
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. Update all images together, including the UI. No configuration changes. With API keys and user auth on, the assistant is available in the UI only.
</Update>

<Update label="v1.37.1" description="September 24, 2026">
  **Installs and upgrades work again after MinIO closed its public image registry.**

  * **MinIO images now come from Chainguard.** MinIO withdrew its images from Docker Hub earlier this month and, as of September 24, put its quay.io images behind a login, so every fresh install and every `docker compose pull` failed with "unauthorized: access to the requested resource is not authorized". The Compose files now pull a multi-arch MinIO build from Chainguard's public registry, pinned to a digest. Running deployments were never affected. The installer, the standalone Compose file and a fresh clone all pick this up automatically, and existing object-storage data is kept.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`.
</Update>

<Update label="v1.37.0" description="September 24, 2026">
  **Agents get live, validated reads from any source, and only land what's worth keeping.**

  * **The scratch destination is now called Live Read.** Same behaviour, same configuration; only the name changed in the wizard, the Pipelines list, the run detail page, the docs, the Assistant and the CLI. The first mention on each surface still says which JSON key it maps to, so nothing you have written needs to change. See [Live Read](/destinations/live-read).
  * **The Assistant offers Live Read.** When it asks where data should go, it now names Live Read alongside the structured destinations, with a one-line description, and still never picks it unless you do. It leaves Live Read out when you have asked for a schedule.
  * **The docs page moved.** Live Read has its own page under Destinations; the old address redirects.
  * **The CLI signs in to the MCP server.** On installs with API keys turned on, every CLI command now sends the key you already export for `datris doctor`. A missing key gives a one-line message instead of a connection error, and a rejected key gives a one-line message and a non-zero exit instead of raw output and success. The query-string form still works. See [CLI](/cli).
  * **A clean exit on every command.** The CLI no longer prints a stray traceback after a successful command, and commands that make several calls no longer reuse a stale connection.
  * **MCP clients see the real version.** The MCP server now reports its own version instead of the library it is built on.
  * **Claude Opus 5.5 and Grok 4.7.** Both are in the model catalog. Opus 5.5 is the recommended and first-boot default for code generation on Anthropic and Bedrock; Grok 4.7 is the recommended and first-boot default for Grok. The chat default is unchanged.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. Update all images together, including the UI and the MCP server. Existing installs keep their models; the new defaults apply only on first boot. If you use API keys and run the CLI, export `DATRIS_API_KEY` once.
</Update>

<Update label="v1.36.0" description="September 23, 2026">
  **Taps handle sources of tens of millions of rows: faster batches, honest timeouts, and an estimate before anything is built.**

  * **A tap can yield batches.** `fetch()` may yield DataFrames or Arrow record batches as well as single records. Batches are written natively, roughly ten times faster than one record at a time, and land exactly what yielding the same rows one by one would land: integers stay integers, nulls stay null, timestamps keep their precision. Frames read with the pyarrow backend are the fastest of all. Scripts that yield or return records are unchanged. See [Taps](/taps).
  * **Tests preview, runs run.** A tap test started from the Assistant, an MCP client or the CLI now stops after 20 records, exactly like the UI's Test button, instead of streaming the whole source and timing out. The tools say plainly that a test's record count is not what a run will produce. Pass a limit of 0 to test the whole source.
  * **Real runs get their own timeout.** Tests still stop at five minutes; real and scheduled runs now get an hour by default. Both are settable, and a timed-out run says which mode it ran in and which setting to raise. A mistyped value warns at startup and falls back instead of stopping the server.
  * **A timed-out run keeps its logs.** The error now ends with how many records were streamed before the kill and the last lines the script printed, and the run log holds the script's full output. A script that yields prints progress every 100,000 records, so a slow-but-healthy run reads as slow, not broken.
  * **Agents estimate before they build.** For a large source the Assistant and the MCP tools show rows, bytes per record, disk and minutes, compare them against the budgets actually set on your install rather than the documented defaults, and when one is exceeded stop and offer three choices: raise it, narrow the window, or chunk the range. Column projection is named as the lever that cuts both numbers.
  * **Malformed schedules are refused at save.** A 5-field Unix cron is rejected with the 6-field equivalent to use, with weekday numbers translated. Taps already stored with a bad schedule log once instead of every 30 seconds.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. Update all images together, including the UI and the MCP server. Two new optional settings, `TAP_SCRIPT_TIMEOUT_SECONDS` (300) and `TAP_RUN_TIMEOUT_SECONDS` (3600); see [Configuration](/configuration-reference).
</Update>

<Update label="v1.35.0" description="September 21, 2026">
  **GPT-6 Astra support, and an Assistant that stays connected through long tool calls.**

  * **GPT-6 Astra works everywhere on the OpenAI provider.** Pick or type `gpt-6-astra` for the Assistant or for code generation and it works for chat, tap script generation, AI data-quality rules, AI transformations, schema generation, brainstorming and natural-language queries; previously every non-chat call to it failed. It appears in the model dropdowns; GPT-5.6 Sol remains the recommended OpenAI model. See [AI Configuration](/ai-configuration).
  * **OpenAI reasoning models show their reasoning.** Every Assistant turn on an OpenAI reasoning model now asks for a reasoning summary, so a thinking block appears where before there was none. Models or accounts that decline the summary fall back to the previous behaviour.
  * **The Assistant no longer stops silently during long tool calls.** A tool call that ran for several minutes could end the conversation mid-turn with no message. The connection now stays alive for as long as the work runs, and if it is ever cut, the chat says so. The same applies to the Ops, Catalog and Search chats.
  * **Tap scripts stream their sources instead of downloading them.** The AI helpers, the Assistant and the documentation now state that a tap has only a small amount of scratch space and must read a remote file directly rather than save it first, and that a run's size should be estimated against the per-run disk budget before anything is built. See [Taps](/taps).
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. Update all images together, including the UI and the MCP server. No configuration changes required.
</Update>

<Update label="v1.34.1" description="September 20, 2026">
  **Taps stream large sources, and failures say what went wrong.**

  * **A tap can yield its records.** `fetch()` may return a generator or any iterator instead of a list; records are written to disk as they are produced, so a tap over millions of rows runs in a few tens of megabytes of memory. Scripts that return a list are unchanged. See [Taps](/taps).
  * **A tap killed for memory says so.** The error explains that the script was killed, almost certainly out of memory, and that `fetch()` should yield records instead of building the whole result. The Assistant and MCP guidance carry the same rule, stop to report after two failed tests, and no longer probe the runner environment.
  * **Rows returned as lists land in CSV pipelines.** A tap returning each row as a list with a header row now loads into a CSV pipeline with the header applied and names normalised, or fails with a message naming the problem, instead of corrupting the pipeline schema.
  * **A bad first line no longer changes the pipeline.** A CSV upload or object-store pickup whose header does not parse fails before schema evolution, leaving the schema and version untouched. Quoted header names resolve to their plain names.
  * **A blank schedule is no schedule.** Saving a tap with an empty schedule stores none and the scheduler stays quiet; taps already saved that way are quiet on upgrade.
  * **Testing a tap that yields stops at the test limit** instead of draining the source.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. Update all images together, including the MCP server. No configuration changes required.
</Update>

<Update label="v1.34.0" description="September 20, 2026">
  **A pipeline run is no longer limited by memory.**

  * **Payload size no longer limits a run.** Files uploaded over the API, archives uploaded as a batch, files picked up from the object store and records returned by taps all stream to disk as they arrive instead of being held in the server's memory. A multi-GB upload or tap run now completes on the default 2 GB heap, and memory use no longer grows with the size of the data passing through.
  * **The tap output cap becomes a per-run disk budget.** `PIPELINE_MAX_PAYLOAD_MB` (default 4096 MB, `0` = unlimited) is the most one run may stage on disk; a run that exceeds it stops with a message naming the variable and leaves nothing behind. Installs that never set the old `TAP_MAX_OUTPUT_MB` go from a 100 MB in-memory ceiling to 4 GB of disk. The old name is honoured for one more release, with a warning at startup. See [Taps](/taps).
  * **Uploads honour the same budget.** An uploaded file, or the combined contents of a batch archive, above the run budget fails with the same message. The upload ceiling stays at 1 GB by default and is now configurable with `DATRIS_MAX_UPLOAD_MB`.
  * **Features that must read the whole payload say so.** Deduplication, JavaScript row functions, JSON/XML schema validation, single-call REST preprocessors and destinations, and single-message Kafka, ActiveMQ and scratch destinations keep working exactly as before below `PIPELINE_MATERIALIZE_MAX_MB` (256 MB). Above it the run stops with a message naming the feature and the setting, instead of running out of memory. CodeGen rules and transformations, provenance stamping and every database, warehouse and object-store destination stream and are never subject to it.
  * **REST preprocessors and destinations can batch.** A `batchSize` sends the rows in batches of that size, each call carrying its batch number and the batch count; a REST row function in batch mode accepts a batch size as its sixth parameter. Without a batch size, both keep today's single call. See [Preprocessor](/preprocessor).
  * **Record counts are right for more shapes of data.** Newline-delimited JSON input reports its record count instead of 1; a quoted value with an embedded newline counts and loads as one row, including through a CodeGen transformation; a quoted value containing the delimiter lands in one column of an object-store destination instead of being split.
  * **Bad input fails at the door.** Invalid JSON on a JSON pipeline is rejected when it is received, before a run is created; a file whose header contains none of the pipeline's columns fails with a message naming both sides instead of loading blank rows.
  * **Testing a tap returns a sample.** A tap test returns the first 20 records and says whether more were cut, instead of every record.
  * **`datris doctor` watches the staging area.** Two new checks report whether the staging area is writable with room for one run, and whether run directories left behind by a crash are accumulating. See [Doctor](/doctor).
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. Update the server and tap-runner images together; an older tap-runner still works but keeps its previous in-memory path. The compose file adds one named volume, `datris-staging`, for staged payloads — it is created on first start and needs no action. The bundled compose file now needs Docker Compose v2. No configuration changes are required; if `.env` sets `TAP_MAX_OUTPUT_MB`, rename it to `PIPELINE_MAX_PAYLOAD_MB` before the next release.
</Update>

<Update label="v1.33.1" description="September 18, 2026">
  **A schedule only ever runs code that has passed a test.**

  * **Test before you schedule.** A tap can carry a schedule only once the exact script that will run on it has passed a server-side test run. Saving a schedule on a never-tested or just-edited script is refused with a message naming the tap and the remedy: save unscheduled, test, then add the schedule. Cadence changes, clearing the schedule, and metadata edits on a tested tap need no re-test. See [Taps](/taps).
  * **Editing a scheduled tap never repoints the schedule at untested code** — by hand, by an agent over MCP, through the AI generate, fix, review and optimize helpers, or by restoring an earlier version. The scheduled script keeps running as it was until the new one is tested and the schedule re-applied.
  * **The wizard handles it for you.** Saving a scheduled tap saves it unscheduled, runs the test, and applies the schedule on success; on failure the tap is kept without its schedule and the wizard says so.
  * **Existing scheduled taps are untouched.** They keep running and stay editable; their first schedule or script change is accepted if the tap has ever tested or run successfully.
  * **Fix:** the CLI works on Python 3.14 (every command previously failed with an event-loop error).
  * **Fix:** upload, generate and profile return a plain error message instead of a stack trace.
  * **Fix:** the MCP Registry listing now names the settings the server actually reads; a Glama directory listing was added.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. Agents editing a scheduled tap's script must clear the schedule first and re-test (the tools explain this on refusal). CLI users on pip or Homebrew: upgrade for the Python 3.14 fix.
</Update>

<Update label="v1.33.0" description="September 18, 2026">
  **Run a pipeline and get the rows back, without landing them.**

  * **Scratch destination.** For when you, or an agent, want an answer now rather than a dataset: what a source returns for these ids, whether today's file passes its rules, what a transformation produces. The run goes through the same preprocessing, data quality and transformation chain as any other pipeline; the rows are held briefly so the caller can read them back. Nothing is landed, catalogued or added to lineage, and the result expires after a retention window you control. Structured sources only; cannot be combined with another destination. See [Scratch](/destinations/live-read).
  * **Reading the result.** The first rows ride on the run's status. Larger results page from the run detail page's **View result** panel, from the CLI (`datris pipeline result`), and from the new `get_pipeline_result` MCP tool. Paging never re-runs the source.
  * **Agents know when to keep and when to scratch.** The MCP server and the Assistant follow a keep-or-scratch rule: data to be kept, observed, queried later or refreshed on a schedule goes to a real destination; a one-off answer goes to scratch. Fetch-only taps target a scratch pipeline; promote it later by changing the destination — the tap, its schedule and its incremental position do not move.
  * **Scratch in the UI.** The pipeline wizard offers Scratch under Destination; the Pipelines list badges scratch pipelines.
  * **Fix:** editing a secret keeps the values you did not change — saving one field no longer blanks the masked sensitive fields you left alone, in the UI and over the API.
  * **Fix:** the destination's authoritative flag set over the REST API was silently ignored; it is now applied and validated.
  * **Hardening.** S3 object-store endpoints pass the same network safeguards as other outbound connections; pipelines writing to the built-in object store can be restricted to an allowlist of buckets (off by default, reported by the doctor); query and status error responses no longer include internal diagnostic detail.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. Scratch results are held for 24 hours by default; shorten it on shared hosts. CLI users on pip or Homebrew: upgrade to pick up `datris pipeline result`.
</Update>

<Update label="v1.32.0" description="September 17, 2026">
  **Pipelines can now land data as Apache Iceberg tables.**

  * **Iceberg as an output format.** Set the file format to `iceberg` on an [object-store](/destinations/object-store) or [S3](/destinations/s3) destination and each run commits a real Iceberg table at the pipeline's location: Parquet data files plus the table metadata any Iceberg-aware engine reads. One atomic commit per run; a failed run leaves the previous snapshot intact. No new destination type, no extra service. See [Iceberg](/destinations/iceberg).
  * **Upserts by key.** A new `merge` write mode updates rows matching the configured key fields and inserts the rest; re-running the same data leaves the row count unchanged.
  * **Partitioning and schema evolution.** Partition by columns; new nullable columns are added automatically, while type changes, dropped columns, and new required columns are refused before anything is written.
  * **Readable without Datris.** Standard Iceberg tooling opens the table by path. Time travel, catalog registration, table maintenance, and Iceberg as a tap source come later; tables written today will not need rewriting.
  * **Snapshot in query results** for Iceberg tables; the wizard, MCP tools, and Assistant know the new format and modes; lineage labels object-store destinations with their format.
  * **Fix:** querying an object-store pipeline that has never run returns an empty result instead of an error (a pipeline whose bucket does not exist also reads as empty).
  * **Overlapping runs** writing the same object-store location now run one after the other, for every object-store format.
  * **S3 credential changes apply immediately.** Correcting or rotating S3 credentials in a secret now takes effect on the next run without a restart.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required; existing Parquet and ORC pipelines are unaffected. Switching an existing pipeline to Iceberg in place is refused unless `deleteBeforeWrite` is set; use a new prefix to keep the old files. The server image is about 50 MB larger.
</Update>

<Update label="v1.30.1" description="September 15, 2026">
  **A smaller install, and docs written for the way people search.**

  * **Minimal install.** The [installation guide](/installation) now leads with a minimal path that skips the bundled embedding server and bundled PostgreSQL and fits in a 4 GB Docker memory allocation; the full stack needs about 8 GB. Numbers measured on a live stack.
  * **OpenAI embeddings by default when you have a key.** Fresh installs with an OpenAI key present use OpenAI embeddings instead of the local embedding server, whichever provider handles chat. Existing installs are unchanged; set the embedding provider explicitly to keep embeddings local.
  * **New recipe:** [Give Claude Code a safe way to load data into PostgreSQL](/recipes/claude-code-postgres) — MCP connection, approval gates, a load, and the audit and lineage records it leaves.
  * **Docs pages named for the task.** Destination, MCP, policy, audit, lineage, and Airflow pages are titled for what you are trying to do. URLs unchanged.
  * **More reliable release builds** for the arm64 UI image.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required.
</Update>

<Update label="v1.30.0" description="September 14, 2026">
  **The doctor now runs itself.**

  * **Upgrades check themselves first.** Re-running the installer on an existing install runs the pre-upgrade self-check before pulling, when the `datris` CLI is on the machine. A warning continues; an error stops the upgrade with the fix on screen (and can be bypassed deliberately). Without the CLI the installer says so and continues as before.
  * **Continuous checks.** Set an interval and the server re-runs its self-checks on that cadence, posting to the incident webhook when a check flips to an error or recovers. Off by default; AI probes never run on the timer. See [Doctor](/doctor).
  * **The recovery agent can consult the doctor.** When an incident looks like the platform rather than the tap, the agent runs the quick self-check and hands the incident to a person with the doctor's fix.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. **CLI users: upgrade to 1.29 or later** (`pip install -U datris-mcp-server` / `brew upgrade datris`) — the installer's pre-upgrade check needs it and will keep reminding you until you do.
</Update>

<Update label="v1.29.1" description="September 14, 2026">
  **Installs and upgrades work again.**

  * **MinIO images now come from quay.io.** MinIO removed its images from Docker Hub, so every fresh install and every `docker compose pull` failed with "pull access denied for minio/minio". The Compose files now pull the identical images from quay.io; running deployments were never affected. The installer, the standalone file and a fresh clone pick this up automatically.
  * **Docs: the production trail.** A new Production section walks a deployment from first install to a governed, audited, agent-operated setup; the Quick Start now ends on an approval and a provenance lookup.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. Compose recreates `minio` on the new image name; the data volume re-attaches. No configuration changes required.
</Update>

<Update label="v1.29.0" description="September 10, 2026">
  **Datris can now diagnose its own deployment.**

  * **Doctor.** A built-in operational self-check finds the problems that used to be discovered only after an outage: a Vault token about to expire or silently capped, an AI provider secret that is missing or incomplete so the server would not come back after an upgrade, an embedding model that was never pulled, a disk about to fill, and a mixed-version stack. Every finding comes with the command that fixes it; nothing is changed automatically.
  * **Four places to run it.** The cheap checks run at every server start as `DOCTOR` log lines (a healthy install logs no warnings). The full set runs on demand from Configuration → Doctor, from the new `run_doctor` MCP tool, and from the new `datris doctor` CLI command.
  * **Host checks from the CLI.** On the machine running Docker, `datris doctor` also catches data on an anonymous volume, data left behind on a dangling volume, a container still running with an old `.env`, containers from a previous Compose file, a stale from-source build, and an unreachable MCP server. `datris doctor --pre-upgrade` works with the server stopped and is step 0 of the upgrade guide.
  * **Optional AI probe.** On request, the doctor sends a minimal request through each configured AI model to confirm the key and model id still work. Never run automatically.
  * **Version stamps.** The server reports the exact build it is running and the UI image carries its version, so drift between components is visible.
  * **Fresh-install AI defaults refreshed.** New installs seed Claude Fable 5.1 (Anthropic assistant), GPT-5.6 Sol (OpenAI) and Claude Sonnet 5 (overload fallback); code generation stays on Claude Opus 5. Existing installs are unchanged.
  * Upgrade: `datris doctor --pre-upgrade`, then `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.29.0.
</Update>

<Update label="v1.28.2" description="September 10, 2026">
  **Failed loads now say which destination failed, and why.**

  * **Destination failures are named in the job status.** When a run fails while writing to a destination, the job's error identifies that destination and carries its own error message instead of a generic orchestration failure with a stack trace. The full stack trace remains in the run's event stream.
  * **Agents and the CLI act on the real cause.** The Assistant, MCP clients and the recovery agent read the same job status field, so they see the failing destination directly. The `datris` CLI prints the failing destination with the error on a failed upload.
  * **Multi-destination runs** still report as a single job; when one destination fails and another succeeds, the error names the one that failed.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.28.2.
</Update>

<Update label="v1.28.1" description="September 8, 2026">
  **Security and documentation maintenance release.**

  * **Security update.** The embedded web server that serves the API is updated to close three recently disclosed critical advisories affecting authentication and authorization handling. The UI and tap-runner images pick up the latest OS package security fixes. No functional changes.
  * **Hardening switches now take effect.** `DATRIS_ENV`, `DATRIS_ALLOW_PLAINTEXT_DB`, `DATRIS_ALLOW_PRIVATE_EGRESS` and `TAP_MAX_OUTPUT_MB` were described in `.env.example` but not passed through to the server by the bundled Compose files. They are now honored and documented in the Configuration Reference.
  * **Documentation accuracy pass.** Every page on docs.datris.ai was checked against the current release and corrected, including the MCP tool list, CLI flags, Assistant defaults, tap runtime behaviour, ingestion and destination options, the API reference and OpenAPI spec, and the examples.
  * **Examples.** Two examples that depended on files no longer in the repository were removed.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.28.1.
</Update>

<Update label="v1.28.0" description="September 4, 2026">
  **Lineage now reflects what actually ran, not just what is configured — with an interactive graph and an agent tool to traverse it.**

  * **Run-level lineage.** Every pipeline run is recorded as it completes: what it read, which destinations it wrote — each with its own success or failure — how many records, and the definition version it ran under. Automatic for every pipeline, independent of provenance stamping, and never fails or delays a run.
  * **History in the graph.** A dataset a pipeline used to land into under an earlier configuration stays in the lineage graph marked historical, with the runs that wrote it.
  * **Lineage graph.** Catalog → Lineage opens an interactive Source → Tap → Pipeline → Dataset → Catalog view, filterable by catalog, tag or name. Click any node for what feeds it, what depends on it, freshness and recent runs, with links to the tap or pipeline.
  * **Recent runs per node**, plus upstream-only / downstream-only / bounded-depth traversal on the lineage neighborhood endpoint.
  * **New MCP tool: `get_lineage`** for impact analysis before changing or deleting something, and for seeing what a pipeline actually wrote lately.
  * **Real sources in lineage.** A tap's source node names where the data actually comes from: the tap's new optional `source` field, else the endpoint host, else the host its script calls most. Taps sharing a provider share one source node.
  * **Evidence on every lineage hop**: runs, records, failures and last run per edge (last 90 days); claims without runs draw faint; numbers on the panel chips and in `find_data`.
  * **Source of authority**: datasets labeled system of record / derived / undeclared, declared never inferred; single destinations authoritative by default; `authoritative: false` for derived copies, `destination.authoritative` for multi-destination pipelines; find\_data prefers the authoritative copy; "Canonical only" graph filter.
  * **Column-level lineage** in the graph's side panel and via a new endpoint: exact schema-derived mappings always, system columns for provenance stamps, and an opt-in AI-inferred tier for pipelines with a transformation that returns only evidenced mappings, cached per definition version.
  * **Fix:** AI transformations that add, remove or reorder CSV columns now land correctly; the pipeline picks up the transformed column list instead of projecting by the old one.
  * **Provenance resolution** for recorded runs now includes per-destination outputs and resolves the definition version from the record when the row's stamp is unavailable.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. Runs are recorded from the first run after upgrading. CLI users on pip or Homebrew: upgrade to 1.28.0.
</Update>

<Update label="v1.26.1" description="September 2, 2026">
  **Claude Fable 5.1 is now the recommended chat model, plus security and dependency updates.**

  * **Claude Fable 5.1 support.** Anthropic's newest model is selectable for both the Assistant and code generation, on the direct Anthropic provider and on Amazon Bedrock. It is the recommended chat model on both; code generation keeps Claude Opus 5. Fresh Bedrock installs default to Fable 5.1; existing installs keep their configured models. On Bedrock it needs the same one-time data-retention opt-in as Fable 5.
  * **Security update.** The UI's build tooling picks up fixes for several recently disclosed high-severity advisories in a third-party component. No functional changes.
  * **UI framework update** to the latest Angular patch release.
  * **CLI and MCP server via pip.** The published package now pins the MCP SDK to the supported major version, so a fresh `pip install datris-mcp-server` no longer pulls an incompatible SDK that fails at startup.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.26.1.
</Update>

<Update label="v1.26.0" description="September 2, 2026">
  **Provenance on your data: any row can say which run, which configuration, which script, and which source produced it — and agents can find datasets by meaning.**

  * Opt-in **provenance stamping** per pipeline: every row, document, message, and vector chunk landed carries the run that produced it, when it landed, the definition version, the tap run that fed it, the exact script commit, and the declared source. Existing rows are never rewritten — provenance starts with the first stamped run; SQL destinations gain the columns automatically.
  * Every stamp resolves back to its origin — the run, the feeding tap run, the script commit, the definition version with its change note, and the source — via the platform or the new **get\_provenance** agent tool. Vector search results carry provenance per chunk so RAG answers can cite the run and source behind each passage.
  * A **lineage view** shows source → tap → pipeline → landed dataset → catalog, derived from configuration alone. Pipeline detail pages show what feeds them, where they land, and how fresh they are, using the same staleness judgment as the Activity dashboard.
  * **Tags** on taps and pipelines: free-form labels shown in the catalog and ranked by discovery.
  * **find\_data**: agents discover datasets by meaning — the best-matching pipelines their key can read, each with location, freshness, provenance handles, and a ready-to-run query hint. Discovery only: the query stays the agent's own call, under its own permissions. AI answers can carry the provenance handles along.
  * Deleting a tap or pipeline from the catalog now shows progress while the platform finishes cleaning up.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required; provenance stamping is off by default — enable it per pipeline in the pipeline editor's Destination step.
</Update>

<Update label="v1.25.0" description="September 1, 2026">
  **The recovery agent: Datris notices broken data flows, diagnoses them, and — within limits you set — repairs and verifies them.**

  * The opt-in recovery agent opens an **incident** when a scheduled tap keeps failing after its automatic retries, a pipeline load errors, a tap goes quiet past its cadence, or a pipeline's volume swings against its baseline — then diagnoses it with the same judgment as the Ops chat and closes it with a step-by-step story.
  * The [Agent Policy](/agent-policy) gains a **recovery mode**: `off`, `propose` (every repair waits for one-click approval), or `autopilot` (repairs follow your per-action policy) — with per-tap / per-pipeline overrides.
  * Repairs are narrow and verified: re-run, script fix (as a new version), test — never deletes, secrets, or schema changes. Every fix must prove itself with a clean real run, or the platform reverts it and says so. Budgets are enforced by the platform, not a prompt.
  * A new **Incidents** panel on the Activity page shows state, classification, and timelines with approval links; the Ops chat explains open incidents instead of re-diagnosing; two new agent tools read them; the audit log carries the full per-incident ledger; an optional webhook reports milestones.
  * The MCP page's tool catalog now lists every available tool, including the destination-typing pair missing since v1.22.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required; set `USE_AGENT_POLICY=true` + `RECOVERY_AGENT_ENABLED=true` and pick a recovery mode to opt in.
</Update>

<Update label="v1.24.1" description="August 31, 2026">
  **Security update for the UI and MCP server images.**

  * The `datris-ui` and `datris-mcp-server` container images ship with updated OpenSSL packages, picking up upstream fixes for several recently disclosed issues, including one rated high severity. No functional changes.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required.
</Update>

<Update label="v1.24.0" description="August 31, 2026">
  **Agent Policy: decide what agents may do on their own, what waits for your approval, and what is refused.**

  * A new opt-in Agent Policy lets an administrator set, per action — run a tap, delete a pipeline, migrate destination column types, write a secret, and every other change an agent can make — whether agents do it unattended, queue it for a person to approve, or are refused. People using the UI are never gated; they are the approvers. Nothing changes until you set a rule, and the policy applies to every agent: the built-in Assistant and any connected MCP client alike.
  * Queued actions appear under Activity → Approvals with who asked, the reason they gave, exactly what would run, and one-click Approve / Reject. Approving performs the original request on your behalf and records the whole chain — request, decision, and result — in the audit log. Approvals go stale if the pipeline or tap changed in the meantime, expire on their own, and can never be approved by an agent.
  * Agents are told the truth: a gated action comes back "waiting for approval" (never silently done), a refused one says so plainly, and the chat shows matching status cards. Agents can read the policy up front, list what they have queued, and poll for your decision; every change-making agent tool now accepts a short reason that shows up on the approval card and in the audit log.
  * Configuration → Agent Policy manages it all, with a recommended starting point and per-pipeline / per-tap overrides that can only tighten. The Configuration → Environment tab has been removed.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required; set `USE_AGENT_POLICY=true` to turn the agent policy on.
</Update>

<Update label="v1.23.0" description="August 28, 2026">
  **Audit log: who did what, by login or API key.**

  * A new opt-in Audit Log records every create, change, run, delete, login, and denied request — humans by their login, agents by their API key, scheduled runs as system. Admins read it under Configuration → Audit Log with filters by time, category, actor, and outcome, a detail view per entry, and CSV export. Entries are also written to the server log so an existing log aggregator picks them up.
  * Actions the Assistant takes for you are attributed to you — in the audit log and in pipeline and tap version history — instead of to the platform's internal key. Every issued API key now carries a stable id, so a key that is revoked and later re-issued under the same name is never confused with its predecessor.
  * Reads are left out by default to keep the trail focused on changes; turn them on to see every query and search an agent runs. Reading a secret is always recorded.
  * Tap scripts that read platform data now work when API keys are required. Each run gets a short-lived, read-only credential that is attached automatically — scripts need no changes and no platform credential in their secret — and the audit log names the tap that made each call.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required; set `USE_AUDIT_LOG=true` to turn the audit log on.
</Update>

<Update label="v1.22.0" description="August 27, 2026">
  **Set real column types on pipelines that landed as text.**

  * Pipelines created through agents store every destination column as text. The Catalog now marks such pipelines (PostgreSQL, Snowflake, and Databricks destinations) with a quiet "text" badge — click it to review proposed column types and apply them. The same action is available from the pipeline view.
  * Proposed types are inferred from the data already loaded, and every column shows real sample values so you can check the proposal at a glance. When a column stays text because of a stray value, the dialog shows the offending value and the type it blocked (for example: found "N/A", would otherwise be a number) so overriding is an informed choice.
  * Applying is safe by design: every loaded value is validated first, the destination table is retyped, and the pipeline definition is updated as a new version. A value that won't convert fails the whole apply with the column named and nothing changed. From then on, incoming data is type-checked on every load.
  * Agents get the same capability through two new MCP tools, `get_dest_types` and `apply_dest_types` — applying always requires the user's explicit approval.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required.
</Update>

<Update label="v1.21.0" description="August 26, 2026">
  **Live progress feedback across the Assistant and tap building.**

  * The Assistant shows what it's thinking about while it reasons — a live summary streams next to the progress indicator with an elapsed timer, instead of silent dots. The full reasoning stays available in the expandable thinking block.
  * AI tap-script generation shows live progress: the tap wizard and the Assistant's tool cards report what the generation is doing and how long it has been running.
  * The Assistant sets time expectations before long-running operations, and when AI script generation fails for a well-known API it writes the script directly instead of retrying — turning a multi-minute failure into a quick recovery.
  * The Assistant only reports an action as completed when it actually ran and observed the result — tightened guarantees against overstated progress reports.
  * Dependency updates across the UI and build tooling.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required.
</Update>

<Update label="v1.20.2" description="August 25, 2026">
  **Security refresh across all container images.**

  * All four container images rebuilt on updated bases with refreshed OS packages and Python tooling, clearing every published high- and medium-severity vulnerability finding against the images.
  * Every container now reports its own health status, so `docker ps` (and any orchestrator) shows healthy/unhealthy per service out of the box.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. No configuration changes required.
</Update>

<Update label="v1.20.1" description="August 25, 2026">
  **MCP tool catalog now matches API-key permissions.**

  * With API keys enabled, each MCP session's tool list is filtered to what its key is actually allowed to do — agents no longer see tools that could only fail with a permission error at call time. Full-access keys and installs without API keys keep the complete catalog.
  * The `rag-builder` key template gained the read permissions its workflow was missing (list taps, manage its own tap credentials). Existing issued keys are unchanged — re-issue or edit rag-builder keys to pick up the additions.
  * With API keys enabled, MCP sessions that present no key are refused instead of silently connecting. Installs without API keys are unaffected.
  * Permission denials return a clear, structured error instead of a generic server error, so agents recognize a permission boundary rather than retrying.
  * Tightened API-key permission enforcement on the server.
  * Upgrade: `docker compose pull && docker compose up -d --force-recreate`. If API keys are enabled, ensure MCP clients send their key (header for remote connections, `DATRIS_API_KEY` environment variable for stdio).
</Update>

<Update label="v1.20.0" description="August 24, 2026">
  **Opt-in Postgres TLS enforcement.**

  * Deployments marked as production now refuse to start when the platform's Postgres connection points at an external host without TLS (`sslmode=require` or stricter), with an explicit opt-out for deployments that intentionally use plaintext. The bundled in-network Postgres is exempt; existing deployments are unaffected unless production mode is enabled. Plaintext connections to external databases now log a startup warning either way.

  **Observability: Prometheus metrics and structured logs.**

  * Prometheus metrics are now exposed for scraping, reachable only from inside the deployment's network (the bundled edge proxy blocks them publicly). See [Audit Log](/audit-log) for the metrics endpoint.
  * Production deployments emit one JSON log line per event for SIEM / log-aggregator ingestion; development logs stay human-readable.

  **Tap isolation on by default for docker compose.**

  * All bundled Docker Compose configurations now default to `USE_TAP_RUNNER=true`. The runner token is minted automatically on first boot (including upgrades where it was missing or still the placeholder value). The server fails fast if isolation is on and no valid token exists — a minted token is accepted.
  * Running the server outside Compose (no runner sidecar) stays in-process and logs a warning. Isolated taps still cannot open direct connections to internal DB / object store / Vault — they return records; this is existing behavior, called out because compose now isolates by default.
  * Set `USE_TAP_RUNNER=false` to keep in-process on compose. No host allowlist in this change.
</Update>

<Update label="v1.19.4" description="August 21, 2026">
  **Dependency security cleanup across the server and UI.**

  * Updated server and UI dependencies to resolve every remaining published medium-severity advisory (and one low). One low-severity advisory remains open upstream with no fixed release available. No functional changes intended.
  * The Activity tab's charts moved to the current major version of the charting library as part of the security updates — same charts, slightly refreshed default colors.
  * Standard upgrade: `docker compose pull && docker compose up -d`. Existing pipelines, taps, and schedules are unaffected.
</Update>

<Update label="v1.19.3" description="August 21, 2026">
  **Server security updates and a friendlier vector-store default in the installer.**

  * Upgraded the server's embedded web framework to its current supported release line, resolving all remaining published high-severity advisories against server dependencies (and a number of medium- and low-severity ones). No functional changes intended.
  * **The installer now presents pgvector as the default vector store.** It ships inside the bundled Postgres, so vector search works out of the box with no extra container. Additional vector stores (Qdrant, Weaviate, Chroma, Milvus) remain opt-in additions at install time. Existing installs are unaffected — prompts and install summary only.
  * Standard upgrade: `docker compose pull && docker compose up -d`. Existing pipelines, taps, and schedules are unaffected.
</Update>

<Update label="v1.19.2" description="August 20, 2026">
  **Security updates across the UI and server, and a pipeline-delete fix.**

  * Upgraded the web UI's underlying framework to the current long-term-support line, resolving all published critical- and high-severity advisories affecting the UI. No visual or functional changes intended.
  * Updated bundled server dependencies to resolve the remaining published high-severity advisories against third-party libraries. No functional changes.
  * **Deleting a pipeline now also removes its data from the built-in object store destination**, matching every other destination type. Previously the files were left behind, and recreating a pipeline with the same output prefix silently picked them up. Shared output prefixes are detected and safely skipped. Files orphaned by deletes on earlier versions are not removed retroactively.
  * Hardened the UI container image configuration. No behavior change.
  * Standard upgrade: `docker compose pull && docker compose up -d`. Existing pipelines, taps, and schedules are unaffected.
</Update>

<Update label="v1.19.1" description="August 20, 2026">
  **Dependency security updates and an install fix.**

  * Updated bundled server dependencies to resolve all critical published advisories against third-party libraries, and removed an unused legacy dependency tree from the server entirely. No functional changes.
  * Fresh installs now create the default object-store output bucket automatically — previously the first pipeline writing to the built-in object store with default settings failed until the bucket was created by hand.
  * Standard upgrade: refresh your checkout (or re-download `docker-compose.standalone.yml`), then `docker compose pull && docker compose up -d`. If you update images without refreshing the Compose files, create a bucket named `{environment}-data` (default: `oss-data`) once in the MinIO console.
</Update>

<Update label="v1.19.0" description="August 19, 2026">
  **Security hardening.**

  * A broad security pass across authentication, authorization, credential handling, and input validation, plus updated bundled dependencies for published advisories. A standard upgrade needs no configuration changes.
  * **Initial admin login** (when user login is enabled): the first admin password is now generated and printed once to the server logs — `docker compose logs datris`, sign in, and change it.
  * **Taps calling internal addresses** are blocked by default, with an opt-out for endpoints that legitimately run on an internal network (see the [Tap Execution & Isolation](/tap-execution-isolation) guide).
  * **Bundled Vault** uses a unique per-install token; for manual `vault` commands read it with `docker compose exec -T datris cat /vault-token/token`.
  * **Fix:** the default Docker Compose config pulls the published images again (a recent change had it building from source).
  * Standard upgrade: `docker compose pull && docker compose up -d`.
</Update>

<Update label="v1.18.0" description="August 18, 2026">
  **Build taps in any language — HTTP taps.**

  * **HTTP taps.** A tap can now be a service you host — Rust, Go, TypeScript, a serverless function, or an existing internal system. Datris calls your endpoint on every run and everything else works exactly like a Python tap: schedules, per-run params, incremental sync state, run history, retries. Your service keeps its own upstream credentials, and no tap code runs on the platform. Wire contract and a complete Rust example: [HTTP Tap Contract](/tap-http-contract). Create one from the wizard, the CLI, or an agent over MCP.
  * **A more focused tap wizard.** Two choices: paste your own Python script, or point at an HTTP endpoint. AI generation stays available through the Assistant and connected agents. Structured vs. document output is detected automatically from the test run, and a passing script is kept exactly as tested — the automatic performance-rewrite pass is gone.
  * **GitHub-stored taps open at the latest code.** The editor loads the newest repo version on open (runs keep the previously saved version until you test and save). False "script was changed" warnings caused by unrelated repository commits are fixed, and the AI test diagnosis now works for repository-stored scripts.
  * **Catalog quality of life.** Drag-resizable columns on the taps and pipelines tables (layout remembered), and background refresh no longer interrupts column resizing, move-to-catalog menus, or delete confirmations.
  * Standard upgrade: `docker compose pull && docker compose up -d` — everything is additive; existing taps and schedules work unchanged.
</Update>

<Update label="v1.17.0" description="August 17, 2026">
  **New AI provider: Grok (xAI).**

  * **Grok for chat and CodeGen.** Grok joins Anthropic Claude and OpenAI as a third model family — the Assistant, tap script generation, AI data quality rules, transformations, and natural-language → SQL can all run on it. Configure it in Configuration → AI Providers with one key from console.x.ai; Grok 4.6 is the recommended model, with Grok Code Fast 1 available for CodeGen. Mix providers freely across sections. See [AI Configuration](/ai-configuration#grok-xai-path).
  * **Install-time support.** The installer prompts for an xAI key alongside Anthropic and OpenAI, and first-boot `.env` seeding supports Grok as the chat/CodeGen provider.
  * **Embeddings unaffected.** xAI has no embeddings API, so semantic search stays on OpenAI, Azure, or the bundled local model — a Grok-only setup works out of the box with the bundled embedder.
  * Standard upgrade: `docker compose pull && docker compose up -d` — no configuration changes required.
</Update>

<Update label="v1.16.3" description="August 14, 2026">
  **Assistant fix: taps that read data already in Datris no longer ask for database credentials.**

  * **No more credentials form for platform data.** When a tap's fetch logic depends on data a pipeline already maintains (e.g. an id list kept fresh in a Datris table), the Assistant previously asked for database credentials or offered to freeze a snapshot of the data into the script. It now builds the tap to read the platform data live on every run — no extra credentials, and the tap's secret stays limited to the external source's API key.
  * **Same guidance for MCP agents.** Agents connected over MCP (Claude Desktop, Claude Code, and others) receive the same instructions.
  * Standard upgrade: `docker compose pull && docker compose up -d`. Start a new Assistant conversation after upgrading — existing conversations keep the older behavior until reopened.
</Update>

<Update label="v1.16.2" description="August 13, 2026">
  **Azure OpenAI without API keys — Microsoft Entra ID authentication.**

  * **Keyless auth for Azure OpenAI.** Required for resources with `disableLocalAuth` (API keys disabled by policy), and useful anywhere you'd rather rotate credentials centrally. Chat, CodeGen, embeddings, and the Assistant all honor it. See [AI Configuration](/ai-configuration#keyless-auth-microsoft-entra-id).
  * **Service principal mode.** Enter your Entra Tenant ID, Client ID, and Client Secret once in Configuration → AI Providers; tokens are acquired and refreshed automatically.
  * **Managed identity mode.** Zero stored secrets when Datris runs on Azure compute (VM, AKS, App Service) — the server authenticates as its managed identity.
  * **Safe mode switching.** A stored API key always takes precedence, so the auth-mode selector clears the mode you're leaving — a stale key can never shadow Entra auth.
  * **Install-time support.** Keyless setup works from `.env` too: provide the service-principal values, or nothing at all on Azure compute. Grant the identity the "Cognitive Services OpenAI User" role on the resource.
  * Existing Azure OpenAI API-key setups keep working exactly as before — no configuration changes on upgrade.
</Update>

<Update label="v1.16.1" description="August 12, 2026">
  **Reliability: long AI generations complete, and the Assistant shows live progress.**

  * **Large script generations finish reliably.** Generating a big tap script can legitimately take more than five minutes — previously these could time out or hang partway. Generation responses now stream under the hood with a much larger time budget, so long generations run to completion.
  * **Live progress in the Assistant.** Running steps show an elapsed-time counter once they take more than a few seconds, plus a note after about a minute explaining that long generations are normal and temporary provider errors retry automatically.
  * **Unresponsive AI providers no longer hang requests.** AI calls fail fast during a provider outage and feed the automatic retry ladder, instead of waiting indefinitely on a response that will never come.

  Standard upgrade: `docker compose pull && docker compose up -d` — no configuration changes required.
</Update>

<Update label="v1.16.0" description="August 11, 2026">
  **Amazon Bedrock support — run Claude through your AWS account.**

  * **New AI provider.** Amazon Bedrock joins Anthropic, OpenAI, Azure OpenAI, and Ollama for the chat assistants and CodeGen — Claude models served from your own AWS account with IAM authentication and AWS billing. Configure it in Configuration → AI Providers. See [AI Configuration](/ai-configuration#amazon-bedrock-path).
  * **IAM-native auth.** Enter AWS credentials once, or leave them blank on AWS infrastructure to use the server's IAM role — zero stored secrets.
  * **Live model discovery.** The model dropdown shows what your AWS account and region can actually invoke — including cross-region inference profiles — rather than a hardcoded list.
  * **Claude Fable 5 support.** Chat recommends Claude Fable 5; CodeGen recommends Claude Opus 5. Fable 5's one-time AWS data-retention opt-in is explained in-product and in the docs.
  * **Deliberate opt-in.** Bedrock is only ever selected explicitly — AWS credentials used for S3 destinations never silently become your AI provider. The installer offers Bedrock when you don't have a direct provider key.
  * **Clearer credentials panel.** The Configuration → AI Providers key panel is now grouped per provider with a short explainer; credentials hide immediately after saving.
  * **Fix: provider-scoped model lists.** A model saved under one provider no longer appears in another provider's dropdown after switching a section between providers.
  * **Refreshed recommendations.** Claude Fable 5 is now the recommended chat model for the Anthropic provider.
</Update>

<Update label="v1.15.0" description="August 10, 2026">
  **Azure OpenAI support — run Datris AI on your Azure resource.**

  * **New AI provider.** Azure OpenAI joins Anthropic, OpenAI, and Ollama as a first-class provider for all three AI slots (AI Provider, CodeGen, Embedding) — mix it freely with the others. Configure it in Configuration → AI Providers with your key, resource endpoint, and deployment name.
  * **Deployment-name aware.** Azure addresses models by your deployment names; name deployments after the model they serve so model-specific handling applies automatically.
  * **Installer support.** The one-command installer offers Azure OpenAI when you don't have a direct Anthropic or OpenAI key, and first-boot seeding configures everything from your Azure details. See [AI Configuration](/ai-configuration#azure-openai-path).
  * **Built for how Azure behaves.** Responses stream under the hood so long reasoning calls complete reliably; both current and legacy Azure endpoint styles work.
  * **Fix: session-expiry login.** Signing back in after a session expires no longer requires a full page refresh.
  * **Fix: per-provider key updates.** Updating one provider's API key through the API no longer clears the other providers' stored keys.
  * **Fix: agent-uploaded content.** Data uploaded through the AI assistants can no longer be silently garbled when the model sends plain text where encoded content was expected.
</Update>

<Update label="v1.14.1" description="August 5, 2026">
  **Fix: the tap sync-state view is now reachable.**

  * **Sync-state button restored.** v1.14.0's per-tap sync-state view (current bookmark, commit time, one-click reset) was missing its button on the Catalog page, so it couldn't be opened in the UI. Each tap row now has a bookmark icon that opens it. The MCP sync-state tools were unaffected.
</Update>

<Update label="v1.14.0" description="August 4, 2026">
  **Incremental taps — scheduled taps now fetch only what's new.**

  * **Taps remember where they left off.** After each successful run the platform stores a bookmark; the next run pulls only records added or changed since. Recurring syncs run at constant size instead of re-fetching everything.
  * **No gaps, no duplicates.** The bookmark only advances when a run succeeds — failed runs leave it in place so the retry re-fetches the same window; key-field destinations absorb overlap by updating in place.
  * **Visible and resettable.** Each tap's sync state can be viewed and reset from the Taps page; agents get the same via two new MCP tools. Manual backfills with an explicit date range never move the bookmark.
  * **AI-built taps are incremental by default.** Described a recurring source? The generated script now uses change-detection automatically when the source supports it (modified-since filters, ID watermarks, cursors, or content hashing). See the [Taps guide](/taps#incremental-sync-state).
  * **MongoDB upsert.** MongoDB pipelines can declare key fields, so revised records update in place instead of duplicating — previously blocked by over-strict validation.
  * **MCP playground fixed.** All tools in the MCP tab's playground now execute (28 previously errored when run there); 63 tools total including the two new sync-state tools.
</Update>

<Update label="v1.13.1" description="July 28, 2026">
  **Store your tap scripts in GitHub — every save is a commit.**

  * **GitHub script storage.** Connect a repository under Configuration → Code Repository and tap Python scripts are stored there instead of built-in storage — every save from the wizard, the Assistant, or the API becomes a reviewable commit.
  * **Runs pinned to the saved version.** Scheduled runs execute the exact commit that was saved, from a local cache — they keep working even if GitHub is unreachable, and each run records which version it executed.
  * **Edit in your own IDE.** Commit changes to the repository directly; Datris detects the newer version when the tap is opened and offers to load it. Conflicting edits are stopped with a clear message instead of overwritten.
  * **Move taps either way, one click.** Existing taps stay on built-in storage until moved; each Taps row shows where its script lives. Fine-grained token scoped to the one repository; connection test built in; GitHub Enterprise Server supported. See the [GitHub Script Storage guide](/tap-github-storage).
</Update>

<Update label="v1.12.2" description="July 27, 2026">
  **Claude Opus 5 support, and a fix for a month-after-install credential expiry.**

  * **Claude Opus 5.** Anthropic's newest model is fully supported and is now the recommended Anthropic model for chat and code generation — same price as Opus 4.8 with better results. Fresh installs default to it; existing installs keep their configured model.
  * **Fix: internal credential expired a month after install.** An internal service credential was silently issued with a 32-day lifetime, after which restarting the platform could fail. Credentials are now long-lived and refreshed on every startup. Applies automatically on upgrade.
</Update>

<Update label="v1.12.1" description="July 24, 2026">
  **Curate the data sources your assistant recommends.**

  * **Approved data sources registry.** A new Configuration → Data Sources tab lets you maintain a curated list of approved external data sources in one editable document. When the AI assistant or the tap brainstormer suggests where to pull data from, registry sources are offered first, by name; anything outside the registry is clearly flagged. Leave it empty and behavior is unchanged.
  * **You stay in control.** Registry entries don't skip confirmation — the assistant still confirms source, scope, and destination before building anything.
</Update>

<Update label="v1.12.0" description="July 23, 2026">
  **Failed scheduled runs now retry themselves — and tell you how to fix what they can't.**

  * **Automatic retry for scheduled runs.** Transient failures (source briefly unreachable, rate limits, network blips) are re-run automatically a few times with a short delay before being flagged. Only runs that are provably safe to repeat are retried, so a retry can never duplicate data. On by default and configurable; manual runs pause the retry ladder.
  * **AI fix suggestions, right where failures show.** Persistent failures get a one-line suggested fix directly on the failed run in Ops → Ingestion and tap Run History, with the full diagnosis and concrete change one click away. Advisory only — nothing changes unless you apply it.
  * **Agents see suggestions too.** Agents polling pipeline status get the same one-line fix suggestion for failed jobs.
  * **Sign-in session fix.** With user login enabled, an expired session now ends cleanly and returns you to the login screen right away, instead of requiring a page refresh.
</Update>

<Update label="v1.11.3" description="July 22, 2026">
  **Search shows only what's installed, plus internal consolidation.**

  * **The Search tab now lists only active data stores.** Stores that aren't installed or reachable no longer appear in the query-type menu, and the list recovers automatically after a server restart.
  * **Clearer errors for unavailable stores.** Searching a store your deployment doesn't have returns a clear message naming the valid options.
  * **Internal consolidation.** Vector-store loading and search, AI provider integration, and tap AI workflows were restructured onto shared foundations — same behavior, less duplication, faster future vector-destination support.
</Update>

<Update label="v1.11.2" description="July 21, 2026">
  **Reliability and hardening.**

  * **Faster repeated queries.** Postgres connections are now pooled and reused across pipeline loads and interactive queries instead of being opened fresh each time.
  * **Connection details are masked in logs.** Database connection information no longer appears in logs or job status output.
  * **More stable long-running deployments.** Fixed several resource leaks in file ingestion and data-reading paths that could slowly accumulate open connections.
  * **Clearer diagnostics.** Many background failures that were previously silent are now logged with context.
  * **Every release is now test-gated.** An automated test suite runs on every change and before every release build.
</Update>

<Update label="v1.11.1" description="July 14, 2026">
  **Installer hardening.**

  * **Clear guidance when a previous installation is in the way.** If containers from an earlier Datris install (in another directory) would block a new one, the installer detects this before downloading anything and prints exactly what is blocking, where it lives, and how to remove or upgrade it.
  * **API keys are no longer echoed.** The installer reads provider keys with input hidden and confirms with a masked preview, so keys never land in your terminal scrollback.
  * **Best model out of the box.** Fresh installs choosing Anthropic default to Claude Opus 4.8 for chat, NL→SQL, and analysis — matching code generation. Existing installs keep their configured model.
</Update>

<Update label="v1.11.0" description="July 13, 2026">
  **Choose your databases at install — run less, connect what you already have.**

  * **Pick and choose at install.** The installer now asks which databases and stores you want instead of installing everything. Each can run bundled, connect to a service you already operate, or be skipped — and pressing Enter at every prompt still gives the standard full install.
  * **Bring your own infrastructure.** Point Datris at an existing Postgres, Kafka, or managed vector store (Qdrant, Weaviate, Chroma, Milvus) during install, and optionally store Snowflake or Databricks destination credentials up front so pipelines can use them on day one.
  * **Lighter installs.** Skipping the bundled semantic-search server avoids a multi-gigabyte model download — the installer recommends OpenAI embeddings, which need no local container and cost pennies. Vector stores and a local Kafka test broker are now a one-line opt-in instead of commented-out YAML.
  * **Both AI providers, each at its best.** The installer asks for an Anthropic key (chat, code generation, AI data quality) and an OpenAI key (embeddings) — both optional, with sensible behavior when only one is present.
  * **Everything knows what's installed.** The pipeline wizard greys out destinations that aren't available, the AI assistant offers only the destinations your deployment actually has, and creating a pipeline against a missing database returns a clear explanation instead of a timeout.
  * **Install-time validation.** After first boot the installer checks every store you configured — bundled or external — and reports each by name, so a mistyped hostname or bad credential surfaces immediately.
  * **Change your mind anytime.** Enable or disable any store after install with a one-line edit and a restart. Disabling keeps the data; re-enabling brings it back.
  * **Safer upgrades.** Upgrading an existing install never changes which services run — everything you had keeps running, no action required. Data services now use durable named storage, so containers can be removed and recreated without losing data.

  **Upgrading:** Re-run the installer, or `docker compose pull && docker compose up -d --remove-orphans` after refreshing the compose file. Your current services and data are preserved automatically. See [Installation](/installation) for the new options.
</Update>

<Update label="v1.10.0" description="July 9, 2026">
  **Databricks destination — load, upsert, and query your Databricks workspace.**

  * **Load into Databricks.** Point any structured pipeline at your own Databricks workspace and Datris loads it as governed Delta tables in Unity Catalog: tables are created automatically, the schema evolves as new columns appear, and natural-key upserts keep scheduled re-runs duplicate-free. Loaded tables are immediately queryable across the workspace — notebooks, dashboards, and lineage all see them natively. Available in the pipeline wizard and through the AI assistant.
  * **Secure by default.** Connects as a dedicated service principal you control, with least-privilege access to a single catalog; credentials live in a platform secret, never in pipeline configs. The [Databricks destination guide](/destinations/databricks) includes a copy-paste grant script. Works out of the box with serverless SQL warehouses — nothing runs in your cloud account and there's no cluster to manage.
  * **Safe full refreshes.** Pipelines that replace their table on each run swap the contents atomically — readers never see a half-loaded or empty table, and a failed run leaves the previous data intact.
  * **Forgiving setup.** Pasted workspace URLs and warehouse connection paths are cleaned up automatically in whatever shape they arrive, and connection problems come back as plain-language errors that say exactly what to fix.
  * **The assistant can verify and query Databricks.** Ask it to confirm a load landed or answer questions over the data — read-only, using the pipeline's own credentials — and it can browse your catalogs, schemas, tables, and columns along the way.
  * **Cleaner deletes.** Removing a pipeline together with its data also cleans up the table it created in Databricks.
  * **CLI catch-up.** `datris ingest` can now target Snowflake and Databricks directly, and pipeline listings label Snowflake and Databricks destinations.

  **Upgrading:** Existing installs: `docker compose pull && docker compose up -d`. The CLI: `brew upgrade datris`. Using Databricks requires a one-time setup in your workspace — see the [Databricks destination guide](/destinations/databricks).
</Update>

<Update label="v1.9.0" description="July 8, 2026">
  **Snowflake destination — load, upsert, and query your Snowflake account.**

  * **Load into Snowflake.** Point any structured pipeline at your own Snowflake account and Datris bulk-loads it: tables are created automatically, the schema evolves as new columns appear, and natural-key upserts keep scheduled re-runs duplicate-free. Available in the pipeline wizard and through the AI assistant.
  * **Secure by default.** Connects with key-pair authentication under a dedicated service user and least-privilege role you control; credentials live in a platform secret, never in pipeline configs. The [Snowflake destination guide](/destinations/snowflake) includes a copy-paste setup script that creates everything — role, warehouse, database, service user — in one run.
  * **Forgiving setup.** Pasted private keys and account URLs are cleaned up automatically in whatever shape they arrive, and connection problems come back as plain-language errors that say exactly what to fix.
  * **The assistant can verify and query Snowflake.** Ask it to confirm a load landed or answer questions over the data — read-only, using the pipeline's own credentials — and it can browse your databases, tables, and columns along the way.
  * **Watch the assistant work.** Long assistant turns now show live progress: a size counter while it composes larger pieces of work, a running glimpse of its reasoning, and a continuous activity indicator between steps — no more silent pauses.
  * **Faster pipeline creation.** The assistant sends only a small schema sample when creating pipelines, trimming minutes off longer build conversations.
  * **Cleaner deletes.** Removing a pipeline together with its data now also cleans up the table it created in Snowflake.

  **Upgrading:** Existing installs: `docker compose pull && docker compose up -d`. The CLI: `brew upgrade datris`. Using Snowflake requires a one-time setup in your Snowflake account — see the [Snowflake destination guide](/destinations/snowflake).
</Update>

<Update label="v1.8.12" description="July 1, 2026">
  **Claude Sonnet 5 support.**

  * **Choose Claude Sonnet 5.** Anthropic's new Claude Sonnet 5 model is now available as a choice for the AI assistant and code generation. Select it in the Configuration tab like any other model.

  **Upgrading:** Existing installs: `docker compose pull && docker compose up -d`. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.11" description="June 29, 2026">
  **Switch AI providers freely, and an assistant that finishes the job.**

  * **Switch providers without losing your keys.** Your Anthropic and OpenAI API keys are now kept per provider, so you can change the provider for any section — AI, CodeGen, or Embedding — and switch back later without re-entering the key you already saved. Enter each provider's key once and mix providers however you like.
  * **The assistant runs to completion.** Longer assistant replies that used to stop partway — leaving you to type "continue" — now finish on their own.

  **Upgrading:** Existing installs: `docker compose pull && docker compose up -d`. Keys you've already entered carry over; no re-entry needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.10" description="June 24, 2026">
  **Your secrets and configuration now persist across restarts and rebuilds.**

  * **Settings that stick.** Connection secrets, AI provider and model choices, and other changes you make in the Configuration tab now survive a rebuild (`docker compose up --build`), a restart, and host reboots. Previously some of these could quietly revert on a rebuild because they were held only in memory — now they're stored durably, and the Configuration tab is the source of truth.

  **Upgrading — one-time, please read:** with this release the secrets store moves from in-memory to durable on-disk storage. Any secret you had set **only** in the app's UI — **including tap connection secrets** — does not carry over and must be entered once more after upgrading. From then on your secrets persist across every future rebuild and restart. (Settings supplied through your environment configuration are restored automatically.) Existing installs: `docker compose pull && docker compose up -d`. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.9" description="June 22, 2026">
  **Version history for taps and pipelines, plus a faster assistant.**

  * **Version history for taps and pipelines.** Every create or change to a tap or pipeline is now saved as a snapshot. Open Version History to see what changed and when, compare any two versions side by side, and roll back — or forward — to an earlier version. Nothing is overwritten; restoring simply adds a new version. AI agents can review, compare, and restore versions too.
  * **Configuration that sticks.** AI provider and model changes in the Configuration tab now save reliably and persist across restarts.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. Existing taps and pipelines are seeded into version history automatically on first start; no manual migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.8" description="June 18, 2026">
  **Stronger isolation for tap scripts, plus a catalog readability fix.**

  * **Optional sandboxed execution for taps.** Tap scripts can run in a dedicated, isolated runner that's cut off from the platform's credentials and internal services — defense-in-depth for the code your taps execute. Opt-in; see the Tap Execution & Isolation guide.
  * **Tighter defaults for tap scripts.** Platform credentials are kept out of tap code, and any extra packages a tap needs install in isolation rather than into the shared environment.
  * **Catalog: full names on hover.** Long tap and pipeline names in the catalog are no longer cut off — hover to see the full name.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.7" description="June 14, 2026">
  **Drop a file into the Assistant and it builds the pipeline for you.**

  * **Drag a file into the chat.** Attach a CSV, JSON, XML, or document right in the Assistant conversation. The assistant reads a sample, works out the shape, and proposes where to put it — then, once you confirm, creates the pipeline, loads your data, and reports how many rows landed. No wizard, no manual schema step.
  * **It confirms before it builds.** The assistant picks a sensible default destination for your file and names the alternatives, so you can steer it to Postgres, MongoDB, an object store, or a vector store before anything is created — and it asks about an upsert key if that makes sense for your data.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.6" description="June 12, 2026">
  **A more decisive assistant, and taps that tell you when a credential is missing.**

  * **No more "type continue to keep going."** The in-product assistants (build, ops, catalog, and search) used to sometimes end a turn by announcing the next step and then stopping — making you nudge them to carry on. Now, when the assistant has already decided what to do next, it just does it, while still pausing where it genuinely needs your decision or approval.
  * **Taps surface missing credentials instead of silently returning nothing.** If a tap's API key or other credential has been deleted or is missing the field the tap needs, the run is now reported as a clear failure naming what's missing — rather than quietly completing with zero records and hiding the real cause. Runs that legitimately have no new data are unaffected.
  * **Clearer MCP activity graph.** The connection graph in the MCP Activity monitor now renders at a readable size.
  * **"Move to catalog" shows everything.** The per-item move menu in the Catalog now lists all your catalogs, not just the first one.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.5" description="June 11, 2026">
  **Claude Fable 5 — Anthropic's most capable model, now selectable.**

  * **New model option.** Claude Fable 5 now appears in the model picker for both the in-product assistant and code generation, alongside the existing Claude and OpenAI choices. Pick it from Configuration when you want Anthropic's most capable model for demanding reasoning and long-horizon work.
  * **A note on data retention.** Fable requires standard (30-day) data retention on the Anthropic account and isn't available on zero-data-retention organizations. If you bring your own Anthropic key on a zero-data-retention plan, choose a different model — you'll now see a clear message explaining why, instead of a generic error.
  * **Clearer model errors.** When a model declines a request — wrong account settings, a rejected key, or unsupported options — the assistant now surfaces a plain-language explanation naming the model, rather than a raw provider error.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.4" description="June 7, 2026">
  **A domain-neutral assistant — guidance that fits whatever data you work with.**

  * **No more finance-flavored examples.** The in-product assistant, the tap generator, and the prompt suggestions used to lean on stock-market wording — tickers, symbols, financial filings — in their examples and defaults. They're now domain-neutral, so the guidance and sample values match the data you're actually working with, whatever the domain.
  * **Generic search defaults.** Vector search now defaults to a `documents` collection (and a `Documents` class) instead of finance-specific names. If you already pass your own collection, table, or class name, nothing changes — only the placeholder defaults differ.
  * **Refreshed starter prompt fragments.** The built-in tap prompt fragments now describe general patterns — rate-limited APIs and APIs that require a User-Agent header — rather than specific named data providers.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.3" description="June 4, 2026">
  **Organize your catalog by chatting with it.**

  * **Catalog assistant.** The Catalog tab has a new side-panel assistant for tidying things up. Ask it to group your taps and pipelines into catalogs, move items around, or suggest a cleaner structure — it lays out a plan first and only makes changes once you approve them, and the catalog updates as it works. It sticks to organizing: questions about your data go to Search, and running or fixing pipelines stays in Ops.
  * **Right where you're working.** The assistant lives in a collapsible right rail (toggle it with Cmd/Ctrl + backslash) so the catalog stays in view while you reorganize. Clicking **Describe to Assistant** on a catalog now opens this panel in place instead of jumping to another tab, and the conversation sticks around as you navigate away and back.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.2" description="June 3, 2026">
  **Ask your data a question — conversational search comes to the Search tab.**

  * **Chat search.** The Search tab has a new **Chat** mode, with a **Traditional** toggle for the structured query UI you already know. Ask a question in plain language and Datris finds the answer across all your pipelines and taps — cataloged or not — querying tables, searching documents, and replying with citations to where each answer came from. It's read-only: it looks, it never changes anything.
  * **Scope to a catalog.** Narrow a chat to a single catalog (or to Uncataloged data) from the dropdown, or leave it on All to search everything.
  * **Conversations survive a refresh.** Your Search chat and Assistant conversations now persist across a browser refresh, so reloading the page no longer clears the transcript.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.8.1" description="June 2, 2026">
  **Orchestrate Datris taps from Apache Airflow.**

  * **Run taps from Airflow.** A new `airflow-provider-datris` package adds an operator that triggers a tap, waits for the pipeline to finish, streams Datris logs into the Airflow task log, and reports run tokens and row counts back to Airflow. Cancelling the DAG run cancels the Datris job.
  * **Date-windowed backfills.** Taps can now take per-run parameters, so an Airflow DAG can pass its logical date (or any window) into the tap for that run — backfills and incremental loads work without editing the tap.
  * **No double-firing.** A tap is scheduled by Datris or Airflow, never both: if a tap has a Datris cron, the Airflow operator declines to trigger it. To drive a tap from Airflow, leave its cron empty.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. Install the provider where Airflow runs: `pip install airflow-provider-datris`.
</Update>

<Update label="v1.8.0" description="May 29, 2026">
  **Write to AWS S3, query Parquet and ORC from the Assistant and Search, and stop chats actually stop.**

  * **AWS S3 as a first-class destination.** The Object Store destination now writes Parquet or ORC directly to S3 alongside the built-in MinIO. Pick *Object Store (MinIO or S3)* in the pipeline wizard, point at your bucket, and reference a credentials secret you've created in Configuration → Secrets → Platform. Multiple S3 destinations with different IAM keys coexist in the same deployment — each pipeline carries its own credential reference, applied per bucket at write time.
  * **Region lives with the credential.** AWS credentials and region travel together in the credentials secret rather than on the pipeline config. One source of truth, one place to rotate, no more silent `us-east-1`-says-the-config-but-the-key-is-`us-west-2` failures. Field names are flexible — `accessKey` / `AWS_ACCESS_KEY` / `AWS_ACCESS_KEY_ID` all work.
  * **Query Parquet and ORC from the Assistant.** "Show me the weather data" now works against pipelines whose destination is Object Store. The Assistant resolves the bucket and credentials from the pipeline config, reads the columnar files, and returns rows in chat.
  * **Search tab gains an Object Store option.** Pick a pipeline from the dropdown, see the resolved bucket/prefix/format, set a limit, hit Execute. Works for both MinIO and S3 destinations.
  * **Assistant offers all three structured destinations.** When you ask for structured data and no pipeline yet covers it, the Assistant now mentions MongoDB, PostgreSQL, **and** Object Store as choices instead of silently defaulting to one.
  * **Assistant can discover destination credentials it shouldn't create.** Platform-tab secrets are now visible to the Assistant for reading (names and field shape only — never values). When a pipeline destination needs a credentials reference, it lists what's available, verifies the field shape, and points you at the Secrets tab to create one when nothing fits.
  * **Stop actually stops.** Clicking Stop in the Assistant now halts the in-flight chat within a fraction of a second instead of waiting for the upstream model to finish generating. Cancelled responses cost only what was already streamed.
  * **Pipeline failures surface fast.** A failed pipeline now flips to Error in the Ops dashboard within seconds instead of staying stuck in Processing for up to ten minutes.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. Existing MinIO pipelines work unchanged. For AWS S3, create a Platform-tab secret with `accessKey`, `secretKey`, `region` (and optionally `sessionToken`), then reference it by name from your pipeline.
</Update>

<Update label="v1.7.9" description="May 28, 2026">
  **Ask the Ops assistant about a failing pipeline without leaving the dashboard.**

  * **New Ops chat side panel.** A collapsible chat lives on the right side of Ops → Activity. The assistant has the current failures, stale taps, and volume anomalies in mind — ask "why did `X` fail?" and it pulls the root cause; ask it to re-run a tap and it runs and reports the outcome. The panel stays mounted as you switch between Activity and Ingestion, so the conversation survives the tab change.
  * **"Ask" buttons on failure and volume rows.** Click "Ask" next to a row to seed the chat with a row-specific question so you don't have to retype the tap or pipeline name.
  * **Successes are expandable like Failures.** Click any row in the Successes pane to see the same event trail (begin → processing → end) you get from the Failures pane. Only one row across either pane is open at a time so the layout stays compact.
  * **Claude Opus 4.8 is the new recommended CodeGen model.** New installs seed Opus 4.8 as the codegen default. Existing installs can pick it from the model dropdown in Configuration. The older Opus versions remain selectable for anyone who wants to pin a specific version.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.8" description="May 27, 2026">
  **The Assistant stops second-guessing itself when the data is already there.**

  * **No more apology loops.** When the Assistant verifies platform state and the pipeline / tap it created earlier is present in the list, it now treats that as evidence the work was done — instead of retracting a prior turn's "done" claim and rebuilding from scratch.
  * **"Show me X" goes straight to the data.** When you ask to see / list / show data and a matching pipeline already exists, the Assistant now jumps to the destination's query tool (Mongo / Postgres / vector search) and returns the actual rows or documents. It no longer asks you which sources, providers, or schedules to use for a pipeline you already have.
  * **Long catalogs no longer hide existing resources.** The tools the Assistant uses to inventory pipelines and taps now return a compact summary with names at the top. Previously, on environments with many pipelines, an entry near the end of the list could slip past the Assistant's scan — the new shape makes every name impossible to miss.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.7" description="May 27, 2026">
  **Ops Activity gains a Successes pane, long lists scroll in place, and the dashboard's auto-refresh no longer yanks you back to the top.**

  * **Successes pane on Ops Activity.** A new pane below Failures lists every pipeline that ran successfully in the selected window, with the run count, items processed, and last-run time. Click a row to jump to that pipeline.
  * **Long lists scroll inline.** Failures, Successes, and the Per-pipeline volume table all cap at \~10 rows of height and scroll internally instead of stretching the page. Per-pipeline volume column headers stick to the top so you don't lose context as you scroll.
  * **Auto-refresh preserves scroll position and expansion state.** The 30-second refresh no longer scrolls a long pipeline-volume list back to the top or collapses an open failure detail. Expand a failure, scroll where you want, leave the tab open — it stays put.
  * **Numeric column headers aligned with their data** on the Per-pipeline volume table — Today, 7d avg, vs avg now line up with the numbers underneath.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.6" description="May 26, 2026">
  **A new Ops Activity dashboard, Postgres pipelines learn upsert, per-run tap parameters, and a safer agent workflow.**

  * **Ops Activity dashboard — at-a-glance ingestion health.** A new tab under **Ops** pulls every tap and pipeline run in a rolling window (24h / 7d / 30d) into KPI tiles, time-series charts of runs and items ingested, a per-pipeline 7-day volume table, and a Failures pane that dedupes by item with attempt counts. Each unrecovered row has a **Re-run** button — for pipeline failures with an upstream tap, the button re-runs that tap to retry the load so you don't have to hunt for it in the Catalog.
  * **Postgres pipelines upsert on conflict when `keyFields` is set.** Matches the semantics Mongo has always had. Backfills over already-loaded dates, incremental taps with overlap, and "load again with the same key" flows now upsert instead of failing with a duplicate-key error. If you retrofit `keyFields` onto an existing table, the platform adds the matching unique index for you on the next load — or surfaces a clear remediation message if existing data violates the proposed key.
  * **Per-run tap parameters.** Run a tap with caller-supplied values — date ranges, ticker lists, page cursors, batch sizes — without rewriting the script or the secret. Pass `params` to `run_tap` and the script reads them as env vars for that one run only. Scheduled cron runs see an empty params bag and fall back to script defaults, so the same script handles both manual ad-hoc calls and unattended schedules without branching.
  * **"No records" is no longer a failure.** A tap that runs cleanly and returns zero rows — a polling tap on a quiet day, an incremental tap that's caught up, a market tap on a weekend — now records a distinct `no_records` status with a neutral badge. Doesn't count in the Failures tile, doesn't fire bogus "recovered" badges, doesn't train agents to interpret "no new data" as "platform broken."
  * **Large tap outputs fail fast with an actionable error.** A backfill that exceeds the size cap (default 100 MB) now stops with a clear message telling you and the Assistant to chunk the source range smaller — instead of OOM-killing the server. Multiple smaller runs all land in the same destination pipeline; with `keyFields` set, overlapping ranges upsert safely.
  * **Concurrent tap runs no longer race.** Two pipelines that loaded data in the same millisecond previously risked landing each other's records in the wrong destination under specific timing. Fixed at the source; no action required.
  * **Secret values stay masked in Configuration.** The Secrets tab now masks any field whose name suggests a credential — passwords, tokens, API keys, signing keys, certificates, and named variants of those (e.g. `*_API_KEY`, `*_SECRET_*`). No action required.
  * **Tighter Assistant workflow.** Three new disciplines:
    * **Scheduling lives on the tap.** Say "every morning" or "at market open" and the Assistant sets the CRON expression on the tap itself — instead of handing back a `cron` line for you to wire up yourself.
    * **Test before first run.** A newly-created or just-edited script is validated before any real run, and before being put on a schedule — no more "guaranteed-bad nightly run" the next time the cron fires.
    * **No confabulated progress.** If the Assistant intended to do N things and only did M of them, it tells you which M happened rather than narrating all N as complete.
      The credential form the Assistant pops up now asks only for true secrets — not for configuration values you already typed into chat.
  * **Agent Monitor stays responsive during long tool calls.** Running a slow tool no longer freezes the Connections viz or the Activity Log — they keep streaming. Returning to the Agent Monitor tab refreshes the log to current server state instead of showing a stale empty view.
  * **Configurable JVM heap.** The bundled `datris` service has a sensible default heap size for small hosts (8 GB) and a clean way to bump it on larger machines via `.env`. See [Installation → JVM Heap Sizing](/installation#jvm-heap-sizing) for the suggested sizings per host RAM.
  * **Ops tab moved after Search** in the top navigation. Mostly cosmetic; route URLs are unchanged so bookmarks and deep links still work.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. Existing Postgres pipelines with `keyFields` automatically pick up the upsert path on the next load. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.5" description="May 22, 2026">
  **Embedding pipelines stop failing on oversized chunks, the Assistant carries long runs to completion, and the Catalog gets the missing delete buttons and run-history detail.**

  * **Embedding pipelines no longer fail when a single chunk is too big.** Every embedding call is now guarded by a token-aware safety net that splits any chunk over the model's input cap before sending — so a 10-Q ingest with one dense table can't take down the whole batch. Works the same for OpenAI, Cohere, Voyage, BGE-M3, Nomic, Mistral, or anything else, with built-in caps for the common models and a conservative default for the rest. OpenAI families get exact token counts; everyone else gets a heuristic.
  * **Token-aware chunking.** A new `maxChunkTokens` option on the chunking config tells the chunker to stop merging segments before they cross a token estimate — the safety net above becomes a true last resort. Recommended for any new vector pipeline.
  * **The Assistant carries pipeline runs to completion.** When the agent kicks off a tap or upload, it now polls the run to a terminal outcome with exponential backoff instead of summarizing "still running — check back later." You'll see a one-line progress update each cycle and the final outcome reported in chat, including any per-document failures.
  * **Run History now shows per-document outcomes.** Expanding a tap run that fed a vector pipeline lists every document the pipeline processed with its own status, elapsed time, and — for failures — the specific stage and error. A run that fetched 28 documents but failed on 1 now shows the failure inline instead of a misleading green "success."
  * **Catalog: delete buttons work on individual taps and pipelines.** The trash icon now shows an inline confirm right in the row — Delete / × for taps, Config & Data / Data Only / × for pipelines (Data Only wipes rows but keeps the config so the next ingest fills it again).
  * **Catalog: last-run timestamp is back above the status badge.**
  * **Agent Monitor fits the viewport.** Both Connections and Activity Log panes now size to the visible area on first load and reflow when you resize the window, not just in the pop-out.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui mcp-server`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.4" description="May 21, 2026">
  **Streamlined navigation and a Catalog-centric workflow — fewer tabs, richer Catalog, and the Assistant front-and-center.**

  * **Five top-level tabs instead of ten.** Datris now opens to Assistant, MCP, Catalog, Data, Configuration — plus the Help dropdown. Same capabilities, organized around how you actually work.
  * **Catalog is the home for taps and pipelines.** Each catalog card embeds the full tap and pipeline tables with inline rename and a move-to-catalog dropdown on every row. Uncataloged is always shown so day-1 users have a place to start.
  * **Bulk move at the catalog level.** Move every tap and pipeline from one catalog into another in one click. The destination auto-expands so you see the items land.
  * **Describe to Assistant from any catalog.** A button on each catalog card opens the Assistant with a fresh chat and the catalog pre-filled — the Assistant assigns the right catalog to whatever it creates.
  * **Wizards link back to where you came from.** Tap and pipeline edit screens now show the item's name in the title, a "Back to Catalog" link at the top, and primary action buttons at both top and bottom of the form. Pipeline wizard's JSON-review step is gone — Save fires from the Destination step.
  * **Pop out the Agent Monitor.** A new icon opens both Connections and Activity Log in a separate browser window — park it on a second monitor and watch tool calls stream while you work in the Assistant.
  * **Catalog state persists across navigation.** Open catalogs and expanded sub-sections survive refreshes and tab switches.
  * **Fixed: tap rename no longer breaks scripts.** Renaming a tap inline used to leave the new tap pointing at a deleted script file.
  * **Fixed: catalogs containing only pipelines could lose their assignment** when the pipeline edit wizard was opened.
  * **Discovery tab removed.** The Assistant covers its workflow.
  * **Getting Started tab removed.** First-run guidance lives in the Assistant's starter prompts and inline empty states.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.3" description="May 19, 2026">
  **Scoped API keys with per-agent permissions, plus Assistant resilience and smarter onboarding.**

  * **New API-Keys tab in Configuration.** Issue a dedicated key per agent, CLI, or integration with an explicit list of what it's allowed to do — read pipelines, run taps, upload documents, query data, and so on. Each key is its own identity in the request log and can be rotated or revoked independently. Five starting templates: read-only, rag-builder, reporting, ops, and full-access. The tab only appears when you've set `USE_API_KEYS=true` in `.env` — issuing keys is pointless until the validation layer is on.
  * **Keys actually constrain.** When an external agent (Claude Desktop, Cursor, the CLI) connects with a scoped key and tries something outside its bundle, the platform refuses the call and tells the agent why in plain JSON — so the agent doesn't keep retrying alternate paths. Agents stay productive within their lane; you don't have to trust them not to wander.
  * **UI no longer asks for a key when user authentication is on.** With login enabled, your session cookie is the only thing the browser needs — paste-the-key flow goes away. The Assistant runs under your identity, audit logs show you as the actor, and your role determines what it can do (admin = full access, editor = data writes, viewer = read-only).
  * **Assistant rides through Anthropic overload.** When Claude Opus is rate-limit-shedded, the Assistant retries with backoff and, if needed, transparently switches to Sonnet for the rest of the turn — with a small inline note so you know it happened. Conversations that previously errored out now keep moving.
  * **Assistant checks the platform before suggesting external sources.** On any data-related ask ("I'm looking for X"), the agent now lists your existing pipelines and taps first, then either points you to what already exists or asks before adding more. Avoids the "let me enumerate seven public APIs" detour.
  * **Assistant auto-runs newly created taps that have no schedule.** When you build a one-shot tap, the Assistant kicks off the first run so you see real data instead of an empty pipeline. For scheduled taps it asks first, since the cron will fire on its own.
  * **Health and version endpoints are public.** Container orchestrators, status pages, and the UI's connection check no longer trip 500s when API-key auth is required.
  * **Configuration → Taps sub-tab removed.** Prompt Fragments are unchanged and still apply to tap creation, brainstorm, auto-fix, and Discovery — they're now managed via the API instead of a dedicated UI page.
  * **Tap wizard pipeline link is clickable.** The "Linked to: \<pipeline>" pill in step 4 now navigates straight to the pipeline editor.

  **Upgrading:** `docker compose pull && docker compose up -d --force-recreate datris ui`. Existing API keys keep working — they're treated as full-access until you replace them with scoped keys from the new tab. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.2" description="May 13, 2026">
  **Polish for the Assistant, AI Configuration, and MCP tabs.**

  * **Assistant: keep typing without clicking.** After you send a question, the cursor stays in the composer — so as soon as the agent finishes (or even while it's still working), you can type your next prompt without reaching for the mouse.
  * **AI Configuration: provider switches no longer wipe your overrides.** Switching the primary AI, codegen, or embedding provider used to clear the saved provider/model/endpoint on the next page load, so you had to re-pick them every time. Your selections now persist correctly through a provider switch (you still re-enter the API key when changing providers — that's intentional).
  * **Connect Your Agent: paste-and-go for local.** The MCP tab's generated config no longer asks for an API key for the default local setup — paste the snippet into your agent and it just works. A key is still required when you point the snippet at a remote instance.
  * **Hosted trial provisioning tools removed from the agent toolset** (hosted platform since retired). The Getting Started tab has also been refreshed to point at the Assistant for the common onboarding path.

  **Upgrading:** `docker compose pull && docker compose up -d`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.7.0" description="May 12, 2026">
  **A new in-product Assistant — chat your way from "I need data" to a working pipeline.**

  * **The Assistant tab.** A new top-level tab opens an in-product agent that finds an external data source, builds the tap, creates the pipeline, runs it, and shows you the result — all in one chat. You watch the model's reasoning, every tool it calls, and the live status, then click straight into the tap or pipeline it created. See [Assistant](/assistant).
  * **Real-time visibility while the agent works.** Streaming reasoning, inline tool cards with friendly labels (*"Searching the web for …"*, *"Creating tap …"*), live success/error status, and a Stop button that aborts the loop at the next checkpoint. Conversations survive navigating to other tabs and back.
  * **Credentials never enter the chat.** When the agent needs an API key or other credentials, it opens an inline credentials form right in the chat. Values go straight to the vault — they don't appear in the conversation log, your screenshots, or the model's context. The form also lets you reuse an existing tap secret instead of creating a new one.
  * **Pipelines can be edited in place.** Calling pipeline-create again with the same name now upserts the configuration without dropping the destination data. Two new knobs in the same call: a natural-key list for dedupe/upsert on every run, and a flag to wipe the destination before each run for full-snapshot workflows. Both work on PostgreSQL and MongoDB destinations.
  * **External agents get the same new tools.** Claude Desktop, Cursor, and any other MCP client connected to your Datris server now see two new tools for discovering existing tap secrets (names and field shapes, never values) plus the new dedupe and reset knobs on pipeline create.
  * **Pipeline delete now describes what it actually does.** The tool description used to claim it kept your destination data; it never has. It now correctly says it removes both the configuration and the destination data, with an explicit opt-in flag for the "keep the schema, clear the data" reset case.
  * **MCP transport upgraded — both protocols at once.** The bundled MCP server now serves the streamable-HTTP transport alongside the existing SSE transport on the same port, so the in-product Assistant and external agents connect to one running process. No configuration change needed.
  * **Discovery tab is hidden in this release.** The Assistant supersedes the Discovery tab for the common path.

  **Upgrading:** `docker compose pull && docker compose up -d`. No data migration needed. The CLI: `brew upgrade datris`. The Assistant uses your existing codegen AI configuration — with Anthropic you'll see full chain-of-thought reasoning streamed inline; on OpenAI you'll see reasoning summaries instead.
</Update>

<Update label="v1.6.21" description="May 11, 2026">
  **Web search for AI tap workflows, plus simpler MCP authentication.**

  * **AI tap workflows can consult the live web.** When enabled in Configuration → AI Providers, tap brainstorm, dataset discovery, tap diagnosis, and tap auto-fix look up current API documentation, free-tier limits, current package names, and recent deprecation notices before recommending sources or generating fixes. Pick your web-search provider independently of AI Primary — the platform uses each provider's native search tool and routes accordingly. First-pass tap script generation stays fast and uses the model's training data only. See [AI Configuration](/ai-configuration) for setup.
  * **AI Configuration changes survive Docker restarts.** Saving from the Configuration UI now mirrors the relevant keys back to your `.env` file so changes aren't lost when the local Vault container restarts. Provider switches also clear any stale credential preserved from the prior provider, so a wrong-provider key can't silently 401 the next call.
  * **Cleaner MCP authentication.** The bundled MCP server is now a transparent forwarder — each connecting agent provides its own API key per session and the MCP server passes it through to the Datris REST API on every tool call. The Configuration → Connect Your Agent panel generates the new configuration snippet automatically; paste your key from the Configuration UI into your agent's MCP config. Existing remote-instance setups see no change in behavior.
  * **Tap brainstorm asks about sources first.** When you describe data without naming where to fetch it from, the AI now lists 3-5 candidate sources (with free vs paid and key-required info) before drilling into parameters — instead of asking for filtering details up front.
  * **Tap script generation is more resilient.** The platform now validates that a generated script actually defines a `fetch()` function before storing it, and the JSON extractor handles model responses that contain narrative braces without falling back to the raw-text path.

  **Upgrading:** `docker compose pull && docker compose up -d`. No data migration needed. The CLI: `brew upgrade datris`. If you run the MCP server standalone outside Docker, the connection-target environment variable was renamed for consistency — see the [MCP server docs](/mcp-server) for the new variable name.
</Update>

<Update label="v1.6.20" description="May 6, 2026">
  **Reliable job-status polling, re-ingest that doesn't overwrite your config, and first-class catalog tooling.**

  * **Polling an upload's job status now gives you a clear answer.** Previously, when a file finished processing successfully, the response was a raw stream of progress events with no terminal status — agents and scripts couldn't tell "still running" from "done." Job status now returns a rollup with a single `allDone` flag and an aggregate outcome (`success`, `warning`, `error`), plus per-job error detail when something fails. Poll the rollup; act on the outcome.
  * **Re-ingesting a file preserves your pipeline's config.** `datris ingest` against an existing pipeline used to silently rewrite the config from CLI flags only — wiping out the catalog, custom validation rules, and any other fields you'd set through the UI or via an agent. Re-ingest now uploads into the existing pipeline as-is. To start over with a different config, delete the pipeline first.
  * **Catalogs without the read-modify-write dance.** New `--catalog` flag on `datris ingest` for new pipelines, new optional `catalog` argument on the `create_pipeline` MCP tool, and a new `set_catalog` MCP tool that retags an existing pipeline or tap in one call. Empty catalog clears the label back to Uncataloged. See [Data Catalog](/data-catalog).

  **Upgrading:** `docker compose pull && docker compose up -d`. No data migration needed. The CLI: `brew upgrade datris`.
</Update>

<Update label="v1.6.19" description="May 5, 2026">
  **Choose your embedding provider independently of your chat provider.**

  * **Mix-and-match AI providers.** The embedding slot is now configured separately from the chat and code-generation slots, so you can keep Claude for chat and code generation while pointing embeddings at OpenAI (or vice-versa). Useful when the bundled embedder is too heavy for your host, or when you want a different model family for vector quality vs chat quality. See [AI Configuration](/ai-configuration) for the full list of options.
  * **Existing installs keep their current behavior.** If you don't set an embedding override, the embedding slot continues to follow your chat provider exactly as before — Claude installs keep using the bundled embedder, OpenAI installs keep using OpenAI embeddings. The override is purely opt-in.

  **Upgrading:** `docker compose pull && docker compose up -d --remove-orphans`. No data migration needed. If you switch embedding providers on an existing deployment, vector destinations built on the previous embedder will fail-fast with a dimension-mismatch message on the next run — drop the affected destination tables or collections and re-ingest.
</Update>

<Update label="v1.6.18" description="May 5, 2026">
  **User authentication, roles, and an admin-only Configuration tab.**

  * **Optional username/password login.** Datris can now require a login before any tab is reachable. Three roles ship out of the box — **admin** (full access), **editor** (read + edit pipelines, taps, secrets), and **viewer** (read-only). Off by default, so existing installs are unaffected. See the new [User Authentication](/user-auth) doc to enable it.
  * **Configuration is admin-only.** When auth is on, only admins see the Configuration tab (Secrets, AI Providers, Taps, Users, Environment). Editors and viewers continue to use everything else.
  * **New Users sub-tab.** Admins can add, remove, and reassign roles. A built-in 16-character password generator with a reveal toggle makes handing out credentials painless. The last admin can't be deleted.
  * **Self-service password change.** Users can change their own password from the top-right user menu.
  * **Reveal toggle on the login screen.** Easier to see what you're typing on a new device.
  * **Clear the Agents activity log.** The trash icon now wipes the server-side activity buffer (with an inline confirm) so the cleared state survives a refresh.

  **Upgrading:** `docker compose pull && docker compose up -d`. No data migration needed; auth defaults to off.
</Update>

<Update label="v1.6.17" description="May 4, 2026">
  **AI-agent RAG ingestion no longer burns through the conversation context.**

  * **Creating a vector-store pipeline no longer requires sample content.** Asking an agent to ingest a PDF into pgvector previously forced it to base64-encode the entire document just to register the pipeline — wasting tens of thousands of tokens before any work began. Vector pipelines now register from a name plus destination alone, freeing budget for the actual upload.
  * **Agents are guided to send each document in a single upload.** The MCP server now makes explicit that vector destinations chunk server-side, preventing agents from needlessly splitting documents into many small uploads.
  * **Clearer Claude setup docs.** The "Configuring Claude" guide shows the recommended SSE / mcp-remote setup first, and steers large-file ingestion to the CLI rather than dragging files directly into the chat — which can overflow the conversation context on sizable PDFs.
  * **README accuracy pass.** Corrected tool counts, AI-provider model defaults, and license badge.

  **Upgrading:** `docker compose pull && docker compose up -d`. No data migration needed.
</Update>

<Update label="v1.6.16" description="May 4, 2026">
  **UI cleanup, friendlier Help menu, and a more reliable bundled embedding service.**

  * **Bundled embedding handles large ingest batches without errors.** The bundled embedding service no longer rejects large batches submitted by the platform, which previously surfaced as ingestion failures on long documents.
  * **Secrets is now a tab inside Configuration.** Instead of a separate top-level tab, Secrets lives under Configuration alongside AI Providers, Taps, and Environment. Existing `/secrets` links continue to work — they redirect to the new location.
  * **Help menu in the top bar.** The Docs link is replaced with a Help dropdown that exposes both the docs and a direct link to file an issue on GitHub.
  * **Easier setup with Claude.** A new "Configuring Claude" page in the docs walks through Claude Desktop and Claude Code setup end-to-end, including a first-prompts walkthrough for an empty install.
  * **Structured issue reporting.** GitHub issues now use forms that capture version, component, deployment mode, and reproduction steps, making bug reports easier to triage and faster to fix.

  **Upgrading:** `docker compose pull && docker compose up -d`. No data migration needed.
</Update>

<Update label="v1.6.15" description="April 30, 2026">
  **Smaller, faster default install. Lighter download, opt-in Kafka, vector ingestion fixes.**

  * **\~58% smaller `docker compose pull`.** The bundled platform now downloads roughly 11 GB less out of the box. Fresh installs come up dramatically faster.
  * **Bundled embeddings are faster on the same `bge-m3` model.** No configuration changes needed; existing vector collections built with the previous bundled embedder continue to work without re-embedding.
  * **Kafka is now opt-in.** Most local installs don't need it, so Kafka, Zookeeper, and the Kafka UI ship commented out in `docker-compose.yml`. Uncomment the bundled blocks (and the related volumes at the top of the file) to enable them. Pipelines that point at external Kafka brokers are unaffected.
  * **Vector ingestion no longer fails with a "duplicate key" error when many documents land at once.** Concurrent document loaders previously raced on creating the `pgvector` extension; the race is now serialized.
  * **Vector ingestion no longer fails on embedding providers that limit batch size.** The chunk batch size is now configurable per embedding secret (`batchSize`), with a cross-provider-safe default. OpenAI users who want to maximize throughput can set this higher.
  * **Configuration tab clarifies optional providers.** The bundled embedding option is labeled `bge-m3 (bundled)`. The AI Provider, CodeGen Provider, and Embedding Provider dropdowns each indicate that local Ollama is opt-in.
  * **Service Health no longer shows "Down" for optional services that were never enabled.** Kafka and the optional vector databases (Qdrant, Weaviate, Milvus, Chroma) now correctly report "Not Configured" until you turn them on. Existing installs may still show "Down" until their stale Vault secrets are removed.

  **Upgrading from earlier versions:** the bundled embedding service moved off Ollama on the same host port `11434`, so existing installs must pass `--remove-orphans` to release the port: `docker compose pull && docker compose up -d --remove-orphans`. Without it the upgrade fails with `Bind for 0.0.0.0:11434 failed: port is already allocated`. Your data and the cached Ollama model are preserved.
</Update>

<Update label="v1.6.14" description="April 29, 2026">
  **Reliable run-completion signals, smarter agent calls, cleaner run history.**

  * **Agents get a single "are we done?" signal when watching a pipeline load.** Polling a publisher token returns a rollup with a clear `allDone` boolean and a per-job outcome (success / warning / error / processing / timed out), so agents no longer have to interpret the raw event stream to figure out whether a run is complete.
  * **Pipeline status by publisher token works reliably for completed runs.** A storage path that occasionally hid completed runs from the publisher-token query is fixed; the query is now backed by an indexed top-level field, with a fallback for older rows so existing data resolves without a migration.
  * **`run_tap` no longer ships the records array back to the agent.** A push run returns `recordCount`, `publisherToken`, and the `persisted` / `persistedReason` flags — enough to verify ingestion via `get_pipeline_status` without bloating the agent's context. Use `test_tap` to preview a script's output (capped at 20 sample rows with a `recordsTruncated` flag).
  * **Duplicate `run_tap` calls are suppressed.** The agent skips a `run_tap` for a tap that's already in flight in the same session, and the platform debounces push runs to one per tap per 5 seconds. Prevents accidental duplicate ingestion from parallel tool calls, double-clicks, and transport retries (`persistedReason: already_running` or `debounced`).
  * **Every tap run now produces visible logs.** The script wrapper emits start / fetch / record-count lifecycle lines on every run, so run history shows useful output even when the user's script never calls `print()`.
  * **Secret values are masked in stored tap logs and exception messages.** A tap script that incidentally printed an API key or Vault-loaded credential would previously have surfaced the raw value in run history; those values are now redacted in the persisted log and the error string.
  * **Deleting a tap now cleans up its run history.** Previously, run-history rows accumulated indefinitely and could resurface under a recreated tap with the same name.
  * **Agent activity log shows full requests and responses.** The expanded view in the agent monitor no longer truncates request arguments or response bodies.
</Update>

<Update label="v1.6.13" description="April 24, 2026">
  **MCP tab and documentation sync.**

  * **MCP tab now lists every agent tool.** The in-app MCP reference was missing `get_pipeline_status`, `create_tap_secret`, `delete_tap_secret`, `get_tap_ledger`, and `query_natural`. All five are now in the catalog and the Try-It playground, and `create_tap` exposes the `tap_type` parameter.
  * **Recommended Agent Workflow rewritten to match the platform's actual flow.** The in-app workflow and the MCP docs had drifted — they started with `profile_data` (which the platform explicitly says not to use for pipeline generation), skipped the `persisted` / `persistedReason` check after `run_tap`, and didn't show the `publisherToken` poll that confirms records actually landed. All three are now canonical: check-before-create, verify-via-publisher-token, and tap credentials managed via `create_tap_secret`.
  * **Documentation: updated agent workflow examples.** RAG over external documents is now shown as a document tap (`tap_type="document"` + `get_tap_ledger`), onboarding an external source uses `create_tap_secret` for credentials, and quality monitoring of scheduled taps shows the `get_tap_logs` → `get_pipeline_status(publisher_token=...)` pivot.
</Update>

<Update label="v1.6.12" description="April 24, 2026">
  **Tap wizard reliability, iteration history, and cleaner vector-search errors.**

  * **Tap wizards learn from their own retries.** When the AI fixes, optimizes, or reviews a tap script, it now carries forward up to the last three attempts — what was tried, what went wrong, and what changed — into the next call. The wizard stops cycling through the same failed approaches.
  * **Saved tap scripts always match the tap.** Saving a tap now pushes the in-memory script to object storage before writing the tap config, and the create/update call verifies the stored script is actually there. No more "missing script" banners from an interrupted save, and auto-revert no longer strands a tap with a deleted script.
  * **Run Tap stays on the page when nothing was ingested.** If a manual run finishes without persisting records, the wizard keeps you on the run step and shows an inline reason (test mode, no records, run error) instead of navigating away and hiding the diagnostic.
  * **Tap logs now carry the publisher token.** Every tap run that submitted records records its publisher token in the log. Agents reading `get_tap_logs` can pivot directly to `get_pipeline_status` to confirm a scheduled run actually landed in the destination — not just that the script ran.
  * **Vector search fails cleanly when the embedding dimension doesn't match the collection.** If you change embedding providers on a pipeline whose vector collection already has vectors of a different dimension, search queries now return a clear 400 with a user-actionable message instead of leaking a JVM stack trace.
  * **Sturdier local-dev startup.** Kafka and Zookeeper now use named volumes (no more corruption races on rebuild), Kafka waits for Zookeeper's request processor to actually be ready (not just its listener bound), and Vault init picks up an explicit AI-provider override so a stray shell env var can't silently flip providers.
</Update>

<Update label="v1.6.11" description="April 22, 2026">
  **Agent-native tap observability, scheduler fix, and agent-owned tap secrets.**

  * **Agents can watch a tap load to completion.** Running a tap now reports back whether the data was actually persisted — and names the reason when it wasn't (test mode, no target pipeline, no records, run error). Every persisted run returns a single *publisher token* covering the whole run, even for document taps that spawn many ingestion jobs. A new `get_pipeline_status` MCP tool lets an agent poll that one token until the entire load reaches its final state, so it can report "done" with real numbers instead of guessing from a response body.
  * **Scheduled taps no longer need a manual kickoff.** Taps saved with a cron schedule now fire on their next scheduled time automatically. Previously, a newly saved scheduled tap would wait indefinitely until you ran it once by hand.
  * **Self-diagnosing tap scripts.** If a tap's generated script goes missing from object storage, the Edit Tap page now shows an amber banner explaining the state and pointing you to Regenerate, instead of a cryptic mid-run "key does not exist" error. Test Tap surfaces the same state with actionable wording.
  * **Agents can manage their own tap secrets.** Via MCP, agents can now create and delete the secrets their taps need (API keys, tokens). Scope is strictly tap-owned — agents cannot create, overwrite, or delete human-owned Platform secrets (DB creds, AI keys, vector-store creds).
  * **Secrets page split into Platform and Taps sub-tabs.** Platform lists the built-in Datris secret slots; Taps lists agent-authored tap secrets. Creating a secret from the Taps sub-tab auto-tags it so it stays agent-editable.
  * **Honest Test Tap banner.** The run-result banner on Test Tap now reflects what actually happened on the server — "sent to pipeline" only when the run was truly persisted, otherwise "not persisted" with the reason — rather than whatever the pre-request checkbox said.
  * **BYO-code taps can declare pip dependencies.** If you paste your own fetch script into Create Tap, you can now list the Python packages it needs. Previously only AI-generated taps could declare dependencies.
  * **Example agent refactored onto taps.** The bundled market-macro-agent example now drives ingestion through taps instead of ad-hoc fetch scripts, demonstrating the full agent-native tap flow (provisioning, secrets, publisher-token watching).
</Update>

<Update label="v1.6.10" description="April 21, 2026">
  **Tap prompt fragments, post-run script review, BYO code, and Configuration page reorg.**

  * **Tap prompt fragments (new).** A new Configuration → Taps sub-tab lets you save reusable context snippets — things like API conventions, required headers, rate limits, and preferred libraries for a given source. When the key or any of its aliases appears in a Create Tap description, brainstorm, auto-fix, optimize, or Discovery chat, the fragment's content is automatically added to the system prompt. Includes an AI Suggest button, a Load Examples catalog (AWS, Polygon, Stripe, SEC EDGAR), JSON import/export, and an "Extra context applied" chip row in the tap wizard showing which fragments hit.
  * **Post-run script review.** After a tap's first successful test, the AI now scans the captured stderr/stdout for signals that the *script* should change — rate-limit or burst warnings, deprecation hints, pagination cues, schema drift, auth warnings — and regenerates the script if needed. On a rewrite the wizard auto-retests; the performance optimizer runs only when the logs are clean. The optimizer's prompt was also tightened so rate-limit markers push it toward throttling instead of more concurrency.
  * **I Have My Own Code (new).** A third Tap Type on the Create Tap wizard lets you paste a fetch() script directly instead of having AI generate one. Step 1 switches to a code textarea with a Use My Code button; after upload the button flips to Re-upload My Code and Step 2 gates on the text matching what's on disk, so edits force a fresh upload before running the test.
  * **Configuration page reorganized into three sub-tabs** — Environment, AI Providers, and Taps — with a prominent "Highly recommended: Anthropic with the latest coding model" tip on the CodeGen Provider section.
  * **Tap Name collision warning.** If the name you type in the Create Tap wizard matches an existing tap, an amber banner appears under the field warning that continuing will overwrite the existing tap's configuration and script.
  * **Auto-fix retries bumped to 3.** When a tap script fails its first test, the AI now gets up to three repair attempts (was two) before giving up.
  * **Cron Custom preset no longer blocked by AI formatting.** AI-generated cron expressions wrapped in code fences, brackets, or quotes are now cleaned automatically, so the Next button is enabled on valid output.
</Update>

<Update label="v1.6.9" description="April 20, 2026">
  * **OpenAI Codex models now work for code generation.** Previously, selecting a codex-family model for CodeGen (e.g., the recommended GPT-5.3-Codex) caused tap-script generation and AI data quality / transformation to fail immediately with a 404. Datris now routes codex models to the right OpenAI endpoint automatically.
  * **Vector-store dimension changes now fail fast with a clear message.** If you switch embedding providers (for example, Ollama bge-m3 → OpenAI text-embedding-3-small) on a pipeline whose destination table or collection already has vectors of the old dimension, the job stops up front and tells you exactly what to do instead of blowing up mid-ingest with a cryptic database error. Applies to pgvector, Qdrant, Weaviate, Milvus, and Chroma destinations.
  * **Configuration save is honest about missing API keys.** Changing the AI Provider to Anthropic or OpenAI without entering that provider's API key no longer silently skips the save while reporting success — the Configuration page now flags the missing key and tells you which one to add.
  * **Create Tap brainstorm wraps up sooner.** For document taps, the AI assistant no longer drills for optional date filters once you've supplied the source, auth, and a broad scope — the tap ledger already dedupes by content, so those extra questions were just noise.
</Update>

<Update label="v1.6.8" description="April 20, 2026">
  * **OpenAI GPT-5 and reasoning-model support.** AI-backed features (Discovery chat, Create Tap brainstorm, AI data quality, AI transformation, script generation) now work with OpenAI's newer model families. Previously, selecting one of these models caused chat panels and wizards to hang without a response.
</Update>

<Update label="v1.6.7" description="April 19, 2026">
  **Document Taps — a new tap type for feeding vector stores with unstructured files.**

  * **Document Taps (new).** A purpose-built tap for ingesting PDFs, Word docs, HTML, and other unstructured files into a vector-store pipeline. Describe the source in plain English ("ingest all PDFs from our SharePoint legal folder", "pull every DOCX from `legal-contracts/2026/` in S3") and the generated tap discovers the files and hands their raw bytes to the pipeline. Text extraction, chunking, embedding, and loading are all handled downstream — you don't configure any of that on the tap.
  * **Tap type toggle in Create Tap.** Choose *Document Ingestion* or *Structured/Semi-Structured* on the first step of the wizard; the prompts, placeholders, and example instructions adapt to the choice.
  * **Ingestion ledger.** Every discovered file is tracked by URI and content hash. Re-running the tap skips files that are already up to date — no re-embedding, no duplicates. Changed files flow through normally.
  * **Pre-flight validation.** Document taps linked to a pipeline are checked at save time to confirm the pipeline is shaped for document ingestion (unstructured source, vector-store destination). Misconfigurations surface as an actionable error instead of a cryptic mid-run failure.
  * **Safe defaults for local paths.** Document taps refuse to silently walk arbitrary host directories if a requested path isn't mounted into the container — they fail loudly instead of ingesting unintended files.
  * **Faster embeddings.** The bundled Ollama sidecar now handles concurrent embedding requests in parallel and keeps the embedding model warm between pipeline runs, eliminating cold-start delays when a pipeline resumes after an idle period.
  * **More Python libraries pre-installed for taps.** AWS S3, Google Cloud Storage, Azure Blob, Excel, YAML, and date/timezone helpers are now baked into the image. Taps that fetch from these sources no longer need a per-run `pip install`.
</Update>

<Update label="v1.6.6" description="April 18, 2026">
  **Live MCP agent monitor and pipeline status self-healing.**

  * **Agents tab** — new live view of connected MCP agents. See every tool call as it happens, with agent name, arguments, record count, response size, status, and latency. Click any row to expand the full request and response.
  * **Pipeline status** now self-heals when a job completes but the summary gets stuck showing "processing" — completed, warned, and errored jobs resolve to their correct final state.
  * **Example agent** (market-macro-agent) automatically reconnects with backoff if the MCP connection drops, and degrades gracefully while offline instead of crashing.
</Update>

<Update label="v1.6.5" description="April 17, 2026">
  * New AI models from OpenAI and Anthropic now appear in the Configuration dropdowns automatically, without a Datris upgrade.
  * **Claude Opus 4.7** is the recommended Anthropic model for CodeGen (tap script generation, AI data quality rules, AI transformations, JSON Schema / XSD generation, and natural-language → SQL).
  * Fresh instances are seeded with the latest recommended models by default.
</Update>

<Update label="v1.6.4" description="April 16, 2026">
  **Two-pass AI tap optimization and Data Catalog UX.**

  * **AI optimize pass.** After a tap script passes its initial test, the platform sends it back to the LLM for a performance rewrite and re-tests. Auto-reverts on regression (>=20% slower) or failure. Also available through the API; tap test results now report run duration.
  * **Discovery wizard.** Auto-optimize per tap with before/after timing banner, per-row stop button, on-demand AI fix panel for failed items, chat auto-scroll, and re-shown Discover Datasets button after new messages.
  * **Create Tap.** Configurable test sample size (defaults to 20 records).
  * **Data Catalog.** Kebab menu on Uncataloged items with Edit, Delete, and Move to Catalog (with name-clash detection).
</Update>

<Update label="v1.6.3" description="April 15, 2026">
  **Database lockdown, tap script hardening, Create Tap UX.**

  * **Server-controlled database name.** The UI no longer edits the Postgres or MongoDB database name — the server reports the configured names and the UI submits them unchanged.
  * **MongoDB internal vs user split.** `mongodb.database` (user-facing, default `datris`) holds pipeline data; `mongodb.internalDatabase` (default `oss`) holds platform state. Existing installs keep `oss` for platform state; new user pipelines land in `datris`.
  * **Unlimited reads for tap scripts.** Tap scripts can now request every matching row from the MongoDB and Postgres query APIs instead of a capped preview. Preview defaults (20 Mongo / 100 Postgres) are unchanged for UI/MCP callers.
  * **Tap test sampling.** New "Limit test sample to 20 records" checkbox in Create Tap step 2 caps how much a test run reads. Cron and manual runs read everything.
  * **Codegen + diagnosis hardening.** Generated scripts treat platform response shapes as contractual (no shape-probing, no candidate-key iteration). Diagnosis quotes the actual traceback and respects in-script guards.
  * **Create Tap UX.** Ask button, auto-apply diagnosis (capped at 2 attempts), Stop Test, copy-to-clipboard, scrollable-JSON test results, destination collision check on Generate Pipeline, full destination shown on step 5.
  * **Data Catalog.** One-click delete of the Uncataloged group; per-item trash icon on taps and pipelines inside every catalog.
</Update>

<Update label="v1.6.2" description="April 14, 2026">
  **Input sanitization hardening and minor UI polish.**

  * Pipeline creation sanitizes destination identifiers before writing the config — Postgres `dbName`/`schema`/`table`, MongoDB `dbName`/`table`, Kafka `topic`, ActiveMQ `queueName`, vector `collectionName`/`tableName`/`schemaName`, and the DQ schema name.
  * Name sanitization is now applied consistently across the UI, so labels and identifiers are cleaned the same way everywhere.
</Update>

<Update label="v1.6.1" description="April 14, 2026">
  **Discovery wizard, Data Catalog, and bring-your-own AI keys.**

  * **Discovery wizard.** Six-step AI-guided onboarding that turns "yfinance daily prices for the S\&P 500" into running taps and pipelines. Available from the UI, the API, and the new `discover_source` MCP tool.
  * **Data Catalog.** Group related taps and pipelines into named catalogs. New Data Catalog tab with expandable contents and an Uncataloged group.
  * **Per-pipeline DQ + transformation editor.** Inline editor inside the pipeline view.
  * **Bring-your-own AI keys.** Hosted trial signups could supply their own Anthropic or OpenAI key, which took effect on the next AI call (hosted platform since retired).
</Update>

<Update label="v1.6.0" description="April 11, 2026">
  **Hosted platform release.**

  * Hosted dedicated-instance provisioning and hosted-aware Configuration UI (since retired).
  * Improved multi-user session handling.
</Update>

<Update label="v1.5.9" description="April 11, 2026">
  **Full tap MCP tool suite and user-supplied tap scripts.**

  * Four new MCP tools: `get_tap`, `test_tap`, `update_tap`, `get_tap_logs`.
  * `create_tap` accepts an optional `script` parameter for user-supplied Python `fetch()`, plus `secret_name` for Vault-injected credentials.
  * CLI: `datris tap create --script path/to/script.py` and new `datris tap show`.
</Update>

<Update label="v1.5.8" description="April 10, 2026">
  **Ollama for all AI slots and hot-reload on save.**

  * Configuration UI offers Ollama (local) as a provider for AI Primary, CodeGen, and Embedding. Bundled Ollama sidecar option pre-fills `bge-m3` for embedding.
  * Saving AI configuration takes effect immediately — no container restart required.
</Update>

<Update label="v1.5.7" description="April 9, 2026">
  **Hosted trial hardening.**

  * Hosted trial cost tuning and Configuration tab polish (since retired). Self-hosted installs continue to recommend Opus for codegen on their own keys.
</Update>

<Update label="v1.5.6" description="April 8, 2026">
  **Breaking: AI configuration restructured into three independent Vault secrets.**

  * `ai.aiSecretName` and `ai.provider` are replaced by three top-level slots — `ai.aiPrimary.secretName`, `ai.codegen.secretName`, `ai.embedding.secretName`.
  * Each Vault secret is self-describing (`provider`, `endpoint`, `model`, `apiKey`, optionally `version`). No path derivation from YAML.
  * v1.5.x deployments must update their server configuration and re-seed Vault on upgrade.
</Update>

<Update label="v1.5.5" description="April 8, 2026">
  **Onboarding and tap scheduling UX.**

  * **Getting Started tab** as the new first-run landing page; Docs ↗ top-nav link.
  * Inline cron editor on the Taps page: preset buttons, AI prompt field ("Every weekday at 4pm"), Quartz field validation, human-readable description.
  * Truncate-before-run toggle in the tap wizard (sets `truncateBeforeWrite: true` on the generated destination).
  * Hosted trial bring-your-own AI key banner on the Configuration tab (since retired).
  * **Fix:** scheduled taps now honor the configured date timezone.
</Update>

<Update label="v1.5.4" description="April 7, 2026">
  **Taps (Beta).** AI-generated Python scripts that fetch data on a schedule and stream it into a pipeline.

  * 4-step tap creation wizard (Describe → Edit & Test → Schedule → Review) with brainstorm chat and proactive env-var suggestions.
  * AI script generation with JSON-parse retry and raw-script fallback; AI diagnosis with explicit (a)/(b)/(c) options; "Apply Diagnosis" rewrites the script in place.
  * Tap secrets stored in Vault and tagged as tap-owned; per-run history is retained and viewable.
  * CRON scheduling (Quartz format), AI-generated from natural language.
  * JSON-to-CSV pipeline feed (union of keys) with column-name normalization via spell-out table (`%` → `percent`, `#` → `num`, `&` → `and`, …).
  * Pipelines page shows the feeding tap; pipeline wizard has "Create from Tap".
</Update>

<Update label="v1.5.3" description="April 3, 2026">
  **Schema Evolution.**

  * Additive evolution: new CSV columns are auto-added as `string`, `schemaVersion` increments, and `ALTER TABLE` runs on Postgres.
  * Dropped columns are left out of the Postgres load so they default to `NULL` (previously failed on typed columns).
  * Missing key fields raise a clear error.
  * Schema evolution now behaves identically for streamed and file-based ingestion.
  * Query results now include `NULL` columns.
</Update>

<Update label="v1.5.2" description="April 3, 2026">
  **Remote MCP endpoint.**

  * Per-session API key forwarding on the MCP server, with an option to reject unauthenticated remote connections.
  * Hosted trial signup and dedicated-upgrade agent tools and remote registry entry (since retired).
</Update>

<Update label="v1.5.1" description="April 3, 2026">
  * Hosted trial instances: pipeline wizard auto-populated database names (since retired).
</Update>

<Update label="v1.5.0" description="April 2, 2026">
  **Shared-instance hosting.**

  * Hosted multi-tenant isolation on a shared instance (since retired).
  * Vector DB secret isolation across Qdrant, Weaviate, Milvus, Chroma, pgvector.
  * Batch upload for compressed files (`.zip`, `.gz`, `.tar`, `.jar`) processes contents inline — no MinIO webhook dependency.
</Update>

<Update label="v1.4.4" description="March 30, 2026">
  * Default AI provider changed to Anthropic (Claude Sonnet 4.6). Customers can still switch to OpenAI or Ollama in configuration.
  * Generated Python scripts for DQ and transformation are logged in full to the server logs for debugging.
</Update>

<Update label="v1.4.3" description="March 29, 2026">
  * New unified `datris analyze` command replaces `ask-sql` and `ask`. Auto-picks the right approach based on `--dest` (Postgres → SQL generation, MongoDB → document fetch, vector stores → RAG).
  * `--ai-analyze` flag on ingest.
</Update>

<Update label="v1.4.2" description="March 27, 2026">
  **Mintlify documentation and accuracy pass.**

  * Migrated docs from `.md` to `.mdx` with a two-tab Mintlify layout.
  * Removed deprecated docs (row rules, column rules, JavaScript row functions, REST transformations, deduplication, column trimming).
  * Page-by-page accuracy review against the codebase.
</Update>

<Update label="v1.4.1" description="March 26, 2026">
  * Server fails fast at startup with a clear error if an AI provider is not configured (CodeGen DQ and transformation require it).
  * Removed stale NEVER rules and deprecated-feature references from the MCP server instructions.
</Update>

<Update label="v1.4.0" description="March 26, 2026">
  **CodeGen DQ and transformation.**

  * `aiRule` replaces the prior AI DQ approach: the LLM generates a self-contained Python validation script from a plain-English instruction, which runs locally. Cost drops from $25–40/file to ~$0.003/rule.
  * `aiTransformation` uses the same CodeGen approach.
  * Works for CSV, JSON, and XML.
  * **Removed:** `columnRules` (regex), JavaScript row rules, REST endpoint row rules, and the legacy AI data-quality path. JavaScript row functions and REST transformations removed from UI and CLI.
</Update>

<Update label="v1.3.0" description="March 26, 2026">
  * **Atomic `create_pipeline`.** `generate_schema` is removed; `create_pipeline` accepts base64-encoded sample data, auto-detects schema, and creates the pipeline in one call.
  * Content-based uploads across `upload_data` (renamed from `upload_file`), `profile_data`, `upload_config`.
  * `update_secret` MCP tool for AI provider keys only.
  * All-string schema on MCP pipeline creation; pipeline registration verified by read-back.
</Update>

<Update label="v1.2.2" description="March 25, 2026">
  * `update_secret` MCP tool added (scoped to AI secrets: anthropic, openai, ollama, embedding).
  * Published to the MCP Registry (`io.github.datris/datris`) and PyPI (`datris-mcp-server`).
</Update>

<Update label="v1.2.0" description="March 24, 2026">
  **New UI tabs.**

  * **MCP Tab** — agent view, service health for 10 backend services, browsable tool grid, config generator for Claude Desktop/Code and Cursor, and a Tool Playground that executes against the live API.
  * **Secrets Tab** — full CRUD for Vault secrets with sensitive-field masking.
  * New REST endpoints for vector store metadata discovery (Qdrant collections and friends).
</Update>


# Datris CLI
Source: https://docs.datris.ai/cli

Command-line interface reference

The Datris CLI (`datris`) is a command-line interface for the Datris Data Platform. It communicates with the platform via the MCP server, providing a simple way to ingest data, run queries, search vector stores, and manage pipelines — all from the terminal.

## Installation

```bash theme={null}
brew tap datris/tap
brew install datris
```

## Configuration

The CLI connects to the MCP server via SSE. Set the server URL with an environment variable:

```bash theme={null}
export MCP_SERVER_URL=http://localhost:3000/sse    # default
```

### Authentication

When `USE_API_KEYS=false` (the OSS default) no key is needed. When [`USE_API_KEYS=true`](/api-keys), export the key and the CLI sends it as the `x-api-key` header on every request to the MCP server:

```bash theme={null}
export DATRIS_API_KEY=<your key>
```

If the key is missing or wrong, commands stop with a one-line message telling you to set `DATRIS_API_KEY`.

<Note>
  Older CLI versions do not read `DATRIS_API_KEY`. For those, put the key on the MCP URL's query string instead, which the MCP server still accepts: `export MCP_SERVER_URL="http://localhost:3000/sse?api_key=<your key>"`.
</Note>

Issue a key dedicated to the CLI from the UI: **Configuration → API-Keys → Issue new key**. See [API Keys](/api-keys) for the full pattern.

## JSON Output

Every command supports `--json` to return raw JSON instead of human-readable output. This is useful for scripting and programmatic use.

```bash theme={null}
datris pipelines --json
datris query "SELECT * FROM trades" --json
datris analyze "top 5 stocks" --table trades --json
datris health --json
```

## Commands

### `datris help`

Show all available commands and options.

```bash theme={null}
datris --help
```

***

### `datris pipelines`

List all registered pipelines.

```bash theme={null}
datris pipelines
```

***

### `datris ingest`

Create a pipeline and ingest a data file in one step. The schema is auto-detected from the file.

If the named pipeline already exists, the existing config is preserved and the file is uploaded into it — `--dest`, `--table`, `--ai-validate`, `--ai-transform`, and `--catalog` are ignored on re-ingest. Delete the pipeline first (`datris delete <name>`) if you need a fresh config.

```bash theme={null}
datris ingest <FILE> [OPTIONS]
```

| Option | Description |
| - | - |
| `--pipeline, -p` | Pipeline name (default: derived from filename) |
| `--dest, -d` | Destination: `postgres`, `mongodb`, `snowflake`, `databricks`, `qdrant`, `weaviate`, `milvus`, `chroma`, `pgvector` (default: `postgres`) |
| `--table, -t` | Table/collection name (default: pipeline name) |
| `--database` | Database name (default: `datris`). For `snowflake`: the Snowflake database; for `databricks`: the Unity Catalog name — required for both |
| `--schema` | Destination schema (`snowflake` default: `PUBLIC`; `databricks` default: `default`) |
| `--warehouse` | Snowflake warehouse name, or Databricks SQL warehouse ID (required for Snowflake; for Databricks, optional when the secret has a `warehouse` field) |
| `--credentials-secret` | Platform secret holding destination credentials (required for `snowflake` and `databricks`) |
| `--ai-validate` | AI data quality rule — plain-English instruction |
| `--ai-transform` | AI transformation instruction — plain-English instruction |
| `--ai-analyze` | Ask a question about the data after ingestion completes |
| `--catalog` | Catalog label to group this pipeline with related pipelines (e.g. `openclaw`). Free-form — no need to pre-create. Only applied when creating a new pipeline. |
| `--json` | Return raw JSON |

**Examples:**

```bash theme={null}
# Basic ingestion — auto-derives pipeline name from filename
datris ingest sales-data.csv --dest postgres

# With explicit pipeline name
datris ingest sales-data.csv --pipeline sales --dest postgres

# With AI validation
datris ingest trades.csv --dest postgres --ai-validate "all prices must be positive and dates must be YYYY-MM-DD"

# With AI transformation
datris ingest trades.csv --dest postgres --ai-transform "convert all date columns to YYYY/MM/DD format"

# Both validation and transformation
datris ingest trades.csv --dest postgres \
  --ai-validate "all prices must be positive" \
  --ai-transform "convert dates to YYYY/MM/DD and uppercase all ticker symbols"

# Ingest into MongoDB
datris ingest events.json --dest mongodb --database analytics

# Ingest into Snowflake (credentials come from a Platform secret)
datris ingest trades.csv --dest snowflake --database ANALYTICS \
  --warehouse COMPUTE_WH --credentials-secret snowflake-creds

# Ingest into a vector store for RAG
datris ingest manual.pdf --dest pgvector

# Group related pipelines under a catalog
datris ingest legal-2026.pdf --pipeline legal_2026 --dest pgvector --catalog legal

# Ingest + analyze in one command
datris ingest trades.csv --dest postgres --ai-analyze "What are the top 5 stocks by volume?"

# Ingest a document and ask a question about it
datris ingest annual-report.pdf --dest pgvector --ai-analyze "What was the company's revenue?"

# Ingest + analyze, raw JSON output for scripts
datris ingest trades.csv --dest postgres --ai-analyze "top 5 by volume" --json
```

***

### `datris query`

Execute a read-only SQL SELECT query against PostgreSQL.

```bash theme={null}
datris query "SELECT * FROM public.sales LIMIT 10"
datris query "SELECT symbol, close FROM public.trades WHERE close > 100" --limit 50
```

| Option | Description |
| - | - |
| `--limit` | Max rows returned (default: 100; pass `-1` for no limit) |
| `--json` | Return raw JSON |

***

### `datris search`

Semantic search across a vector database.

```bash theme={null}
datris search "What is the return policy?" --store pgvector --collection support_docs
```

| Option | Description |
| - | - |
| `--store`, `-s` | Vector store: `qdrant`, `weaviate`, `milvus`, `chroma`, `pgvector` (default: `pgvector`) |
| `--collection`, `-c` | Collection/table name (required) |
| `--top-k`, `-k` | Number of results (default: 5) |
| `--json` | Return raw JSON |

***

### `datris analyze`

Ask a question about your data using AI. Works with any destination type — auto-picks the right approach based on `--dest`.

```bash theme={null}
datris analyze <QUESTION> --table <TABLE> [OPTIONS]
```

| Option | Description |
| - | - |
| `--table, -t` | Table/collection name (required) |
| `--dest, -d` | Data source type: `postgres`, `mongodb`, `snowflake`, `databricks`, `qdrant`, `weaviate`, `milvus`, `chroma`, `pgvector` (default: `postgres`) |
| `--pipeline, -p` | Pipeline name (required for `snowflake`/`databricks` — the pipeline selects the connection) |
| `--top-k, -k` | Number of search results for vector stores (default: 5) |
| `--json` | Return raw JSON instead of AI narrative |

**Examples:**

```bash theme={null}
# Analyze PostgreSQL data — AI generates SQL, executes it, returns AI answer
datris analyze "What are the top 5 stocks by volume?" --table trades

# Analyze MongoDB data
datris analyze "How many events occurred in March?" --table events --dest mongodb

# Analyze a Snowflake or Databricks destination — pipeline selects the connection
datris analyze "Which region had the most sales?" --table sales --dest snowflake --pipeline sales

# Analyze vector store data (RAG) — semantic search + AI answer
datris analyze "What is the return policy?" --table support_docs --dest pgvector
datris analyze "What was quarterly revenue?" --table financial_docs --dest qdrant

# Raw JSON output for scripts
datris analyze "top 5 stocks" --table trades --json
```

**How it works by destination:**

* **PostgreSQL** — AI generates a SQL query from your question, executes it, then summarizes the results in a natural language answer
* **MongoDB** — fetches documents from the collection, then AI answers the question based on the data
* **Snowflake / Databricks** — previews rows from the pipeline's destination table (connection and credentials resolved server-side from the pipeline config), then AI answers the question based on the data
* **Vector stores** — performs semantic search to find relevant chunks, then AI generates an answer from the retrieved context

***

### `datris query-mongo`

Query a MongoDB collection with optional filter and projection.

```bash theme={null}
datris query-mongo events
datris query-mongo events --filter '{"status": "active"}' --limit 20
datris query-mongo events --projection '{"name": 1, "status": 1}'
```

| Option | Description |
| - | - |
| `--filter`, `-f` | MongoDB filter JSON (default: `{}`) |
| `--projection` | Fields to include/exclude |
| `--limit` | Max documents (default: 20) |
| `--json` | Return raw JSON |

***

### `datris status`

Get the latest job status for a pipeline.

```bash theme={null}
datris status my_pipeline
```

***

### `datris delete`

Delete a pipeline configuration together with its destination data.

```bash theme={null}
datris delete my_pipeline
```

| Option | Description |
| - | - |
| `--json` | Return raw JSON |

Deleting a pipeline always deletes its destination data. `--keep-data` is not supported: it exits with an error and deletes nothing. To keep the data, copy it or point a new pipeline at the same prefix before deleting. Any warnings the server returns (for example, a Unity Catalog table an admin still has to drop) are printed after the success line, one per line, prefixed with `⚠`.

***

### `datris health`

Check the health of all backend services.

```bash theme={null}
datris health
```

***

### `datris secrets`

List all configured secrets.

```bash theme={null}
datris secrets
```

***

### `datris taps`

List all taps.

```bash theme={null}
datris taps
```

***

### `datris tap create`

Create a tap. The `instruction` argument (a plain-English description) is optional — provide it to have AI generate the fetch script, or omit it and supply your own script with `--script` (or create a config-only tap with neither).

```bash theme={null}
datris tap create ["<instruction>"] --pipeline <name> [--name <tap-name>] [--script <file>] [--cron "<expression>"] [--type structured|document] [--kind python|http --endpoint-url <url>]
```

| Argument | Description |
| - | - |
| `instruction` | Optional plain-English description of the data to fetch. When provided, AI generates the script. Omit when using `--script` or creating a config-only tap |

| Option | Description |
| - | - |
| `--pipeline`, `-p` | Target pipeline name |
| `--name`, `-n` | Tap name (default: derived from pipeline or instruction) |
| `--cron` | CRON expression for scheduling (Quartz format) |
| `--secret` | Vault secret name for credentials injected as env vars |
| `--script` | Path to a Python script file with a `fetch()` function (skips AI generation) |
| `--type` | `structured` (default) returns rows of records; `document` returns `{uri, filename, content}` dicts destined for a vector-store pipeline. See [Document Taps](/taps#document-taps) |
| `--kind` | Tap implementation kind: `python` (default) runs a script on the platform; `http` calls a user-hosted endpoint speaking the tap HTTP contract (no instruction or `--script`) |
| `--endpoint-url` | For `--kind http`: absolute http(s) URL Datris POSTs the run context to on each run (required with `--kind http`) |
| `--json` | Return raw JSON |

**Examples:**

```bash theme={null}
# Structured: rows into a Postgres/Mongo pipeline
datris tap create "Fetch daily stock prices for S&P 500 from yfinance" --pipeline stocks --cron "0 0 0 * * ?"
datris tap create "Get weather data from Open-Meteo API" --pipeline weather --name weather-tap

# Document: raw file bytes into a vector-store pipeline
datris tap create "Discover all PDFs in the S3 bucket 'contracts' under prefix 2026/" \
  --pipeline contracts-vec --type document --secret aws-creds
```

Document taps require a target pipeline whose source is `unstructuredAttributes` and whose destination is a vector store (qdrant, pgvector, weaviate, milvus, or chroma). The server rejects `tap create --type document` against a structured pipeline with HTTP 400.

***

### `datris tap run`

Run a tap manually. Output reflects whether the fetched records actually landed in the target pipeline:

```bash theme={null}
$ datris tap run stock-prices
  Running tap 'stock-prices'...
  ✓ success — 33 records fetched
    → persisted to stock-prices
    → watch: datris pipeline status --publisher 5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e
```

Or if the run succeeded but records weren't persisted (missing target pipeline, test mode, script error, zero records), the CLI tells you exactly why:

```bash theme={null}
  ✓ success — 33 records fetched
    → not persisted (no_target_pipeline)
```

Pass `--json` to get the full response including `mode`, `persisted`, `persistedReason`, `publisherToken`, and `pipelineTokens`.

***

### `datris tap show`

Show full details of a tap, including its script. Prints the name, description, target pipeline, schedule, enabled state, secret, and last run/test results, followed by the stored script.

```bash theme={null}
datris tap show <name>
```

| Option | Description |
| - | - |
| `--json` | Return raw JSON |

***

### `datris tap test`

Test-run a tap without pushing the fetched records to the target pipeline. Reports the run status, record count, and data type.

```bash theme={null}
datris tap test <name>
```

| Option | Description |
| - | - |
| `--json` | Return raw JSON |

***

### `datris tap logs`

Show the run history for a tap — each entry's timestamp, status, mode (run or test), record count, duration, and any error.

```bash theme={null}
datris tap logs <name>
```

| Option | Description |
| - | - |
| `--json` | Return raw JSON |

***

### `datris tap update`

Update a tap's configuration without regenerating its script. Specify at least one option.

```bash theme={null}
datris tap update <name> [--enabled | --disabled] [--cron "<expression>"] [--pipeline <name>] [--description "<text>"] [--endpoint-url <url>]
```

| Option | Description |
| - | - |
| `--enabled` / `--disabled` | Enable or disable the tap |
| `--cron` | CRON expression for scheduling (Quartz format) |
| `--pipeline`, `-p` | Target pipeline name |
| `--description`, `-d` | New description |
| `--endpoint-url` | New endpoint URL (HTTP taps only) |
| `--json` | Return raw JSON |

***

### `datris tap delete`

Delete a tap.

```bash theme={null}
datris tap delete <name>
```

***

### `datris catalog export`

Download a catalog as a zip archive: every tap and pipeline in it, each tap's script and incremental state, and the validation schemas its pipelines reference. Secret values, destination data, run history and version history are never included. The command prints how many taps and pipelines the archive holds and which secrets the catalog needs.

```bash theme={null}
datris catalog export <NAME> [OPTIONS]
```

| Option | Description |
| - | - |
| `--out` | Archive file to write (default: `<NAME>.zip`) |
| `--json` | Print the raw tool result (includes the archive as base64) |

**Example:**

```bash theme={null}
datris catalog export sales --out sales-backup.zip
```

Uncataloged is not a catalog and cannot be exported. On an error nothing is written and the command exits 1.

***

### `datris catalog restore`

Restore an archive made by `datris catalog export`, under the same name or a new one. A restore never merges into an existing catalog: it is refused when the catalog already exists or when any archived tap or pipeline name already exists anywhere on this install, and each clash is printed. Restored taps start disabled; test each one and then enable it. Secrets are never written: the command names any secret the restored items need that does not exist yet.

```bash theme={null}
datris catalog restore <FILE> [OPTIONS]
```

| Option | Description |
| - | - |
| `--as` | Catalog name to restore into (default: the name stored in the archive). Letters, digits, `_` and `-` only |
| `--overwrite` | Replace the items already in the target catalog instead of creating a new one. Asks you to type the catalog name to confirm |
| `--yes` | Skip the `--overwrite` confirmation prompt (for scripts) |
| `--no-state` | Skip each tap's incremental state, so a clone starts from scratch instead of from the source's cursor |
| `--json` | Print the raw tool result |

**Examples:**

```bash theme={null}
# Restore under a new name
datris catalog restore sales-backup.zip --as sales_restored

# Clone without the source taps' incremental state
datris catalog restore sales-backup.zip --as sales_copy --no-state

# Replace the contents of an existing catalog
datris catalog restore sales-backup.zip --as sales --overwrite
```

Tap and pipeline names are unique across the install, so restoring onto the same install under a new name is refused while the source catalog still exists: delete or rename the source first. The command exits 1 on an error, on a refusal, and when some items could not be restored (each is printed with its reason).

***

### `datris version`

Get the server version.

```bash theme={null}
datris version
```

***

### `datris doctor`

Operational self-check: the server's own checks (Vault token expiry, AI slot secrets, embedding model loaded, disk, version skew) plus host checks that need Docker (anonymous volumes, container env drift, orphaned containers, stale source build). Every finding prints the command that fixes it; nothing is changed. See [Doctor](/doctor) for the full check list.

```bash theme={null}
datris doctor                   # human-readable report
datris doctor --json            # the same report as JSON
datris doctor --probes ai       # also confirm each AI model answers (spends a few tokens)
datris doctor --pre-upgrade     # host checks only — run before every upgrade
```

| Option | Description |
| - | - |
| `--pre-upgrade` | Host checks plus the AI slot secrets via the vault container; works with the server stopped. Prints the upgrade command when clean. |
| `--probes ai` | Opt in to the AI reachability probe |
| `--json` | Print the merged report as JSON |
| `--project-dir DIR` | Directory holding `docker-compose.yml` and `.env` (default: current directory) |
| `--compose-file FILE` | Compose file to inspect (e.g. `docker-compose.standalone.yml`) |

Unlike the other commands, `doctor` calls the server's REST API directly rather than going through the MCP server (`DATRIS_URL`, default `http://localhost:8080`). It reads the same `DATRIS_API_KEY` as every other command when API keys are on — a dead MCP server is itself a finding, not a reason to fail.

Exit codes: `0` all ok, `1` any warning, `2` any error, `3` server unreachable (host checks still ran).

## Pipeline Name Auto-Detection

When `--pipeline` is not specified, the CLI derives the pipeline name from the filename:

* `sales-data.csv` → `sales_data`
* `Q1 Revenue Report.csv` → `q1_revenue_report`
* `trades.json` → `trades`

The extension is stripped, hyphens and spaces are replaced with underscores, and the name is lowercased.


# Configuration Assistant
Source: https://docs.datris.ai/configuration-assistant

Read and change AI providers, secrets, users, API keys and agent policy from a chat panel on the Configuration tab. Every change waits for your confirmation; secrets go in a form, never in chat.

The Configuration assistant is a chat panel on the **Configuration** tab. Ask it what a setting is, or tell it what to change, and it reads the same settings the sub-tabs show and proposes the change for you to confirm.

## What it is

A narrow **Ask** rail sits at the right edge of the Configuration tab. Click it, or press `Cmd+\`, to open a panel titled **Configuration assistant**. Press `Cmd+\` again, or use the collapse arrow, to close it.

The panel stays open while you move between sub-tabs, and it knows which sub-tab you are looking at, so "what is set here?" means the sub-tab in front of you. The conversation is kept when you leave Configuration and come back; the **+** button in the header starts a new one.

An empty panel offers four starting points:

| Button | What it asks |
| - | - |
| What is each AI slot using? | List each AI slot with the provider and model it is using right now. |
| Add a secret for a new tap | Help me add a secret for a new tap and tell me which fields it needs. |
| Who has admin? | Which users have the admin role? |
| Run doctor | Run doctor and summarize anything that needs attention. |

## What it can read and change

It covers the same ground as the sub-tabs, one row per sub-tab:

| Sub-tab | Reads | Changes |
| - | - | - |
| AI Providers | Each AI slot (primary, code generation, embedding, web search) with its provider and model, and which provider credentials are set | Point a slot at a provider and model; enter a provider's credentials in a masked form. Not offered on trial or hosted installs, matching the sub-tab |
| Secrets | Secret names and types, and the field names of one secret. Values are never returned | Create or replace a secret; delete a secret |
| Data Sources | The data-sources prompt fragment and whether it is on | Replace the fragment and turn it on or off |
| Code Repository | The repository settings used to store generated scripts; a connection test | Update the settings; store an access token through a masked form; delete a stored token |
| Users | Users and their roles. Only when the Users sub-tab is shown (`USE_USER_AUTH=true`) | Create a user, change a role, reset a password, delete a user. The `admin` user cannot be deleted, matching the sub-tab |
| API-Keys | Keys by label with their capabilities, the capability templates and the full capability list. Only when the API-Keys sub-tab is shown (`USE_API_KEYS=true`). Key values are never returned | Issue a key from a template or a capability list; rotate a key; revoke a key |
| Agent Policy | Whether the policy is on, the current policy, the recommended policy and the known actions | Change actions, overrides or limits; switch to the recommended policy |
| Audit Log | Read-only: search entries by time, category, action, actor, outcome or resource, and list the values present for each filter | Nothing |
| Doctor | Read-only: run the platform health checks, optionally with a small test request to each configured AI provider | Nothing |

So you can ask the audit log a question in plain language ("what changed yesterday?", "who rotated a key this week?") and get an answer from the entries themselves, and you can run [Doctor](/doctor) from the chat and get a summary of what needs attention.

## Every change is confirmed

The assistant proposes one change at a time. Each proposal appears in the panel as a card with a short summary of what will happen and two buttons, **Confirm** and **Cancel**.

* Nothing is written until you click **Confirm**.
* **Cancel** does nothing, and the assistant says the change was cancelled.
* A confirmation works once. It also expires after ten minutes, so an old card left in the conversation cannot be confirmed later; ask again for a fresh one.

Before it proposes removing a user, an API key, a secret or a repository token, the assistant reads the current list and names the exact one it will remove. If your request could mean more than one, it asks which.

Changes you confirm here are carried out as your own actions. The [Agent Policy](/agent-policy) gates agents and API-key clients, not the Configuration assistant, so a confirmed change is not queued for approval: your Confirm is the approval.

## Secrets never go in the chat

For AI provider credentials, code repository access tokens and secret values, the panel opens a masked form on the card. The value goes from your browser straight to the platform; the assistant only learns that it was provided.

The assistant will not ask you to paste a key, token or password into the conversation. If an answer shows a value as `••••••••`, the value is set, not missing: masked values are never revealed.

## Shown once

Two kinds of value are shown once, on the card, with a **Copy** button:

* the temporary password for a new user;
* the value of a newly issued or rotated API key.

They appear on that card and nowhere else, and the assistant does not repeat them in later answers. Copy the value before you move on. Resetting a password does not produce a value; the user sets a new one at next login, as described in [User Authentication](/user-auth).

## Who can use it

* With user authentication on, the panel is shown to admins only.
* With user authentication off, it is shown to everyone who can open Configuration.
* On a trial install it is not shown.

With user authentication on, the assistant needs a signed-in admin session. A client that authenticates with an API key, including the CLI and MCP agents, cannot use it, and the MCP tool catalog does not gain any of these abilities: agents still see Platform-tab secrets as read-only (see [the Assistant](/assistant#platform-tab-secrets-are-visible-but-read-only)) and still need a scoped key for everything else (see [API Keys](/api-keys)).

## Audited as you

With user authentication on and the audit log on, every change the assistant makes lands in the [audit log](/audit-log#who-is-recorded) (best effort by default; see [strict mode](/audit-log#strict-mode)) attributed to the signed-in admin, exactly as if you had made it in the sub-tab, with `via: config-chat` in the entry detail so you can tell which changes came through the panel. With user authentication off there is no signed-in user, and the entry is recorded like any other UI action.

## Refresh and Show me

When a change succeeds, the sub-tab that owns the setting reloads on its own, so what you see matches what was saved. A **Show me** link on the card switches to that sub-tab.

Links that open Configuration on a given sub-tab now work for Code Repository too (`?tab=code-repo`).

## What it leaves to other panels

The Configuration assistant handles settings. Other questions go elsewhere, and it will say which panel to use:

* building taps and pipelines, including Live Read, belongs to the [Assistant](/assistant);
* runs, failures, retries and recovery belong to the Ops panel (see [Incidents](/incidents));
* questions about the data itself belong to the Search panel.

The settings those rely on (AI providers, secrets, the code repository, users, API keys and agent policy) are the Configuration assistant's.


# Configuration Reference
Source: https://docs.datris.ai/configuration-reference

Complete configuration reference

The pipeline server is configured via `application.yaml` (or `application.properties`). This page documents all available properties.

## Changing Settings After Install

Datris settings fall into **two buckets**, and which bucket a setting is in determines how you change it after the first install. Editing the wrong place — or restarting the wrong way — is the most common source of "I changed it and nothing happened."

### Bucket 1 — Configuration tab (Vault-backed)

AI provider / model / API keys, the embedding slot, and all connection secrets (Postgres, MongoDB, MinIO, vector stores, API keys) are stored in Vault and **edited in the Configuration tab**.

`.env` seeds these **only on the very first boot**. After that, Vault persists on disk and the **Configuration tab is the source of truth** — editing the corresponding `.env` value later has **no effect**. Saves in the tab go straight to Vault and take effect right away (AI config hot-reloads in-process; other secrets are picked up on next use), with no restart needed.

To deliberately throw away the persisted config and re-seed from `.env`, wipe the Vault volume — see [How Configuration Persists](/ai-configuration#how-configuration-persists).

### Bucket 2 — Container environment variables (`.env`, read every boot)

The deployment-level settings in the [Container Environment Variables](#container-environment-variables) table — `JAVA_OPTS`, `PIPELINE_MAX_PAYLOAD_MB`, `USE_USER_AUTH`, `USE_API_KEYS`, `USE_AUDIT_LOG`, `AUDIT_LOG_STRICT`, `PROVENANCE_STRICT`, `CAPABILITY_ENFORCEMENT`, `TAP_SCRIPT_TIMEOUT_SECONDS`, `TAP_RUN_TIMEOUT_SECONDS`, the tap-runner vars, etc. — are **not** Vault-seeded. They're plain container env vars read **fresh on every boot**, so changing them in `.env` after install works normally:

```bash theme={null}
# 1. Edit the value in .env, then:
docker compose up -d datris
```

`up -d` recreates the `datris` container with the new environment. (Use `--force-recreate datris` if Compose doesn't detect the change.)

<Warning>
  Do **not** use `docker compose restart` to apply an `.env` change. `restart` reuses the existing container with its **old** environment, so your edit is silently ignored. Always use `up -d` (which recreates the container) or a `down` / `up`.
</Warning>

**Enabling `USE_USER_AUTH` is a clean toggle.** When the user table is empty the server seeds an `admin` user with a random bootstrap password and prints it once, in the log of that container only (`docker compose logs datris | grep "Bootstrap login"`). Read and save it **before** you recreate the container to turn the flag on, because recreating discards that log; your first login is then `admin` with that password, and there's no separate provisioning step. If the line is already gone, follow [Recovering admin access](/user-auth#recovering-admin-access). Change the password after you log in. See [User Authentication](/user-auth) for the full walkthrough.

## Full Reference

### Spring Boot

| Property | Default | Description |
| - | - | - |
| `spring.servlet.multipart.max-file-size` | `${DATRIS_MAX_UPLOAD_MB:1024}MB` | Maximum upload file size (1 GB unless `DATRIS_MAX_UPLOAD_MB` is set) |
| `spring.servlet.multipart.max-request-size` | `${DATRIS_MAX_UPLOAD_MB:1024}MB` | Maximum request size (same variable) |
| `spring.servlet.multipart.location` | `${DATRIS_TEMP_DIR:/tmp/datris-staging}/_multipart` | Where an upload is spooled while it is received — on the staging volume, not the container's writable layer |
| `spring.server.tomcat.connection-timeout` | `600000` | Tomcat connection timeout (ms) |

### Logging

| Property | Default | Description |
| - | - | - |
| `logging.level.root` | `INFO` | Root log level |
| `logging.level.org.springframework.web` | `INFO` | Spring web log level |
| `logging.level.ai.datris` | `INFO` | Pipeline log level |

### Scheduling

| Property | Default | Description |
| - | - | - |
| `schedule.checkFileNotifierQueue` | `5000` | Polling interval for file notification queue (ms) |
| `schedule.findJobsToStart` | `5000` | Interval to check for queued jobs (ms) |
| `schedule.checkDatabaseSourceQueries` | `30000` | Interval to check for database pulls (ms) |
| `schedule.checkTapSchedules` | `30000` | Interval to check for taps with CRON schedules due to run (ms) |
| `schedule.incidentSweep` | `300000` | Recovery-agent sweep interval (ms): resumes approved actions and raises stale/volume signals |
| `schedule.doctorTick` | `60000` | How often (ms) the periodic [doctor](/doctor) checks whether `doctor.intervalMinutes` has elapsed |

### Pipeline

| Property | Default | Description |
| - | - | - |
| `environment` | `oss` | Environment name. Used as prefix for bucket names (`oss-raw`, `oss-data`, etc.) and table names |
| `useApiKeys` | `false` | Enable API key authentication. See [Authentication](#authentication) and the [API Keys guide](/api-keys) |
| `multiTenant` | `false` | Enable per-request tenant resolution. When true, the postgres database is overridden per-request with the tenant name |
| `sendPipelineNotifications` | `true` | Enable pipeline event notifications |
| `ttlFileNotifierQueueMessages` | `60` | Days to retain processed message IDs for deduplication |
| `tapScriptTimeoutSeconds` | `300` | Wall-clock ceiling for a tap **test** (`mode=test`), in seconds. Set it from `TAP_SCRIPT_TIMEOUT_SECONDS` in `.env` |
| `tapRunTimeoutSeconds` | `3600` | Wall-clock ceiling for a real or scheduled tap **run**, in seconds. When unset it resolves to `max(3600, tapScriptTimeoutSeconds)`, so an install that raised the old single ceiling never gets a shorter one. Set it from `TAP_RUN_TIMEOUT_SECONDS` in `.env` |
| `versionCap` | `50` | Definition versions retained per tap / pipeline before older snapshots (and their pinned scripts) are pruned |
| `cron.retry.enabled` | `true` | Automatically retry failed scheduled (CRON) tap runs that fed nothing downstream |
| `cron.retry.cap` | `3` | Maximum automatic retries per failed scheduled tap run |
| `cron.retry.backoffMinutes` | `5,15` | Comma-separated backoff ladder (minutes) between automatic retries; the last value repeats |
| `pipelineMaxPayloadMB` | `4096` | Per-run payload disk budget in megabytes: the most one run — a tap run, an upload or an object-store pickup — may stage on disk (`DATRIS_TEMP_DIR`) before it is stopped with an actionable error naming `PIPELINE_MAX_PAYLOAD_MB` (raise it or split the source; for a tap, chunk the source range via `run_tap` `params`). `0` = unlimited. Records stream to the staged file and never sit in JVM heap, so this bounds disk, not memory. Size it from the data: a staged record costs about 26 bytes × column count for numeric/timestamp/short-string data (JSON keys repeat on every row), so 20,000,000 rows × 25 columns stage roughly 13 GB. Override per deployment with the `PIPELINE_MAX_PAYLOAD_MB` env var. |
| `tapMaxOutputMB` | unset | **Deprecated** alias of `pipelineMaxPayloadMB`, honoured for one release only when `pipelineMaxPayloadMB` is unset; a warning is logged at boot when it is set. |
| `scratchInlineRows` | `200` | For Live Read pipelines (the `scratch` destination): how many result rows ride inline on the run status (`resultPreview`), and the hard maximum page size for `GET /api/v1/pipeline/result`. Override per deployment with the `SCRATCH_INLINE_ROWS` env var. |
| `scratchRetentionHours` | `24` | How long a Live Read result object is kept before the hourly sweep deletes it; reads after expiry return 410 and the caller runs the pipeline again. A Live Read result holds whatever the source returned, so retention is the mitigation — keep it short. `0` expires results immediately. Override per deployment with the `SCRATCH_RETENTION_HOURS` env var. |
| `datris.agent.toolResultMaxChars` | `32000` | In-app assistants (Assistant, Ops, Catalog, Configuration and Search chats, recovery agent): the most characters of one tool result passed back to the model. The chat always shows the full result; past the cap the model sees a note saying how much was cut. |
| `datris.agent.toolResultMaxCharsScript` | `131072` | The same cap for tools that return code (`get_tap`, `get_tap_version`, `diff_tap_versions`, and any tool whose name ends in `_script`), so an assistant can read a whole tap script. Never lower than `datris.agent.toolResultMaxChars`. |

### Authentication

All four governance controls (`USE_USER_AUTH`, `USE_API_KEYS`, `USE_AUDIT_LOG`, `USE_AGENT_POLICY`) default to `false` and are switched on for production together — see [Switch on the governance controls](/quick-start#2-switch-on-the-governance-controls) for the installer option and the `.env` block; the `governance.controls` [doctor](/doctor) check reports any that is off.

Two independent flags control access. Set them together — see [the supported matrix](/api-keys#supported-configurations) — and use [User Authentication](/user-auth) and [API Keys](/api-keys) for the full setup walkthroughs.

| Env var | Default | Description |
| - | - | - |
| `USE_USER_AUTH` | `false` | Require login for the UI; gate Configuration and Users tabs to `admin`. See [User Authentication](/user-auth) |
| `USE_API_KEYS` | `false` | Require `x-api-key` on every REST and MCP request. Maps to `useApiKeys` above |
| `MULTI_TENANT` | `false` | Advanced: per-tenant database isolation. Requires `USE_API_KEYS=true`; each request is routed to the tenant environment its API key belongs to |
| `OIDC_ENABLED` | `false` | `oidc.enabled`: single sign-on through your identity provider ("Sign in with SSO" on the login page). Only takes effect with `USE_USER_AUTH=true`; a wrong or incomplete setting switches SSO off (ERROR in the log) and never blocks startup or password login. See [Single sign-on (OIDC)](/user-auth#single-sign-on-oidc) |
| `OIDC_ISSUER` | empty | `oidc.issuer`: the provider's issuer URL; must equal the `issuer` in its discovery document exactly |
| `OIDC_CLIENT_ID` | empty | `oidc.clientId`: the client id registered at the provider. The client secret is not an env var: it lives in Vault under `oss/oidc` (`oidc.secretName`), key `clientSecret` |
| `OIDC_REDIRECT_URI` | empty | `oidc.redirectUri`: `https://<ui-host>/api/v1/auth/oidc/callback`, registered at the provider |
| `OIDC_SCOPES` | `openid email profile` | `oidc.scopes`: scopes requested at sign-in |
| `OIDC_USERNAME_CLAIM` | `email` | `oidc.usernameClaim`: the ID-token claim that becomes the Datris username |
| `OIDC_DEFAULT_ROLE` | empty | `oidc.defaultRole`: role for a person with no Datris user. Empty refuses them; `viewer` or `editor` creates them. `admin` is not accepted and counts as empty |
| `CAPABILITY_ENFORCEMENT` | `enforce` | Default: scoped keys missing the required capability get HTTP 403. Override to `log-only` to emit `would-deny` log lines without rejecting — useful only when iterating on a new scope policy and you want a trial run before flipping enforcement on. Legacy `*:*` keys are unaffected either way |
| `DATRIS_ALLOW_PRIVATE_EGRESS` | `false` | HTTP taps and REST endpoint destinations refuse loopback, link-local, private, and cloud-metadata addresses. Set `true` only if your endpoints legitimately live on an internal network. See [Tap Execution & Isolation](/tap-execution-isolation#always-on-protections). |
| `USE_CODEGEN_RUNNER` | `true` (compose) / unset=`false` (sbt) | Run the Python scripts generated for AI data-quality rules and AI transformations in the isolated `datris-codegen-runner` sidecar (no platform secrets, no server files, no internet). With it on, a stopped or unreachable runner fails the rule or transformation; there is no in-process fallback. Set `false` to run them in-process (warned at startup, per run, and by `datris doctor`). See [Tap Execution & Isolation](/tap-execution-isolation#generated-data-quality-and-transformation-scripts). |
| `CODEGEN_RUNNER_URL` | `http://datris-codegen-runner:8090` | Address of the CodeGen runner service. Shares `TAP_RUNNER_TOKEN` with the tap runner. |
| `CODEGEN_RUNNER_API_BLOCK` | `true` | With `USE_CODEGEN_RUNNER=true`, the server answers `403` to every request from the CodeGen runner (its `codegen-net` interface and the runner's address). Set `false` only if the guard refuses traffic it should not. See [Tap Execution & Isolation](/tap-execution-isolation#generated-data-quality-and-transformation-scripts). |
| `CODEGEN_SCRATCH_DIR` | unset (compose sets `/scratch` on `datris-codegen-runner`) | Set on the runner container, not on `datris`. Directory where the runner keeps each script's input and output (the `codegen-scratch` volume in compose). Setting it is what turns on the cleanup controlled by the next two variables. |
| `CODEGEN_SCRATCH_DEDICATED` | `1` (image default) | Runner container. When `1` and `CODEGEN_SCRATCH_DIR` is set, that directory is treated as used for nothing else: **everything in it** is removed after every run and when the runner starts. Never point `CODEGEN_SCRATCH_DIR` at a shared path with this on. Set any other value to remove only the runner's own `cg_*` run directories. |
| `CODEGEN_TMP_DIR` | `/tmp` (image default) | Runner container. When `CODEGEN_SCRATCH_DIR` is set, this directory (the runner's small `/tmp`) is emptied after every run and at start, so a script cannot leave files there for the next run. |
| `DATRIS_TAP_SECRET_SCOPE` | `tap` | Which secrets a tap may use. `tap` (default): only secrets tagged `_type=tap`; saving a tap on any other secret returns HTTP 400, and a tap saved earlier on a platform secret fails its run with the same message before any field is injected. `any`: taps may also use platform secrets (the behaviour before this setting existed), which lets anyone who can create and run a tap read those values. Any other value means `tap` and is logged once. The server-managed secrets (field protection keys, API-key stores) are refused either way. Affected taps are named in a startup warning and by the `tap.secret_scope` [doctor](/doctor) check. See [Taps](/taps#taps-can-only-use-tap-secrets). |
| `DATRIS_AI_SAMPLE_VALUES` | `true` | Whether the helpers that write configuration or code send row values to a model: schema generation from a file, data profiling, JSON Schema / XSD generation, Assistant attachments, the sample rows in CodeGen data-quality and transformation prompts, and the error text in AI fix suggestions and recovery diagnosis for failed pipeline runs. `false`: none of them sends a row value; each works from column names, types, counts and lengths (or a value-free skeleton for JSON and XML), error text is reduced to exception classes and fixed messages, and helper responses carry `valuesWithheld: true`. Any other value means `true` and is logged once. Not affected: Search chat, the answer endpoint, the Assistant reading query results or a Live Read preview, the Assistant or recovery agent reading run status, logs or job details through tools, tap script generation, review and diagnosis and fix suggestions for failed tap runs, embeddings, and the run-time header check (`validateFileHeader`); see [Data Flows](/production/data-flows) for the full list. Reported by the `ai.sample_values` [doctor](/doctor) check. See [Field Protection](/transformation/field-protection#what-still-sees-raw-data). |
| `DATRIS_ENV` | *(unset)* | Set to `production` to require TLS (`sslmode=require` or stricter) on pipeline loads to external Postgres hosts. The bundled and local Postgres services are exempt. |
| `DATRIS_ALLOW_PLAINTEXT_DB` | `false` | Opt-out for the `DATRIS_ENV=production` TLS check: the load proceeds and a warning is logged. |
| `DATRIS_UNITY_CATALOG_SYNC` | `true` | Install-wide switch for Unity Catalog sync on pipelines that set `unityCatalog.enabled` or that `DATRIS_UNITY_CATALOG_DEFAULT` turned on: the metadata push and lineage on Databricks pipelines, and Iceberg table registration on object store pipelines. Set `false` to skip it (each opted-in run logs one line saying it is off). An object store pipeline with `catalogMode: "rest"` writes by path while it is off, unless its table has already been committed through the catalog, in which case the run fails rather than split the table's history. A pipeline with `catalogMode: "managed"` fails every run while it is off, because it never writes by path. Beats `DATRIS_UNITY_CATALOG_DEFAULT`: while it is `false`, nothing is written even for pipelines the default turned on. Has no effect on pipelines where Unity Catalog is off. See [Databricks](/destinations/databricks#unity-catalog-metadata) and [Register Iceberg tables](/destinations/unity-catalog#register-iceberg-tables). |
| `DATRIS_UNITY_CATALOG_DEFAULT` | `disabled` | Install-wide default for Databricks pipelines that have no `unityCatalog` block. Set `enabled` and such a pipeline behaves as if it had `"unityCatalog": {"enabled": true}` (comments, tags, properties and lineage all on); its Unity Catalog state reports `enabledBy: "default"`. A pipeline's own block always wins, so `"unityCatalog": {"enabled": false}` opts it out. `DATRIS_UNITY_CATALOG_SYNC=false` beats it. Object store Iceberg pipelines are never defaulted (registration needs the pipeline's own `credentialsSecret` and `catalog`). Any value other than `enabled` means `disabled`. See [Databricks](/destinations/databricks#unity-catalog-metadata). |
| `DATRIS_UNITY_CATALOG_SEARCH_TIMEOUT_SECONDS` | `45` | Per-secret time limit for `find_data` with `includeUnityCatalog`. A Databricks secret whose warehouse does not answer in time (for example a classic warehouse still starting) is listed under `unityCatalog.skipped` as `timed out after Ns` instead of holding the call. Blank or non-positive values use the default. See [Unity Catalog](/destinations/unity-catalog). |
| `DATRIS_OBJECTSTORE_BUCKET_ALLOWLIST` | *(unset)* | Comma-separated list of buckets on the built-in MinIO that a pipeline's `destinationBucketOverride` may target. Unset means any bucket (today's behaviour, with a warning logged on save). When set, an override outside the list is rejected on save and refused at run, query and delete time; the environment default bucket `{environment}-data` is always allowed. `provider: s3` buckets are exempt. See [Object Store](/destinations/object-store#restricting-bucket-overrides). |
| `USE_AUDIT_LOG` | `false` | Record who created, changed, ran, or deleted what — readable by admins under Configuration → Audit Log and mirrored to the server log. Restart to change. See [Audit Log](/audit-log) |
| `AUDIT_LOG_RETENTION_DAYS` | `90` | Days to keep audit entries in the config store; `0` keeps them forever. Maps to `auditLog.retentionDays` |
| `AUDIT_LOG_READS` | `false` | Also record reads — queries, searches, listings, metadata. Off by default because they dwarf the write trail. Maps to `auditLog.logReads` |
| `AUDIT_LOG_STRICT` | `false` | Strict evidence mode for the audit log (needs `USE_AUDIT_LOG=true`). `false`: best effort, an entry the config store cannot take is dropped and counted in `datris_audit_dropped_total`. `true`: nothing is dropped; while the audit queue is backed up, requests that would be audited get HTTP 503 before they run. Restart to change. Maps to `auditLog.strict`. See [Audit Log](/audit-log#strict-mode) |
| `PROVENANCE_STRICT` | `false` | Strict evidence mode for provenance stamping. `false`: a run with `provenance.stamp` on whose data cannot be stamped loads unstamped. `true`: that run fails before anything is loaded, and unknown `provenance.fields` names are refused on save. XML sources are exempt: they are never stamped and still load unstamped. Does not turn stamping on. Restart to change. Maps to `provenance.strict`. See [Provenance](/provenance#strict-mode) |
| `USE_AGENT_POLICY` | `false` | Per-action auto / approve / deny for agent-initiated changes, with a human approval queue under Activity → Approvals. With it on and no rules set, nothing changes. Restart to change. See [Agent Policy](/agent-policy) |
| `RECOVERY_AGENT_ENABLED` | `false` | The platform opens, diagnoses and — within the agent policy's recovery mode — repairs incidents for failed, stale or anomalous data flows. Needs `USE_AGENT_POLICY=true`. See [Incidents](/incidents) |
| `INCIDENT_WEBHOOK_URL` | *(empty)* | Optional URL receiving a JSON POST on incident open / awaiting-approval / resolved / failed, and on doctor checks flipping to error when the periodic run is on |
| `DOCTOR_ONSTARTUP` | `true` | Log the cheap [doctor](/doctor) checks at boot as `DOCTOR` lines. Maps to `doctor.onStartup` |
| `DOCTOR_INTERVAL_MINUTES` | `0` | Re-run the server-side doctor checks every N minutes and POST checks that flip to error (or recover) to `INCIDENT_WEBHOOK_URL`. `0` = off. AI probes never run on the timer. Maps to `doctor.intervalMinutes` |
| `DATRIS_UPDATE_CHECK` | `1` | Host-side only, read by the `datris doctor` CLI from the project `.env` or its environment (not by any container). `0`, `false` or `off` stops the `version.update` check from asking datris.ai whether a newer version exists; with it on, a full `datris doctor` run sends the server version, OS and CPU architecture. See [Doctor](/doctor#update-check) |

One further audit setting lives only in `application.yaml`: `auditLog.emitLogLine` (default `true`; mirror each entry to the `ai.datris.audit` logger for SIEM pickup).

With `USE_USER_AUTH=true` + `USE_API_KEYS=true` (recommended), the UI authenticates via session cookie — no `x-api-key` value to seed or paste. Issue programmatic-client keys from **Configuration → API-Keys**.

For `USE_USER_AUTH=false` + `USE_API_KEYS=true` (legacy paste-and-go), the seeded fallback value `default-ui-key` is what the user pastes into the Connect prompt on first load. Rotate it any time from **Configuration → API-Keys**.

### CORS

Cross-Origin Resource Sharing controls which browser origins can call the Datris API directly. The default `*` allows any origin and is appropriate for local development. **In production, lock this down to your real frontend origin(s).**

| Property | Default | Description |
| - | - | - |
| `cors.allowedOrigins` | `*` | Comma-separated list of allowed origins. Use `*` for any origin (development only), or specific URLs like `https://app.example.com,https://admin.example.com`. Applied globally to all `/api/**` endpoints. |

In the deploy config, this reads from the `CORS_ALLOWED_ORIGINS` environment variable so you can change it without rebuilding the image:

```yaml theme={null}
cors:
  allowedOrigins: ${CORS_ALLOWED_ORIGINS:*}
```

### Date / Timezone

All display timestamps across the platform (pipeline status, tap run history, etc.) are formatted using these settings.

| Property | Default | Description |
| - | - | - |
| `dateFormat` | `yyyy-MM-dd HH:mm:ss z` | Java `SimpleDateFormat` pattern. Use `z` to print the timezone abbreviation (e.g., `UTC`, `EDT`, `EST`) |
| `dateTimezone` | `America/New_York` | IANA timezone ID (e.g., `UTC`, `America/New_York`, `Europe/London`). When the format includes `z`, daylight saving is handled automatically |

**Example — Eastern time with auto-DST:**

```yaml theme={null}
dateFormat: "yyyy-MM-dd HH:mm:ss z"
dateTimezone: "America/New_York"
# Displays: 2026-04-05 14:30:00 EDT (summer) or 2026-11-05 14:30:00 EST (winter)
```

### MinIO (Object Store)

| Property | Description |
| - | - |
| `minio.server` | MinIO endpoint URL (e.g., `http://localhost:9000`) |

MinIO credentials are stored in Vault under the secret specified by `secrets.minIOSecretName`:

```json theme={null}
{
  "accessKey": "minioadmin",
  "secretKey": "minioadmin"
}
```

### AWS S3 (per-pipeline credentials)

S3 destinations don't use a global secret. Each pipeline that writes to S3 references a **Platform-tab secret** by name via the objectStore config's `credentialsSecret` field. Create the secret in the UI under Configuration → Secrets → Platform with the fields below:

```json theme={null}
{
  "accessKey": "AKIA…",
  "secretKey": "…",
  "region": "us-east-1",
  "sessionToken": "…"
}
```

Required: `accessKey`, `secretKey`, `region`. Optional: `sessionToken` (for temporary STS credentials). Field-name lookups are case-insensitive and accept the `AWS_ACCESS_KEY` / `AWS_SECRET_KEY` / `AWS_REGION` style as well.

Region lives in the credentials secret rather than on the pipeline config so the credential and its scope travel together — one place to rotate, no silent mismatches. See [S3 Destination](/destinations/s3) for the full pipeline-config shape.

### Secrets (HashiCorp Vault)

| Property | Description |
| - | - |
| `secrets.apiKeysSecretName` | Vault path for API keys |
| `secrets.postgresSecretName` | Vault path for PostgreSQL credentials |
| `secrets.minIOSecretName` | Vault path for MinIO credentials |
| `secrets.activeMQSecretName` | Vault path for ActiveMQ credentials |
| `secrets.mongoDbSecretName` | Vault path for MongoDB credentials |
| `secrets.kafkaProducerSecretName` | Vault path for Kafka producer credentials |
| `secrets.qdrantSecretName` | Vault path for Qdrant connection |
| `secrets.weaviateSecretName` | Vault path for Weaviate connection |
| `secrets.milvusSecretName` | Vault path for Milvus connection |
| `oidc.secretName` | Vault path for the OIDC single sign-on client secret (default `oss/oidc`, key `clientSecret`). A platform secret: no tap or pipeline can reference it |
| `secrets.chromaSecretName` | Vault path for Chroma connection |
| `secrets.pgvectorSecretName` | Vault path for pgvector PostgreSQL connection |

Vault connection is configured via environment variables:

* `VAULT_ADDR` - Vault server URL (e.g., `http://vault:8200`)
* `VAULT_TOKEN_FILE` - Path to a file holding the auth token (preferred; the bundled Compose writes a random per-install token here). Takes precedence over `VAULT_TOKEN`.
* `VAULT_TOKEN` - Authentication token, used when `VAULT_TOKEN_FILE` is unset (local dev, custom setups)

### ActiveMQ (Queue & Notifications)

| Property | Description |
| - | - |
| `activemq.server` | ActiveMQ broker URL (e.g., `tcp://localhost:61616`) |

ActiveMQ credentials are stored in Vault under `secrets.activeMQSecretName`:

```json theme={null}
{
  "username": "admin",
  "password": "admin"
}
```

### MongoDB (NoSQL Store)

| Property | Description |
| - | - |
| `mongodb.connectionString` | MongoDB connection URI (e.g., `mongodb://localhost:27017`) |
| `mongodb.database` | **User-facing** database name (default: `datris`). Pipelines write here, the UI Data tab reads here (semantic-search panel), and tap scripts get this as `DATRIS_MONGODB_DATABASE`. In multi-tenant mode the tenant environment name is used instead. |
| `mongodb.internalDatabase` | **Platform-internal** database name (default: `oss`). Holds pipeline/tap configs, run status, job queues — never surfaced in the UI. Keep this distinct from `mongodb.database` so user data and platform state don't mix. |

### PostgreSQL

| Property | Default | Description |
| - | - | - |
| `postgres.database` | `datris` | Default database name used by `/api/v1/query/postgres` and `/api/v1/metadata/postgres/*` when no `database` parameter is supplied. Also injected into tap scripts as the `DATRIS_POSTGRES_DATABASE` environment variable. In multi-tenant mode this value is automatically overridden per-request with the tenant name. |

PostgreSQL connection details (`username`, `password`, `jdbcUrl`) are stored in Vault under `secrets.postgresSecretName` — see the Vault Secret Formats section below.

### Kafka Consumer (Optional)

| Property | Default | Description |
| - | - | - |
| `kafkaConsumer.enabled` | `false` | Enable Kafka topic consumption |
| `kafkaConsumer.bootstrapServers` | `localhost:9092` (`kafka:9092` in Docker) | Kafka broker address |
| `kafkaConsumer.groupId` | `oss-group` | Consumer group ID |
| `kafkaConsumer.topicPollingInterval` | `500` | Topic polling interval (ms) |
| `kafkaConsumer.topicPrefix` | `oss` | Prefix for topic names |

### AI (Required)

AI configuration is split into four independent slots (three seeded at first boot, plus optional web search), each pointing at its own self-describing Vault secret. The resolver reads whatever it finds in the secret — `provider`, `endpoint`, `model`, `apiKey`, and (optionally) `version` — so the YAML side never needs a `provider` field.

| Property | Default | Description |
| - | - | - |
| `ai.enabled` | `true` | Enable AI features (required for the platform to start) |
| `ai.aiPrimary.secretName` | `oss/ai-primary` | Vault secret for the main AI model used for general reasoning (NL→SQL, search answers, etc.). Seeded with the recommended chat model — Anthropic gets `claude-opus-5-5`, OpenAI gets `gpt-5.6-sol`. |
| `ai.codegen.secretName` | `oss/codegen` | Vault secret for the code-generation model (tap scripts, AI DQ, AI transformations, schema generation). Seeded with the strongest code-generation model — Anthropic gets `claude-opus-5-5`, OpenAI gets `gpt-5.6-sol`. |
| `ai.embedding.secretName` | `oss/embedding` | Vault secret for the embedding model used by vector destinations (Chroma, Qdrant, Milvus, Weaviate, pgvector) and search. For Anthropic-only deployments this is seeded to point at the bundled TEI sidecar serving `BAAI/bge-m3` (1024-dim), so vector destinations work out of the box without an OpenAI key. |
| `ai.webSearch.secretName` | `oss/web-search` | Vault secret for the optional live web-search provider used during tap script generation and diagnosis. Not seeded at first boot — configure it from the Configuration tab. |
| `ai.extendedThinking` | `true` | Enable extended thinking on the Assistant tab's agent loop (Anthropic models only); the model's reasoning is shown above the visible response. Set to `false` to disable platform-wide. |

Each Vault secret is self-describing and looks like:

```bash theme={null}
vault kv put secret/oss/ai-primary \
  provider="anthropic" \
  endpoint="https://api.anthropic.com/v1/messages" \
  model="claude-opus-5-5" \
  apiKey="sk-ant-..." \
  version="2023-06-01"
```

`docker/vault-init.sh` seeds all three secrets automatically based on which key is present in `.env` (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `XAI_API_KEY`, or the `AZURE_OPENAI_*` trio) or the explicit `AI_PROVIDER=bedrock` selection. For multi-tenant deployments, per-tenant override secrets live at `{env}/ai-primary`, `{env}/codegen`, `{env}/embedding`.

These `.env` variables tune what gets seeded. They apply on the **first boot only** — after that, Vault persists and the Configuration tab is the source of truth:

| Variable | Default | Description |
| - | - | - |
| `AI_PROVIDER` | auto | Explicit provider pick (`anthropic`, `openai`, `azure`, `bedrock`, or `grok`) for the `ai-primary` and `codegen` slots. When unset, whichever API key is present wins (OpenAI wins ties; azure and grok are auto-picked only when they hold the sole key; bedrock is never auto-picked). |
| `ANTHROPIC_MODEL` | `claude-opus-5-5` | Model seeded into `oss/ai-primary` on the Anthropic path. |
| `OPENAI_MODEL` | `gpt-5.6-sol` | Model seeded into `oss/ai-primary` on the OpenAI path. Any OpenAI chat model id is accepted, including `gpt-6-astra`. |
| `AZURE_OPENAI_ENDPOINT` | — | Azure resource base URL (`https://YOUR-RESOURCE.openai.azure.com`). Required on the Azure path. |
| `AZURE_OPENAI_MODEL` | — | Chat deployment name seeded into `oss/ai-primary` on the Azure path. Required on the Azure path. |
| `AZURE_TENANT_ID` / `AZURE_CLIENT_ID` / `AZURE_CLIENT_SECRET` | — | Entra ID service principal for keyless Azure OpenAI auth (no API key stored anywhere). Single-tenant only; the Configuration tab's Azure credentials section is the equivalent UI path. On Azure compute, leave all auth unset to use the server's managed identity. |
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | — | AWS credentials for the Bedrock path (`AI_PROVIDER=bedrock`). Optional — leave unset to use the server's IAM role / default credential chain. Never selects Bedrock by itself. |
| `AWS_REGION` | — | AWS region for the Bedrock path (e.g. `us-east-1`). |
| `BEDROCK_MODEL` | `anthropic.claude-opus-5-5` | Invokable Bedrock model id seeded into `oss/ai-primary` on the Bedrock path. |
| `XAI_API_KEY` | — | Grok (xAI) API key from console.x.ai, for the Grok path (`AI_PROVIDER=grok`, or auto-picked when it's the sole key). |
| `GROK_MODEL` | `grok-4.7` | Grok model seeded into `oss/ai-primary` on the Grok path. |
| `CODEGEN_MODEL` | `claude-opus-5-5` or `gpt-5.6-sol` | Model seeded into `oss/codegen` (Anthropic and OpenAI defaults; the Azure path defaults to `AZURE_OPENAI_MODEL`; the Bedrock path defaults to `anthropic.claude-opus-5-5`; the Grok path defaults to `grok-4.7`). |
| `EMBEDDING_PROVIDER` | follows provider | `openai`, `azure`, `tei`, or `ollama`. When unset: `openai` when the AI provider is OpenAI, otherwise the bundled TEI sidecar (Azure embeddings are opt-in only). |
| `EMBEDDING_MODEL` | provider default | `text-embedding-3-small` (openai; also the azure default deployment name), `BAAI/bge-m3` (tei), `bge-m3` (ollama). |
| `EMBEDDING_ENDPOINT` | provider default | Endpoint override for the `tei`, `ollama`, and `azure` providers — useful when the service runs on a separate host or a non-standard path. |

All AI calls (`callAI`, `callAIWithSystem`, `callAIWithMessages`) share a unified retry helper that automatically retries on transient `429` (rate limited), `503` (service unavailable), and `529` (overloaded) responses with linear backoff (5s, 10s, 15s, 20s, 25s) for up to 5 attempts. This applies uniformly across all configured providers.

See [AI Configuration](/ai-configuration) for full setup details.

## Vault Secret Formats

### PostgreSQL

```json theme={null}
{
  "username": "postgres",
  "password": "password",
  "jdbcUrl": "jdbc:postgresql://localhost:5432"
}
```

### MySQL

```json theme={null}
{
  "username": "root",
  "password": "password",
  "jdbcUrl": "jdbc:mysql://localhost:3306"
}
```

### Kafka Producer

```json theme={null}
{
  "bootstrapServers": "kafka:9092",
  "username": null,
  "password": null
}
```

### MongoDB

```json theme={null}
{
  "connectionString": "mongodb://localhost:27017"
}
```

### Embedding

```json theme={null}
{
  "endpoint": "https://api.openai.com/v1/embeddings",
  "model": "text-embedding-3-small",
  "apiKey": "sk-...",
  "batchSize": "32",
  "maxTokens": "8192",
  "tokenizer": "openai",
  "tokensPerCharRatio": "2.0",
  "oversize": "split",
  "retryIndividualOnFailure": "false"
}
```

`endpoint`, `model`, `apiKey` are required; `batchSize` defaults to 32. The remaining fields tune the server-side token-count guard that prevents oversized chunks from failing an embedding batch:

| Field | Default | Description |
| - | - | - |
| `maxTokens` | model-default | Per-model input cap. Built-in defaults cover common OpenAI / Cohere / Voyage / BAAI / Nomic / E5 / Mistral models; unknown models get 6000. A 90% safety margin is applied automatically. |
| `tokenizer` | auto | Token-count strategy. Auto picks `openai` (exact, via jtokkit) when the model name matches an OpenAI family, otherwise `heuristic`. Set explicitly to override. |
| `tokensPerCharRatio` | `2.0` | Chars-per-token for the heuristic counter. Lower is more conservative (more splits); raise to `~3.5` for predictable Latin prose. Ignored when `tokenizer=openai`. |
| `oversize` | `split` | What to do with chunks over the cap: `split` (lossless — fan one input into N sub-chunks, token-boundary with `openai`, char-boundary with heuristic), `truncate` (lossy — drop the tail), or `fail` (raise an error naming the chunk index). |
| `retryIndividualOnFailure` | `false` | When the embedding API rejects a batch with a 4xx, retry each chunk one-at-a-time so a single poison chunk doesn't lose the whole batch. Costs N HTTP calls in the failure path. |

The guard always runs. When `maxChunkTokens` is set on the destination's `chunking` config (see [Pipeline configuration](/pipeline-configuration#chunking-config)), the chunker stops merging segments before they cross the token cap, leaving the guard as a true safety net rather than the primary defender.

## MinIO Buckets

The following buckets are created automatically by the `minio-init` container:

| Bucket | Purpose |
| - | - |
| `{environment}-raw` | File upload staging |
| `{environment}-temp` | Temporary processing files |
| `{environment}-data` | Object store destination output |
| `{environment}-config` | Configuration files (validation schemas) |

## MongoDB Collections

| Collection | Purpose |
| - | - |
| `{environment}-pipeline` | Pipeline configurations |
| `{environment}-pipeline-status` | Job processing status |
| `{environment}-archived-metadata` | File ingestion metadata |
| `{environment}-file-notifier-message` | Processed message deduplication |
| `{environment}-data-pull` | Database pull scheduling state |
| `{environment}-pipeline-version` | Append-only pipeline definition versions (pruned per `versionCap`) |
| `{environment}-tap` | Tap configurations |
| `{environment}-tap-version` | Append-only tap definition versions (pruned per `versionCap`) |
| `{environment}-tap-state` | Per-tap incremental-sync state (bookmarks / cursors) |
| `{environment}-tap-log` | Tap run history |
| `{environment}-tap-ledger` | Document-tap ledger of processed documents |
| `{environment}-tap-prompt` | Tap prompt fragments |
| `{environment}-code-repo` | Git repository connections for repo-backed tap scripts |
| `{environment}-codegen-scripts` | Index of each pipeline's stored CodeGen scripts (AI rule, AI transformation): where the script is stored, fingerprint, model, generated-at, ready or pending. The script text is in the object store under `pipeline-scripts/<pipeline>/`, or in the code repository under `<pathPrefix>pipelines/<pipeline>/` with the recorded commit |
| `{environment}-run-lineage` | One document per pipeline run (what it read, wrote, and how it ended) |
| `{environment}-column-lineage` | AI-inferred column mappings per pipeline version |
| `{environment}-user` | User accounts (when `USE_USER_AUTH=true`) |
| `{environment}-user-session` | Login sessions (when `USE_USER_AUTH=true`) |
| `{environment}-audit-log` | Audit entries (when `USE_AUDIT_LOG=true`); TTL-expired per `AUDIT_LOG_RETENTION_DAYS` |
| `{environment}-agent-policy` | The [agent policy](/agent-policy) document (when `USE_AGENT_POLICY=true`) |
| `{environment}-pending-action` | Agent actions queued for approval; TTL-expired at their own expiry |
| `{environment}-incident` | Recovery-agent incidents (when `RECOVERY_AGENT_ENABLED=true`) — the operational history behind Activity → Incidents |

## Container Environment Variables

These environment variables are read by `docker-compose.yml` and passed into the `datris` container. They tune deployment-level concerns rather than business logic. Override in your `.env` file. The four governance controls below (`USE_USER_AUTH`, `USE_API_KEYS`, `USE_AUDIT_LOG`, `USE_AGENT_POLICY`) stay `false` by default; for production switch them on together as described in [Switch on the governance controls](/quick-start#2-switch-on-the-governance-controls).

| Variable | Default | Description |
| - | - | - |
| `JAVA_OPTS` | `-Xms512m -Xmx2g` | JVM heap sizing for the datris-server process. The default is a 2 GB max heap: the minimal install (`TEI_ENABLED=0`, `POSTGRES_ENABLED=0`) fits a 4 GB Docker memory allocation, and the full stack with the bundled TEI embedder needs about 8 GB. Bump on larger hosts (e.g. `-Xms1g -Xmx8g` for 24 GB). See [Installation → JVM Heap Sizing](/installation#jvm-heap-sizing) for the full sizing table and Docker Desktop ceiling notes. |
| `PIPELINE_MAX_PAYLOAD_MB` | `4096` | Override `pipelineMaxPayloadMB`: the per-run payload disk budget (MB) for tap runs, `0` = unlimited. Bounds what one run may write to the `datris-staging` volume, not JVM heap. Size it from the data: about 26 bytes × column count per staged record for numeric/timestamp/short-string data. Raise for large single-shot backfills on hosts with the disk to hold them; lower on disk-constrained deployments to fail earlier. Compose forwards it without a default, so leaving it out of `.env` lets the server apply 4096 (or the deprecated `TAP_MAX_OUTPUT_MB` when that is set) and report which one was in force as `pipelineMaxPayloadMBSource` on `GET /api/v1/version`. |
| `TAP_MAX_OUTPUT_MB` | unset | **Deprecated** alias of `PIPELINE_MAX_PAYLOAD_MB`, honoured for one release when `PIPELINE_MAX_PAYLOAD_MB` is unset; the server logs a deprecation warning at boot while the alias is set. |
| `TAP_SCRIPT_TIMEOUT_SECONDS` | `300` | Override `tapScriptTimeoutSeconds`: how long a tap **test** may run before it is killed, in seconds. Keep it short so a broken script fails in seconds. A timed-out test names this variable in its error. Compose forwards it without a default, so leaving it out of `.env` lets the server apply 300 and report `tapScriptTimeoutSecondsSource` as `default` on `GET /api/v1/version`. |
| `TAP_RUN_TIMEOUT_SECONDS` | `3600` (or `tapScriptTimeoutSeconds` when higher) | Override `tapRunTimeoutSeconds`: how long a real or scheduled tap **run** may take, in seconds. Raise it when a large streaming source needs more than an hour to land in one run; a timed-out run names this variable in its error. Compose forwards it without a default, so leaving it out of `.env` resolves to `max(3600, tapScriptTimeoutSeconds)` rather than a flat 3600. |
| `TAP_RUNNER_QUEUE_TIMEOUT_SECONDS` | `600` | How long a tap run or generated script waits for a free runner slot when the runner is busy, in seconds. A tap run that waited shows `queued N s` in its run history; one that still has no slot after this fails naming `TAP_RUNNER_MAX_CONCURRENT`. See [Overload protection](/tap-execution-isolation#overload-protection). |
| `TAP_RUNNER_MAX_CONCURRENT` / `CODEGEN_RUNNER_MAX_CONCURRENT` | `4` / `4` | Passed to the `datris-tap-runner` / `datris-codegen-runner` containers (not to `datris`): the most scripts each runs at once. |
| `TAP_RUNNER_SCRIPT_MEMORY_MB` / `CODEGEN_RUNNER_SCRIPT_MEMORY_MB` | `1024` / `1024` | Passed to the runner containers: address-space limit per script, in MB (`0` = no limit). |
| `TAP_RUNNER_CPUS`, `TAP_RUNNER_MEMORY`, `TAP_RUNNER_PIDS` | `1`, `2g`, `256` | CPU, memory and process ceilings of the `datris-tap-runner` container. Set `TAP_RUNNER_CPUS=2` or more on hosts with the cores; Docker refuses a value above the host's CPU count. |
| `CODEGEN_RUNNER_CPUS`, `CODEGEN_RUNNER_MEMORY`, `CODEGEN_RUNNER_PIDS` | `1`, `1g`, `128` | CPU, memory and process ceilings of the `datris-codegen-runner` container. |
| `DATRIS_MAX_UPLOAD_MB` | `1024` | Largest single file `POST /api/v1/pipeline/upload` accepts, in MB (`spring.servlet.multipart.max-file-size` / `max-request-size`). Deliberately separate from `PIPELINE_MAX_PAYLOAD_MB`: the upload is received before the API key is checked, so this ceiling bounds pre-auth disk exposure and stays small. Raise it to accept multi-GB uploads; the run itself is then bounded by `PIPELINE_MAX_PAYLOAD_MB`. |
| `DATRIS_TEMP_DIR` | `/tmp/datris-staging` | Local directory where a pipeline run's payload is staged on disk (one sub-directory per run, removed when the run ends; stale runs are swept at boot). Backed by the named `datris-staging` volume in the default compose file. |
| `PIPELINE_MATERIALIZE_MAX_MB` | `256` | Largest staged payload (MB) a pipeline feature that still reads the whole payload into memory may materialize: deduplication, JavaScript row functions, JSON/XML schema validation, and a REST preprocessor or `restEndpoint` row function in batch mode with no batch size (one call carrying every row). Above it the run fails with an error naming the feature and this variable — `<feature> reads the whole payload into memory (N bytes) and PIPELINE_MATERIALIZE_MAX_MB is 256 MB. Raise it for this install, or use a CodeGen rule/transformation, which streams.` — instead of exhausting the JVM heap. Loaders, provenance stamping, CodeGen rules/transformations and batched REST calls stream the staged file and are never subject to it. Raise together with `JAVA_OPTS` on larger hosts. |
| `SCRATCH_INLINE_ROWS` | `200` | Override `scratchInlineRows`: rows inlined on a Live Read run's status and the maximum `limit` per `/pipeline/result` page. |
| `SCRATCH_RETENTION_HOURS` | `24` | Override `scratchRetentionHours`: hours a Live Read result object survives before the hourly sweep deletes it. Live Read results hold whatever the source returned, so shorten this rather than lengthen it on shared hosts. |
| `USE_USER_AUTH` | `false` | Enable username/password login + admin gating. See [User Authentication](/user-auth). |
| `OIDC_ENABLED` | `false` | Single sign-on through your identity provider; needs `USE_USER_AUTH=true`. With `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_REDIRECT_URI`, `OIDC_SCOPES` (`openid email profile`), `OIDC_USERNAME_CLAIM` (`email`) and `OIDC_DEFAULT_ROLE` (empty). Client secret in Vault, `oss/oidc` key `clientSecret`. See [Single sign-on (OIDC)](/user-auth#single-sign-on-oidc). |
| `USE_API_KEYS` | `false` | Enable per-tenant API-key gating on the REST/MCP API. See [API Keys](/api-keys). |
| `MULTI_TENANT` | `false` | Advanced: switch to per-tenant database isolation, with each request routed by its API key. Requires `USE_API_KEYS=true`. |
| `ANTHROPIC_OVERLOAD_FALLBACK_MODEL` | `claude-sonnet-5` | Model used for the current request when Anthropic returns a sustained `overloaded_error` on an Opus call (after the automatic retries). Only consulted when the configured model is an Anthropic Opus variant — other providers ignore it. Bump when a newer Sonnet ships to keep the fallback current. |
| `AI_HTTP_MAX_PER_ROUTE` | `96` | Maximum concurrent connections the server's shared AI-provider and MCP clients open to one host. Every chat panel and the Assistant stream to the same provider host, so this is the limit concurrent chats hit first. The default covers the five chat panels at full load (80 concurrent streams) with headroom; when it is exhausted, the AI clients wait up to 10 seconds for a free connection and then fail with `Timeout waiting for connection from pool`. A blank, non-numeric or non-positive value uses the default. |
| `AI_HTTP_MAX_TOTAL` | `192` | Maximum concurrent connections across all hosts for the same clients. Never lower than `AI_HTTP_MAX_PER_ROUTE`; a smaller value is raised to match. A blank, non-numeric or non-positive value uses the default. |
| `CAPABILITY_ENFORCEMENT` | `enforce` | `enforce` returns HTTP 403 on policy violations by scoped keys. Set to `log-only` to emit would-deny log lines without rejecting — useful when iterating on a new scope policy. |
| `USE_AUDIT_LOG` | `false` | Enable the [Audit Log](/audit-log) — who did what, by login or API key. |
| `AUDIT_LOG_RETENTION_DAYS` | `90` | Days to keep audit entries; `0` = forever. |
| `AUDIT_LOG_READS` | `false` | Also record reads (queries, searches, listings). |
| `AUDIT_LOG_STRICT` | `false` | Refuse audited requests (503) instead of dropping audit entries when the config store cannot keep up. Needs `USE_AUDIT_LOG=true`. See [Audit Log](/audit-log#strict-mode). |
| `PROVENANCE_STRICT` | `false` | Fail a stamping pipeline's run instead of loading unstamped rows. XML sources are exempt and load unstamped. See [Provenance](/provenance#strict-mode). |
| `USE_AGENT_POLICY` | `false` | Enable the [Agent Policy](/agent-policy) — auto / approve / deny per agent action, with a human approval queue. |
| `RECOVERY_AGENT_ENABLED` | `false` | Enable the [recovery agent](/incidents) — platform-opened incidents with a bounded repair loop. |
| `INCIDENT_WEBHOOK_URL` | *(empty)* | Optional JSON webhook for incident lifecycle events and periodic-doctor flips. |
| `DOCTOR_ONSTARTUP` | `true` | Boot-time [doctor](/doctor) lines. |
| `DOCTOR_INTERVAL_MINUTES` | `0` | Periodic doctor run; `0` = off. |


# Configuring Claude
Source: https://docs.datris.ai/configuring-claude

Connect Claude Desktop and Claude Code to the Datris MCP server

This page covers the client-side configuration for Anthropic's Claude products to talk to the Datris MCP server. For server-side details (architecture, transports, available tools), see [MCP Server (AI Agent Integration)](/mcp-server).

## Authentication

The MCP server is a transparent forwarder — it has no API key of its own. When auth is enabled, the client sends an `x-api-key` header on every session and the MCP server forwards that key to the Datris REST API on every tool call.

* **Single-tenant OSS with `USE_API_KEYS=false`** (default): no key needed. The snippets below work as-is.
* **Single-tenant OSS with `USE_API_KEYS=true`**: append `--header x-api-key:<your-key>` to the `args` array. Issue and rotate keys from **Configuration → API-Keys** in the UI — the value shown when you create a key is what your client sends.

The Configuration UI's **Connect Your Agent** panel pre-generates these snippets — paste your API key into the field there and the `--header` flag gets added automatically.

## Configuring Claude Desktop

Edit `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows), then quit and relaunch Claude Desktop.

### Recommended: mcp-remote bridge (Docker stack)

Run the bundled MCP server through the [mcp-remote](https://www.npmjs.com/package/mcp-remote) stdio bridge. It transparently reconnects when the server restarts and lets every desktop agent (Claude Desktop, Cursor, Claude Code) share one snippet shape:

```json theme={null}
{
    "mcpServers": {
        "datris": {
            "command": "npx",
            "args": [
                "-y",
                "mcp-remote",
                "http://localhost:3000/sse",
                "--transport",
                "sse-only"
            ]
        }
    }
}
```

The underlying connection is SSE to port 3000, so sessions register with the Datris server and the **Agent Monitor** tab in the Datris UI shows live tool-call streaming. `--transport sse-only` pins the protocol so the bridge doesn't try its streamable-HTTP fallback.

Requires Node.js on your `PATH` (`brew install node`). On macOS, Claude Desktop is a GUI app and doesn't inherit your shell `PATH`, so use the full path to `npx` (e.g. `/opt/homebrew/bin/npx`) if `npx` alone doesn't resolve.

If your instance requires an API key (`USE_API_KEYS=true`), append two more args:

```json theme={null}
"args": [
    "-y", "mcp-remote", "http://localhost:3000/sse",
    "--transport", "sse-only",
    "--header", "x-api-key:<paste your Datris API key here>"
]
```

### Alternative: stdio via uvx (no Docker)

If you're not running the Docker stack — for example, you only want the MCP server and run the Datris API separately — use the PyPI package directly:

```json theme={null}
{
    "mcpServers": {
        "datris": {
            "command": "uvx",
            "args": ["datris-mcp-server"],
            "env": {
                "DATRIS_API_URL": "http://localhost:8080"
            }
        }
    }
}
```

Requires `uv` / `uvx` (`brew install uv`). On macOS, Claude Desktop is a GUI app and doesn't inherit your shell PATH, so use the full path to `uvx` (e.g. `/Users/you/.local/bin/uvx`) if `uvx` alone doesn't resolve.

<Note>
  The stdio transport has no header layer, so the API key comes from the environment instead: if your Datris instance has `USE_API_KEYS=true`, add `"DATRIS_API_KEY": "<your-key>"` to the `env` block (or use the mcp-remote option above). With `USE_API_KEYS=false` (single-tenant OSS default), stdio works fine without a key, but tool calls won't appear in the Agent Monitor tab.
</Note>

## Configuring Claude Code

Add to `.mcp.json` in your project root. The same two options apply.

### Recommended: mcp-remote bridge (Docker stack)

```json theme={null}
{
    "mcpServers": {
        "datris": {
            "command": "npx",
            "args": [
                "-y",
                "mcp-remote",
                "http://localhost:3000/sse",
                "--transport",
                "sse-only"
            ]
        }
    }
}
```

To add an API key (`USE_API_KEYS=true`), append `"--header", "x-api-key:<your-key>"` to the args array.

### Alternative: stdio via uvx (no Docker)

```json theme={null}
{
    "mcpServers": {
        "datris": {
            "command": "uvx",
            "args": ["datris-mcp-server"],
            "env": {
                "DATRIS_API_URL": "http://localhost:8080"
            }
        }
    }
}
```

## Give Claude the Datris skill

Connecting the MCP server gives Claude the tools. The [datris-platform skill](/agent-skill) tells Claude when to use them during ordinary project work: it triggers on data ingestion, storage, search, scheduling and provenance tasks and steers Claude to Datris instead of hand-written loaders, cron jobs or vector-store code. Copy it into `.claude/skills/` in the project, or into `~/.claude/skills/` for every project.

## Talking to Claude

Once the MCP server is configured, just talk to Claude naturally. Claude sees every Datris tool and picks the right one based on your request.

### First prompts (empty Datris)

If you've just installed Datris and have no data yet, these prompts exercise the full create → query loop without needing files on hand. Watch the **Agent Monitor** tab in the Datris UI as you go — you'll see each tool call stream in real time.

**1. Confirm discovery works (returns empty, but proves the round-trip):**

> "List my Datris pipelines and tell me what's there."

**2. Have Claude generate sample data and ingest it:**

> "Generate 50 rows of sample stock price data (symbol, date, open, close, volume) and ingest it into PostgreSQL as a pipeline called `stock_prices`."

This single prompt exercises `create_pipeline` (with destination schema), the actual ingest, and several supporting tools — multiple tool calls light up the activity log.

**3. Query the data you just created:**

> "Show me the top 10 rows in `stock_prices` by volume."

**4. Add AI-generated data quality rules:**

> "Profile the `stock_prices` pipeline and suggest data quality rules I should add."

This exercises the AI/CodeGen path — Claude writes Python validators, Datris runs them locally.

**5. Read the pipeline configuration resource:**

> "Read the Datris pipeline configuration reference and summarize what destination types are supported."

Confirms Claude can pull MCP resources (not just tools) — useful to verify your client supports both.

For the RAG path, ingest the document **outside** the chat (see the warning below), then ask Claude to search it:

```bash theme={null}
# from your terminal
datris ingest annual-report.pdf --dest pgvector --pipeline reports
```

> "Search the `reports` pipeline for \[topic from the PDF]."

### Ingest data

> "Create a pipeline called stock\_prices and ingest this CSV into PostgreSQL"

> "Ingest this sales data into PostgreSQL and validate that all prices are positive"

<Warning>
  **Don't drag large local files into the chat to ingest them.** When you say "ingest this PDF" with a file attached, Claude has to base64-encode the bytes and pass them as a string argument to the `upload_data` tool — that string traverses the conversation context, and a 300KB PDF alone consumes \~90K tokens before any work happens. Conversations hit "too long to continue" almost immediately.

  Use the right tool for each shape of input:

  * **Local files** of any size — use the CLI: `datris ingest <file> --dest <destination>`. No LLM, no context cost.
  * **Files at a URL** (S3, MinIO, HTTPS) — ask Claude to create a Datris **tap** that fetches them. Bytes are pulled server-side; the agent only handles the URL.
  * **Small inline data** (a few rows of CSV, a JSON snippet, generated sample data) — fine to drop into the chat.

  The chat path is for *natural-language operations* (querying, profiling, pipeline management, search), not for funneling local-disk files through a remote conversation.
</Warning>

### Query data

> "Show me the top 10 stocks by volume from the stock\_prices table"

> "What's the average closing price for AAPL?"

> "How many orders were placed in March?"

### Search documents (RAG)

> "Search my financial documents for information about quarterly revenue"

> "What does the employee handbook say about the return policy?"

### Profile data

> "Profile this CSV file and tell me what validation rules I should add"

> "What does this data look like? Give me a summary"

### Manage pipelines

> "List all my pipelines"

> "Delete the test\_pipeline"

> "What's the status of the stock\_prices pipeline?"

### Explore databases

> "What tables are in my PostgreSQL database?"

> "Show me the columns in the orders table"

> "What vector collections do I have?"

### Health and diagnostics

> "Are all the backend services running?"

> "Check the health of the platform"

You can drag small inline files (a CSV with tens of rows, a short JSON snippet) directly into the chat for ingestion or profiling. For larger or binary files, use the CLI or a tap (see the warning above).


# Configuring OpenClaw
Source: https://docs.datris.ai/configuring-openclaw

Use the Datris Platform as OpenClaw's long-term memory and semantic search layer

This page covers connecting [OpenClaw](https://openclaw.ai) to a Datris MCP server so the agent can use Datris as its **long-term semantic memory** — local memory files stay canonical, while Datris becomes the retrieval/index layer.

For server-side details (architecture, transports, available tools), see [MCP Server (AI Agent Integration)](/mcp-server).

## Why route OpenClaw memory through Datris

This isn't a "no memory vs. memory" comparison. OpenClaw already has memory. It's a comparison about *enforcement, scale, and reuse*.

The short version:

1. **OpenClaw's built-in memory is best-effort** — the model decides when to save and when to look — so long sessions, restarts, and context compaction quietly erode recall. Datris moves memory out of the prompt and enforces it at the system layer, so your agent stops forgetting.
2. **The same MCP connection that gives OpenClaw enforced semantic memory also exposes 70+ Datris tools** — pipelines, ingestion, SQL, vector search — so you're not bolting on a memory product, you're handing your agent a full data platform.

### vs. OpenClaw's default memory (best-effort, not enforced)

OpenClaw ships with persistent memory out of the box — daily memory files, a curated `MEMORY.md`, and semantic search over archived sessions. It works. The catch is that everything is advisory: the model decides when to save, what to save, and whether to search before answering. Long sessions, context compaction, and restarts all chip away at recall. You'll see it fail the same way every time — a fact you established in week one stops surfacing in week three, not because it's gone, but because the model didn't think to look for it.

Datris-backed memory enforces capture and retrieval at the system layer instead of leaving it to the prompt. Memory is written outside the agent's session, and relevant context is reintroduced on every turn. Restarts don't matter. Long conversations don't matter. The agent reasons with the same facts every time.

### vs. flat file-based memory (raw markdown, JSON blobs)

Hand-rolled file memory — a `PREFERENCES.md` here, a `CONTEXT.md` there, a `notes/` folder with grep on top — is the path most people start down. It works for a week. Past a few dozen entries the agent ends up either loading whole files into context or substring-matching for keywords that may not appear in the original note. A question phrased as *"what did we decide about the pipeline architecture?"* misses notes filed under "ingestion topology" or "data flow design."

Datris stores memories in pgvector (or Qdrant, Weaviate, Milvus, Chroma — your call) and retrieves by meaning, not exact wording. The same question finds the right note regardless of how it was originally phrased.

### vs. MemGPT/Letta-class agent runtimes

MemGPT (now Letta) does semantic memory well — that's the system that popularized the LLM-as-OS framing for agent memory. The trade-off is that Letta is a runtime, not a memory layer. Adopting it means running your agent inside their framework: their agent loop, their tool execution, their state model. That's a fine choice from a clean slate. It's a heavy migration if you already have an agent you like — including OpenClaw.

Datris is a memory layer exposed over MCP. OpenClaw stays OpenClaw. Your agent loop, your tools, your skills — all unchanged. Point your MCP client at Datris and the agent gains enforced, semantic, persistent memory without rewriting anything.

### What you actually get

Indexing local memory files into Datris gives you the best of both worlds — markdown stays human-editable and version-controllable, while Datris handles retrieval at scale.

* **Agent-runtime-agnostic.** Datris exposes memory through MCP, so OpenClaw, Claude Desktop, Cursor, or any custom MCP client can point at the same instance and share the same memory. No bespoke integration per agent, no rewriting your agent loop to match someone else's runtime.
* **Tool-rich, not just retrieval.** The same MCP connection that gives OpenClaw memory also exposes pipeline management, ingestion, and SQL and vector query tools — over 70 in total. Memory-only products stop at recall; Datris hands the agent the rest of a data platform on the same connection.
* **Auditable.** The **Agent Monitor** tab in the Datris UI shows OpenClaw's memory tool calls as they happen, so you can see exactly what the agent searched for, what came back, and what it stored.
* **Semantic recall, not substring search.** Ask in natural language and get the right snippet back even when the wording doesn't match — *"what did we decide about the pipeline architecture?"* finds the note filed under "ingestion topology."
* **Scales past the context window.** Large memory corpora that would never fit into a single prompt become retrievable on demand. The agent pulls only what's relevant for the current turn.
* **Files remain the source of truth.** For memory you author yourself, the markdown files are yours — Datris is a derived index. Edit locally, re-ingest when ready. No vendor lock-in on your notes.

## Step 1 — Set the embedding model to OpenAI `text-embedding-3-small`

Before ingesting any memory, pick the embedding model the pipeline will use to encode it. For long-term memory we recommend **OpenAI** with **`text-embedding-3-small`** — it's the best balance of recall quality, latency, and cost for this use case in our testing.

In the Datris UI:

1. Open the **Configuration** tab, then the **AI Providers** sub-tab.
2. In the **Embedding Provider** section, choose **OpenAI** as the provider and **`text-embedding-3-small`** as the model.
3. Enter your OpenAI API key in the right-hand **API Keys** panel (if you haven't already).
4. Click **Save Configuration**. Changes take effect immediately — no restart required.

<Warning>
  **Set the embedder before you create the memory pipeline.** Vector dimensions are pinned at pipeline-creation time — `bge-m3` (the bundled default) is 1024-dim, `text-embedding-3-small` is 1536-dim. Switching embedders after ingestion will fail-fast with a dimension-mismatch error and you'll have to drop the destination collection and re-ingest. See [AI Configuration](/ai-configuration) for the full provider matrix.
</Warning>

## Step 2 — Install the Datris CLI

The memory skill prefers MCP tools but falls back to the `datris` CLI for health checks, stalled-job recovery, and operating when the MCP server is unreachable. It also declares `datris` as a required binary in its install metadata, so it has to be on your `PATH` before the skill will activate cleanly.

Install via Homebrew:

```bash theme={null}
brew tap datris/tap
brew install datris
```

Verify:

```bash theme={null}
datris health
```

You should see every backend service report healthy. See [docs.datris.ai/cli](/cli) for the full command reference.

## Step 3 — Register the Datris MCP server with OpenClaw

Point OpenClaw at the local Datris MCP server (the SSE endpoint exposed by the standard Docker stack on port 3000):

```bash theme={null}
openclaw mcp set datris '{"url":"http://localhost:3000/sse"}'
```

Verify the registration:

```bash theme={null}
openclaw mcp show datris
```

You should see the `datris` server listed with its URL. SSE registration also means OpenClaw will appear in the Datris UI's **Agent Monitor** tab with live tool-call streaming — useful for watching what the agent is actually doing during ingestion and retrieval.

## Step 4 — Install the Datris memory skill

A one-shot prompt isn't enough — memory work is ongoing (bootstrap, incremental sync, retrieval defaults, handling renames and deletes), and the agent needs persistent operating instructions, not just a single bootstrap run. OpenClaw uses **skills** — markdown files under `.openclaw/workspace/skills/` — for exactly this. Install the Datris memory skill once and OpenClaw auto-triggers it whenever the user mentions saving, recalling, ingesting, syncing, or searching memory (and even on memory-shaped questions like *"what did I note about X"* that don't say "Datris" out loud).

You have two options for installing the skill.

### Option A: install via the OpenClaw CLI (recommended)

```bash theme={null}
openclaw skills install datris-memory
```

This pulls the latest published version of the skill, drops it at `~/.openclaw/workspace/skills/datris-memory/SKILL.md`, and you're done. Skip ahead to Step 5.

### Option B: install manually

Create the skill directory:

```bash theme={null}
mkdir -p ~/.openclaw/workspace/skills/datris-memory
```

Then create `~/.openclaw/workspace/skills/datris-memory/SKILL.md` with the contents below:

```markdown theme={null}
---
name: datris-memory
description: Use the Datris Platform as a long-term semantic memory layer through the Datris MCP server. Local markdown memory files (MEMORY.md, memory/*.md) stay canonical; Datris is the retrieval index built from them. Use this skill any time the user mentions saving, recalling, ingesting, syncing, or searching memory through Datris — and any time they ask a memory-shaped question ("what did I note about X", "find anything related to Y", "summarize what I've written down about Z") even when they don't explicitly say "Datris." Trigger on any work involving long-term memory ingestion, retrieval, or sync. Datris MCP is the only memory backend — do not substitute a built-in `memory_search`, `recall`, or host-provided knowledge tool, even when one is available.
---

# Datris memory layer

Datris is the long-term semantic memory layer. Local memory files are the source of truth. Datris is rebuilt from them — never the other way around.

## Rules

- **Datris MCP is the only memory backend.** If the host exposes a built-in memory tool (`memory_search`, `recall`, an internal RAG store, a sibling skill's memory index, etc.), do not use it for memory-shaped queries. Those tools index a different corpus and produce divergent results — the local markdown files are the source of truth and Datris is the only retrieval index built from them. There is no third channel.
- Prefer Datris MCP tools over shell or CLI for memory operations.
- **One file in = one `upload_data` call.** Never concatenate, bundle, or consolidate multiple memory files into a single corpus upload. Each file's name is its provenance — it must round-trip cleanly into Datris and back out in retrieval results. Bundling looks like an optimization and is not one: it permanently breaks provenance, blocks per-file incremental sync, and makes resets harder.
- **Source filenames preserve the full relative path.** Use `MEMORY.md` and `memory/2026-05-06.md` as source filenames — the directory prefix is part of the provenance and must round-trip into retrieval results. Some Datris builds reject path separators at the temp-file staging layer; if uploads fail on slashes, encode the separator reversibly (e.g., `memory__2026-05-06.md`) and tell the user what encoding was applied. Never silently flatten to a basename — that strips the directory and creates ambiguity if two memory subdirectories ever contain files with the same name.
- Upload each file as-is. Let Datris chunk server-side. Do not pre-chunk. Do not use a document tap for local files. The only time to split a single file is when that one file genuinely exceeds the upload limit; in that case, split it with explicit provenance markers in the chunk filenames (e.g. `MEMORY.md.part1`, `MEMORY.md.part2`).
- **Run uploads concurrently across files.** Each `upload_data` call returns a job token immediately — collect all the tokens first, then poll. Use a bounded concurrency limit (8–16 parallel uploads is safe for most Datris builds) rather than unbounded fan-out. On bootstrap, where there are no pre-existing chunks to remove, this is straight parallelism.
- **Re-uploads are delete-then-upload, serialized per file.** When re-ingesting a file that already exists in the pipeline, delete its existing chunks first, wait for the delete to complete, then upload. Do not rely on upsert behavior — some Datris builds append duplicates rather than overwriting on filename collision, and a delete-then-upload pair is build-agnostic and idempotent. Within a single file, delete→upload is serial. Across files, multiple delete→upload pairs run concurrently up to the same bounded limit.
- Poll job status to completion before claiming any ingestion succeeded. Polling is for verification across the batch, not for gating cross-file uploads.
- Verify retrieval with a semantic search after every ingestion run.
- Memory pipelines must target a vector destination (pgvector, Qdrant, Weaviate, Milvus, or Chroma).
- The embedding model is pinned at pipeline-creation time. Vector dimensions cannot change after the fact. Confirm the embedder matches the pipeline before ingesting. Switching embedders means dropping and recreating the destination collection.
- **Place every created resource in the `openclaw` data catalog by default.** Pipelines, taps, secrets, destination collections — anything the agent creates in Datris on OpenClaw's behalf goes in the catalog named `openclaw` unless the user explicitly directs otherwise. This keeps OpenClaw's footprint cleanly separated from other Datris workloads on the same instance, makes cleanup and auditing trivial, and lets multiple OpenClaw users share an instance without colliding. If an existing resource the agent wants to reuse lives in a different catalog, do not migrate it silently — surface the mismatch to the user and ask before continuing.
- **Return to prescribed concurrency after debugging.** When isolating a failure — a Datris bug, a tool returning unexpected output, a transient error — it is reasonable to drop temporarily to single-file sequential operations to get a clean signal. Once the issue is identified and resolved, return to the concurrency model in the rules above before processing the rest of the batch. Do not carry a debug-mode sequential pattern into subsequent ingestion or sync runs. Note in the audit log when this happens, including the reason and what was done.

## First-run bootstrap

1. Read the Datris MCP resources and tool descriptions. Understand the pipeline, upload, job-status, and search workflows before acting.
2. Check service health.
3. Reuse an existing vector pipeline for memory in the `openclaw` catalog if one exists. Otherwise create one in the `openclaw` catalog — pgvector is fine. If a memory pipeline exists outside the `openclaw` catalog, surface that to the user before reusing or migrating it.
4. Ingest `MEMORY.md` and each `memory/*.md` file via its own `upload_data` call — one upload per file, no exceptions. Do not concatenate them into a single corpus document, even if the total set is small. Fire the uploads in parallel and collect all the job tokens before polling.
5. Poll all jobs concurrently until every one reports completion.
6. Verify with two or three representative semantic queries. Confirm both that (a) the expected content comes back, and (b) results show real source filenames (`MEMORY.md`, `memory/2026-05-06.md`, etc.) — not a consolidated corpus filename or any other synthetic name. The filename round-trip check is the early-warning signal that the one-file-per-upload rule is being followed; if results show a corpus filename, stop and apply the remediation workflow below.
7. Record the run in `memory/<today>.md`: pipeline used, files ingested, verification queries and results, any failures.
8. Propose an incremental sync strategy for future edits.

## Ongoing sync

Memory files change continuously — the agent writes to them during sessions, the user edits them in their editor between sessions, and new dated files appear over time. Sync is incremental and runs lazily, in three modes:

### When to sync

1. **Periodic background sync.** A timer-driven sweep runs every 30 minutes by default — diff memory files by `mtime` against the last sync record, upload anything stale. This runs out-of-band: it never blocks an agent response or a user query. Cadence is configurable: faster (5–10 min) for users actively editing memory between sessions, slower (hourly or more) for read-mostly use. The right value is whatever keeps the staleness window short enough that the user rarely needs to force a sync.

2. **End of any agent response that wrote to memory.** When the agent edits or creates a memory file during a turn, flush those uploads before the response is considered complete — including polling to completion. This puts the cost on the response that did the writing, not on later queries, and guarantees that a follow-up question in the same conversation can retrieve what the agent just wrote. Never let an agent-authored memory write wait for the next timer tick.

3. **On explicit user request.** Phrases like "sync memory," "save what we discussed," "update Datris with my recent notes" — full diff sweep across all memory files, immediate sync. This is the user's escape valve for the case where they just edited a file in their editor and want it queryable right now rather than waiting for the next timer tick.

### Staleness window — and why it's acceptable

Memory edits made outside of an agent session — for example, the user editing `MEMORY.md` in their editor between conversations — may be up to one timer interval behind in retrieval results. That is the explicit trade. Query latency stays predictable, ingestion is invisible, and the user has a one-line escape hatch ("sync memory") when freshness matters. Do not try to close the staleness window by syncing on every retrieval; that path puts ingestion cost on the user's wait time and is the design this skill replaces.

### Detecting what changed

Compare each memory file's filesystem `mtime` against the most recent sync timestamp recorded for that file in the `memory/<date>.md` audit logs. Three cases:

- **No sync record exists** → treat as a new file, ingest it.
- **`mtime` newer than last-sync timestamp** → re-upload.
- **`mtime` unchanged** → skip. Do not re-ingest unchanged files; the point of incremental sync is to do less work.

A content hash is more reliable than `mtime` if the user touches files without editing them (some editors do this on save), but `mtime` is sufficient as a default. Switch to hashing only if redundant uploads start showing up in the audit log.

### Sync workflow

Classify each changed file into one of four cases and act accordingly:

- **Edited file** → delete existing chunks for that source filename, wait for the delete to complete, then upload via `upload_data`. Per-file serial; multiple files' delete→upload pairs run concurrently.
- **New file** (no prior sync record) → upload via `upload_data` directly. No delete step needed since there are no existing chunks for this filename.
- **Renamed file** → delete chunks for the old filename, then upload the new file. Same per-file serialization as edits, same cross-file concurrency.
- **Deleted file** → delete its chunks from the destination collection. Do not leave orphans.

Then:

- Wait for all jobs to complete, polling concurrently.
- Verify with one or two semantic queries that touch the changed content. Confirm filenames in results reflect the post-sync state — no stale entries from before a rename or delete, no consolidated-corpus names, no duplicate chunks (same content under the same filename appearing more than once would indicate the delete step was skipped).
- Append a per-file entry to today's audit log: filename, change type (edit / new / rename / delete), timestamp, verification result.

## Inheriting a consolidated-corpus pipeline

If the existing memory pipeline was bootstrapped with a single consolidated upload (a `*-corpus-*.md` source file, or any upload whose filename doesn't match a real memory file), the pipeline has broken provenance and cannot be incrementally synced cleanly. Do not patch around it. Reset and re-ingest:

1. Confirm with the user before resetting — destination collections may contain manual edits.
2. Drop the destination collection (or recreate the pipeline with the same name).
3. Re-ingest each canonical memory file individually per the bootstrap workflow above.
4. Verify with semantic queries that retrieval results now show real source filenames (`MEMORY.md`, `memory/2026-05-06.md`, etc.) rather than a corpus filename.

## Retrieval

When the user asks a memory-shaped question:

1. Call `vector_search` against the Datris memory pipeline (or `ai_answer` for synthesis).
2. If — and only if — Datris MCP is unreachable, grep local memory files as a fallback. Tell the user you fell back and why.

**Do not** call:

- Built-in agent memory tools (`memory_search`, `recall`, etc.)
- Host-provided knowledge stores or RAG indexes
- Sibling skills' memory features

These index a different corpus than Datris and will return inconsistent results. Before sending your answer, name the tool you actually called in your own reasoning — if it isn't `vector_search`/`ai_answer` against the Datris memory pipeline, stop and re-route.

## Reporting

After any bootstrap or sync, report:

- Pipeline used or created.
- Files ingested.
- Verification queries and whether they returned the expected content.
- Anything that failed and why.
```

### What this skill enforces

* **Auto-triggers on memory intent** — the skill's `description` covers explicit phrasing ("sync memory," "save what we discussed") *and* implicit memory-shaped questions ("what did I note about X"), so the user never has to remember to invoke it.
* **Datris MCP is the only memory backend** — built-in agent memory tools, host-provided knowledge stores and sibling skills' memory features are never used for memory-shaped queries; local files are grepped only if Datris MCP is unreachable, and the agent says so.
* **One file in = one upload** — no bundling, no concatenation. Filenames (including their directory prefix) round-trip into retrieval results so provenance is preserved and per-file incremental sync stays clean.
* **Bounded parallel uploads, then poll** — fires `upload_data` calls concurrently (8–16 at a time), collects job tokens, polls concurrently. Re-uploads are delete-then-upload, serialized per file.
* **Catalog isolation** — every resource the agent creates lands in the `openclaw` data catalog by default, keeping OpenClaw's footprint separate from other workloads on the same Datris instance.
* **Three-mode incremental sync** — periodic timer sweep (default 30 min), end-of-response flush for agent-authored writes, and explicit user "sync memory" trigger. Renames go through delete-then-add so the index never accumulates orphans.
* **Self-healing for inherited corpora** — if a previous setup bundled everything into one file, the skill includes the reset-and-re-ingest workflow rather than papering over broken provenance.

## Step 5 — Restart the OpenClaw gateway

Skills are discovered at gateway startup, so the new `datris-memory` skill won't be active until you reload:

```bash theme={null}
openclaw gateway restart
```

After the restart, OpenClaw will pick up `~/.openclaw/workspace/skills/datris-memory/SKILL.md` and auto-trigger it on any memory-shaped request.

## Step 6 — Kick off the bootstrap, then use it

With the skill loaded, point OpenClaw at it once so it runs the first-time bootstrap — health check, pipeline creation, parallel ingestion of every existing memory file, and verification. A simple opener works:

> "You now have the `datris-memory` skill installed. Use it to ingest my existing memory into Datris."

Watch the **Agent Monitor** tab in the Datris UI as it runs — you'll see each `upload_data` call, `get_job_status` poll, and `search_pgvector` verification stream in real time. When the bootstrap finishes, OpenClaw will write a summary into today's `memory/<date>.md` audit log.

After that, just talk normally. The skill auto-triggers on any memory-shaped request and reaches for Datris semantic search instead of grepping local files:

> "What did I note about my preferred code review process?"

> "Find anything in my memory related to onboarding new teammates."

> "Summarize what I've written down about meeting cadence."

## Advantages of this setup

Beyond the per-feature wins listed at the top, the architectural payoff of putting Datris between OpenClaw and your memory files is:

* **Recall quality scales with corpus size** — adding more memory files makes retrieval *better*, not slower or noisier, because semantic search ranks by relevance instead of dumping everything into context.
* **Context budget stays tight** — only the top-matching snippets land in the prompt, so long-running OpenClaw sessions don't burn their window loading memory the agent doesn't need.
* **One memory layer, many agents** — point Claude Desktop, Claude Code, OpenClaw, or any other MCP-aware client at the same Datris pipeline and they all share the same long-term memory. No per-tool re-indexing.
* **Local-first, not cloud-locked** — your markdown stays on disk, the index lives in your own Datris stack. You can wipe and rebuild the index any time without losing knowledge.
* **Memory becomes queryable, not just retrievable** — because it's a real Datris pipeline, you can run vector search, AI answer, and even SQL-shaped questions against your memory corpus from the CLI or REST API, not only from inside the agent chat.
* **Future memory is incremental** — the skill's three-mode sync (timer sweep, end-of-response flush, explicit user trigger) re-ingests only the files that changed, never the whole corpus.

The net effect: your agent stops forgetting, your memory files stop being a flat pile of markdown, and you get a real semantic layer over the knowledge you've been writing down all along.


# Data Catalog
Source: https://docs.datris.ai/data-catalog

A logical grouping for related taps and pipelines

<Note>
  **Beta.** Data Catalogs may evolve as we add features like sharing and access control.
</Note>

A **Data Catalog** is a named bucket that groups related [taps](/taps) and [pipelines](/pipeline-configuration). It's pure metadata — assigning a tap to a catalog doesn't change how the tap runs or where its data lands. Catalogs exist to make a busy platform browsable: instead of a flat list of 40 taps, you see "Stock Prices (12)", "Weather (6)", "Earnings (8)" and so on.

A Datris catalog is not a Databricks Unity Catalog catalog. On a Databricks pipeline, `destination.database.dbName` names the Unity Catalog catalog the table lands in; the Datris catalog is a separate grouping label, carried to Databricks as the `datris_catalog` tag when the Unity Catalog metadata push is on. See [Unity Catalog](/destinations/unity-catalog).

## When to use a catalog

Use one whenever you have more than a handful of related taps that should travel together — same source, same domain, same project.

You don't have to use catalogs. Anything you create outside a catalog falls into the **Uncataloged** group, which works fine for one-off taps and pipelines.

## How catalogs are stored

Catalogs aren't a separate table or external system. Each tap and pipeline carries a nullable `catalog` string field; the platform groups by that field. Two consequences:

* A catalog "exists" as soon as anything is assigned to it. There's no separate create step required to *use* a name.
* Renaming a catalog means updating the `catalog` field on every tap and pipeline in it. The Catalog tab's rename control does this in one step, or call `rename_catalog` over MCP.

When you create a catalog explicitly via the UI, Datris persists the name by writing a placeholder tap named `__catalog__<name>` (description "Catalog placeholder", `enabled=false`). This makes empty catalogs survive across UI reloads. The placeholder is hidden from the Taps list and ignored by the runtime.

## The Catalog tab

The **Catalog** tab in the UI shows one card per catalog plus an **Uncataloged** card.

* Each card shows the catalog name and counts of taps + pipelines.
* **Rename catalog** — the pencil icon on a named catalog's card turns the name into an editable field. Press Enter (or the check mark) to save, Escape to cancel. The new name may use letters (upper or lower case), digits, `_` and `-` only; anything else is converted when you leave the field (spaces become `_`, other characters are dropped). Capitalisation is kept. Every tap and pipeline in the catalog moves to the new name in one step.
  * A rename cannot target a catalog that already exists: it is refused and both catalogs stay as they were. To combine two catalogs, use **Move all contents** (the move icon on the source catalog's card) instead.
  * If some items could not be moved, the page lists each one with the reason; the rest are renamed. If the catalog was renamed or deleted from somewhere else after the page loaded, the rename or delete changes nothing, says the catalog was not found, and refreshes the list. An item that someone moved to another catalog while the rename ran is reported as no longer in the catalog; refresh the page to see where it is now.
  * API keys whose capabilities are scoped to the old catalog name keep that scope and no longer match the renamed items. After such a rename a banner names those keys; update them under **Configuration → Keys**.
  * Catalog names may be mixed case (for example `DatrisFund` or `Sales_Q3`), and they are matched case-sensitively. A rename to a name that differs from another existing catalog only by case (for example `datrisfund` when `DatrisFund` exists) is refused with a message that the catalog already exists with different capitalisation, because it would create a second catalog next to it. A catalog can still be renamed to a different capitalisation of its own name.
* Click a card to expand it. Inside you'll see:
  * Each tap with its enabled/disabled status, last-run outcome, and record count.
  * Each pipeline with a quick-jump link to its config.
* **+ Create Catalog** — make a new empty catalog. The name is sanitized to letters, digits, `_` and `-` (spaces become `_`); capitalisation is kept. A name that differs from an existing catalog only by capitalisation is refused, here and in the new-catalog field of the tap and pipeline forms; typing an existing name exactly in those forms selects that catalog.
* **Delete catalog, keep items** — the delete icon on a named catalog's card opens a confirm with two choices. **Keep items (move to Uncataloged)** is the default: the catalog is removed and every tap and pipeline in it moves to Uncataloged. Nothing is deleted.
* **Delete catalog, delete items and their data** — the second choice deletes the catalog **plus every tap and pipeline inside it, including the data already written to their destinations**. This is destructive and not reversible, so **Delete** stays disabled until you type the catalog name exactly (case-sensitive). If some items could not be deleted, the page lists each one with the reason.
* **Download catalog** — the download icon on a named catalog's card saves the catalog as a zip archive (see [What an archive contains](#what-an-archive-contains)). The archive holds no secret values and no data, so it is safe to keep alongside other configuration backups.
* **Import catalog** — the **Import catalog** button next to **+ Create Catalog** restores an archive. Pick the zip (the panel shows its name and size), enter the catalog name to restore into (it defaults to the file name), and choose:

  * **Create new catalog (refuse if it exists)** — the default. The restore is refused when the catalog already exists or any archived tap or pipeline name already exists on this install; the page lists each clash and nothing changes. A restore never merges into an existing catalog.
  * **Overwrite catalog** — replaces the items already in that catalog. **Restore** stays disabled until you type the catalog name exactly.
  * **Skip incremental tap state (for a clone)** — restores the taps without the source's incremental cursor, so a copy starts from scratch.

  After the restore the page shows how many taps and pipelines came back, names any secrets they need that do not exist yet, lists items that could not be restored with the reason, and expands the restored catalog. Restored taps are disabled: test each one, then enable it. When some restored taps read tables produced by other restored taps, the result says how many: run their sources first, in the [enable order](/taps#dependencies-between-taps).
* **Enable order hints** — in an expanded catalog, each disabled tap shows its level in the catalog's enable order (`Level 1` reads no other tap's tables), or `needs: x, y` while the taps it reads from have not landed data. The **Disabled** badge menu names the sources to run first and offers **Run now** for a tap whose last test passed: one real run that lands its data without enabling it.
* **Enable in order** — on a catalog with at least one disabled tap, the play-list icon on the card shows the catalog's enable order level by level, and **Start** enables the whole catalog in that order in one job. For each tap the job runs a test if its last test did not pass, enables it, does one real run and waits until that data has landed before moving to the taps that read it. A tap whose source failed or landed nothing is skipped with the reason, so you see which source to fix. A run that returns no records fails the tap only when another tap in the enable order reads from it; a tap nothing depends on (for example one that is fed through run parameters) stays enabled and is marked landed with "no records returned; nothing depends on this tap". Progress shows on the card, one row per tap, and **Cancel** stops the job after the tap in progress (its run is not interrupted; the remaining taps are marked cancelled). While the job runs, the catalog cannot be renamed, deleted or restored over. After a restore, the result banner offers the same **Enable in order** link when restored taps depend on each other.
* **Uncataloged is not a catalog** — its card has no rename, delete or download control. Delete uncataloged items one at a time (below).
* **Delete individual tap or pipeline** — expand any catalog (including Uncataloged) and use the trash icon on a specific row to delete just that item. Clicking the trash icon swaps the row's action icons for an inline confirm: **Delete** / **×** for a tap; **Config & Data** / **Data Only** / **×** for a pipeline (Data Only wipes the rows but keeps the pipeline config so the next ingest fills it again).
* **Row actions menu (Uncataloged)** — each item in the Uncataloged group has a ⋮ kebab menu after the trash icon with three actions:
  * **Edit** — opens the tap or pipeline editor.
  * **Delete** — same as the trash icon (inline confirm).
  * **Move to catalog** — submenu listing every other catalog. Picking one sets the item's `catalog` field and reloads the view. Before persisting, the UI checks the target catalog for a tap or pipeline of the same name; on a clash the move is blocked and a red banner explains the collision ("A tap named 'foo' already exists in catalog 'reporting'. Rename one of them first.").

## Assigning a catalog

Four ways to put a tap or pipeline into a catalog:

1. **Tap editor** — when creating or editing a tap manually, set the **Catalog** field.
2. **Pipeline editor** — same field on the pipeline form.
3. **CLI** — pass `--catalog <name>` to `datris ingest` when creating a new pipeline. Re-ingesting an existing pipeline preserves its current catalog (and the rest of the config) — use the `set_catalog` MCP tool to change a catalog after the fact.
4. **MCP agent** — call `set_catalog(pipeline="<name>", catalog="<label>")` or `set_catalog(tap="<name>", catalog="<label>")` to set the label on an existing item. Pass an empty string (or omit `catalog`) to clear it back to Uncataloged. New pipelines created via `create_pipeline` accept an optional `catalog` argument directly.

Leave the field blank to keep the item Uncataloged.

## API and MCP

Catalogs are a field on the Tap and Pipeline configs. You read and write single items through the existing endpoints:

* `GET /api/v1/taps` and `GET /api/v1/pipelines` return the `catalog` field on each item.
* `POST` / `PUT` to those endpoints accept a `catalog` field. To move an item, send a request with the new value.

Two endpoints act on a whole catalog:

* `PUT /api/v1/catalog/{name}` with body `{"newName": "<label>"}` renames the catalog. If `newName` already exists the request is refused with 409 and nothing changes; move items to combine catalogs. The response lists the items `renamed`, any `failed` items with their error, and `affectedKeys`: API keys still scoped to the old name.
* `DELETE /api/v1/catalog/{name}?mode=detach` removes the catalog and moves its items to Uncataloged. `mode=cascade&confirm=<name>` deletes the items and their data as well; `confirm` must equal the catalog name. The cascade response's `warnings` lists each deleted pipeline whose Unity Catalog table Datris did not drop, as `{pipeline, message}` with the pipeline name and the table an admin has to drop in Unity Catalog; the Data Catalog page shows these in a banner until dismissed. Uncataloged cannot be renamed or deleted.

From an MCP agent:

* `set_catalog` wraps the read-modify-write on a single pipeline or tap, so you don't have to round-trip the full config yourself.

* `rename_catalog` renames a whole catalog in one call, with the same rules and affected-keys behaviour as the Catalog tab: the new name must not already exist.

* `delete_catalog` removes a catalog and moves its items to Uncataloged. It never deletes items or data; deleting a catalog together with its items is available from the UI or the API only.

* `catalog_enable_order` returns the catalog's taps in dependency levels with, per tap, whether its last test passed, whether a real run has landed data and which sources still block it (`GET /api/v1/catalog/{name}/enable-order`). Read-only. See [Dependencies between taps](/taps#dependencies-between-taps).

* `enable_catalog_ordered` starts the same job as **Enable in order** (`POST /api/v1/catalog/{name}/enable-ordered`, no body; 202 with the job id and plan). It needs `tap:run`, and the key must be allowed to update and run every tap in the catalog. Progress is the `job` block of `catalog_enable_order` (or `GET /api/v1/catalog/{name}/enable-ordered`); a finished job stays readable for an hour. Cancel with `DELETE /api/v1/catalog/{name}/enable-ordered` or from the card; there is no MCP cancel tool. Under an [agent policy](/agent-policy) the action is `tap:run:enable-ordered`, so one approval covers the whole job. How long the job waits for one tap's data to land is `datris.catalog.enableWaitMs` (default 30 minutes).

* `export_catalog` returns a catalog as a zip archive (base64) with its manifest; `restore_catalog` restores one under the same or a new name, with the same rules as **Import catalog**. From a terminal, `datris catalog export` and `datris catalog restore` do the same (see [CLI](/cli)).

For browsing, list taps or pipelines and group client-side by `catalog`.

## What an archive contains

A catalog archive holds:

* the definition of every tap and pipeline in the catalog,
* each tap's script,
* each tap's incremental state (its cursor),
* the validation schemas the pipelines reference.

It never holds secret values, destination data, run history or version history. The archive lists the names of the secrets the catalog needs; create any that are missing on the target install. Restored taps are disabled until you enable them.

To promote a catalog from one install to another, for example from dev to QA to production, see [Dev, QA and production](/production/environments).

Things to know when restoring:

* A restore never merges. Tap and pipeline names are unique across the install, so restoring onto the same install under a new name is refused while the source catalog still exists. Delete or rename the source catalog first.
* Inline REST bearer tokens and API keys on a pipeline are removed from the archive. Re-enter them after the restore; the result names each field that needs it.
* An API key without unscoped secret read access gets a "not checked" warning for each required secret instead of a list of missing secrets.
* To make a clone that starts from scratch, skip the incremental tap state (the checkbox in the import panel, or `--no-state` on the CLI).

## What catalogs don't do

* They don't isolate access — anyone with API access to a tap can see it regardless of catalog.
* They don't affect tap or pipeline behavior, scheduling, or destination.
* They aren't permissioned — catalogs are a flat namespace per tenant.

If you need true tenant isolation, that's handled at the Datris environment level, not via catalogs. See [tenant isolation](/configuration-reference) in the configuration reference.


# Data Quality — AI Rule
Source: https://docs.datris.ai/data-quality/ai-rules

AI-powered data quality rules

The `aiRule` validates data using a plain-English instruction. Datris generates a Python validation script from your instruction via an LLM (single API call) when the pipeline is saved, stores it, and runs the stored script against all data on every run, in an isolated runner container.

## Configuration

Define the AI rule in the `aiRule` field within the `dataQuality` block of the pipeline configuration.

| Field | Type | Description |
| - | - | - |
| `instruction` | string | A natural language description of all validation criteria. Multiple checks can be combined in one instruction. |
| `onFailureIsError` | boolean | If `true`, failures abort processing. If `false`, failures are logged as warnings. |

***

## How it works

1. **Prompt generation** — At pipeline save, Datris combines the source schema (delimiter, column names and types) with your instruction and sends a single prompt to the configured AI model. When a run has to generate the script instead (see [When the script is generated](#when-the-script-is-generated)), it uses the column names and a few sample rows from that run's data.
2. **Script generation** — The AI model generates a self-contained Python 3 validation script (stdlib only, no pip packages).
3. **Local execution** — On every run, the stored script is written to a temp file and executed via `python3` against the full data file. Runs make no model call.
4. **Result parsing** — The script outputs a JSON array of failures: `[{"index": <row_number>, "reason": "..."}]`. Datris parses this and maps failures back to the original data. The `index` is **0-based**: the first record is index `0`, the second is index `1`, and so on.
5. **Cleanup** — Temp files are deleted after execution.

### Example

```json theme={null}
{
  "dataQuality": {
    "aiRule": {
      "instruction": "Validate that all email addresses are properly formatted, all phone numbers contain 7-15 digits, all dates are in YYYY-MM-DD format, and prices are positive and not exceeding $1,000,000",
      "onFailureIsError": true
    }
  }
}
```

The generated Python script handles all checks in one pass.

***

## Works with all file types

This approach works for:

* **CSV/delimited files** — The script reads the CSV with the appropriate delimiter and validates each row.
* **JSON files** — The script parses the JSON array and validates each record.
* **XML files** — The script uses `xml.etree.ElementTree` to parse and validate each element.

For all formats, the instruction is the same plain-English description. The LLM generates the appropriate parser for the file type.

***

## When the script is generated

The script is generated once and stored, not on every run:

* **At save** — Saving a pipeline with an AI rule (`POST /api/v1/pipeline`, the MCP `create_pipeline` tool, or the UI wizard) generates the script from the source schema: the delimiter and the ordered field names and types. No sample rows are available when a pipeline is saved, so state any format the rule depends on in the instruction (for example "dates are YYYY-MM-DD"). The script is stored in the built-in object store under `pipeline-scripts/<pipeline>/`, or committed to the code repository when one is enabled (see [Pipeline scripts](/tap-github-storage#pipeline-scripts)).
* **Every run** — Each run executes the stored script. A run makes no model call, gives the same result for the same data, and keeps working if the model is unreachable. The status shows "Running stored CodeGen script generated at ...".
* **Re-saving** — Saving again regenerates only when the instruction or the source schema (delimiter, field names, types or order) changed. Re-saving an unchanged pipeline makes no model call. A changed CodeGen model does not regenerate: the script keeps the model that wrote it.
* **Pending** — If the script cannot be generated at save (no AI key, model unreachable, AI disabled), the pipeline is still saved, the save response carries a warning and `codegenScripts` lists the script as `pending` with the reason. JSON and XML pipelines have no column schema to generate from, so they are always saved pending. The first run generates the script from its data, stores it, and later runs reuse it.
* **Header differs** — If the header that reaches the rule differs from the one the script was written for (for example a column was removed by field protection or is missing from the file), that run generates a script for itself only and keeps the stored one. The status says why.
* **Failures** — A stored script that exits with an error, times out, or prints output that is not a JSON array fails the run. It is not regenerated automatically: the error names when the script was generated and the regenerate endpoint.

To read the stored script, call `GET /api/v1/pipelines/{name}/codegen-scripts`. To replace it (for example to adopt a new CodeGen model, or after a wrong script), call `POST /api/v1/pipelines/{name}/codegen-scripts/dataQuality/regenerate`, which needs the `pipeline:create` capability. If generation fails there, the current script stays in place.

With a code repository configured under **Configuration > Code Repository**, the script for a new pipeline is committed to it as `<pathPrefix>pipelines/<pipeline>/data-quality.py` and every run executes the recorded commit. You can review it, blame it and edit it in the repository; an edit is used only after you adopt it with `POST /api/v1/pipelines/{name}/codegen-scripts/dataQuality/pull`. See [Pipeline scripts](/tap-github-storage#pipeline-scripts) for commit messages, conflicts, and what happens when the repository is unreachable.

Pipelines created before stored scripts generate their script on the first run after the upgrade and reuse it after that.

## Values withheld

When the server sets `DATRIS_AI_SAMPLE_VALUES=false`, the prompt carries the column names and schema types (or a value-free skeleton of a JSON or XML file) and the line "Sample rows withheld by configuration" instead of sample rows. The generated script still runs on the full data. Because the model cannot see what the values look like, state any format the rule depends on in the instruction (for example "dates are YYYY-MM-DD"). See [Field Protection](/transformation/field-protection#what-still-sees-raw-data).

## Behavior

* An `aiRule` whose `instruction` is empty or only whitespace is ignored: no script is generated at save and the run skips the stage with a status line saying so.
* The AI rule's Python script is generated when the pipeline is saved (or by the first run that needs one) and reused by every run until the instruction or source schema changes
* The generated script runs via `python3` with a 5-minute timeout, in the isolated `datris-codegen-runner` container on a compose install (see below)
* Scripts are constrained to Python standard library only (no external packages)
* Rules with `onFailureIsError: true` are counted as errors; `onFailureIsError: false` are warnings
* All rows are evaluated; the abort message lists the first **100** errors and a count of any remaining
* After all rows are evaluated, if **any errors** exist, processing is aborted
* Warnings are summarized at the end of validation

## Where the script runs

On a compose install the generated script runs in `datris-codegen-runner`, an isolated container with no platform secrets, no access to the server's files, and no route to the internal services or the internet. The server sends it the script and the data file and reads back the result. If the runner is stopped or unreachable the rule fails with an error naming it; it never falls back to running the script in the server. See [Generated data-quality and transformation scripts](/tap-execution-isolation#generated-data-quality-and-transformation-scripts).

## Requirements

* `ai.enabled: true` must be set in `application.yaml`
* The Vault secret for the AI provider must be configured (see [AI Configuration](/ai-configuration))


# CSV Header Validation
Source: https://docs.datris.ai/data-quality/header-validation

Validate CSV headers

Header validation uses an AI model to check that the CSV file header matches the expected field names defined in the pipeline schema. The validation is fuzzy — it allows case differences, underscores vs spaces, abbreviations, and minor naming variations. Column order does not matter. All schema columns must be present; extra columns in the file are OK.

## Requirements

* The source file must be CSV format.
* The pipeline must be configured with `header: true` (i.e., the first row contains column names).
* `ai.enabled: true` must be set in `application.yaml`.

## Configuration

Enable header validation by setting `validateFileHeader` to `true` in the `dataQuality` block:

```json theme={null}
{
  "dataQuality": {
    "validateFileHeader": true
  }
}
```

## Behavior

1. The pipeline reads the first row of the CSV file (the header) and the schema field names.
2. Both are sent to the AI model, which evaluates whether they match.
3. The AI allows fuzzy matching: `"First Name"` matches `"first_name"`, `"qty"` matches `"quantity"`, etc.
4. Column order does not matter — the pipeline uses the header to map columns to schema fields by name.
5. All schema columns must be present in the header. Missing columns fail validation.
6. Extra columns in the header beyond the schema are accepted.
7. If validation fails, a clear error message explains which columns are missing or unmatched.

## When to Use

Use header validation to catch files with missing columns or unexpected column names. Because matching is fuzzy and order-independent, minor formatting differences won't cause false rejections.


# JSON/XML Schema Validation
Source: https://docs.datris.ai/data-quality/schema-validation

Validate data against JSON/XML schemas

Schema validation checks incoming JSON or XML files against a formal schema definition before processing. This catches structural issues, missing required fields, and type mismatches early in the pipeline.

## Requirements

* The source file must be JSON or XML format.
* The validation schema file must be stored in MinIO in the `{environment}-config` bucket under `validation-schema/{filename}`.

## Configuration

Set the `validationSchema` field in the `dataQuality` block to the filename of the schema. The pipeline resolves the file from the MinIO config bucket at `{environment}-config/validation-schema/{filename}`. A full `s3://bucket/key` URL is also accepted and used as-is.

```json theme={null}
{
  "pipelineName": "product_catalog",
  "sourceFileFormat": "JSON",
  "dataQuality": {
    "validationSchema": "product_catalog_schema.json"
  }
}
```

## Example Schema

Validation uses the Everit `org.everit.json.schema` library (1.5.1), which implements **JSON Schema Draft 4 only**. The `$schema` URI is ignored, and Draft 07-only keywords (`const`, `if`/`then`/`else`, `propertyNames`, numeric `exclusiveMinimum`/`exclusiveMaximum` as values) are silently ignored rather than enforced. Write schemas to the Draft 4 specification.

A JSON Schema file stored in MinIO at `{environment}-config/validation-schema/product_catalog_schema.json`:

```json theme={null}
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "required": ["product_id", "name", "price"],
  "properties": {
    "product_id": {
      "type": "string",
      "pattern": "^PRD-[0-9]{6}$"
    },
    "name": {
      "type": "string",
      "minLength": 1
    },
    "price": {
      "type": "number",
      "minimum": 0
    },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    }
  }
}
```

## Behavior

1. The pipeline loads the schema file from the configured path in the environment's config bucket.
2. The whole file is validated as a single document against the schema. A JSON file containing an array of records therefore needs a schema with `"type": "array"` and `items` describing each record; the example above validates a file that holds a single object.
3. If any record fails validation, the file is rejected and processing stops.
4. Validation errors are logged with details about which fields failed and why.

## When to Use

Use schema validation when you need to enforce a strict contract on the structure of incoming JSON or XML data. This is particularly useful when data is provided by external systems where you do not control the format directly.


# Publish pipeline output to ActiveMQ
Source: https://docs.datris.ai/destinations/activemq

Write validated pipeline output as messages to an ActiveMQ queue.

Use this destination when downstream systems consume from ActiveMQ and you want the messages validated before they are queued.

The ActiveMQ destination writes messages to an ActiveMQ queue.

## Message Format

The payload format depends on the source schema:

* **Structured data** -- When the schema contains regular typed columns, each row is serialized as a JSON object and sent as a text message.
* **JSON or XML** -- When the schema contains a single `_json` or `_xml` field, the raw string value is sent directly as the message body without additional wrapping.

## Configuration Example

```json theme={null}
{
  "name": "orders_activemq",
  "source": { "..." : "..." },
  "destination": {
    "activeMQ": {
      "queueName": "orders.inbound"
    }
  }
}
```

ActiveMQ connection credentials are configured globally in `application.yaml` via `secrets.activeMQSecretName`. See [Configuration Reference](/configuration-reference) for Vault secret format.

### Field Reference

| Field | Required | Default | Description |
| - | - | - | - |
| `queueName` | yes | | Target queue name |

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Embed documents into Chroma for RAG
Source: https://docs.datris.ai/destinations/chroma

Chunk, embed, and upsert documents into a Chroma collection for retrieval-augmented generation with a single-container vector store.

Use this destination when you want the simplest possible RAG index, built and refreshed by a repeatable pipeline rather than a notebook.

Ingest unstructured documents into [Chroma](https://www.trychroma.com), a lightweight, developer-friendly open-source vector database. The pipeline extracts text from the document, chunks it using a configurable strategy, generates vector embeddings, and upserts the chunks with metadata into a Chroma collection via its REST API.

Chroma is the simplest vector database to set up — a single Docker container with no external dependencies.

## Configuration

```json theme={null}
"source": {
    "fileAttributes": {
        "unstructuredAttributes": {
            "fileExtension": "pdf",
            "preserveFilename": true
        }
    }
},
"destination": {
    "chroma": {
        "collectionName": "financial_documents",
        "chunking": {
            "strategy": "recursive",
            "chunkSize": 500,
            "chunkOverlap": 50
        },
        "metadata": {
            "company": "Apple Inc",
            "documentType": "10-Q",
            "filingDate": "2026-01-30"
        },
        "chromaSecretName": "oss/chroma"
    }
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `collectionName` | string | (required) | Chroma collection name. Auto-created with cosine distance if it doesn't exist. |
| `chunking` | object | recursive, 500/50 | Chunking strategy configuration (see below). |
| `metadata` | map | `{}` | Static key-value metadata stored on every chunk. Used for filtered search. |
| `embeddingSecretName` | string | *server default* | Embedding Vault secret name. Only consulted when the server-level `ai.embedding.secretName` is unset; with the default configuration (`oss/embedding`, seeded automatically) the server-level secret is always used and this field is ignored. |
| `chromaSecretName` | string | (required) | Vault secret name for Chroma connection. |

## Supported File Types

| Format | Description |
| - | - |
| PDF (`.pdf`) | Text extracted via Apache PDFBox |
| Word (`.doc`) | Text extracted via Apache POI (legacy format) |
| Word (`.docx`) | Text extracted via Apache POI (modern format) |
| PowerPoint (`.ppt`) | Text extracted via Apache POI (legacy format) |
| PowerPoint (`.pptx`) | Text extracted via Apache POI (modern format) |
| Excel (`.xls`, `.xlsx`) | Cell values extracted via Apache POI |
| HTML (`.html`, `.htm`) | Text extracted via JSoup (tags stripped) |
| RTF (`.rtf`) | Text extracted via javax.swing RTF parser |
| Email (`.msg`) | Subject, from, to, and body extracted via Apache POI |
| Email (`.eml`) | Subject, from, and body extracted via Jakarta Mail |
| EPUB (`.epub`) | XHTML content extracted and parsed via JSoup |
| Plain text (`.txt`, `.md`, `.csv`, `.json`, `.xml`) | Content used directly |

For structured data (CSV, JSON, XML), use database destinations (PostgreSQL, MongoDB) instead.

## Vault Secrets

### Chroma connection

```bash theme={null}
vault kv put secret/oss/chroma \
  host="host.docker.internal" \
  port="8000"
```

| Field | Description |
| - | - |
| `host` | Chroma server hostname. Use `host.docker.internal` for local Chroma. |
| `port` | REST API port (default 8000). |

Chroma does not require authentication by default. For Chroma Cloud or authenticated instances, add token-based auth to the loader as needed.

### Embedding API

The embedding secret is server-level (`ai.embedding.secretName`, default `oss/embedding`) and is seeded automatically by `docker/vault-init.sh`. You only need to write it manually if you want to point at a different embedding provider. See [AI Configuration](/ai-configuration) for the full picture.

```bash theme={null}
vault kv put secret/oss/embedding \
  provider="openai" \
  endpoint="https://api.openai.com/v1/embeddings" \
  model="text-embedding-3-small" \
  apiKey="sk-..."
```

For Anthropic-only deployments, the bundled TEI sidecar serves `bge-m3` (1024-dim) and `vault-init.sh` seeds the embedding secret to point at it — no OpenAI key required.

## Chunking Strategies

Documents are split into chunks before embedding. Each chunk becomes a separate entry in the Chroma collection with the document's metadata plus `chunk_index`, `filename`, and `source_pipeline` fields.

```json theme={null}
"chunking": {
    "strategy": "recursive",
    "chunkSize": 500,
    "chunkOverlap": 50
}
```

| Strategy | Description |
| - | - |
| `none` | No chunking — one embedding per document. Only for very short documents. |
| `fixed` | Split by character count. Fast but may cut mid-sentence. |
| `sentence` | Split on sentence boundaries (`.` `!` `?`). Preserves semantic units. |
| `paragraph` | Split on double newlines. Ideal for structured documents with clear sections. |
| `recursive` | Try `\n\n`, then `\n`, then `.`, then space — best general-purpose default. |

* `chunkSize` (default 500): maximum characters per chunk
* `chunkOverlap` (default 50): characters of overlap between consecutive chunks

## Metadata

Static metadata is stored on every chunk in the Chroma collection:

```json theme={null}
"metadata": {
    "company": "Apple Inc",
    "documentType": "10-Q",
    "filingDate": "2026-01-30"
}
```

This allows filtered queries combining similarity with metadata predicates using Chroma's `where` clause.

Every entry automatically includes:

* `text` — the chunk text (stored as Chroma document)
* `chunk_index` — position of the chunk in the document
* `filename` — original uploaded filename
* `source_pipeline` — pipeline name

## How It Works

1. **Upload** — an unstructured file is uploaded via `POST /api/v1/pipeline/upload`
2. **Extract** — text is extracted from the document
3. **Chunk** — text is split into chunks using the configured strategy
4. **Embed** — each chunk is sent to the embedding API to generate a vector
5. **Upsert** — vectors are upserted into the Chroma collection via REST API with metadata
6. **Notify** — a pipeline notification is published on completion

The collection is auto-created on first upsert with cosine distance. No Java client library is needed — the loader communicates with Chroma's REST API directly via HTTP.

## Running Chroma

Chroma ships as a bundled **opt-in** Compose service. Enable it with one line in `.env`:

```bash theme={null}
COMPOSE_PROFILES=chroma
```

Then start the stack:

```bash theme={null}
docker compose up -d
```

Combine profiles with commas (e.g., `COMPOSE_PROFILES=chroma,kafka`). The `install.sh` installer can also enable Chroma at install time — it writes the profile and seeds the `oss/chroma` secret for you. For the bundled container, use the compose service name as the host in the secret (`host="chroma"`, port `8000`).

To use an existing external instance instead, skip the profile and point the `oss/chroma` secret's `host`/`port` at it — the secret fields are the same either way. You can also run Chroma standalone:

```bash theme={null}
docker run -d --name chroma -p 8000:8000 chromadb/chroma:latest
```

Either way there are no external dependencies like etcd or MinIO — Chroma is the simplest vector database to run.

## Verifying

```bash theme={null}
# Check Chroma is running
curl http://localhost:8000/api/v2/heartbeat

# List collections
curl http://localhost:8000/api/v2/tenants/default_tenant/databases/default_database/collections

# Get collection details
curl http://localhost:8000/api/v2/tenants/default_tenant/databases/default_database/collections/financial_documents

# Count entries in collection (replace COLLECTION_ID with the actual UUID)
curl http://localhost:8000/api/v2/tenants/default_tenant/databases/default_database/collections/COLLECTION_ID/count
```


# Load data into Databricks Delta tables from an AI agent or pipeline
Source: https://docs.datris.ai/destinations/databricks

Load validated data into Unity Catalog managed Delta tables through a SQL warehouse with COPY INTO, MERGE upserts, and per-pipeline credentials.

Use this destination when you want an agent or a scheduled pipeline to land validated data in Databricks without giving it workspace credentials directly. Each pipeline references its own secret, and every load is atomic, audited, and lineage-tracked.

The Databricks destination loads data into **Unity Catalog managed Delta tables** through a SQL warehouse: the pipeline output is staged into a Unity Catalog volume, then loaded with a single atomic SQL statement — `COPY INTO` for inserts, `MERGE` when `keyFields` is set, or `INSERT OVERWRITE` when `truncateBeforeWrite` is set. Every load is one Delta commit, so a failed run never leaves the table half-written or empty. Tables land governed and immediately queryable in the workspace — notebooks, dashboards, Genie, and lineage all see them natively.

Unlike the bundled PostgreSQL and MongoDB destinations, Databricks is an external target: credentials are per-pipeline, referenced by name through `credentialsSecret` — the same model as the [Snowflake](/destinations/snowflake) and [S3](/destinations/s3) destinations. Different pipelines can write to different Databricks workspaces.

Serverless SQL warehouses work out of the box and are the recommended compute — nothing runs in your cloud account and there is no cluster to manage.

## One-time Databricks setup

This is the recommended long-term setup for connecting Datris to a company Databricks workspace: a dedicated **service principal** with **OAuth machine-to-machine (M2M)** authentication and least-privilege grants on one catalog. A service principal is Databricks' recommended identity for unattended integrations — it is not tied to any employee's account and its access is scoped by Unity Catalog grants you control.

### 1. Create a service principal and OAuth secret

In the workspace, go to **Settings → Identity and access → Service principals** (or the account console for account-wide principals):

1. **Add service principal** — name it e.g. `datris-writer`.
2. Open it and generate an **OAuth secret**. Copy the **Client ID** (the principal's application ID) and the **Secret** — the secret is shown only once.

### 2. Note the SQL warehouse ID and grant access

Open **SQL Warehouses → your warehouse → Connection details**. The **warehouse ID** is the trailing segment of the **HTTP path** — a 16-character hex string, **not** the warehouse's display name:

```
HTTP path:     /sql/1.0/warehouses/a1b2c3d4e5f67890
Warehouse ID:                      a1b2c3d4e5f67890
```

This ID goes in the destination's `warehouse` field, or in a `warehouse` field on the platform secret (step 4) so every pipeline using that secret picks it up. (Pasting the whole HTTP path into Datris also works — the ID is extracted automatically.)

On the warehouse's **Permissions**, grant the service principal **Can use**.

A serverless warehouse with a short auto-stop (5–10 minutes) is ideal: it starts in seconds when a load arrives and costs nothing while idle.

### 3. Grant catalog access

Catalog names vary by workspace: long-standing accounts often have a catalog named `main`, but new and trial/express workspaces typically ship one named `workspace` instead — check what yours has before granting:

```sql theme={null}
SHOW CATALOGS;
```

The script below creates a dedicated `datris` catalog so the grants don't depend on what already exists (a new catalog automatically contains a `default` schema, which Datris loads into). Run it in the **SQL editor** as a workspace admin (or metastore admin) — every statement is idempotent, so re-running is safe.

One gotcha: `GRANT` identifies a service principal by its **application ID** — the same UUID as the client ID from step 1 — *not* by its display name. Granting to `datris-writer` fails with `PRINCIPAL_DOES_NOT_EXIST` even though the principal exists. Replace `<sp-application-id>` below with the UUID:

```sql theme={null}
-- Dedicated catalog for Datris pipeline output. Requires metastore default
-- storage (standard on serverless/express workspaces). If you'd rather load
-- an existing catalog — or you lack CREATE CATALOG rights — skip this line
-- and replace `datris` below with a name from SHOW CATALOGS.
CREATE CATALOG IF NOT EXISTS datris;

-- <sp-application-id> = the service principal's application/client ID (UUID),
-- shown on its page under Settings → Identity and access → Service principals.
GRANT USE CATALOG ON CATALOG datris TO `<sp-application-id>`;
GRANT USE SCHEMA, CREATE TABLE, CREATE VOLUME, SELECT, MODIFY
  ON SCHEMA datris.default TO `<sp-application-id>`;

-- Optional: lets Datris auto-create additional schemas beyond `default`
GRANT CREATE SCHEMA ON CATALOG datris TO `<sp-application-id>`;

-- Optional: needed only for Unity Catalog metadata (`unityCatalog.enabled`);
-- without it, tagging is reported as a warning on each run
GRANT APPLY TAG ON SCHEMA datris.default TO `<sp-application-id>`;

-- Optional: needed only for Unity Catalog lineage (`unityCatalog.lineage`,
-- on by default when `unityCatalog.enabled`). A separate, metastore-admin-only
-- step; without it, lineage is reported as a warning on each run
GRANT CREATE EXTERNAL METADATA ON METASTORE TO `<sp-application-id>`;

-- Recommended: let your own users read what Datris loads. Tables are OWNED
-- by the service principal that created them, and Unity Catalog gives no
-- implicit access to anyone else — not even workspace admins — so without
-- this, your first SELECT fails with INSUFFICIENT_PERMISSIONS. Granting at
-- the schema level covers current AND future tables. Use a specific email
-- instead of `account users` to scope it down.
GRANT USE CATALOG ON CATALOG datris TO `account users`;
GRANT USE SCHEMA, SELECT ON SCHEMA datris.default TO `account users`;
```

Tables and the staging volume auto-created by Datris are owned by the service principal, so no further grants are needed. If you manage tables yourself (`manageTableManually: true`), also pre-create the staging volume and grant access:

```sql theme={null}
CREATE VOLUME IF NOT EXISTS datris.default.datris_staging;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME datris.default.datris_staging TO `<sp-application-id>`;
```

### 4. Create the platform secret in Datris

On **Configuration → Secrets → Platform**, create a secret (for example `databricks`) with:

| Field | Required | Notes |
| - | - | - |
| `host` | yes | Workspace hostname, e.g. `dbc-a1b2c3d4-e5f6.cloud.databricks.com`. Pasting the full workspace URL also works — Datris strips the protocol and path |
| `clientId` | yes\* | The service principal's client ID (application ID) |
| `clientSecret` | yes\* | The OAuth secret generated in step 1 |
| `token` | no | Personal access token — fallback when no `clientId`/`clientSecret` is present, see below |
| `warehouse` | no | SQL warehouse ID (or its HTTP path). Used when a pipeline's destination leaves `warehouse` out, and by [Unity Catalog](/destinations/unity-catalog) browse. `httpPath`, `http_path` and `DATABRICKS_WAREHOUSE` also work |

\*Either the `clientId`/`clientSecret` pair or `token` must be present.

The `host` lives in the secret rather than the pipeline config so it stays bound to the credential that authorizes it. If a required field is missing, the load fails at resolve time with an explicit error naming the secret and the field. With a `warehouse` field on the secret, pipelines (and agents creating them) can leave the destination's `warehouse` out.

### Secret rotation

OAuth secrets have a fixed lifetime (up to two years; you choose at creation). A service principal can hold up to five secrets at once, so rotation has no downtime: generate a new secret, update the Datris secret's `clientSecret`, then delete the old one in Databricks.

### Alternative: personal access token (PAT)

For a quick evaluation you can skip the service principal: generate a PAT (**Settings → Developer → Access tokens**) and store it in the secret's `token` field. Be aware of the trade-offs before relying on one long-term: a PAT authenticates as *you*, inherits *your* permissions, and expires. A service principal with scoped grants is the right identity for an unattended integration.

## Configuration

```json theme={null}
{
  "name": "orders_pipeline",
  "source": { "..." : "..." },
  "destination": {
    "database": {
      "useDatabricks": true,
      "credentialsSecret": "databricks",
      "warehouse": "1a2b3c4d5e6f7a8b",
      "dbName": "datris",
      "schema": "default",
      "table": "orders",
      "keyFields": ["order_id"]
    }
  },
  "unityCatalog": { "enabled": true }
}
```

### Field reference

| Field | Required | Default | Description |
| - | - | - | - |
| `useDatabricks` | yes | `false` | Must be `true` to enable the Databricks destination |
| `credentialsSecret` | yes | | Name of the Platform secret holding `host` + auth |
| `warehouse` | if not on secret | | SQL warehouse **ID** — the 16-char hex trailing segment of the HTTP path in **SQL Warehouses → Connection details** (`/sql/1.0/warehouses/<id>`), not the warehouse's display name. A pasted full HTTP path is accepted too. Optional when the platform secret has a `warehouse` field; the pipeline's value wins when both are set, and a load with neither fails with an error naming both fixes |
| `dbName` | yes | | Target **Unity Catalog catalog** |
| `schema` | yes | | Target schema |
| `table` | yes | | Target table |
| `manageTableManually` | no | `false` | If `true`, skip auto-creation of the schema, table, and staging volume |
| `truncateBeforeWrite` | no | `false` | Replace the table contents atomically on each run |
| `keyFields` | no | | Natural-key columns; switches the load to a `MERGE` upsert |

Note that `dbName` names the Unity Catalog **catalog** — Databricks uses three-level names (`catalog.schema.table`), and `dbName`/`schema`/`table` map onto them in order.

Identifiers follow Unity Catalog's normal resolution rules: names are case-insensitive and stored lowercase, so `Orders` and `orders` are the same table. Names containing characters beyond letters, digits, and underscores are backtick-quoted automatically; prefer simple underscore names.

### Unity Catalog metadata

Set the top-level `"unityCatalog": { "enabled": true }` and every successful load also annotates the table in Unity Catalog, so warehouse users can see in Catalog Explorer which Datris pipeline loaded a table and how to trace it:

* **Comments:** a table comment naming the Datris pipeline and its source (the tap, or `file upload`), plus a fixed comment on each `_datris_*` provenance column the load carries (see [provenance stamping](/provenance)).
* **Tags:** `datris_pipeline`, `datris_catalog` (only when the pipeline belongs to a catalog), `datris_dq_status` (`pass` or `warn`) and `managed_by = datris`.
* **Table properties:** `datris.pipeline`, `datris.configVersion`, `datris.lastRunId`, `datris.lastRunAt`, `datris.environment`, `datris.lineagePath` (the Datris lineage API path for the pipeline) and `datris.lastSource` (the tap or the uploaded file name of the last run).

**On by default for the install:** set `DATRIS_UNITY_CATALOG_DEFAULT=enabled` on the Datris server and every Databricks pipeline without a `unityCatalog` block behaves as if it had `{"enabled": true}`, with all the knobs below on. `GET /api/v1/pipelines/{name}/unity-catalog` then reports `enabledBy: "default"` and the pipeline page marks the card "enabled by default". A pipeline's own block always wins, so to opt one pipeline out set:

```json theme={null}
"unityCatalog": { "enabled": false }
```

`DATRIS_UNITY_CATALOG_SYNC=false` still switches everything off. The default is `disabled`, and it never applies to object store Iceberg pipelines.

| Field | Default | Description |
| - | - | - |
| `enabled` | `false` (`true` for a pipeline with no block when `DATRIS_UNITY_CATALOG_DEFAULT=enabled`) | Turn the metadata push on. Databricks destinations only; any other destination is rejected on save |
| `comments` | `true` | Write the table and provenance column comments |
| `tags` | `true` | Write the four tags |
| `properties` | `true` | Write the table properties |
| `lineage` | `true` | Publish the tap or upload, the pipeline and the table as external lineage, with column mappings (see [Lineage in Unity Catalog](/destinations/unity-catalog#lineage-in-unity-catalog)) |

The `credentialsSecret`, `catalog`, `schema` and `register` fields of the block apply to object store Iceberg tables only (see [Register Iceberg tables](/destinations/unity-catalog#register-iceberg-tables)); a Databricks pipeline ignores them. For an object store Iceberg table governed by Databricks Unity Catalog, use `catalogMode: "managed"`: Databricks chooses the table's location in the pipeline's S3 bucket and Datris commits every write through the catalog (see [Databricks: catalogMode managed](/destinations/unity-catalog#databricks-catalogmode-managed)). Databricks has no `register` call and ignores a requested location, so `register` and `rest` do not keep the pointer there.

Comments and tags are only re-issued when they change; properties are written on every run. With `manageTableManually: true`, if you drop and recreate the table yourself, Datris re-applies comments and tags only when their content changes. Tagging needs the `GRANT APPLY TAG ON SCHEMA` line from the setup script. Lineage needs the metastore-level `GRANT CREATE EXTERNAL METADATA ON METASTORE` line, which only a metastore admin can run. A metadata failure (a missing grant, a warehouse hiccup) is recorded as a warning on the run and in the audit log; the load itself still succeeds, and the next run retries the failed part. The pipeline detail page shows the last sync or the last error, also available from `GET /api/v1/pipelines/{name}/unity-catalog`. Set `DATRIS_UNITY_CATALOG_SYNC=false` to switch the push off for the whole install.

To see what a Databricks secret can already reach in Unity Catalog, or to search warehouse tables from `find_data`, see [Unity Catalog](/destinations/unity-catalog).

`useTransaction` has no effect on Databricks — Delta has no multi-statement transactions. It doesn't need one: every load path here is a single atomic Delta commit, which is a stronger guarantee than a transaction around a multi-statement load.

## Staging Volume

Loads stage through a Unity Catalog volume named `datris_staging`, auto-created in the target schema on first run. Each run uploads one uniquely-named file and removes it when the load finishes — the volume is empty between runs. The volume is shared by every pipeline loading that schema and is left in place when a pipeline is deleted. With `manageTableManually: true` the volume is not auto-created; pre-create it as shown in the setup section.

## Table Management

When `manageTableManually` is `false` (the default), the pipeline auto-creates the schema, staging volume, and table if they don't exist. When `keyFields` is set, key columns are created `NOT NULL` with an informational `PRIMARY KEY` constraint (Unity Catalog primary keys document the key but are not enforced — the `MERGE` provides the actual dedupe). Schema changes are additive: new columns in the pipeline schema are added with `ALTER TABLE ... ADD COLUMN`; columns are never dropped or retyped.

**Who can read the tables:** tables are owned by the service principal that created them, and Unity Catalog grants no implicit access to anyone else — not even workspace admins. If your own `SELECT` fails with `INSUFFICIENT_PERMISSIONS`, grant your users read access at the schema level (covers current and future tables): `GRANT USE SCHEMA, SELECT ON SCHEMA <catalog>.<schema> TO ...` — included as the "Recommended" block in the setup script above.

## Type Mapping

| Source Type | Databricks Type |
| - | - |
| `string` | `STRING` |
| `int` / `integer` / `tinyint` / `smallint` | `INT` |
| `bigint` | `BIGINT` |
| `float` | `FLOAT` |
| `double` | `DOUBLE` |
| `boolean` | `BOOLEAN` |
| `date` | `DATE` |
| `timestamp` | `TIMESTAMP` |
| `_json` | `VARIANT` |
| `_xml` | `STRING` |

Other types (`decimal(p,s)`) pass through as-is. Semi-structured `_json` columns land as `VARIANT` (parsed with `PARSE_JSON` during the load), so JSON payloads are natively queryable with Databricks' `:` path syntax rather than stored as text.

## Truncate Before Write

Set `truncateBeforeWrite` to `true` to replace the full table contents on each run. The loader uses `INSERT OVERWRITE`, which swaps the contents in **one atomic Delta commit**: readers never observe an empty table mid-load, and if the load fails the previous rows survive untouched. (Previous versions of the data also remain available through Delta time travel, subject to the table's retention settings.)

## Primary Key and Upsert

Specify one or more columns in `keyFields` to define the natural key. When set (and `truncateBeforeWrite` is `false`), the loader issues a `MERGE` on the key columns directly from the staged file: matched rows are updated, unmatched rows are inserted. The same rows can be loaded across runs without duplicates.

Upsert semantics match the PostgreSQL and Snowflake destinations: on match, non-key columns are overwritten in full, including `NULL`s. If a single batch contains duplicate key values, Delta rejects the `MERGE` as non-deterministic — deduplicate upstream.

When `keyFields` is empty, the loader uses a straight `COPY INTO` — the fastest path, for append-only ingestion. The run status reports the inserted-row count when the warehouse returns one and otherwise shows it as unknown; the load itself is a single `COPY INTO` commit either way.

## Querying from Datris

The Assistant (and any MCP client) can read the destination back with the `query_databricks` tool — useful for verifying a load landed and for answering questions over the data without leaving Datris. Queries are pipeline-scoped: the tool takes a pipeline name, and the server connects with that pipeline's credentials secret, warehouse, and catalog. Credentials never leave the server, and only read-only statements are accepted — `SELECT` (with `LIMIT` applied automatically), plus `SHOW` and `DESCRIBE` for discovering catalogs, schemas, tables, and columns.

Queries run on the pipeline's configured SQL warehouse and consume Databricks compute like any other query. Results are capped at 100 rows by default.

## Warehouse Behavior

A stopped SQL warehouse auto-starts when a load or query arrives. Serverless warehouses start in seconds; classic warehouses can take several minutes, during which the connection waits. The run's status log notes when a start may be in progress. Loads are batch-shaped and infrequent, so a short auto-stop keeps warehouse cost minimal.

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Write Apache Iceberg tables to MinIO or S3 from a pipeline
Source: https://docs.datris.ai/destinations/iceberg

Land validated pipeline output as an Apache Iceberg table on MinIO or AWS S3 with atomic commits, upsert by key, and schema evolution that any Iceberg-aware engine can read by path.

Use this format when you want pipeline output to land beside your lake as a real table rather than a directory of loose files: one atomic commit per run, upsert by natural key, new columns added without breaking readers, and a location your own query engines open directly, with no Datris in the read path.

Iceberg is a table format, not a file format. The data files it writes are still Parquet; what it adds is a `metadata/` directory next to them that records exactly which files make up the table at each commit. In Datris it is selected the same way as `parquet` or `orc`: set `fileFormat` to `iceberg` on an [Object Store](/destinations/object-store) or [S3](/destinations/s3) destination. There is no new destination type and no extra service to run.

## What an Iceberg table gives you over loose files

| Concern | Loose Parquet and ORC files | Iceberg table |
| - | - | - |
| A run fails half-way | Partial files are left under the prefix and readers scan them | Nothing is committed; readers keep seeing the previous snapshot |
| Loading the same rows again | Duplicates, unless you overwrite the whole prefix | `writeMode: merge` upserts on `keyFields` |
| Adding a column to the destination schema | Readers see a directory of mixed schemas | The column is added to the table; older rows read as null |
| What the lake sees | A path to glob | A table with a schema, a partition spec, and a history of snapshots |

## Configuration

```json theme={null}
{
  "destination": {
    "objectStore": {
      "prefixKey": "sales/orders",
      "fileFormat": "iceberg",
      "writeMode": "merge",
      "keyFields": ["order_id"],
      "partitionBy": ["region"]
    }
  }
}
```

For AWS S3 add `provider`, `destinationBucketOverride` and `credentialsSecret` exactly as on the [S3 page](/destinations/s3); the Iceberg-specific fields are the same for both providers.

### Field reference

| Field | Required | Default | Description |
| - | - | - | - |
| `prefixKey` | yes | | Key prefix under the bucket. The Iceberg table lives at this prefix: `metadata/` and `data/` are created under it |
| `fileFormat` | no | `parquet` | Must be `iceberg` for this page. `parquet` and `orc` keep writing loose files |
| `writeMode` | no | `append` | `append`, `overwrite`, `merge`, `ignore`, or `errorifexists`. `merge` is only valid with `iceberg` |
| `keyFields` | with `merge` | | Natural-key columns for `merge`; each must be a column in the destination schema. Not accepted with any other write mode |
| `partitionBy` | no | | Columns to partition the table by (identity partitioning) |
| `deleteBeforeWrite` | no | `false` | Remove everything under the prefix before writing, which drops the table and recreates it. Applies on every run while it stays set, so clear it once the table exists |
| `destinationBucketOverride` | for S3 | `{environment}-data` | Bucket to write to; required when `provider` is `s3` |
| `provider` | no | `minio` | `minio` (built-in) or `s3` (AWS S3) |
| `credentialsSecret` | for S3 | | Name of the Platform secret holding `accessKey`, `secretKey` and `region` for the S3 bucket |

`writeToTemporaryLocation` is not accepted together with `iceberg`: commits are already atomic, so it would have nothing to add.

## Write modes

Every run that writes rows produces exactly one new snapshot. Readers see either the snapshot before the run or the one after it, never an in-between state.

| Mode | Behaviour |
| - | - |
| `append` | Adds the run's rows to the table as one snapshot. Two runs of the same data give twice the rows |
| `overwrite` | Replaces the table's contents in one snapshot. On a partitioned table only the partitions present in the run are replaced; untouched partitions keep their rows |
| `merge` | Upserts on `keyFields`: a row whose key matches an existing row replaces it, every other row is inserted. Re-running the same data leaves the row count unchanged and adds one snapshot |
| `ignore` | Does nothing if the table already exists |
| `errorifexists` | Fails the run if the table already exists |

`merge` is a true upsert: all non-key columns of the incoming row overwrite the existing row, including nulls. This matches `keyFields` on the PostgreSQL, MongoDB, Snowflake and Databricks destinations, so an agent that already knows the concept can reuse it here. If your source emits partial rows, coalesce them upstream.

## Partitioning

`partitionBy` creates an identity partition on each named column. Data files land under `data/<column>=<value>/` inside the prefix, and engines that understand Iceberg partition pruning skip partitions your query does not touch. Every column in `partitionBy` must exist in the destination schema.

Adding partitioning to, or removing it from, an existing pipeline is refused because the table would have to be rebuilt: delete the pipeline's destination data first, then re-register it with the new `partitionBy`.

## Schema evolution

Before each write the destination schema is compared with the table's current schema:

* **New nullable columns are added automatically.** Existing rows read as null for the new column. The column is committed before the data is written, so if the data write then fails, the column exists but has no values yet; the failure message says so.
* **Type changes are refused.** A column whose type differs from the one already in the table stops the run with a message naming the column and both types.
* **Dropping columns is refused.** A column present in the table but missing from the destination schema stops the run. Remove it from the table with `deleteBeforeWrite` (which recreates the table) or keep it in the schema.
* **New required columns are refused.** A new column that is not nullable stops the run, since existing rows would have no value for it.

When a change is refused nothing is committed: the table stays at its previous snapshot and the run reports the reason.

## Reading the table from outside Datris

The table is fully described by the files under the prefix. `metadata/` holds one metadata file per commit plus a `version-hint.text` naming the current one; `data/` holds the Parquet files. Any Iceberg-aware engine that can open the location by path reads the current snapshot without contacting Datris. For example, with the Python Iceberg library:

```python theme={null}
from pyiceberg.table import StaticTable

table = StaticTable.from_metadata("s3://analytics-data/sales/orders/metadata/v3.metadata.json")
print(table.scan().to_arrow())
```

Engines with a path-based Iceberg reader take the prefix itself, for example `iceberg_scan('s3://analytics-data/sales/orders')` in SQL, or `spark.read.format("iceberg").load("s3a://analytics-data/sales/orders")`. Point the engine at the same bucket credentials the pipeline uses; for the built-in MinIO that is the MinIO endpoint with path-style access.

Datris itself reads the table through the same query surface as the other formats: the Search tab, the Assistant, the object-store query endpoint, and the `query_objectstore` MCP tool. For an Iceberg table the result also carries `snapshotId` (the snapshot the rows were read from, returned as a string because the id is a 64-bit number) and `snapshotTimestamp` (ISO-8601). Both are null for `parquet` and `orc`. Querying a pipeline that has never run returns zero rows rather than an error.

## Switching an existing pipeline to Iceberg

A prefix that already holds loose Parquet files or ORC files cannot be converted in place. Changing `fileFormat` to or from `iceberg` on an existing pipeline is refused unless `deleteBeforeWrite` is set, and a write that finds non-Iceberg objects at the prefix stops with a message asking you to set `deleteBeforeWrite` or use a new prefix. Either:

* point the pipeline at a **new `prefixKey`**, so the old files stay where they are; or
* set **`deleteBeforeWrite: true`**, which removes everything under the prefix and creates the table fresh on the next run. It applies to every run while it stays set: clear it after the first Iceberg run, or each run will drop the table again (re-registering with it cleared passes validation because the pipeline is already `iceberg` on both sides).

Existing `parquet` and `orc` pipelines are unaffected; `fileFormat` still defaults to `parquet`.

## Concurrent runs

Two runs of the same pipeline that would write the same output path are serialised: the second waits for the first to commit, then runs on top of it. This applies to every object-store format, not only Iceberg.

## Deleting the pipeline

Deleting the pipeline with its destination data removes the prefix, which drops the table (metadata and data together).

## Not yet available

* **Catalog registration.** The table is path-based. Registering it in a REST, AWS Glue, or JDBC catalog so that catalog-only engines can query it by name is planned; the data written today does not need to be rewritten when that lands.
* **Time travel.** Queries read the current snapshot only; reading a table as of an earlier snapshot or timestamp is planned.
* **Table maintenance.** Expiring old snapshots, compacting small files, and removing orphan files are not yet performed by Datris. Long-lived tables fed by frequent small runs accumulate snapshots and small files; run your engine's maintenance procedures against the path in the meantime.
* **Partition transforms** such as day, month, or bucket partitions; only identity partitioning is supported.
* **Iceberg as a source.** Taps do not read Iceberg tables yet.

## Completion notification

A pipeline notification is published on completion, as for the other object-store formats. See [Notifications](/notifications).


# Publish pipeline output to Kafka
Source: https://docs.datris.ai/destinations/kafka

Write validated pipeline output to a Kafka topic with keyed partitioning, structured or raw payloads, and SASL/PLAIN auth.

Use this destination when downstream consumers read from Kafka and you want the messages validated before they are published.

The Kafka destination writes messages to a Kafka topic. It supports keyed messages for partition control, structured and raw payload formats, and SASL/PLAIN authentication.

## Message Format

The payload format depends on the source schema:

* **Structured data** -- When the schema contains regular typed columns, each row is serialized as a JSON object and sent as the message value.
* **JSON or XML** -- When the schema contains a single `_json` or `_xml` field, the raw string value is sent directly as the message value without additional wrapping.

## Message Key

Set `keyField` to the name of a column whose value should be used as the Kafka message key. Messages with the same key are routed to the same partition, preserving ordering for that key. If `keyField` is omitted, messages are distributed across partitions using the default partitioner.

## Authentication

The destination authenticates to the Kafka cluster using SASL/PLAIN. Credentials are retrieved from Vault using the secret name specified in `kafkaProducerSecretName`. The secret must contain:

| Key | Description |
| - | - |
| `bootstrapServers` | Kafka bootstrap server list |
| `username` | SASL username |
| `password` | SASL password |

## Bootstrap Server Override

To override the bootstrap servers from the Vault secret on a per-pipeline basis, set `overrideBootstrapServers` in the destination configuration.

## Configuration Example

```json theme={null}
{
  "name": "stock_price_kafka",
  "source": { "..." : "..." },
  "destination": {
    "kafka": {
      "topic": "order-events",
      "keyField": "order_id",
      "overrideBootstrapServers": "broker1:9092,broker2:9092",
      "timeoutMs": 10000
    }
  }
}
```

Kafka credentials (`bootstrapServers`, `username`, `password`) are configured globally in `application.yaml` via `secrets.kafkaProducerSecretName`. See [Configuration Reference](/configuration-reference) for Vault secret format.

### Field Reference

| Field | Required | Default | Description |
| - | - | - | - |
| `topic` | yes | | Target Kafka topic |
| `keyField` | no | | Column name to use as the message key |
| `overrideBootstrapServers` | no | | Override bootstrap servers from the Vault secret |
| `timeoutMs` | no | `10000` | Producer send timeout in milliseconds |

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Live Read: run a pipeline and get the rows back without landing them
Source: https://docs.datris.ai/destinations/live-read

Use Live Read (the scratch destination) to validate, transform, or fetch data through a pipeline and read the result back — nothing is landed, results expire.

<Note>
  Agents get live, validated reads from any source, and only land what's worth keeping.
</Note>

Use Live Read (the `scratch` destination) when you (or an agent) want an answer now — what a source returns for these ids, whether today's file is clean, what a transformation produces — and have no reason to keep the rows. The run goes through the same chain as any other pipeline (preprocessor, data quality, transformation), and instead of a loader writing to a table, the rows are held for a short time so the caller can read them back.

Live Read is a second lane beside ingestion, not a shortcut around it. Ingestion lands validated data beside your lake where it is observed, catalogued and queryable; Live Read hands rows back once. Live Read results are **not observed**, **not catalogued**, **not queryable later**, and **expire**. If the rows turn out to be worth keeping, switch the pipeline's destination to a real one — the tap, its schedule and its incremental cursor do not move.

## Keep it, or answer now?

| You want to… | Use |
| - | - |
| Keep the data, observe it over time, query it later, or refresh it on a schedule | A real destination — [PostgreSQL](/destinations/postgres), [MongoDB](/destinations/mongodb), [Object Store](/destinations/object-store), [Snowflake](/destinations/snowflake), [Databricks](/destinations/databricks) |
| See what a source returns for some inputs, check a file against its data quality rules, or preview a transformation — and then move on | **Live Read** |
| Preview what a tap script produces before wiring it to anything | The tap test run (20-row sample; see [Taps](/taps)) |

Never create a table just to read rows back once; never use Live Read for data you will want again.

## Configuration

```json theme={null}
"destination": {
  "scratch": {}
}
```

There is nothing to configure per pipeline beyond its presence. Three rules apply:

* **Structured sources only.** CSV, JSON and XML sources; unstructured (document) sources and vector destinations are not accepted.
* **A sample is required at create time.** Like the object store destination, Live Read needs the typed header so data quality rules and transformations can run — send the header row plus a few representative rows, exactly as for a structured destination.
* **Exclusive.** A Live Read pipeline lands nothing anywhere; it cannot be combined with another destination in the same pipeline.

In the pipeline wizard, choose **Live Read** under Destination. It shows no fields.

## What a run produces

Data quality rules behave as everywhere else: a run either yields the whole transformed set (status `success`, or `warning` when rules flagged rows) or nothing (status `error`). There are no rejected rows to read.

When the run finishes, its entry in the status rollup (`GET /api/v1/pipeline/status?…&withrollup=true`, or the MCP `get_pipeline_status` tool) carries the result:

| Field | Description |
| - | - |
| `resultUri` | Where the result is held. Present only on Live Read runs that produced a result. |
| `resultRowCount` | Total rows in the result. |
| `resultExpiresAt` | When the result is deleted. |
| `resultPreview` | The first rows, inline — up to the configured inline cap. |
| `resultTruncated` | `true` when the result has more rows than the preview holds. |

For small results the preview is the whole answer. Page past it only when `resultTruncated` is `true`.

## Reading the result

### Run detail page

Ops → Ingestion → the run. When the rollup carries a result, the page shows a **View result** panel: the record grid, **Prev** / **Next** paging, and a footer reading "showing N of M, expires at …". Paging reads the stored result; it never re-runs the source.

### REST

```
GET /api/v1/pipeline/result?pipelinetoken={token}&offset={offset}&limit={limit}
```

Pass `pipelinetoken` (one job) or `publishertoken` (the job a tap run submitted). `offset` defaults to `0`; `limit` is clamped to the inline cap, so always advance by the page's `returnedCount`, not by the `limit` you asked for.

```json theme={null}
{
  "records": [ { "id": "1", "city": "c1" }, "…" ],
  "rowCount": 300,
  "returnedCount": 200,
  "offset": 0,
  "truncated": true,
  "resultUri": "…",
  "resultExpiresAt": "2026-09-19T10:00:05Z"
}
```

| Status | Meaning |
| - | - |
| `404` | The pipeline has no result — only Live Read pipelines return rows (or no run matched the token). |
| `410` | The result has expired. Run the pipeline again. |

See the [Pipeline Status API](/api-reference/status-api#get-a-live-read-pipelines-result) for the full reference.

### CLI

```bash theme={null}
# First page, with "showing N of M, expires at …"
datris pipeline result <pipeline-token>

# Page explicitly
datris pipeline result <pipeline-token> --offset 200

# Everything, one JSON object per line, paging until the result is exhausted
datris pipeline result <pipeline-token> --out rows.jsonl
```

### MCP

Agents read the first rows straight off the `get_pipeline_status` rollup (`resultPreview`) and call `get_pipeline_result` only when `resultTruncated` is `true`. See [MCP Server](/mcp-server#taps).

## Retention and sensitive data

A Live Read result holds whatever the source returned, after transformation, in the platform's built-in object store alongside staged documents. It is deleted by an hourly sweep once its retention window passes; reads after that return `410`. Retention is the mitigation: keep it short, especially on shared hosts, and shorten it rather than lengthen it when sources carry personal or otherwise sensitive data. `0` expires results immediately.

The inline preview cap and the retention window are deployment settings — see `SCRATCH_INLINE_ROWS` and `SCRATCH_RETENTION_HOURS` in the [Configuration Reference](/configuration-reference).

<Note>
  A Live Read pipeline still records its run history and data quality outcomes like any other pipeline. The rows are transient, and no dataset is registered in the catalog or the lineage graph.
</Note>


# Embed documents into Milvus for RAG
Source: https://docs.datris.ai/destinations/milvus

Chunk, embed, and upsert documents into a Milvus collection for retrieval-augmented generation and semantic search.

Use this destination when you want a Milvus index built and refreshed by a repeatable pipeline, with chunking and embedding handled for you.

Ingest unstructured documents into [Milvus](https://milvus.io), a high-performance open-source vector database built for scalable similarity search. The pipeline extracts text from the document, chunks it using a configurable strategy, generates vector embeddings, and upserts the chunks with metadata into a Milvus collection.

## Configuration

```json theme={null}
"source": {
    "fileAttributes": {
        "unstructuredAttributes": {
            "fileExtension": "pdf",
            "preserveFilename": true
        }
    }
},
"destination": {
    "milvus": {
        "collectionName": "financial_documents",
        "chunking": {
            "strategy": "recursive",
            "chunkSize": 500,
            "chunkOverlap": 50
        },
        "metadata": {
            "company": "Apple Inc",
            "documentType": "10-Q",
            "filingDate": "2026-01-30"
        },
        "milvusSecretName": "oss/milvus"
    }
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `collectionName` | string | (required) | Milvus collection name. Auto-created if it doesn't exist. |
| `chunking` | object | recursive, 500/50 | Chunking strategy configuration (see below). |
| `metadata` | map | `{}` | Static key-value metadata stored as dynamic fields on every chunk. |
| `embeddingSecretName` | string | *server default* | Embedding Vault secret name. Only consulted when the server-level `ai.embedding.secretName` is unset; with the default configuration (`oss/embedding`, seeded automatically) the server-level secret is always used and this field is ignored. |
| `milvusSecretName` | string | (required) | Vault secret name for Milvus connection. |

## Supported File Types

| Format | Description |
| - | - |
| PDF (`.pdf`) | Text extracted via Apache PDFBox |
| Word (`.doc`) | Text extracted via Apache POI (legacy format) |
| Word (`.docx`) | Text extracted via Apache POI (modern format) |
| PowerPoint (`.ppt`) | Text extracted via Apache POI (legacy format) |
| PowerPoint (`.pptx`) | Text extracted via Apache POI (modern format) |
| Excel (`.xls`, `.xlsx`) | Cell values extracted via Apache POI |
| HTML (`.html`, `.htm`) | Text extracted via JSoup (tags stripped) |
| RTF (`.rtf`) | Text extracted via javax.swing RTF parser |
| Email (`.msg`) | Subject, from, to, and body extracted via Apache POI |
| Email (`.eml`) | Subject, from, and body extracted via Jakarta Mail |
| EPUB (`.epub`) | XHTML content extracted and parsed via JSoup |
| Plain text (`.txt`, `.md`, `.csv`, `.json`, `.xml`) | Content used directly |

For structured data (CSV, JSON, XML), use database destinations (PostgreSQL, MongoDB) instead.

## Vault Secrets

### Milvus connection

```bash theme={null}
vault kv put secret/oss/milvus \
  host="host.docker.internal" \
  port="19530" \
  apiKey=""
```

| Field | Description |
| - | - |
| `host` | Milvus server hostname. Use `host.docker.internal` for local Milvus. |
| `port` | gRPC port (default 19530). |
| `apiKey` | API key / token for Milvus Cloud. Empty string for local instances. |

### Embedding API

The embedding secret is server-level (`ai.embedding.secretName`, default `oss/embedding`) and is seeded automatically by `docker/vault-init.sh`. See [AI Configuration](/ai-configuration) and the [Qdrant docs](/destinations/qdrant#embedding-api) for the full picture.

```bash theme={null}
vault kv put secret/oss/embedding \
  provider="openai" \
  endpoint="https://api.openai.com/v1/embeddings" \
  model="text-embedding-3-small" \
  apiKey="sk-..."
```

For Anthropic-only deployments, the bundled TEI sidecar serves `bge-m3` and `vault-init.sh` seeds the embedding secret to point at it — no OpenAI key required.

## Chunking Strategies

Documents are split into chunks before embedding. Each chunk becomes a separate entity in the Milvus collection with the document's metadata plus `chunk_index`, `filename`, and `source_pipeline` fields.

```json theme={null}
"chunking": {
    "strategy": "recursive",
    "chunkSize": 500,
    "chunkOverlap": 50
}
```

| Strategy | Description |
| - | - |
| `none` | No chunking — one embedding per document. Only for very short documents. |
| `fixed` | Split by character count. Fast but may cut mid-sentence. |
| `sentence` | Split on sentence boundaries (`.` `!` `?`). Preserves semantic units. |
| `paragraph` | Split on double newlines. Ideal for structured documents with clear sections. |
| `recursive` | Try `\n\n`, then `\n`, then `.`, then space — best general-purpose default. |

* `chunkSize` (default 500): maximum characters per chunk
* `chunkOverlap` (default 50): characters of overlap between consecutive chunks

## Metadata

Static metadata is stored as dynamic fields on every chunk in the Milvus collection:

```json theme={null}
"metadata": {
    "company": "Apple Inc",
    "documentType": "10-Q",
    "filingDate": "2026-01-30"
}
```

This allows filtered vector search queries combining similarity with metadata predicates.

Every entity automatically includes:

* `id` — deterministic UUID (idempotent upserts)
* `text` — the chunk text
* `chunk_index` — position of the chunk in the document
* `filename` — original uploaded filename
* `source_pipeline` — pipeline name
* `embedding` — float vector for similarity search

## How It Works

1. **Upload** — an unstructured file is uploaded via `POST /api/v1/pipeline/upload`
2. **Extract** — text is extracted from the document
3. **Chunk** — text is split into chunks using the configured strategy
4. **Embed** — each chunk is sent to the embedding API to generate a vector
5. **Upsert** — vectors are inserted into the Milvus collection with metadata
6. **Notify** — a pipeline notification is published on completion

The collection is auto-created on first upsert with a cosine distance index on the embedding field. Dynamic fields are enabled for metadata.

## Running Milvus

Milvus runs outside the pipeline's Docker Compose (unlike Qdrant, Weaviate, and Chroma, which ship as opt-in Compose profiles). Unlike simpler vector databases, Milvus requires etcd and MinIO as internal dependencies, so a simple `docker run` won't work.

**Option 1 — Milvus standalone script (recommended):**

```bash theme={null}
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh
bash standalone_embed.sh start
```

To stop: `bash standalone_embed.sh stop`

**Option 2 — Milvus docker-compose:**

```bash theme={null}
wget https://github.com/milvus-io/milvus/releases/download/v2.4.4/milvus-standalone-docker-compose.yml -O milvus-docker-compose.yml
docker-compose -f milvus-docker-compose.yml up -d
```

This starts Milvus with its required etcd and MinIO services.

For more options, see [milvus.io/docs/install\_standalone-docker.md](https://milvus.io/docs/install_standalone-docker.md).

## Verifying

```bash theme={null}
# Check Milvus is running
curl http://localhost:9091/v1/vector/collections

# List collections
curl http://localhost:9091/v1/vector/collections

# Get collection details
curl http://localhost:9091/v1/vector/collections/describe \
  -H "Content-Type: application/json" \
  -d '{"collectionName": "financial_documents"}'

# Query entities
curl -X POST http://localhost:9091/v1/vector/query \
  -H "Content-Type: application/json" \
  -d '{
    "collectionName": "financial_documents",
    "filter": "chunk_index >= 0",
    "outputFields": ["text", "chunk_index", "filename"],
    "limit": 5
  }'
```


# Load JSON documents into MongoDB from an AI agent or pipeline
Source: https://docs.datris.ai/destinations/mongodb

Write validated JSON documents to a MongoDB collection with upserts, transactions, and per-run lineage.

Use this destination when an agent or pipeline produces JSON documents and you want them written to MongoDB with schema validation, audit logging, and a lineage record for every run.

The MongoDB destination writes JSON documents to a MongoDB collection. It supports upserts, transactions, and two JSON input modes.

## Schema Requirement

The source schema must contain a single field named `_json` of type `string`. Each value in this field is a JSON document that will be written to the target collection.

## JSON Modes

The JSON input mode is controlled by `everyRowContainsObject` in the source `fileAttributes.jsonAttributes`:

| Value | Description |
| - | - |
| `true` | Each line of the file is a separate JSON document (newline-delimited JSON). |
| `false` | The file contains a single JSON array; each element becomes a separate document. |

The effective default depends on whether the `jsonAttributes` block is present: when `fileAttributes.jsonAttributes` is absent altogether, the loader assumes `true` (newline-delimited); when the block is present but `everyRowContainsObject` is omitted, it defaults to `false` (single JSON array). Set it explicitly to avoid surprises.

## Upsert with Compound Keys

Specify one or more fields in `keyFields` to perform an upsert instead of a plain insert. The pipeline matches existing documents by the compound key formed from these fields. If a matching document exists it is replaced; otherwise a new document is inserted.

## Truncate Before Write

Set `truncateBeforeWrite` to `true` to delete all documents from the target collection before writing. This effectively replaces the entire collection on each run.

## Transaction Support

By default (`useTransaction: true`), all write operations execute inside a MongoDB transaction. If any error occurs during the write, the transaction is aborted and no partial data is persisted. Set `useTransaction: false` to disable transactional writes.

## Configuration Example

```json theme={null}
{
  "name": "events_mongodb",
  "source": { "..." : "..." },
  "destination": {
    "database": {
      "dbName": "analytics",
      "table": "events",
      "useMongoDB": true,
      "keyFields": ["event_id", "source"],
      "truncateBeforeWrite": false,
      "useTransaction": true
    }
  }
}
```

MongoDB connection credentials are configured globally in `application.yaml` via `secrets.mongoDbSecretName`. See [Configuration Reference](/configuration-reference) for Vault secret format.

### Field Reference

| Field | Required | Default | Description |
| - | - | - | - |
| `dbName` | yes | | Target database name |
| `table` | yes | | Target collection name |
| `useMongoDB` | yes | `false` | Must be `true` to enable MongoDB destination |
| `keyFields` | no | | Array of field names forming the compound upsert key |
| `truncateBeforeWrite` | no | `false` | Delete all documents before writing |
| `useTransaction` | no | `true` | Wrap all writes in a MongoDB transaction |

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Write Parquet, ORC, or Apache Iceberg tables to MinIO object storage
Source: https://docs.datris.ai/destinations/object-store

Write validated pipeline output as Parquet files, ORC files, or an Apache Iceberg table to the built-in MinIO object store for a lakehouse or query engine to pick up.

Use this destination when you want pipeline output landed as columnar files beside your lake rather than in a database. This is the default file destination that ships with Datris.

The object store destination uses an embedded Apache Spark engine to write columnar files to MinIO-compatible object storage. Output is written as loose Parquet files, loose ORC files, or an [Apache Iceberg table](/destinations/iceberg) at an S3A path.

<Note>
  This page covers the **MinIO** (default) provider — the built-in object store that ships with Datris. To write to **AWS S3**, see [S3 Destination](/destinations/s3) — same code path, same file formats, just per-pipeline credentials instead of the global MinIO ones.
</Note>

## Output Path

Files are written to:

```
s3a://{bucket}/{prefixKey}
```

The default bucket is `{environment}-data`. You can override this per pipeline with the `destinationBucketOverride` field.

### Restricting bucket overrides

By default the override is unrestricted: any pipeline can name any bucket on the built-in MinIO, which is reached with the platform's own MinIO credentials. On a shared MinIO that lets one pipeline read, overwrite, or `deleteBeforeWrite` another environment's data.

To restrict it, set `DATRIS_OBJECTSTORE_BUCKET_ALLOWLIST` in `.env` to a comma-separated list of the buckets pipelines may target:

```
DATRIS_OBJECTSTORE_BUCKET_ALLOWLIST=analytics-data,archive-data
```

With the list set, saving a pipeline whose override is not listed is rejected, and a pipeline saved before the list existed is refused when it runs, is queried, or has its data deleted. The default bucket `{environment}-data` is always allowed even when not listed, so you cannot lock yourself out. Buckets on AWS S3 (`provider: s3`) are your own and are not checked. `datris doctor` reports pipelines that use an override without a list, and any stored override that falls outside the list you set.

## File Formats

| Format | Description |
| - | - |
| `parquet` | Apache Parquet columnar format (default) |
| `orc` | Apache ORC columnar format |
| `iceberg` | Apache Iceberg table at the prefix: atomic commits, `merge` upserts on `keyFields`, schema evolution. See [Iceberg](/destinations/iceberg) |

Set the format with the `fileFormat` field. `parquet` and `orc` write loose files under the prefix; `iceberg` writes a table there that other engines read by path. An Iceberg table can also be registered in Databricks Unity Catalog; see [Register Iceberg tables](/destinations/unity-catalog#register-iceberg-tables).

## Write Modes

| Mode | Behavior |
| - | - |
| `append` | Append new files to the output path (default) |
| `overwrite` | Replace all existing files at the output path |
| `ignore` | Do nothing if data already exists at the output path |
| `errorifexists` | Fail the job if data already exists at the output path |
| `merge` | Upsert on `keyFields`. Only valid with `fileFormat: iceberg` |

## Partitioning

Partition output files by one or more columns using the `partitionBy` array. Spark creates a directory structure based on the distinct values of the specified columns.

## Delete Before Write

Set `deleteBeforeWrite` to `true` to remove all existing objects under the output prefix before writing new data. This is useful when you need a clean target path but want finer control than the `overwrite` write mode provides.

## Type Casting

Column types from the source schema are mapped to Spark types before writing:

| Schema Type | Spark Type |
| - | - |
| `tinyint` | `ByteType` |
| `smallint` | `ShortType` |
| `int` | `IntegerType` |
| `bigint` | `LongType` |
| `float` | `FloatType` |
| `double` | `DoubleType` |
| `decimal(p,s)` | `DecimalType(p,s)` (scale defaults to 0 if omitted) |
| `string` | `StringType` |
| `boolean` | `BooleanType` |
| `date` | `DateType` |
| `timestamp` | `TimestampType` |

## Configuration Example

```json theme={null}
{
  "name": "sales_pipeline",
  "source": { "..." : "..." },
  "destination": {
    "objectStore": {
      "fileFormat": "parquet",
      "writeMode": "append",
      "prefixKey": "sales/daily",
      "destinationBucketOverride": "analytics-data",
      "partitionBy": ["region", "year"],
      "deleteBeforeWrite": false
    }
  }
}
```

### Field Reference

| Field | Required | Default | Description |
| - | - | - | - |
| `prefixKey` | yes | | Key prefix under the bucket |
| `fileFormat` | no | `parquet` | `parquet`, `orc`, or `iceberg` |
| `writeMode` | no | `append` | `append`, `overwrite`, `ignore`, `errorifexists`, or `merge` (iceberg only) |
| `keyFields` | with `merge` | | Natural-key columns for `writeMode: merge` on an Iceberg table |
| `destinationBucketOverride` | no | `{environment}-data` | Override the default bucket name |
| `partitionBy` | no | | Array of column names to partition by |
| `deleteBeforeWrite` | no | `false` | Delete existing objects before writing |

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Embed documents into pgvector for RAG
Source: https://docs.datris.ai/destinations/pgvector

Chunk, embed, and upsert PDFs, Word docs, and text into PostgreSQL with the pgvector extension for retrieval-augmented generation.

Use this destination when you want a RAG index inside PostgreSQL you already run, built by a repeatable pipeline instead of a one-off script.

Ingest unstructured documents into PostgreSQL with the [pgvector](https://github.com/pgvector/pgvector) extension for retrieval-augmented generation (RAG) and semantic search. The pipeline extracts text from the document, chunks it using a configurable strategy, generates vector embeddings, and upserts the chunks with metadata into a PostgreSQL table with a `vector` column.

pgvector uses your existing PostgreSQL infrastructure — no separate vector database server required. Standard SQL can be used to combine vector similarity search with traditional filters.

## Configuration

```json theme={null}
"source": {
    "fileAttributes": {
        "unstructuredAttributes": {
            "fileExtension": "pdf",
            "preserveFilename": true
        }
    }
},
"destination": {
    "pgvector": {
        "tableName": "financial_documents",
        "schemaName": "public",
        "chunking": {
            "strategy": "recursive",
            "chunkSize": 500,
            "chunkOverlap": 50
        },
        "metadata": {
            "company": "Apple Inc",
            "document_type": "10-Q",
            "filing_date": "2026-01-30"
        },
        "postgresSecretName": "oss/pgvector"
    }
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `tableName` | string | (required) | PostgreSQL table name. Auto-created with vector column if it doesn't exist. |
| `schemaName` | string | `"public"` | PostgreSQL schema name. Auto-created if it doesn't exist. |
| `chunking` | object | recursive, 500/50 | Chunking strategy configuration (see below). |
| `metadata` | map | `{}` | Static key-value metadata stored as columns on every chunk. Use snake\_case for column names. |
| `embeddingSecretName` | string | *server default* | Embedding Vault secret name. Only consulted when the server-level `ai.embedding.secretName` is unset; with the default configuration (`oss/embedding`, seeded automatically) the server-level secret is always used and this field is ignored. |
| `postgresSecretName` | string | (required) | Vault secret name for PostgreSQL connection. |

## Supported File Types

| Format | Description |
| - | - |
| PDF (`.pdf`) | Text extracted via Apache PDFBox |
| Word (`.doc`) | Text extracted via Apache POI (legacy format) |
| Word (`.docx`) | Text extracted via Apache POI (modern format) |
| PowerPoint (`.ppt`) | Text extracted via Apache POI (legacy format) |
| PowerPoint (`.pptx`) | Text extracted via Apache POI (modern format) |
| Excel (`.xls`, `.xlsx`) | Cell values extracted via Apache POI |
| HTML (`.html`, `.htm`) | Text extracted via JSoup (tags stripped) |
| RTF (`.rtf`) | Text extracted via javax.swing RTF parser |
| Email (`.msg`) | Subject, from, to, and body extracted via Apache POI |
| Email (`.eml`) | Subject, from, and body extracted via Jakarta Mail |
| EPUB (`.epub`) | XHTML content extracted and parsed via JSoup |
| Plain text (`.txt`, `.md`, `.csv`, `.json`, `.xml`) | Content used directly |

For structured data (CSV, JSON, XML), use the standard PostgreSQL database destination instead.

## Vault Secrets

### PostgreSQL connection (for pgvector)

```bash theme={null}
vault kv put secret/oss/pgvector \
  jdbcUrl="jdbc:postgresql://postgres:5432/datris" \
  username="postgres" \
  password="postgres"
```

| Field | Description |
| - | - |
| `jdbcUrl` | JDBC URL for the PostgreSQL database with pgvector extension installed. |
| `username` | PostgreSQL username. |
| `password` | PostgreSQL password. |

This is separate from the pipeline's standard `oss/postgres` secret so the vector store can target a different database or server.

### Embedding API

The embedding secret is server-level (`ai.embedding.secretName`, default `oss/embedding`) and is seeded automatically by `docker/vault-init.sh`. See [AI Configuration](/ai-configuration) and the [Qdrant docs](/destinations/qdrant#embedding-api) for the full picture.

```bash theme={null}
vault kv put secret/oss/embedding \
  provider="openai" \
  endpoint="https://api.openai.com/v1/embeddings" \
  model="text-embedding-3-small" \
  apiKey="sk-..."
```

For Anthropic-only deployments, the bundled TEI sidecar serves `bge-m3` and `vault-init.sh` seeds the embedding secret to point at it — no OpenAI key required.

## Chunking Strategies

Documents are split into chunks before embedding. Each chunk becomes a row in the PostgreSQL table with the document's metadata columns plus `chunk_index`, `filename`, and `source_pipeline`.

```json theme={null}
"chunking": {
    "strategy": "recursive",
    "chunkSize": 500,
    "chunkOverlap": 50
}
```

| Strategy | Description |
| - | - |
| `none` | No chunking — one embedding per document. Only for very short documents. |
| `fixed` | Split by character count. Fast but may cut mid-sentence. |
| `sentence` | Split on sentence boundaries (`.` `!` `?`). Preserves semantic units. |
| `paragraph` | Split on double newlines. Ideal for structured documents with clear sections. |
| `recursive` | Try `\n\n`, then `\n`, then `.`, then space — best general-purpose default. |

* `chunkSize` (default 500): maximum characters per chunk
* `chunkOverlap` (default 50): characters of overlap between consecutive chunks

## Metadata

Static metadata is stored as dedicated columns in the PostgreSQL table:

```json theme={null}
"metadata": {
    "company": "Apple Inc",
    "document_type": "10-Q",
    "filing_date": "2026-01-30"
}
```

Use snake\_case for metadata keys since they become PostgreSQL column names. This enables powerful combined queries:

```sql theme={null}
SELECT text FROM financial_documents
WHERE company = 'Apple Inc' AND document_type = '10-Q'
ORDER BY embedding <=> '[query_vector]'
LIMIT 5;
```

Every row automatically includes:

* `id` — deterministic UUID (idempotent upserts)
* `text` — the chunk text
* `chunk_index` — position of the chunk in the document
* `filename` — original uploaded filename
* `source_pipeline` — pipeline name
* `embedding` — vector column for similarity search

## How It Works

1. **Upload** — an unstructured file is uploaded via `POST /api/v1/pipeline/upload`
2. **Extract** — text is extracted from the document
3. **Chunk** — text is split into chunks using the configured strategy
4. **Embed** — each chunk is sent to the embedding API to generate a vector
5. **Upsert** — chunks are upserted into PostgreSQL with `INSERT ... ON CONFLICT DO UPDATE`
6. **Notify** — a pipeline notification is published on completion

The pgvector extension and table are auto-created on first upsert. Vector dimension is detected from the embedding model.

## Running PostgreSQL with pgvector

The standard PostgreSQL Docker image does not include pgvector. Use the pgvector image:

```bash theme={null}
docker run -p 5433:5432 -e POSTGRES_PASSWORD=postgres pgvector/pgvector:pg16
```

Port 5433 avoids conflict with the pipeline's PostgreSQL instance on 5432.

To add pgvector to an existing PostgreSQL instance:

```sql theme={null}
CREATE EXTENSION IF NOT EXISTS vector;
```

## Verifying

```bash theme={null}
# Check pgvector extension is installed
psql -h localhost -p 5433 -U postgres -d datris -c "SELECT extname FROM pg_extension WHERE extname = 'vector';"

# List chunks
psql -h localhost -p 5433 -U postgres -d datris -c "SELECT id, chunk_index, filename, company FROM public.financial_documents LIMIT 5;"

# Similarity search (replace [...] with your query vector)
psql -h localhost -p 5433 -U postgres -d datris -c "
  SELECT text, 1 - (embedding <=> '[...]') AS similarity
  FROM public.financial_documents
  ORDER BY embedding <=> '[...]'
  LIMIT 5;
"

# Combined filter + similarity search
psql -h localhost -p 5433 -U postgres -d datris -c "
  SELECT text, chunk_index
  FROM public.financial_documents
  WHERE company = 'Apple Inc'
  ORDER BY embedding <=> '[...]'
  LIMIT 5;
"
```

## Advantages Over Dedicated Vector Databases

* **No separate server** — uses your existing PostgreSQL infrastructure
* **Standard SQL** — combine vector search with traditional WHERE clauses, JOINs, aggregations
* **ACID transactions** — full transactional guarantees on vector data
* **Familiar tooling** — use psql, pgAdmin, any PostgreSQL client
* **No new dependencies** — uses the existing PostgreSQL JDBC driver


# Load data into PostgreSQL from an AI agent or pipeline
Source: https://docs.datris.ai/destinations/postgres

Bulk-load CSV, JSON, and API data into PostgreSQL with COPY, schema validation, upserts, and per-run lineage.

Use this destination when an agent, a tap, or a scheduled pipeline needs to land rows in PostgreSQL and you want the write validated against a schema, recorded in the audit log, and traceable to the run that produced it.

The PostgreSQL destination performs bulk inserts using the PostgreSQL `COPY` command. It streams data directly into the target table, making it significantly faster than row-by-row inserts.

## Connection

Connection credentials are retrieved from Vault using the secret name specified in `secrets.postgresSecretName` in `application.yaml`. The secret must contain the following keys:

| Key | Description |
| - | - |
| `jdbcUrl` | JDBC connection string for the database |
| `username` | Database user |
| `password` | Database password |

## Table Management

When `manageTableManually` is set to `false` (the default), the pipeline auto-creates the target table if it does not exist. The table schema is derived from the pipeline's destination schema if one is defined, otherwise it falls back to the source schema. Types are mapped using the table below.

Set `manageTableManually` to `true` if you want to create and manage the table yourself. In this case the table must already exist and its column types must be compatible with the incoming data.

## Type Mapping

When auto-creating tables, source schema types are mapped to PostgreSQL types as follows:

| Source Type | PostgreSQL Type |
| - | - |
| `tinyint` | `int2` |
| `smallint` | `int2` |
| `float` | `float4` |
| `double` | `float8` |
| `string` | `text` |
| `_json` | `json` |
| `_xml` | `xml` |

All other types (`int`, `bigint`, `boolean`, `date`, `timestamp`, `decimal(p,s)`, `varchar(n)`, `char(n)`) are passed through to PostgreSQL as-is.

## Truncate Before Write

Set `truncateBeforeWrite` to `true` to issue a `TRUNCATE` statement on the target table before loading data. This replaces the full contents of the table on each run.

## Transaction Support

All writes are wrapped in a database transaction by default (`useTransaction: true`). If any error occurs during the `COPY` operation, the transaction is rolled back and no partial data is committed. Set `useTransaction` to `false` to disable transactional writes.

## Primary Key and Upsert

Specify one or more columns in `keyFields` to define the natural key for the target table. `keyFields` controls **two** behaviors:

1. **Table shape.** When `manageTableManually` is `false`, the auto-created table includes a `PRIMARY KEY` constraint on these columns.
2. **Load strategy.** When `keyFields` is set and `truncateBeforeWrite` is `false`, the loader switches from raw `COPY` to an upsert path: `COPY` into a session-local staging table, then `INSERT ... SELECT ... ON CONFLICT (keyFields) DO UPDATE SET <non_key_cols> = EXCLUDED.<non_key_cols>` into the target. Rows that match an existing primary-key tuple replace the existing row; rows that don't match are inserted. The same row can therefore be loaded across multiple runs without violating the unique constraint.

**Retrofitting `keyFields` onto an existing table:** if you add `keyFields` to a pipeline that's been loading without it, the underlying table may not have a matching unique constraint yet. The loader auto-creates a unique index on the keyFields columns the next time it runs. If the existing data already contains rows that violate the proposed uniqueness, the index creation fails with a clear remediation message — deduplicate the existing rows, or set `truncateBeforeWrite: true` to wipe and reload from scratch.

**Upsert semantics — on conflict, non-key columns are overwritten in full, including NULLs.** This matches MongoDB's `keyFields` behavior (true upsert, not a non-null merge). Sources that emit partial rows and don't want NULLs to clobber existing values should coalesce upstream before the pipeline.

**When `keyFields` is empty,** the loader uses raw `COPY` directly into the target table. This is the fastest path; duplicate-key violations against any pre-existing unique constraint fail the load. Use this for append-only ingestion where duplicates aren't expected.

**When `truncateBeforeWrite: true`,** the loader uses raw `COPY` regardless of `keyFields` because truncate already prevents duplicates. The upsert path would just be slower with no benefit.

## Schema Auto-Creation

When `manageTableManually` is `false`, the pipeline also creates the target schema if it doesn't exist (`CREATE SCHEMA IF NOT EXISTS`).

## Custom COPY Options

Use the `options` array to pass additional PostgreSQL `COPY` options. Each entry is appended to the `COPY` command. When no options are specified, the default is `FORMAT csv` — Postgres CSV mode treats unquoted empty fields as `NULL` automatically. If your source uses a non-empty placeholder for missing values (e.g. a literal `.`), normalize it at the source layer or set the `NULL` option explicitly via `options`.

## Configuration Example

```json theme={null}
{
  "name": "orders_pipeline",
  "source": { "..." : "..." },
  "destination": {
    "database": {
      "dbName": "analytics",
      "schema": "public",
      "table": "orders",
      "usePostgres": true,
      "manageTableManually": false,
      "truncateBeforeWrite": true,
      "keyFields": ["order_id"],
      "options": ["NULL 'NA'"]
    }
  }
}
```

Connection credentials are configured globally in `application.yaml` via `secrets.postgresSecretName`, not per pipeline. See [Configuration Reference](/configuration-reference) for Vault secret format.

### Field Reference

| Field | Required | Default | Description |
| - | - | - | - |
| `dbName` | yes | | Target database name |
| `schema` | yes | | Target schema. The literal placeholder `SCHEMA_NAME` is replaced with `public` when the pipeline is saved (likewise `DATABASE_NAME` in `dbName` becomes the configured Postgres database); any other value is used as-is |
| `table` | yes | | Target table name |
| `usePostgres` | yes | `false` | Must be `true` to enable PostgreSQL destination |
| `manageTableManually` | no | `false` | If `true`, skip auto-creation of the table |
| `truncateBeforeWrite` | no | `false` | Truncate the table before loading |
| `keyFields` | no | | Array of column names for the natural key. Sets a `PRIMARY KEY` constraint on auto-created tables AND switches the load strategy to upsert (`INSERT ... ON CONFLICT DO UPDATE`) when `truncateBeforeWrite` is `false`. See [Primary Key and Upsert](#primary-key-and-upsert). |
| `useTransaction` | no | `true` | Wrap the COPY in a database transaction |
| `options` | no | | Array of additional COPY option strings |

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Embed documents into Qdrant for RAG
Source: https://docs.datris.ai/destinations/qdrant

Chunk, embed, and upsert PDFs, text, and HTML into a Qdrant collection for retrieval-augmented generation.

Use this destination when you want a Qdrant index built and refreshed by a repeatable pipeline, with chunking and embedding handled for you.

Ingest unstructured documents (PDFs, text files, HTML) into [Qdrant](https://qdrant.tech), a high-performance vector database for RAG and semantic search applications. The pipeline extracts text from the document, chunks it using a configurable strategy, generates vector embeddings, and upserts the chunks with metadata into a Qdrant collection.

## Configuration

```json theme={null}
"source": {
    "fileAttributes": {
        "unstructuredAttributes": {
            "fileExtension": "pdf",
            "preserveFilename": true
        }
    }
},
"destination": {
    "qdrant": {
        "collectionName": "financial_documents",
        "chunking": {
            "strategy": "recursive",
            "chunkSize": 500,
            "chunkOverlap": 50
        },
        "metadata": {
            "company": "Apple Inc",
            "documentType": "10-Q",
            "filingDate": "2026-01-30"
        },
        "qdrantSecretName": "oss/qdrant"
    }
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `collectionName` | string | (required) | Qdrant collection name. Auto-created if it doesn't exist. |
| `chunking` | object | recursive, 500/50 | Chunking strategy configuration (see below). |
| `metadata` | map | `{}` | Static key-value metadata stored on every chunk as Qdrant payload. Used for filtered search. |
| `embeddingSecretName` | string | *server default* | Embedding Vault secret name. Only consulted when the server-level `ai.embedding.secretName` is unset; with the default configuration (`oss/embedding`, seeded automatically) the server-level secret is always used and this field is ignored. |
| `qdrantSecretName` | string | (required) | Vault secret name for Qdrant connection. |

## Supported File Types

| Format | Description |
| - | - |
| PDF (`.pdf`) | Text extracted via Apache PDFBox |
| Word (`.doc`) | Text extracted via Apache POI (legacy format) |
| Word (`.docx`) | Text extracted via Apache POI (modern format) |
| PowerPoint (`.ppt`) | Text extracted via Apache POI (legacy format) |
| PowerPoint (`.pptx`) | Text extracted via Apache POI (modern format) |
| Excel (`.xls`, `.xlsx`) | Cell values extracted via Apache POI |
| HTML (`.html`, `.htm`) | Text extracted via JSoup (tags stripped) |
| RTF (`.rtf`) | Text extracted via javax.swing RTF parser |
| Email (`.msg`) | Subject, from, to, and body extracted via Apache POI |
| Email (`.eml`) | Subject, from, and body extracted via Jakarta Mail |
| EPUB (`.epub`) | XHTML content extracted and parsed via JSoup |
| Plain text (`.txt`, `.md`, `.csv`, `.json`, `.xml`) | Content used directly |

For structured data (CSV, JSON, XML), use database destinations (PostgreSQL, MongoDB) instead.

## Vault Secrets

### Qdrant connection

```bash theme={null}
vault kv put secret/oss/qdrant \
  host="host.docker.internal" \
  port="6334" \
  apiKey=""
```

| Field | Description |
| - | - |
| `host` | Qdrant server hostname. Use `host.docker.internal` for local Qdrant. |
| `port` | gRPC port (default 6334). |
| `apiKey` | API key for Qdrant Cloud. Empty string for local instances. |

### Embedding API

The embedding secret is server-level (`ai.embedding.secretName`, default `oss/embedding`) and is seeded automatically by `docker/vault-init.sh`. Each Vault secret is self-describing — the resolver reads `provider`, `endpoint`, `model`, `apiKey`, and (optionally) `version` from inside. See [AI Configuration](/ai-configuration) for the full picture.

```bash theme={null}
vault kv put secret/oss/embedding \
  provider="openai" \
  endpoint="https://api.openai.com/v1/embeddings" \
  model="text-embedding-3-small" \
  apiKey="sk-..."
```

| Field | Description |
| - | - |
| `provider` | `openai`, `azure`, `tei`, or `ollama`. (Anthropic has no embeddings API.) |
| `endpoint` | Embedding API URL. Must speak the OpenAI embeddings contract. |
| `model` | Embedding model name. |
| `apiKey` | API key. Omit for local Ollama. |

**Common embedding configurations:**

| Provider | Endpoint | Model | Dimensions |
| - | - | - | - |
| OpenAI | `https://api.openai.com/v1/embeddings` | `text-embedding-3-small` | 1536 |
| OpenAI | `https://api.openai.com/v1/embeddings` | `text-embedding-3-large` | 3072 |
| TEI (bundled sidecar) | `http://tei:80/v1/embeddings` | `BAAI/bge-m3` | 1024 |

For Anthropic-only deployments, the bundled TEI sidecar serves `bge-m3` and `vault-init.sh` seeds the embedding secret to point at it — no OpenAI key required.

## Chunking Strategies

Documents are split into chunks before embedding. Each chunk becomes a separate Qdrant point with the document's metadata plus `chunk_index`, `filename`, and `source_pipeline` fields.

```json theme={null}
"chunking": {
    "strategy": "recursive",
    "chunkSize": 500,
    "chunkOverlap": 50
}
```

| Strategy | Description |
| - | - |
| `none` | No chunking — one embedding per document. Only for very short documents. |
| `fixed` | Split by character count. Fast but may cut mid-sentence. |
| `sentence` | Split on sentence boundaries (`.` `!` `?`). Preserves semantic units. |
| `paragraph` | Split on double newlines. Ideal for structured documents with clear sections. |
| `recursive` | Try `\n\n`, then `\n`, then `.`, then space — best general-purpose default. |

* `chunkSize` (default 500): maximum characters per chunk
* `chunkOverlap` (default 50): characters of overlap between consecutive chunks

## Metadata

Static metadata is attached to every chunk in the Qdrant payload, enabling filtered semantic search:

```json theme={null}
"metadata": {
    "company": "Apple Inc",
    "documentType": "10-Q",
    "filingDate": "2026-01-30"
}
```

This allows queries like "find chunks about revenue from Apple 10-Q filings" by combining vector similarity with metadata filters.

In addition to static metadata, every chunk automatically includes:

* `text` — the chunk text
* `chunk_index` — position of the chunk in the document
* `filename` — original uploaded filename
* `source_pipeline` — pipeline name

## How It Works

1. **Upload** — an unstructured file (PDF, text) is uploaded via `POST /api/v1/pipeline/upload`
2. **Extract** — text is extracted from the document (PDFBox for PDFs, UTF-8 for text files)
3. **Chunk** — text is split into chunks using the configured strategy
4. **Embed** — each chunk is sent to the embedding API to generate a vector
5. **Upsert** — vectors are upserted into the Qdrant collection with metadata payload
6. **Notify** — a pipeline notification is published on completion

The collection is auto-created on first upsert using cosine distance and the embedding dimension detected from the model.

## Running Qdrant

Qdrant ships as a bundled **opt-in** Compose service. Enable it with one line in `.env`:

```bash theme={null}
COMPOSE_PROFILES=qdrant
```

Then start the stack:

```bash theme={null}
docker compose up -d
```

Combine profiles with commas (e.g., `COMPOSE_PROFILES=qdrant,kafka`). The `install.sh` installer can also enable Qdrant at install time — it writes the profile and seeds the `oss/qdrant` secret for you. For the bundled container, use the compose service name as the host in the secret (`host="qdrant"`, gRPC port `6334`).

To use an existing external or Qdrant Cloud instance instead, skip the profile and point the `oss/qdrant` secret's `host`/`port`/`apiKey` at it — the secret fields are the same either way. You can also run Qdrant standalone:

```bash theme={null}
docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant
```

Or install natively from [qdrant.tech/documentation/quick-start](https://qdrant.tech/documentation/quick-start/).

## Verifying

```bash theme={null}
# List collections
curl http://localhost:6333/collections

# Get collection info
curl http://localhost:6333/collections/financial_documents

# Scroll points with payload
curl -X POST http://localhost:6333/collections/financial_documents/points/scroll \
  -H "Content-Type: application/json" \
  -d '{"limit": 5, "with_payload": true, "with_vector": false}'

# Search with metadata filter
curl -X POST http://localhost:6333/collections/financial_documents/points/scroll \
  -H "Content-Type: application/json" \
  -d '{"limit": 5, "with_payload": true, "filter": {"must": [{"key": "company", "match": {"value": "Apple Inc"}}]}}'
```


# POST pipeline output to a REST endpoint
Source: https://docs.datris.ai/destinations/rest-endpoint

Send validated pipeline output to any HTTP endpoint with bearer auth, sync or async delivery, and configurable timeouts.

Use this destination when the system you need to feed has no native connector but does have an HTTP API.

The REST endpoint destination POSTs data to an HTTP endpoint. It supports synchronous and asynchronous modes, bearer token authentication, and configurable timeouts.

## Request

The pipeline sends an HTTP POST request with the following JSON payload:

```json theme={null}
{
  "pipelineToken": "abc-123",
  "pipelineName": "orders",
  "data": {
    "size": 100,
    "header": ["order_id", "amount", "region"],
    "rows": [
      ["1001", "59.99", "us-east"],
      ["1002", "24.50", "eu-west"]
    ],
    "rawData": null
  }
}
```

| Payload Field | Description |
| - | - |
| `pipelineToken` | Token identifying the pipeline run |
| `pipelineName` | Name of the pipeline being sent |
| `data.size` | Number of rows in this batch |
| `data.header` | Array of column names |
| `data.rows` | Array of row arrays, each matching the header order |
| `data.rawData` | Raw string content when sending JSON or XML data instead of rows |

## Expected Response

The endpoint must return a JSON response with the following structure:

```json theme={null}
{
  "data": {
    "size": 100,
    "header": ["order_id", "status"],
    "rows": [
      ["1001", "accepted"],
      ["1002", "accepted"]
    ],
    "rawData": null
  }
}
```

The response data is forwarded to downstream steps in the pipeline.

## Batching

By default (`batchSize` = `0`) the pipeline makes exactly one call carrying every row. Set `batchSize` to a positive number to send row-based (CSV) payloads in batches of that many rows, one call per batch, in source order. Each batched request carries the same envelope plus two extra top-level fields:

```json theme={null}
{
  "pipelineToken": "abc-123",
  "pipelineName": "orders",
  "batch": 1,
  "ofBatches": 3,
  "data": {
    "size": 6983,
    "header": ["order_id", "amount", "region"],
    "rows": ["1001,59.99,us-east", "1002,24.50,eu-west"]
  }
}
```

| Field | Description |
| - | - |
| `batch` | 1-based index of this batch |
| `ofBatches` | Total number of batches in the run (`ceil(rows / batchSize)`) |

Batched requests are only checked for an `{"error": ...}` response; the response data is not forwarded anywhere. JSON and XML payloads (`rawData`) are always sent in a single call regardless of `batchSize`. With `async` set, each batch waits for its own callback before the next batch is sent.

## Sync and Async Modes

Set `async` to control whether the pipeline waits for a response:

| `async` | Behavior |
| - | - |
| `false` | The pipeline waits for the endpoint to return a response before proceeding (default). |
| `true` | The pipeline sends the request, then waits for the endpoint to call back `POST /api/v1/restendpoint/callback` with the same `pipelineToken` and the result payload. If no callback arrives within `timeoutMs`, the job fails. |

## Authentication

Set `bearerToken` to include an `Authorization: Bearer {token}` header on every request. If omitted, requests are sent without authentication.

## Timeout

The `timeoutMs` field controls the maximum time in milliseconds the pipeline will wait — for the HTTP response when `async` is `false`, or for the callback when `async` is `true`. The default is `300000` ms (5 minutes).

## Configuration Example

```json theme={null}
{
  "name": "orders_rest",
  "source": { "..." : "..." },
  "destination": {
    "restEndpoint": {
      "endpoint": "https://api.example.com/ingest",
      "async": false,
      "bearerToken": "eyJhbGciOiJIUzI1NiIs...",
      "timeoutMs": 300000,
      "batchSize": 0
    }
  }
}
```

### Field Reference

| Field | Required | Default | Description |
| - | - | - | - |
| `endpoint` | yes | | Target endpoint URL |
| `async` | no | `false` | If `true`, send the request and wait for a callback to `/api/v1/restendpoint/callback` instead of the HTTP response |
| `bearerToken` | no | | Bearer token added as `Authorization: Bearer` header |
| `apiKey` | no | | API key sent as an `x-api-key` header |
| `timeoutMs` | no | `300000` | Response (sync) or callback (async) timeout in milliseconds |
| `batchSize` | no | `0` | Rows per call for row-based payloads. `0` = one call with every row (the default behaviour); `> 0` adds `batch` / `ofBatches` to each request |

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Write Parquet, ORC, or Apache Iceberg tables to AWS S3 from a pipeline
Source: https://docs.datris.ai/destinations/s3

Write validated pipeline output as Parquet files, ORC files, or an Apache Iceberg table to AWS S3 with per-bucket credentials and region.

Use this destination when your lake lives in S3 and you want validated, lineage-tracked files landed there by an agent or a scheduled pipeline, with credentials scoped per bucket.

The S3 destination writes Parquet files, ORC files, or an [Apache Iceberg table](/destinations/iceberg) directly to AWS S3. It uses the same Spark + S3A code path as the [MinIO destination](/destinations/object-store); the only differences are the credentials, the endpoint, and a few S3A settings applied per-bucket so multiple S3 destinations can coexist with the built-in MinIO output.

## How it differs from MinIO

| Setting | MinIO (default) | S3 |
| - | - | - |
| `provider` | `minio` (or omitted) | `s3` |
| Credentials | Global `MinIOConfig` | Per-pipeline `credentialsSecret` |
| Region | n/a (endpoint is explicit) | Stored in the credentials secret |
| `path.style.access` | true | false |
| `connection.ssl.enabled` | false | true |
| Default bucket | `{environment}-data` | None — `destinationBucketOverride` required |

S3A's per-bucket key syntax (`fs.s3a.bucket.<bucket>.*`) is what allows a single Spark session to write to both MinIO and AWS S3 in the same deployment without one set of credentials clobbering the other.

## Credentials secret

Region lives in the credentials secret next to `accessKey` and `secretKey`. Bundling region with the key it pairs with keeps the credential self-contained and prevents the silent-failure mode where the pipeline config claims `us-east-1` but the key is scoped to `us-west-2`.

Create a Vault secret (Secrets tab in the Datris UI) with these fields:

| Field | Required | Notes |
| - | - | - |
| `accessKey` | yes | AWS access key ID |
| `secretKey` | yes | AWS secret access key |
| `region` | yes | AWS region for the bucket (e.g. `us-east-1`) |
| `sessionToken` | no | Only when using temporary credentials (STS) |

If any of `accessKey`, `secretKey`, or `region` is missing, the write fails at resolve time with an explicit "missing required field" error naming the secret.

### Rotating credentials

To rotate or correct the keys, update the secret in the Secrets tab. Fields you leave masked keep their current value, so you can change `region` or add `sessionToken` without re-entering `accessKey` and `secretKey`. The next run of any pipeline that uses that secret picks up the new keys; no restart is needed. A query against the same bucket that is already running at the moment the keys change can fail with "Filesystem closed" — re-run it.

## Configuration

```json theme={null}
{
  "destination": {
    "objectStore": {
      "provider": "s3",
      "destinationBucketOverride": "acme-analytics-prod",
      "credentialsSecret": "acme-analytics-s3",
      "prefixKey": "events/orders",
      "partitionBy": ["dt"],
      "fileFormat": "parquet",
      "writeMode": "append"
    }
  }
}
```

### Field reference

| Field | Required | Default | Description |
| - | - | - | - |
| `provider` | yes | `minio` | Set to `s3` to select the AWS S3 path |
| `destinationBucketOverride` | yes | | Bucket to write to. No default for S3 |
| `credentialsSecret` | yes\* | | Name of the Platform secret holding `accessKey`/`secretKey`/`region`. Pass the bare name as shown on Configuration → Secrets → Platform; the `{environment}/` Vault prefix is added automatically |
| `endpoint` | no | AWS regional | Override the S3 endpoint URL. Must use `https://` |
| `prefixKey` | yes | | Key prefix under the bucket |
| `fileFormat` | no | `parquet` | `parquet`, `orc`, or `iceberg` |
| `writeMode` | no | `append` | `append`, `overwrite`, `ignore`, `errorifexists`, or `merge` (iceberg only) |
| `keyFields` | with `merge` | | Natural-key columns for `writeMode: merge` on an Iceberg table |
| `partitionBy` | no | | Array of column names to partition by |
| `deleteBeforeWrite` | no | `false` | Delete existing objects under the prefix before writing |

\* The `credentialsSecret` field is optional only when Datris itself runs in AWS with an instance role attached; the AWS default credentials provider chain will pick up credentials and region from instance metadata. For self-hosted Datris (not running on EC2 / EKS / ECS), `credentialsSecret` is effectively required.

## File formats, write modes, partitioning, type casting

These behave the same as the MinIO destination — see the [Object Store Destination](/destinations/object-store) page for the full reference, and the [Iceberg](/destinations/iceberg) page for table commits, `merge`, and schema evolution.

To make an Iceberg table written to S3 findable by name in Databricks, register it in Unity Catalog; see [Register Iceberg tables](/destinations/unity-catalog#register-iceberg-tables).

## Multiple S3 destinations

Per-bucket credential isolation means two pipelines can write to different S3 buckets using different IAM keys in the same deployment. Each pipeline references its own `credentialsSecret`; the credentials are applied to that specific bucket only.


# Load data into Snowflake from an AI agent or pipeline
Source: https://docs.datris.ai/destinations/snowflake

Bulk-load into Snowflake with PUT and COPY INTO, MERGE upserts, key-pair auth, and per-pipeline credentials.

Use this destination when you want an agent or a scheduled pipeline to land validated data in Snowflake without handing it warehouse credentials directly. Each pipeline references its own secret, and every load is audited and lineage-tracked.

The Snowflake destination bulk-loads data using Snowflake's native `PUT` → `COPY INTO` path: the pipeline output is staged onto the target table's internal stage and copied in, with gzip compression handled automatically on the wire. When `keyFields` is set, the loader upserts via `MERGE` instead. No Snowflake stages need to be created — the loader uses the table stage (`@%table`) that every Snowflake table has implicitly.

Unlike the bundled PostgreSQL and MongoDB destinations, Snowflake is an external target: credentials are per-pipeline, referenced by name through `credentialsSecret` — the same model as the [S3 destination](/destinations/s3). Different pipelines can write to different Snowflake accounts.

## One-time Snowflake setup

This is the recommended long-term setup for connecting Datris to a company Snowflake account: a dedicated **service user** with **key-pair authentication**, a least-privilege role, and an isolated warehouse. Key-pair auth is Snowflake's recommended method for programmatic access — it never expires, needs no network policy, and supports zero-downtime rotation.

### 1. Generate a key pair

On any machine you trust (the keys are yours, not Snowflake's):

```bash theme={null}
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
```

To encrypt the private key at rest, drop `-nocrypt` and remember the passphrase — you'll store it in the secret as `privateKeyPassphrase`.

### 2. Create the database, role, warehouse, and service user

In Snowsight (app.snowflake.com), open **Projects → Workspaces** and create a SQL file, paste the script below, replace the public-key placeholder, then **select all and Run All** — the run button alone executes only the statement under the cursor.

The script is self-contained: it creates everything Datris needs, including a dedicated `DATRIS` database. Every statement is idempotent (`IF NOT EXISTS` / `GRANT`), so re-running it is safe. Rename `DATRIS` throughout if you'd rather load an existing database.

```sql theme={null}
-- Everything below needs account-level privileges
USE ROLE ACCOUNTADMIN;

-- Least-privilege role for Datris loads, attached to the admin hierarchy
-- so tables Datris creates stay manageable by your admins
CREATE ROLE IF NOT EXISTS DATRIS_LOADER;
GRANT ROLE DATRIS_LOADER TO ROLE SYSADMIN;

-- Dedicated database for Datris pipeline output
CREATE DATABASE IF NOT EXISTS DATRIS;

-- Dedicated warehouse so Datris load cost is isolated and visible
CREATE WAREHOUSE IF NOT EXISTS DATRIS_WH
  WAREHOUSE_SIZE = XSMALL
  AUTO_SUSPEND = 60
  AUTO_RESUME = TRUE
  INITIALLY_SUSPENDED = TRUE;

GRANT USAGE ON WAREHOUSE DATRIS_WH TO ROLE DATRIS_LOADER;

-- Database/schema privileges
GRANT USAGE ON DATABASE DATRIS TO ROLE DATRIS_LOADER;
GRANT USAGE, CREATE TABLE ON SCHEMA DATRIS.PUBLIC TO ROLE DATRIS_LOADER;
-- Lets Datris auto-create additional schemas beyond PUBLIC:
GRANT CREATE SCHEMA ON DATABASE DATRIS TO ROLE DATRIS_LOADER;

-- Service user: no password, no MFA prompts — key-pair only
CREATE USER IF NOT EXISTS DATRIS_SVC
  TYPE = SERVICE
  DEFAULT_ROLE = DATRIS_LOADER
  DEFAULT_WAREHOUSE = DATRIS_WH
  RSA_PUBLIC_KEY = '<contents of rsa_key.pub without the BEGIN/END lines>';

GRANT ROLE DATRIS_LOADER TO USER DATRIS_SVC;

-- Verify: should show a fingerprint in RSA_PUBLIC_KEY_FP
DESC USER DATRIS_SVC;
```

The final `DESC USER` is the success check — a value in the `RSA_PUBLIC_KEY_FP` row confirms the key registered.

Tables auto-created by Datris are owned by `DATRIS_LOADER`, so no further table grants are needed. If you manage tables yourself (`manageTableManually: true`), also grant `SELECT, INSERT, UPDATE, DELETE` on those tables to `DATRIS_LOADER` — `MERGE` upserts and truncation need all four.

`TYPE = SERVICE` marks the user as a machine identity: password login is disabled and MFA does not apply, which is exactly what you want for an unattended integration.

### 3. Create the platform secret in Datris

On **Configuration → Secrets → Platform**, create a secret (for example `snowflake`) with:

| Field | Required | Notes |
| - | - | - |
| `account` | yes | Account identifier, e.g. `myorg-myaccount` or `xy12345.us-east-1`. Pasting the full account URL (`https://myorg-myaccount.snowflakecomputing.com`) also works — Datris strips the protocol and domain |
| `user` | yes | The service user, e.g. `DATRIS_SVC` |
| `privateKey` | yes | Full contents of `rsa_key.p8`, including the BEGIN/END lines |
| `privateKeyPassphrase` | no | Only if the private key is encrypted |
| `password` | no | Fallback when no `privateKey` is present — see below |

The `account` lives in the secret rather than the pipeline config so it stays bound to the credential that authorizes it. If a required field is missing, the load fails at resolve time with an explicit error naming the secret and the field.

Paste the key file's entire contents into `privateKey` — don't worry about formatting. The secret form is a single-line field that strips line breaks from the paste; Datris reconstructs the strict PEM framing the Snowflake driver requires, so a flattened key (or even just the base64 body without the BEGIN/END lines) works. What it cannot accept is a PKCS#1 key (`BEGIN RSA PRIVATE KEY` — convert with `openssl pkcs8 -topk8 -nocrypt`) or an encrypted key without `privateKeyPassphrase`; both fail with an error saying exactly that.

### Key rotation

Snowflake users hold two public-key slots, so rotation has no downtime: generate a new pair, register it with `ALTER USER DATRIS_SVC SET RSA_PUBLIC_KEY_2 = '<new public key>'`, update the Datris secret's `privateKey` to the new private key, then clear the old slot with `ALTER USER DATRIS_SVC UNSET RSA_PUBLIC_KEY`.

### Alternative: programmatic access token (PAT)

For a quick evaluation you can skip the key pair: generate a PAT in Snowsight (**Settings → Authentication → Programmatic access tokens**) and store it in the secret's `password` field instead of `privateKey`. Be aware of the trade-offs before relying on one long-term: PATs expire (one year at most), and Snowflake refuses PAT authentication unless the account or user has an active **network policy** allowing the caller's IP. Key-pair auth has neither constraint.

## Configuration

```json theme={null}
{
  "name": "orders_pipeline",
  "source": { "..." : "..." },
  "destination": {
    "database": {
      "useSnowflake": true,
      "credentialsSecret": "snowflake",
      "warehouse": "DATRIS_WH",
      "role": "DATRIS_LOADER",
      "dbName": "DATRIS",
      "schema": "PUBLIC",
      "table": "ORDERS",
      "keyFields": ["ORDER_ID"]
    }
  }
}
```

### Field reference

| Field | Required | Default | Description |
| - | - | - | - |
| `useSnowflake` | yes | `false` | Must be `true` to enable the Snowflake destination |
| `credentialsSecret` | yes | | Name of the Platform secret holding `account`/`user`/auth |
| `warehouse` | yes | | Virtual warehouse that runs the `COPY`/`MERGE` |
| `role` | no | | Role to assume; omit to use the user's default role |
| `dbName` | yes | | Target database |
| `schema` | yes | | Target schema |
| `table` | yes | | Target table |
| `manageTableManually` | no | `false` | If `true`, skip auto-creation of the table |
| `truncateBeforeWrite` | no | `false` | Empty the table before loading (see semantics below) |
| `keyFields` | no | | Natural-key columns; switches the load to a `MERGE` upsert |
| `useTransaction` | no | `true` | Wrap the load in a transaction |

`warehouse` and `role` are per-pipeline on purpose: a light feed can run on an extra-small warehouse while a wide table gets a larger one, without touching the credential.

Identifiers follow Snowflake's normal resolution rules: simple names (letters, digits, underscores) are passed unquoted, so `datris` and `DATRIS` both resolve to the uppercase `DATRIS` database — the same behavior as every other Snowflake tool. Names containing other characters (hyphens, spaces) are quoted and matched case-sensitively.

## Table Management

When `manageTableManually` is `false` (the default), the pipeline auto-creates the schema and table if they don't exist, with a `PRIMARY KEY` constraint on `keyFields` when set. Schema changes are additive: new columns in the pipeline schema are added with `ALTER TABLE ... ADD COLUMN`; columns are never dropped or retyped.

## Type Mapping

| Source Type | Snowflake Type |
| - | - |
| `string` | `VARCHAR` |
| `int` / `integer` / `tinyint` / `smallint` / `bigint` | `NUMBER(38,0)` |
| `float` / `double` | `FLOAT` |
| `boolean` | `BOOLEAN` |
| `date` | `DATE` |
| `timestamp` | `TIMESTAMP_NTZ` |
| `_json` | `VARIANT` |
| `_xml` | `VARCHAR` |

Other types (`decimal(p,s)`, `varchar(n)`, `char(n)`) pass through as-is. Semi-structured `_json` columns land as `VARIANT` (parsed with `PARSE_JSON` during the `COPY`), so JSON payloads are natively queryable in Snowflake rather than stored as text.

## Truncate Before Write

Set `truncateBeforeWrite` to `true` to replace the full table contents on each run. The mechanism depends on `useTransaction`:

* **`useTransaction: true` (default):** rows are removed with `DELETE FROM` inside the load transaction. If the load fails, the delete rolls back and the previous rows survive — a failed run never leaves the table empty.
* **`useTransaction: false`:** rows are removed with `TRUNCATE TABLE`, which is faster on very large tables but commits immediately (Snowflake DDL auto-commits) — a failure after the truncate leaves the table empty until the next successful run.

## Primary Key and Upsert

Specify one or more columns in `keyFields` to define the natural key. When set (and `truncateBeforeWrite` is `false`), the loader stages the batch into a session-temporary table and issues a `MERGE` on the key columns: matched rows are updated, unmatched rows are inserted. The same rows can be loaded across runs without duplicates.

Upsert semantics match the PostgreSQL destination: on match, non-key columns are overwritten in full, including `NULL`s. If a single batch contains duplicate key values, Snowflake rejects the `MERGE` as non-deterministic — deduplicate upstream.

When `keyFields` is empty, the loader uses a straight `COPY INTO` — the fastest path, for append-only ingestion.

## Querying from Datris

The Assistant (and any MCP client) can read the destination back with the `query_snowflake` tool — useful for verifying a load landed and for answering questions over the data without leaving Datris. Queries are pipeline-scoped: the tool takes a pipeline name, and the server connects with that pipeline's credentials secret, warehouse, and role. Credentials never leave the server, and only read-only statements are accepted — `SELECT` (with `LIMIT` applied automatically), plus `SHOW` and `DESCRIBE` for discovering databases, schemas, tables, and columns.

Queries run on the pipeline's configured warehouse and consume Snowflake compute like any other query. Results are capped at 100 rows by default.

## Warehouse Behavior

The loader assumes `AUTO_RESUME = TRUE` on the warehouse (Snowflake's default): a suspended warehouse resumes itself when the `COPY`/`MERGE` arrives. Datris never issues `ALTER WAREHOUSE ... RESUME`, so the loader role does not need the `OPERATE` privilege. If your warehouse is deliberately configured with `AUTO_RESUME = FALSE`, resume it before the pipeline runs.

## Completion Notification

A pipeline notification is published to ActiveMQ on completion. See [Notifications](/notifications) for details.


# Discover and annotate Databricks Unity Catalog tables from an AI agent
Source: https://docs.datris.ai/destinations/unity-catalog

Browse the catalogs, schemas, tables and columns a Databricks secret can see, search warehouse tables from find_data, mark the tables Datris loads with comments and tags, and register Iceberg tables in Unity Catalog.

Use this page when an agent works against a Databricks workspace. Datris does two things with Unity Catalog: it **annotates** the tables its [Databricks](/destinations/databricks) pipelines load (comments, tags and table properties), and it lets an agent **discover** what a Databricks secret can already see before it creates a pipeline. It also publishes **lineage**, so Catalog Explorer can trace a table back to the tap or upload that fed it, and it can **register** the Iceberg tables that [object store](/destinations/object-store) pipelines write, so warehouse users find them by name instead of by S3 path. All of this is read-only toward your data; annotations, lineage and table registrations are the only things Datris writes to Unity Catalog.

## Two meanings of "catalog"

Datris and Unity Catalog both use the word *catalog*, and they mean different things:

* A **Datris catalog** is a label on taps and pipelines that groups them in the Datris UI (see [Data Catalog](/data-catalog)). It lives only in Datris and never changes where data lands.
* A **Unity Catalog catalog** is the top level of Databricks' three-level namespace, `catalog.schema.table`. On a Databricks pipeline, `destination.database.dbName` is the Unity Catalog catalog the table lands in.

The two are independent. A pipeline in the Datris catalog `finance` can load into the Unity Catalog catalog `main`. When the metadata push is on, the Datris catalog travels with the table as the `datris_catalog` tag, so you can see it from Databricks too.

## What Datris writes

With `"unityCatalog": { "enabled": true }` on a Databricks pipeline, every successful load annotates the table: a table comment naming the pipeline and source, comments on the `_datris_*` provenance columns, the tags `datris_pipeline`, `datris_catalog`, `datris_dq_status` and `managed_by = datris`, and `datris.*` table properties (Iceberg tables also carry `write.metadata.previous-versions-max=1000`, and an adopted table `datris.adopted-from`). See [Unity Catalog metadata](/destinations/databricks#unity-catalog-metadata) for the fields, the grant it needs, and `GET /api/v1/pipelines/{name}/unity-catalog`. To turn it on for every Databricks pipeline without a block, set `DATRIS_UNITY_CATALOG_DEFAULT=enabled` on the server; a pipeline opts out with `"unityCatalog": { "enabled": false }`.

The `datris_pipeline` tag is what discovery uses to recognize a table Datris loaded.

## Lineage in Unity Catalog

With the metadata push on, every successful load also publishes the pipeline's lineage to Unity Catalog, so the table's **External lineage** graph in Catalog Explorer shows where its data came from without opening Datris. Datris registers three external metadata objects and two lineage relationships:

* **The source:** `datris-tap-<tap>` for a tap-fed pipeline (with the tap description, its declared source, the script identity and, for HTTP taps, the endpoint URL), or `datris-upload-<pipeline>` for a pipeline fed by file uploads.
* **The pipeline:** `datris-pipeline-<pipeline>`, the transform node, with the Datris catalog, the pipeline definition version and the Datris lineage API path as properties.
* **The table:** the Delta table the pipeline loads, `catalog.schema.table`.
* **Source to pipeline, and pipeline to table:** the pipeline-to-table relationship carries the column mappings from Datris [column-level lineage](/provenance#column-level-lineage), one mapping per source column. Mappings Datris inferred with AI (rather than derived from the pipeline definition) are included, and they mark the whole relationship with the properties `datris.confidence = inferred` and `datris.evidence`. Unity Catalog keeps properties per relationship, not per column, so the flag applies to the relationship as a whole. Publishing never runs AI inference; only mappings already computed are sent.
* **Run properties:** both relationships carry `datris.lastRunId`, `datris.recordCount`, `datris.status` (`pass` or `warn`) and `datris.lastRunAt`, overwritten on every run. Unity Catalog shows the latest run, not a history.

Object names replace dots, slashes, spaces and backticks with `_`. The objects and column mappings are only re-sent when the pipeline definition, its column lineage or the tap script changes, or when Datris recreated the table; otherwise a run updates only the run properties.

Lineage uses the workspace REST API with the same Databricks secret as the load, and needs privileges beyond the load grants:

* `CREATE EXTERNAL METADATA` on the metastore, which only a metastore admin can grant: `` GRANT CREATE EXTERNAL METADATA ON METASTORE TO `<sp-application-id>` ``.
* `MODIFY` on the external metadata objects Datris creates (the creator owns them, so this holds unless ownership changes).
* `MODIFY` on the table, for the pipeline-to-table relationship. The setup grants on the schema already cover it.

A missing privilege, a wrong host or a workspace without the external lineage API is recorded as a warning on the run and in the audit log, never a failed load, and the next run retries. The pipeline detail page shows when lineage was last published. To keep the comments, tags and properties but skip lineage, set `"unityCatalog": { "enabled": true, "lineage": false }`. `DATRIS_UNITY_CATALOG_SYNC=false` switches lineage off along with the rest of the push. Pipelines turned on by `DATRIS_UNITY_CATALOG_DEFAULT=enabled` publish lineage too.

## Register Iceberg tables

An [object store](/destinations/object-store) or [S3](/destinations/s3) pipeline with `fileFormat: "iceberg"` can register its table in Unity Catalog. After each successful write, Datris points Unity Catalog at the table's current Iceberg metadata file, so Databricks SQL and any Iceberg REST client can find the table by name.

Three modes decide how the table gets into the catalog: `catalogMode: "register"` (the default) points the catalog at the table Datris writes under the pipeline's prefix, `"rest"` commits every write through the catalog at that prefix, and `"managed"` lets the catalog choose where the table lives and commits every write through it. `register` and `rest` need a catalog that implements the Iceberg REST `register` call and honours the location Datris asks for, such as the Apache Iceberg REST test fixture. Databricks Unity Catalog does neither, so **on Databricks use `catalogMode: "managed"`** (see [Databricks: catalogMode managed](#databricks-catalogmode-managed)).

```json theme={null}
{
  "destination": {
    "objectStore": { "provider": "s3", "destinationBucketOverride": "my-lake", "prefixKey": "orders", "fileFormat": "iceberg", "credentialsSecret": "aws_lake" }
  },
  "unityCatalog": {
    "enabled": true,
    "credentialsSecret": "databricks_prod",
    "catalog": "analytics",
    "schema": "default",
    "catalogMode": "rest"
  }
}
```

* `credentialsSecret` names a Databricks Platform secret with the same fields the [Databricks](/destinations/databricks) destination uses: `host`, plus `clientId`/`clientSecret` or `token`. It is separate from the object store's own `credentialsSecret`.
* `catalog` is the Unity Catalog catalog. It is required and has no default.
* `schema` is the Unity Catalog schema. It defaults to `default`.
* The table is named after the pipeline: `<catalog>.<schema>.<pipeline name>`, lowercased, with dots, slashes, spaces and backticks replaced by `_`.
* `"register": false` keeps the block but skips registration.

The API returns `400` when `credentialsSecret` or `catalog` is missing, or when the object store writes `parquet` or `orc` (only Iceberg tables can be registered).

### Databricks prerequisites

These are the grants `catalogMode: "managed"` (see [Databricks: catalogMode managed](#databricks-catalogmode-managed)) and the [Databricks destination](/destinations/databricks) need.

Registration uses the Unity Catalog Iceberg REST catalog, which needs three things set up once:

1. **External data access** enabled on the metastore. A metastore admin turns it on in Catalog Explorer, under the metastore's settings.
2. **`EXTERNAL USE SCHEMA`** on the target schema for the secret's identity. Only the catalog owner can grant it: `` GRANT EXTERNAL USE SCHEMA ON SCHEMA analytics.default TO `<sp-application-id>` ``.
3. An **external location** that covers the table's S3 path (the bucket and prefix above), with a storage credential that can read it.

<Warning>
  **The target schema must be on Unity Catalog managed cloud storage**, meaning a schema (or its catalog) created with `MANAGED LOCATION` on an external location. A schema on the workspace default storage (DBFS root) cannot be used: Databricks rejects `GRANT EXTERNAL USE SCHEMA` there with `PRIVILEGE_NOT_APPLICABLE_TO_ENTITY` (`SCHEMA_DB_STORAGE`), and the Iceberg REST endpoint does not serve its tables. Create the schema on an external location, then grant the secret's identity what it needs:

  ```sql theme={null}
  -- Only if the catalog itself is on DBFS root:
  CREATE CATALOG <catalog> MANAGED LOCATION 's3://<bucket>/<prefix>/';

  CREATE SCHEMA <catalog>.<schema> MANAGED LOCATION 's3://<bucket>/<prefix>/';

  GRANT USE CATALOG ON CATALOG <catalog> TO `<sp-application-id>`;
  GRANT USE SCHEMA, CREATE TABLE, EXTERNAL USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<sp-application-id>`;
  GRANT CREATE EXTERNAL TABLE, READ FILES, WRITE FILES ON EXTERNAL LOCATION <external-location> TO `<sp-application-id>`;
  ```
</Warning>

A missing prerequisite is a warning on the run that names what to fix: external data access or `EXTERNAL USE SCHEMA` for a permission error, or the S3 path that no external location covers.

### How runs behave

* The first successful write registers the table.
* Later writes leave Unity Catalog alone. When Unity Catalog still points at an older metadata file of the same table, the run gets a warning that its pointer is behind; Unity Catalog readers see the table as of the first registration until that pointer is refreshed. To keep the pointer current on every run, set `catalogMode: "rest"` (see [Keep the catalog current](#keep-the-catalog-current-catalogmode-rest)).
* If a table with that name already exists and points somewhere other than this pipeline's table, Datris refuses to touch it. It never replaces, merges or drops a table. Rename the pipeline or choose another schema.
* Any failure (a missing grant, a wrong host, a network error) is one warning on the run and an entry in the [audit log](/audit-log); the write itself still succeeds.
* The pipeline detail page shows when the table was registered, and whether the pointer is stale or the last attempt failed. `GET /api/v1/pipelines/{name}/unity-catalog` returns `register` (`off`, `never`, `registered`, `stale` or `error`), `registeredMetadataLocation` and `lastRegisterAt`.
* Deleting a registered pipeline together with its data warns that Unity Catalog still has the table registered at a location that no longer exists; an admin drops it.
* `DATRIS_UNITY_CATALOG_SYNC=false` switches registration off for the whole install.

Tables in the built-in MinIO store can be registered, but Unity Catalog cannot read MinIO, so Databricks cannot serve them. Use an S3 bucket for tables you want to query from Databricks.

### Keep the catalog current (`catalogMode: rest`)

By default Datris writes the table by path and registers it once (`catalogMode: "register"`, the behaviour above). With `catalogMode: "rest"`, every write goes through the Unity Catalog Iceberg REST catalog instead: creating the table, appends, overwrites, merges and added columns are all committed through the catalog, so Unity Catalog points at the current metadata file after every run and the stale-pointer warning goes away.

```json theme={null}
"unityCatalog": {
  "enabled": true,
  "credentialsSecret": "databricks_prod",
  "catalog": "analytics",
  "schema": "default",
  "catalogMode": "rest"
}
```

`catalogMode` accepts `register`, `rest` or `managed` (any case) and applies to object store Iceberg pipelines only. The API returns `400` for any other value, for `catalogMode` on a Databricks destination, and for `"register": false` together with `"rest"` or `"managed"`. `rest` needs the same `credentialsSecret` and `catalog` as registration. To convert an existing non-Iceberg pipeline (parquet, orc), take two steps: first switch `fileFormat` to `iceberg` with `deleteBeforeWrite` on and run it once, then remove `deleteBeforeWrite` and add `catalogMode: "rest"`.

The first `rest` run looks at both the catalog and the table already at the pipeline's path:

* **Nothing at either place:** the table is created through the catalog at the pipeline's path.
* **A table Datris wrote at the path, and nothing in the catalog:** Datris registers the table's current metadata file once (the run says `adopted`), then commits through the catalog. The adopted table is stamped with the adoption marker `datris.adopted-from` and a longer metadata history (`write.metadata.previous-versions-max` of 1000), so the catalog stays recognised as Datris' own even if Datris' own record of the adoption is lost (the run says `adopted ... (stamped)`). This needs the `register` call, so it is not possible on Databricks: the run is refused and writes by path, and the warning says to start from a new prefix, or to set `deleteBeforeWrite` for one run (allowed because the table was never committed through the catalog) so Datris can create the table through the catalog. A table that reached the catalog before this marker existed (registered in register mode, or adopted by an earlier Datris version) is stamped once on its next run through the catalog.
* **The catalog already points at the table's current metadata file** (for example, a table registered with `catalogMode: "register"` that has not been written since): Datris commits through the catalog.
* **The catalog points at an older metadata file of the same table:** Datris refuses to use the catalog, because moving the pointer back in line would need the catalog table dropped, and Datris never drops a table. The warning names both files. An admin drops the Unity Catalog table; the next run registers the current file and continues through the catalog.
* **The catalog has a table with that name somewhere else:** Datris refuses to touch it (never merge, never drop). Rename the pipeline or choose another schema.

A refused run still writes the data by path and completes; the warning is recorded on the run and in the [audit log](/audit-log). If another pipeline creates a table with the same name at the same moment, the run that loses the race is refused the same way.

`deleteBeforeWrite` works with `rest` until the catalog holds a table at the pipeline's prefix. After that, a run with `deleteBeforeWrite` fails and names the Unity Catalog table to drop first. A catalog table with the same name somewhere else does not block it: only the pipeline's own prefix is deleted, and the run is then refused and writes by path.

Reads follow the catalog: querying the pipeline's object store (the query API and MCP tool that `find_data` points to) reads a table committed through the catalog at the catalog's current metadata file. If the catalog cannot be reached, the query reads the table as of the last commit Datris recorded.

There are two kinds of failure:

* **Before the commit** (a wrong host, a missing grant, a network error while looking the table up): one warning, an audit entry, and the run writes by path as before. The pipeline page shows `refused` with the reason.
* **During the commit** (the catalog rejects or loses the commit after data files are written): the write fails and so does the run, the same as any failed write.

Once a table has been committed through the catalog, its current metadata is known only to the catalog: path-based readers and writers no longer see the latest version. Starting over at a new prefix also needs an admin to drop the Unity Catalog table, because the table keeps the same name and a catalog entry pointing at the old prefix is refused. From then on:

* If the catalog cannot be reached, or `DATRIS_UNITY_CATALOG_SYNC=false`, the run **fails** instead of writing by path, because a path write would split the table's history. Fix the catalog connection, or point the pipeline at a new prefix and have an admin drop the Unity Catalog table.
* `deleteBeforeWrite` fails the run, whatever the file format, because deleting the files would leave the catalog pointing at a table that no longer exists. To start over, have an admin drop the Unity Catalog table, or use a new prefix.
* Switching `catalogMode` from `rest` back to `register` (or removing it, setting `"register": false` or `"enabled": false`, or removing the `unityCatalog` block) is not supported: the next run fails and says so. Keep `rest`, or start a new prefix and have an admin drop the Unity Catalog table.
* Deleting the pipeline together with its data removes the files but not the Unity Catalog table; an admin drops it. The delete response carries a `warnings` entry naming the table (`Unity Catalog still holds <catalog>.<schema>.<table>; have an admin drop it`), and the [audit log](/audit-log) records it as `unity-catalog` / `pipeline-delete` with outcome `warning`. The CLI prints each warning after the success line.
* If the catalog created a table for the pipeline at a location Datris refused (a Databricks managed table), deleting or resetting the pipeline warns that Unity Catalog still holds it, created by Datris but never written to; an admin drops it.
* Resetting the pipeline (deleting its data but keeping the configuration) warns that the next run will fail until an admin drops the Unity Catalog table, or the pipeline moves to a new prefix.
* A new pipeline created with the name of a deleted one starts clean: leftover Unity Catalog state from the old pipeline is cleared, unless it guards table files that still exist at the new pipeline's prefix, or records a table the catalog created for that name at a refused location (so deleting the new pipeline still warns about it). A run that finds "committed" state while neither the catalog nor the prefix has the table ignores the state with a warning and creates the table. On startup, Datris removes Unity Catalog state left by pipelines that no longer exist, except state that guards catalog-committed table files or records a catalog-created table.
* If the pipeline's Unity Catalog state cannot be read (the config database is unavailable), the run fails rather than risk a path write.

On a pipeline that has never committed through the catalog, `DATRIS_UNITY_CATALOG_SYNC=false` writes by path with one line saying the sync is off.

The pipeline detail page shows a `rest` chip and when the table was last kept current, or `refused` with the reason. `GET /api/v1/pipelines/{name}/unity-catalog` adds `catalogMode`, `restMetadataLocation`, `lastRestCommitAt`, `restRefusedReason` and `restCreatedTable`, and `register` gains the values `rest`, `managed` and `refused`.

`catalogMode: "rest"` works against catalogs that honour the requested location, such as the Apache Iceberg REST fixture. Datris checks where the catalog put the table before writing anything: if a catalog places the table outside the pipeline's prefix (Databricks does, it creates a managed table and ignores the requested location), Datris refuses to write there, warns (naming both locations and the table an admin should drop) and writes by path; the pipeline page shows `refused`. On Databricks, use `catalogMode: "managed"` instead.

Saving the pipeline already warns about this. When the `credentialsSecret` host is a Databricks workspace (`*.databricks.com` or `*.azuredatabricks.net`) and `catalogMode` is `register` (the default) or `rest`, `POST /api/v1/pipeline` answers `200` with a `warnings` entry naming the host: with `register`, that Databricks has no Iceberg REST register call so the table will not be registered; with `rest`, that Databricks creates a managed table at its own location, which Datris refuses. Both point at `catalogMode: "managed"` or the databricks destination. The warning is advisory and the pipeline is saved; the MCP `create_pipeline` response carries the same `warnings`.

**Testing locally.** The Iceberg REST catalog test fixture works as a stand-in for Unity Catalog: run the `apache/iceberg-rest-fixture` image on the Datris network, create its `default` namespace, and use a secret with `host` set to `http://iceberg-rest:8181`, `icebergRestPath` set to `/` and `icebergRestPrefix` set to `-` (see the next section). Point the pipeline at the built-in MinIO store for this test.

### Databricks: `catalogMode: managed`

Databricks Unity Catalog creates Iceberg tables as managed tables: it chooses where the table lives (under the schema's `MANAGED LOCATION`, in a `__unitystorage/` folder) and has no `register` call. With `catalogMode: "managed"`, Datris works the same way: it creates `<catalog>.<schema>.<pipeline>` through the Unity Catalog Iceberg REST catalog without asking for a location, the catalog chooses the location, Datris writes the data files there with the pipeline's own S3 credentials, and every append, overwrite, merge and added column is committed through the catalog. Catalog Explorer, Databricks SQL and the pipeline's object store query all see the same current table.

```json theme={null}
{
  "destination": {
    "objectStore": { "provider": "s3", "destinationBucketOverride": "my-lake", "prefixKey": "orders", "fileFormat": "iceberg", "credentialsSecret": "aws_lake" }
  },
  "unityCatalog": {
    "enabled": true,
    "credentialsSecret": "databricks_prod",
    "catalog": "analytics",
    "schema": "uc",
    "catalogMode": "managed"
  }
}
```

**Prerequisites**

* A schema created with `MANAGED LOCATION 's3://<pipeline bucket>/<prefix>/'` in the pipeline's own bucket (the bucket in `destinationBucketOverride`), for example `CREATE SCHEMA analytics.uc MANAGED LOCATION 's3://my-lake/uc/'`, and the grants from [Databricks prerequisites](#databricks-prerequisites).
* `provider: s3`, with an object store `credentialsSecret` that can read and write that bucket, including the catalog's `__unitystorage/` folder under the schema's location. The pipeline's S3 credentials cover the whole bucket, so they reach wherever the catalog puts the table in it.
* `provider: minio` is accepted only when the Unity Catalog secret has a custom `icebergRestPath`, meaning a non-Databricks catalog such as the test fixture (see [Other Iceberg REST catalogs](#other-iceberg-rest-catalogs)). With a Databricks secret, a MinIO pipeline fails the run before anything is written, because Unity Catalog cannot place a managed table in the built-in MinIO store.

**How runs behave**

* The catalog chooses the location. It must be in the pipeline's bucket: a table the catalog places in another bucket is refused before any data is written, and the run fails, naming both buckets.
* `prefixKey` holds no data in this mode. Datris never writes under it, and queries read the table where the catalog put it. The run log names the catalog location (`created analytics.uc.orders at s3://my-lake/uc/__unitystorage/... through the catalog`).
* Datris only writes to a table it created for this pipeline. If the catalog already has a table with that name that this pipeline did not create, the run fails; have an admin drop it or choose another schema. The one exception is the empty table an earlier `catalogMode: "rest"` run created on Databricks for this pipeline: managed mode adopts it. A table counts as this pipeline's only while it is the exact table the pipeline last committed to: after a change of `catalog` or `schema`, a table of the same name in the new schema is not adopted. If the new schema has no table of that name, the run creates one there and warns that it is forgetting the old table, naming its location so an admin can drop it. A pipeline recreated under the same name with a `unityCatalog` block continues the same managed table, unless an admin dropped it.
* Unlike `rest`, managed mode never falls back to a path write. Any refusal before the commit (a wrong host, a missing grant, a network error, the sync switched off with `DATRIS_UNITY_CATALOG_SYNC=false`) fails the run, because a path table nobody reads would silently strand the rows. A failure during the commit fails the run as in every mode.
* `deleteBeforeWrite` is rejected with `400`: there is nothing at the prefix to delete, and the table's files belong to the catalog.
* Switching a table that has been committed in managed mode to `rest` or `register` (or switching a `rest` table to `managed`) fails the next run. Datris never converts a table between modes; to change, have an admin drop the table first. While runs are refused this way, the state endpoint keeps reporting the managed table (`catalogMode: "managed"`, its location) with `register: "refused"`, `state: "error"` and the reason in `restRefusedReason`. Removing the `unityCatalog` block (or recreating the pipeline without one) is different: the pipeline can no longer reach the table, so the run warns that it is forgetting the managed table, names its location, and writes by path under `prefixKey`. The table stays in Unity Catalog until an admin drops it.
* Deleting or resetting the pipeline leaves the table and its files to Unity Catalog: Datris deletes only the pipeline's prefix, which is empty in this mode. The delete response carries the warning `Unity Catalog owns <catalog>.<schema>.<table> and its files (catalogMode managed); Datris deleted nothing there; drop <catalog>.<schema>.<table> in Unity Catalog to remove the data`.
* The pipeline detail page shows a `managed` chip and when the table was last committed. `GET /api/v1/pipelines/{name}/unity-catalog` returns `catalogMode: "managed"`, `register: "managed"` and `restMetadataLocation` under the catalog's location. After a run that failed before writing, `state` is `error` and `restRefusedReason` says why; the table committed earlier is unchanged. Object store queries and the run's notification report the catalog's location as the table path.

Managed mode has been verified against Databricks Unity Catalog: the table is created in the schema's managed storage and every commit goes through the catalog. Databricks' `DESCRIBE` shows the table's storage root, while Datris reports the Iceberg root (`.../_iceberg`) beneath it.

### Other Iceberg REST catalogs

For testing and for non-Databricks catalogs, the secret can carry two optional fields that change where the register call goes:

| Field | Default | Meaning |
| - | - | - |
| `icebergRestPath` | `/api/2.1/unity-catalog/iceberg-rest` | Path of the Iceberg REST catalog on `host`. `/` means the host root. |
| `icebergRestPrefix` | `catalogs/<catalog>` | Catalog prefix in the REST path. An empty value, or `-`, means no prefix. |

`host` may then include a scheme and port, for example `http://iceberg-rest:8181`, and a secret without `clientId`/`clientSecret` or `token` sends no credentials. Registration, `catalogMode: "rest"` and `catalogMode: "managed"` all work this way.

## Browse what a secret can see

`browse_unity_catalog` (MCP) and `GET /api/v1/unity-catalog/browse` (REST) walk Unity Catalog one level per call, using a Databricks Platform secret. No pipeline is needed.

| Parameters | Returns |
| - | - |
| `secret` | `catalogs`: the catalog names the secret can see |
| `secret`, `catalog` | `schemas` in that catalog |
| `secret`, `catalog`, `schema` | `tables` in that schema. A table with the `datris_pipeline` tag carries `datrisPipeline` |
| `secret`, `catalog`, `schema`, `table` | `columns` (`name`, `type`, `comment`), `tags`, and `datrisPipeline` when present |

```bash theme={null}
curl -s -H 'x-api-key: <your-api-key>' \
  'http://localhost:8080/api/v1/unity-catalog/browse?secret=<secret>&catalog=<catalog>&schema=<schema>'
```

* Find the secret name with `list_platform_secrets`. The secret needs `host` and either `clientId`/`clientSecret` or `token`, the same fields the Databricks destination uses.
* Catalog and schema listings are cached for 5 minutes per secret. Table and column listings are always live. Each level returns at most 1000 entries.
* A stopped SQL warehouse starts on the first call. Serverless warehouses take about 5 to 20 seconds; classic warehouses can take several minutes.
* When the secret cannot read the tag views, the response has `tagsAvailable: false` and no tags. Browsing still works.

Responses: `400` when `secret` is missing, when `schema` or `table` is given without the levels above it, or when no warehouse can be found; `404` when no Databricks Platform secret has that name, or when your API key may not read it (the same response either way, so a key cannot probe for secret names it has no access to; the audit log records the second case as `denied`); `502` when the warehouse connection or a metadata query fails, with the same message the Databricks destination gives. Every call is recorded in the [audit log](/audit-log) as `unity-catalog` / `browse` with the secret name.

Access: the route needs `metadata:read`, and your key must also be able to read the named secret, the same check `list_platform_secrets` applies. Keys without `metadata:read` do not see the MCP tool.

### Which SQL warehouse is used

The secret holds the workspace host and credentials, not a warehouse. Browse picks one in this order:

1. the `warehouse` query parameter (MCP argument `warehouse`)
2. an optional `warehouse` field on the secret (`httpPath`, `http_path` and `DATABRICKS_WAREHOUSE` also work). The value can be the warehouse ID, its HTTP path, or a JDBC URL
3. the warehouse of the first Databricks pipeline whose `credentialsSecret` is this secret and that sets `warehouse` itself (a pipeline that leaves it out uses the secret's field, so it adds nothing here)

If none applies, the call returns `400` listing these three options. Existing Databricks secrets work unchanged; the `warehouse` field is optional.

## Search warehouse tables from find\_data

`find_data` ranks Datris pipelines by default. With `include_unity_catalog: true` (REST: `includeUnityCatalog=true`) it also searches the Unity Catalog tables visible to every Databricks Platform secret your key can read. It matches table names, table comments and column names.

* Datris hits come first and carry `source: "datris"`. Unity Catalog hits follow with `source: "unity-catalog"`, `name` as `catalog.schema.table`, the `secret` that found it, `matchedColumns`, `tags` and a `location` of kind `databricks`.
* A table is **owned** when a Databricks pipeline you can see loads into those exact coordinates, or when its `datris_pipeline` tag names a pipeline you can see. Owned tables have `ownedByPipeline`, and their `howToQuery` points at `query_databricks` with that pipeline. Owned tables come before unowned ones. If the owning pipeline is already a Datris hit, the table is left out because the pipeline hit already covers it.
* Unowned tables get a plain SQL hint (`SELECT * FROM catalog.schema.table LIMIT 100`) to run in your warehouse. `query_databricks` cannot run it, because that tool needs a pipeline.
* The response adds `unityCatalog: { searched: [...], skipped: [{ secret, reason }] }`. A secret with no known warehouse (only options 2 and 3 above apply here) or a failed connection is listed under `skipped`; it never fails the call. With no Databricks secrets, `searched` is empty and you get the Datris hits only.
* `limit` applies to Datris hits and Unity Catalog hits separately: you get up to `limit` Datris hits plus up to `limit` additional Unity Catalog hits, so `limit: 5` can return 10 results. `count` and `totalMatches` include the Unity Catalog hits. The optional AI rerank orders Datris hits only.
* Each secret has its own time limit, 45 seconds by default (`DATRIS_UNITY_CATALOG_SEARCH_TIMEOUT_SECONDS`, see the [configuration reference](/configuration-reference)). A slow warehouse is skipped with the reason `timed out after 45s` rather than blocking the call.

Without the flag, `find_data` output is unchanged: no `source` field and no `unityCatalog` block.

## Grants for discovery

Browse and search run as the secret's identity, so they see only what it is granted:

* `USE CATALOG` and `USE SCHEMA` on what you want to browse, and `SELECT` (or `BROWSE`) to describe tables.
* Search and tag lookups read `system.information_schema` (`columns`, `tables`, `table_tags`). Unity Catalog filters these views to the objects the identity can access, so a table the secret cannot use never shows up. If the identity cannot read the `system` catalog, tag lookups return `tagsAvailable: false` and `find_data` lists the secret under `skipped` with the warehouse's error. Ask a workspace admin to grant the identity access to `system.information_schema`.

## Not covered yet

* By default the Unity Catalog pointer is set once and not kept current; set `catalogMode: "rest"` to keep it current on every run. `register` stays the default.
* There is no Unity Catalog browser in the Datris UI yet; use the API or MCP tool.


# Embed documents into Weaviate for RAG
Source: https://docs.datris.ai/destinations/weaviate

Chunk, embed, and upsert PDFs, Word docs, text, and HTML into a Weaviate class for retrieval-augmented generation.

Use this destination when you want a Weaviate index built and refreshed by a repeatable pipeline, with chunking and embedding handled for you.

Ingest unstructured documents (PDFs, Word docs, text files, HTML) into [Weaviate](https://weaviate.io), an open-source vector database for RAG and semantic search applications. The pipeline extracts text from the document, chunks it using a configurable strategy, generates vector embeddings, and upserts the chunks with metadata into a Weaviate class.

## Configuration

```json theme={null}
"source": {
    "fileAttributes": {
        "unstructuredAttributes": {
            "fileExtension": "pdf",
            "preserveFilename": true
        }
    }
},
"destination": {
    "weaviate": {
        "className": "FinancialDocuments",
        "chunking": {
            "strategy": "recursive",
            "chunkSize": 500,
            "chunkOverlap": 50
        },
        "metadata": {
            "company": "Apple Inc",
            "documentType": "10-Q",
            "filingDate": "2026-01-30"
        },
        "weaviateSecretName": "oss/weaviate"
    }
}
```

| Field | Type | Default | Description |
| - | - | - | - |
| `className` | string | (required) | Weaviate class name. **Must be PascalCase** (e.g., `FinancialDocuments`). Auto-created if it doesn't exist. |
| `chunking` | object | recursive, 500/50 | Chunking strategy configuration (see below). |
| `metadata` | map | `{}` | Static key-value metadata stored on every chunk as Weaviate properties. Used for filtered search. |
| `embeddingSecretName` | string | *server default* | Embedding Vault secret name. Only consulted when the server-level `ai.embedding.secretName` is unset; with the default configuration (`oss/embedding`, seeded automatically) the server-level secret is always used and this field is ignored. |
| `weaviateSecretName` | string | (required) | Vault secret name for Weaviate connection. |

## Supported File Types

| Format | Description |
| - | - |
| PDF (`.pdf`) | Text extracted via Apache PDFBox |
| Word (`.doc`) | Text extracted via Apache POI (legacy format) |
| Word (`.docx`) | Text extracted via Apache POI (modern format) |
| PowerPoint (`.ppt`) | Text extracted via Apache POI (legacy format) |
| PowerPoint (`.pptx`) | Text extracted via Apache POI (modern format) |
| Excel (`.xls`, `.xlsx`) | Cell values extracted via Apache POI |
| HTML (`.html`, `.htm`) | Text extracted via JSoup (tags stripped) |
| RTF (`.rtf`) | Text extracted via javax.swing RTF parser |
| Email (`.msg`) | Subject, from, to, and body extracted via Apache POI |
| Email (`.eml`) | Subject, from, and body extracted via Jakarta Mail |
| EPUB (`.epub`) | XHTML content extracted and parsed via JSoup |
| Plain text (`.txt`, `.md`, `.csv`, `.json`, `.xml`) | Content used directly |

For structured data (CSV, JSON, XML), use database destinations (PostgreSQL, MongoDB) instead.

## Vault Secrets

### Weaviate connection

```bash theme={null}
vault kv put secret/oss/weaviate \
  host="host.docker.internal" \
  port="8079" \
  apiKey=""
```

| Field | Description |
| - | - |
| `host` | Weaviate server hostname. Use `host.docker.internal` for local Weaviate. |
| `port` | REST API port (default 8079 to avoid conflict with pipeline's 8080). |
| `scheme` | Protocol — `http` (default) or `https` for Weaviate Cloud. |
| `apiKey` | API key for Weaviate Cloud. Empty string for local instances. |

### Embedding API

The embedding secret is server-level (`ai.embedding.secretName`, default `oss/embedding`) and is seeded automatically by `docker/vault-init.sh`. See [AI Configuration](/ai-configuration) and the [Qdrant docs](/destinations/qdrant#embedding-api) for the full picture.

```bash theme={null}
vault kv put secret/oss/embedding \
  provider="openai" \
  endpoint="https://api.openai.com/v1/embeddings" \
  model="text-embedding-3-small" \
  apiKey="sk-..."
```

For Anthropic-only deployments, the bundled TEI sidecar serves `bge-m3` and `vault-init.sh` seeds the embedding secret to point at it — no OpenAI key required.

## Chunking Strategies

Documents are split into chunks before embedding. Each chunk becomes a separate Weaviate object with the document's metadata plus `chunk_index`, `filename`, and `source_pipeline` properties.

```json theme={null}
"chunking": {
    "strategy": "recursive",
    "chunkSize": 500,
    "chunkOverlap": 50
}
```

| Strategy | Description |
| - | - |
| `none` | No chunking — one embedding per document. Only for very short documents. |
| `fixed` | Split by character count. Fast but may cut mid-sentence. |
| `sentence` | Split on sentence boundaries (`.` `!` `?`). Preserves semantic units. |
| `paragraph` | Split on double newlines. Ideal for structured documents with clear sections. |
| `recursive` | Try `\n\n`, then `\n`, then `.`, then space — best general-purpose default. |

* `chunkSize` (default 500): maximum characters per chunk
* `chunkOverlap` (default 50): characters of overlap between consecutive chunks

## Metadata

Static metadata is attached to every chunk as Weaviate properties, enabling filtered semantic search:

```json theme={null}
"metadata": {
    "company": "Apple Inc",
    "documentType": "10-Q",
    "filingDate": "2026-01-30"
}
```

This allows queries like "find chunks about revenue from Apple 10-Q filings" by combining vector similarity with metadata filters.

In addition to static metadata, every chunk automatically includes:

* `text` — the chunk text
* `chunk_index` — position of the chunk in the document
* `filename` — original uploaded filename
* `source_pipeline` — pipeline name

## How It Works

1. **Upload** — an unstructured file (PDF, DOC, DOCX, HTML, text) is uploaded via `POST /api/v1/pipeline/upload`
2. **Extract** — text is extracted from the document (PDFBox for PDFs, Apache POI for Word, JSoup for HTML)
3. **Chunk** — text is split into chunks using the configured strategy
4. **Embed** — each chunk is sent to the embedding API to generate a vector
5. **Upsert** — vectors are upserted into the Weaviate class with metadata properties
6. **Notify** — a pipeline notification is published on completion

The class is auto-created on first upsert using HNSW indexing with cosine distance.

## Running Weaviate

Weaviate ships as a bundled **opt-in** Compose service. Enable it with one line in `.env`:

```bash theme={null}
COMPOSE_PROFILES=weaviate
```

Then start the stack:

```bash theme={null}
docker compose up -d
```

Combine profiles with commas (e.g., `COMPOSE_PROFILES=weaviate,kafka`). The `install.sh` installer can also enable Weaviate at install time — it writes the profile and seeds the `oss/weaviate` secret for you. On the Compose network the bundled container is reachable at `weaviate:8080`; on the host it is published at port 8079 to avoid conflict with the pipeline server on port 8080.

To use an existing external or Weaviate Cloud instance instead, skip the profile and point the `oss/weaviate` secret's `host`/`port`/`apiKey` at it — the secret fields are the same either way. You can also run Weaviate standalone:

```bash theme={null}
docker run -p 8079:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:latest
```

Or install from [weaviate.io/developers/weaviate/installation](https://weaviate.io/developers/weaviate/installation).

## Verifying

```bash theme={null}
# List all classes
curl http://localhost:8079/v1/schema

# Get class details
curl http://localhost:8079/v1/schema/FinancialDocuments

# Get objects
curl "http://localhost:8079/v1/objects?class=FinancialDocuments&limit=5"

# Get objects with specific metadata filter
curl -X POST http://localhost:8079/v1/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{ Get { FinancialDocuments(limit: 5, where: {path: [\"company\"], operator: Equal, valueText: \"Apple Inc\"}) { text chunk_index filename company documentType } } }"
  }'
```


# Doctor
Source: https://docs.datris.ai/doctor

Operational self-check for a Datris deployment

`datris doctor` checks a running deployment for the failure modes that are cheap to detect and expensive to miss: a Vault token about to expire, an AI slot secret missing so the server won't boot after an upgrade, an embedding model that was never pulled, a disk about to fill, data sitting on an anonymous Docker volume, a container still running with an old `.env`, a mixed-version stack. Every finding names the command that fixes it. Doctor never changes anything.

## Where it runs

| Surface | What it covers | When |
| - | - | - |
| Server log at boot | The cheap subset (Vault token, AI secrets, embedding model, disk) as `DOCTOR` warn lines | Every start; a fresh install logs nothing |
| **Configuration → Doctor** | Every server-side check, with an optional AI probe | On demand |
| `run_doctor` MCP tool | Same as the UI; for an agent diagnosing a failure | On demand — not part of the normal workflow |
| `datris doctor` CLI | Server-side checks **plus** host checks that need Docker | On demand, and `--pre-upgrade` before every upgrade |
| Installer (`install.sh`) | Runs `datris doctor --pre-upgrade` for you when the CLI is installed | Every upgrade through the installer, before anything is pulled |
| Periodic run | The server-side checks on a timer, posting flips to error to the incident webhook | Opt-in: `DOCTOR_INTERVAL_MINUTES` |
| Recovery agent | Holds `run_doctor` as one of its read-only diagnosis tools | When an incident looks like the platform, not the tap |

The server-side report is also available at `GET /api/v1/doctor` for scripts. It is not a public endpoint — it names environment keys, Vault paths and model ids — so an API key with `config:read` is required when [API keys](/api-keys) are on.

## The checks

| Check | Fires when | Fix it prints |
| - | - | - |
| `vault.token_ttl` | The server's Vault token expires within 7 days (error), or a 10-year periodic token was clamped to Vault's default 32-day ceiling (warn) | Raise `max_lease_ttl` in the Vault config, recreate the vault container, restart the server |
| `vault.ai_slots` | An AI slot secret is missing or lacks provider / endpoint / model / key | The exact `vault kv put` or `vault kv patch` |
| `jdbc.mssql_driver` | A pipeline uses `type: mssql` but the driver is not bundled | Add the driver and rebuild, or change the source type |
| `objectstore.bucket_overrides` | A pipeline overrides the built-in MinIO bucket and `DATRIS_OBJECTSTORE_BUCKET_ALLOWLIST` is unset (warn), or a stored override falls outside the list you set and will be refused at run time (error) | Set the allowlist, or add the bucket / change the pipeline's `destinationBucketOverride` |
| `ai.embedding_model` | The bundled embedding service is serving a different model than the embedding slot names, or Ollama hasn't pulled it | `ollama pull …` or align the slot with the served model |
| `ai.model_reachable` | **Opt-in.** A chat or embedding slot's key is rejected, the model id is unknown, or the endpoint doesn't answer within 10 s | Fix the key / pick a current model in Configuration → AI Providers |
| `disk.usage` | The disk holding the server or Docker's data root is ≥ 85% full (warn) or ≥ 95% (error) | Prune images and builder cache; inspect the pip-cache and object-store volumes |
| `staging.area` | The payload staging area (`DATRIS_TEMP_DIR`, the `datris-staging` volume) is missing, refuses a write, or has under 1 GB free (error); or its free space is below the per-run payload budget `PIPELINE_MAX_PAYLOAD_MB` (warn) | Point `DATRIS_TEMP_DIR` at a writable directory with room for one run, free space on the disk, or lower the budget. The 1 GB "room for one run" threshold is a floor, not a sizing guide: an upload can briefly occupy up to about three times its size (multipart spool, staged upload, the run's own copy) |
| `staging.orphans` | Run directories older than 24 h are still under the staging area — left by a process that died mid-run (warn; error when they hold ≥ 1 GB) | Restart datris (the boot sweep reclaims them) or remove the directories by hand |
| `tap.secret_scope` | A saved tap uses a secret not tagged `_type=tap`: error when taps are limited to tap secrets (the default; the tap fails to run), warn under `DATRIS_TAP_SECRET_SCOPE=any` (the tap can read a platform secret) | Create a tap secret with the fields the tap needs and select it, or add `_type: tap` to a hand-made tap secret. See [Taps](/taps#taps-can-only-use-tap-secrets) |
| `codegen.isolation` | Reports where the Python scripts for AI data-quality rules and AI transformations run. Ok when they run in the `datris-codegen-runner` sidecar and it answers; warn when they run in-process (`USE_CODEGEN_RUNNER` is not `true`: an install on an older compose file, or `sbt` / an IDE where it is expected); error when the runner is required but cannot run a script (stopped, unreachable, an older image without `/execute-file`, token rejected, or its `codegen-scratch` volume not writable). Those rules and transformations fail until it is fixed. In the boot-time lines a runner that is not listening yet is a skip, not an error | Warn: refresh `docker-compose.yml` (or re-download `docker-compose.standalone.yml`) and `docker compose up -d`. Error: `docker compose pull datris-codegen-runner && docker compose up -d datris-codegen-runner`, then read its logs. See [Tap Execution & Isolation](/tap-execution-isolation#generated-data-quality-and-transformation-scripts) |
| `runner.concurrency` | Not part of the boot lines; skipped when taps run in-process (`USE_TAP_RUNNER` is not `true`). Ok names `TAP_RUNNER_MAX_CONCURRENT` and how many of its slots are in use; warns when tap runs waited for a runner slot in the last 24 h (for example "3 runs waited (longest 42 s)") or the runner does not report slot use; errors when a run was turned away after waiting `TAP_RUNNER_QUEUE_TIMEOUT_SECONDS` | Raise `TAP_RUNNER_MAX_CONCURRENT` (and `TAP_RUNNER_MEMORY` with it) in `.env` or stagger tap schedules, then `docker compose up -d datris-tap-runner`. See [Overload protection](/tap-execution-isolation#overload-protection) |
| `ai.sample_values` | Never errors. Reports whether schema, profile, attachment and CodeGen helpers send row values to the model (`DATRIS_AI_SAMPLE_VALUES`, default `true`) or withhold them. Warns when a pipeline protects fields while values are still sampled; with values withheld, warns when a stored source field name does not read as a column name (stored names are always sent), naming up to five pipelines; warns when the pipeline configs cannot be read | Set `DATRIS_AI_SAMPLE_VALUES=false` on the datris service and recreate it. See [Field Protection](/transformation/field-protection#what-still-sees-raw-data) |
| `governance.controls` | Never errors; not part of the boot lines. Warns when any of user login (`USE_USER_AUTH`), API keys (`USE_API_KEYS`), the audit log (`USE_AUDIT_LOG`) or the agent policy (`USE_AGENT_POLICY`) is off, naming the ones that are off and the ones that are on. They ship off and are switched on for production, so a default install shows this warning | Add the missing lines (for example `USE_API_KEYS=true`) to `.env`, then `docker compose up -d --force-recreate datris mcp-server`. See [Quick Start](/quick-start#2-switch-on-the-governance-controls) |
| `audit.strict` | Reports the audit log's evidence mode. Warns when `AUDIT_LOG_STRICT=true` but `USE_AUDIT_LOG` is off (strict mode then does nothing); errors when strict mode is on and the audit log is not accepting entries, so audited requests are being refused with 503 (the config store is not taking writes); ok otherwise | Warn: set `USE_AUDIT_LOG=true` or unset `AUDIT_LOG_STRICT`, then recreate datris. Error: bring MongoDB back (`docker compose ps mongodb`); queued entries are written in order once it recovers. See [Audit Log](/audit-log#strict-mode) |
| `provenance.strict` | Never errors. Reports the provenance evidence mode. With `PROVENANCE_STRICT=true`, warns and names the pipelines that have stamping on and an XML source: XML is exempt and loads unstamped, so those pipelines are outside the strict guarantee. Also warns when the pipeline configs cannot be read | Convert those sources to JSON or CSV if every row must carry provenance. See [Provenance](/provenance#strict-mode) |
| `version.skew` | Server, UI, MCP server and CLI differ in major.minor, or the UI image carries no version stamp | Pull all images; upgrade the CLI |
| `env.seen` | Never — reports which hardening variables reached the server | — |
| `volumes.anonymous` | A stateful service (postgres, mongodb, minio, vault, tei, kafka) stores data on an anonymous volume that `docker compose down` will orphan | Add a named volume and copy the data across |
| `volumes.dangling` | A dangling anonymous volume holds a recognizable Postgres / MongoDB / MinIO data directory — data left behind by a container recreate | Inspect before pruning; copy into the named volume to recover |
| `vault.hcl_drift` | `docker/vault.hcl` changed after the vault container started (Vault reads its config only at start) | `docker compose up -d --force-recreate vault` |
| `env.not_forwarded` | `.env` sets a key that no service in the compose file references, so the value never reaches a container | Add it to the service's `environment:` block |
| `env.container_drift` | A datris container (`datris`, `mcp-server`, `datris-tap-runner` or `datris-codegen-runner`) was created with a different `.env` than the current one (`restart` does not reload env). Error when a credential-looking key differs | `docker compose up -d --force-recreate --no-deps <service>` |
| `compose.orphans` | Containers from a previous compose file are still present and may hold ports | `docker compose up -d --remove-orphans` |
| `build.stale_jar` | Building from source, and the running server's git commit or build time is behind the checkout | Rebuild with `--no-cache` |
| `mcp.reachable` | The MCP server's SSE endpoint doesn't answer within 3 s | Check the `mcp-server` container |
| `version.update` | Never errors. Warns when datris.ai reports a newer release than the running server; ok when current. Skipped when switched off (`DATRIS_UPDATE_CHECK=0`), when the server version is unknown (nothing is sent), or when datris.ai cannot be reached within 3 s. Not run by `--pre-upgrade` | `datris doctor --pre-upgrade && docker compose pull && docker compose up -d --remove-orphans`. See [Upgrading](/production/upgrades) |

Checks below the line run only from the CLI on the machine running Docker; the UI and MCP surfaces report them as not applicable. Secret values never appear in any output — only key names.

## CLI

```bash theme={null}
datris doctor                      # server checks + host checks, human-readable
datris doctor --json               # the same report as JSON
datris doctor --probes ai          # also send a minimal request through each AI slot (spends a few tokens)
datris doctor --pre-upgrade        # host checks only; safe with the server stopped
```

Run it from the directory that holds `docker-compose.yml` and `.env`, or point it there with `--project-dir` (and `--compose-file` for the standalone file). Unlike the other CLI commands, `doctor` talks to the server directly rather than through the MCP server — a dead MCP server is itself a finding. Set `DATRIS_URL` if the server is not on `http://localhost:8080`. When API keys are on, set `DATRIS_API_KEY` — the same variable every other CLI command reads.

Exit codes make it scriptable:

| Code | Meaning |
| - | - |
| 0 | Everything ok |
| 1 | At least one warning |
| 2 | At least one error |
| 3 | Server unreachable — host checks ran, server checks skipped |

Skipped checks (Docker not on this machine, a service not running, datris.ai not reachable) never affect the exit code. A default install, with the governance controls still off, or a deployment that is behind the latest release, exits `1` because of the `governance.controls` or `version.update` warning; switch the controls on and upgrade (or treat `1` as passing) if a script gates on exit `0`.

### Update check

A full `datris doctor` run (not `--pre-upgrade`) makes one request that leaves the machine, so the `version.update` row can tell you whether a newer version exists. It is the only check that does. The request is:

```
GET https://get.datris.ai/version?version=<server version>&os=<os>&arch=<arch>
User-Agent: datris-doctor/<cli version>
```

`os` is `linux`, `darwin` or `windows`; `arch` is the CPU architecture as the OS names it (`x86_64`, `arm64`, `aarch64`). Nothing else is sent: no API key, cookie, body, hostname or identifier. The answer is `{"latest": "1.45.0"}`; anything else, a redirect included (it is not followed), is treated as unreachable. It times out after 3 s, and failure is a skip that never changes the exit code. Nothing is written to disk. The server itself never makes this request, and neither does the periodic run, the UI, the `run_doctor` tool or the installer.

To switch it off, add `DATRIS_UPDATE_CHECK=0` to the project `.env` (doctor reads it from `--project-dir`) or set it in the environment; `0`, `false` and `off` all work, and the environment wins over `.env`. The row then reads "off (DATRIS\_UPDATE\_CHECK=0)" and nothing is sent. See [Security Architecture](/production/security-architecture#no-telemetry).

## Before an upgrade

```bash theme={null}
datris doctor --pre-upgrade
```

This runs the host checks plus a read of the three AI slot secrets through the vault container, so a missing secret is caught **before** the new server crash-loops on it, and an anonymous volume is caught before `--remove-orphans` could drop it. When nothing is at error level it prints the upgrade command. See [Upgrading](/production/upgrades).

**The installer runs it for you.** When you upgrade by re-running `curl -fsSL https://get.datris.ai/install.sh | sh` and the `datris` CLI is on the machine, the installer runs the pre-upgrade check before it pulls anything. A warning is printed and the upgrade continues; an error stops the upgrade with the fix on screen. `DATRIS_SKIP_DOCTOR=1` bypasses it. Without the CLI the installer says so and continues as it always has.

## Periodic run

The boot-time lines catch a problem once, at start. To keep catching them — a Vault token that will expire next month, a disk crossing 85 %, an embedding service that stopped answering — set an interval and the server re-runs the full server-side report on that cadence:

```bash theme={null}
DOCTOR_INTERVAL_MINUTES=60
INCIDENT_WEBHOOK_URL=https://hooks.example.com/datris   # optional
```

Every run logs non-ok checks as `DOCTOR` lines. When a check **flips to error**, or recovers from one, the server POSTs to the incident webhook (the same one the [recovery agent](/incidents) uses, so one receiver handles both):

```json theme={null}
{
  "event": "doctor_error",
  "check": "vault.token_ttl",
  "status": "error",
  "detail": "expires in 3d (period 3650d)",
  "remediation": "…",
  "surface": "server",
  "ts": "2026-09-11T14:00:00Z"
}
```

`event` is `doctor_error` or `doctor_recovered`. Warnings and checks that stay in error are not re-posted; an error already present when the timer starts is posted once. The AI probes never run on the timer. The interval is `0` (off) by default; the timer polls once a minute, so the first run lands within a minute of boot. Each run logs one `DOCTOR periodic run` summary line; a POST that fails logs a `DOCTOR webhook POST failed` warning. Like every outbound call the server makes, the POST is refused when the receiver resolves to a private or loopback address unless `DATRIS_ALLOW_PRIVATE_EGRESS=true` (see [Configuration Reference](/configuration-reference)).

## In the recovery agent

When the [recovery agent](/incidents) diagnoses an incident, `run_doctor` is among its read-only tools. Its instructions tell it to call the quick subset once, without AI probes, only when the failure looks like the platform rather than the tap — a missing secret, a model or embedding service not answering, a full disk, a version mismatch — and, if the doctor reports an error, to hand the incident to a human with the doctor's remediation rather than propose a re-run that will fail the same way. When API keys are on, the agent's `recovery-agent` key is granted `config:read` for this on the first start after upgrading; the grant is audited.

## Report format

All surfaces produce the same JSON:

```json theme={null}
{
  "doctorVersion": 1,
  "ranAt": "2026-09-10T14:00:00Z",
  "mode": "full",
  "surface": { "server": "1.29.0", "cli": "1.29.0" },
  "summary": { "ok": 12, "warn": 1, "error": 0, "skip": 2 },
  "checks": [
    {
      "id": "vault.token_ttl",
      "status": "warn",
      "severity": "warn",
      "detail": "ttl 28d, period 3650d — clamped",
      "remediation": "…",
      "surface": "server",
      "ms": 12
    }
  ]
}
```

`status` is one of `ok`, `warn`, `error`, `skip`; `surface` is `server`, `host`, or `cli`.

## Turning off the boot-time lines

The startup subset only logs; it never blocks a start. To silence it, set `DOCTOR_ONSTARTUP=false` in `.env`. The on-demand surfaces and the periodic run are unaffected.


# Examples
Source: https://docs.datris.ai/examples

Example configurations and use cases

The `examples/` directory contains standalone applications that integrate with and extend the Datris platform. From a full-blown AI agent to simple utility scripts — each demonstrates a different way to use Datris.

## Agent

| Example | Description |
| - | - |
| [Market Macro Agent](/examples/market-macro-agent) | MacroAgent — a real-time financial data pipeline agent. Connects to Datris via MCP, provisions four platform-owned taps (FRED, yfinance, CoinGecko, SEC EDGAR), triggers them to refresh data, and answers market questions grounded in the landed numbers — all visible in a browser UI. Full agentic loop with background refresh. |

## Data Processing

| Example | Description |
| - | - |
| [Preprocessor](/examples/preprocessor) | Flask REST API for preprocessing data before pipeline ingestion. Synchronous and asynchronous modes. |

## Data Ingestion

| Example | Description |
| - | - |
| [Topic Subscriber](/examples/topic-subscriber) | ActiveMQ consumer that listens for pipeline completion notifications to trigger downstream workflows. |

## Search & RAG

| Example | Description |
| - | - |
| [Chat Vector Store](/examples/chat-vector-store) | Unified RAG chat app supporting all 5 vector stores (Qdrant, Weaviate, Milvus, Chroma, pgvector). |

## Testing

| Example | Description |
| - | - |


# Chat Vector Store
Source: https://docs.datris.ai/examples/chat-vector-store

Chat with a vector store example

**Location:** `examples/chat-vector-store/`

A unified RAG chat application that supports all five vector store destinations: Qdrant, Weaviate, Milvus, Chroma, and pgvector. Select the vector store via a command-line argument. Ask questions about documents ingested through Datris pipelines and get AI-powered answers grounded in your data.

## Setup

```bash theme={null}
cd examples/chat-vector-store
pip install qdrant-client weaviate-client psycopg2-binary pymilvus chromadb openai anthropic python-dotenv requests
```

## Run

```bash theme={null}
python app.py                     # default: qdrant
python app.py --store qdrant
python app.py --store weaviate
python app.py --store pgvector
python app.py --store milvus
python app.py --store chroma
python app.py -s chroma            # short flag
```

## Configuration (`.env`)

```bash theme={null}
# Shared
EMBEDDING_PROVIDER=openai
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_ENDPOINT=http://localhost:11434   # only used by the ollama embedding provider
OPENAI_API_KEY=your-key
LLM_PROVIDER=anthropic
LLM_MODEL=claude-sonnet-4-6
ANTHROPIC_API_KEY=your-key
TOP_K=5

# Qdrant
QDRANT_HOST=localhost
QDRANT_PORT=6333
QDRANT_COLLECTION=financial_documents

# Weaviate
WEAVIATE_HOST=localhost
WEAVIATE_PORT=8079
WEAVIATE_GRPC_PORT=50051
WEAVIATE_SCHEME=http
WEAVIATE_API_KEY=            # leave empty for anonymous access
WEAVIATE_CLASS=FinancialDocuments

# Milvus
MILVUS_HOST=localhost
MILVUS_PORT=19530
MILVUS_COLLECTION=financial_documents

# Chroma
CHROMA_HOST=localhost
CHROMA_PORT=8000
CHROMA_COLLECTION=financial_documents

# pgvector
PG_HOST=localhost
PG_PORT=5432
PG_DATABASE=datris
PG_USER=postgres
PG_PASSWORD=postgres
PG_SCHEMA=public
PG_TABLE=financial_documents
```


# Market Macro Agent
Source: https://docs.datris.ai/examples/market-macro-agent

Market macro agent example

**Location:** `examples/market-macro-agent/`

A real-time financial data pipeline agent powered by Datris. MacroAgent acts as a pipeline orchestrator: it connects to the Datris MCP server, provisions four platform-owned taps (FRED, equities, crypto, SEC filings), triggers them to refresh data, and answers market questions grounded in actual numbers. The Datris platform does the fetching; the agent decides when and what to refresh. Everything runs in the browser with a live activity feed.

100% Python — FastAPI backend, vanilla JS frontend, no Node.js required.

## Architecture

```
Browser (vanilla JS)
  │
  ├── GET  /stream/state  (SSE — pipeline tiles, activity feed, row counts)
  │
  └── POST /chat          (SSE — tool calls, partial text, answers)
           │
           └── agent/loop.py
                 │
                 ├── Anthropic API (Claude)
                 │     ↕ tool_use / end_turn
                 │
                 ├── agent/executor.py
                 │     └── MCP tools → Datris MCP Server (port 3000)
                 │           (run_tap, get_pipeline_status, query_postgres,
                 │            list_taps, get_tap_logs, …)
                 │
                 └── agent/scheduler.py → run_tap every N minutes

              ┌──────────────────────────────────────┐
              │ Datris Platform                      │
              │  Tap runner (Docker sandbox)         │
              │    fred_tap → FRED API               │
              │    equities_tap → yfinance           │
              │    crypto_tap → CoinGecko            │
              │    sec_tap → SEC EDGAR               │
              │           ▼                          │
              │   Pipelines (Postgres destinations)  │
              └──────────────────────────────────────┘
```

## How It Works

1. **On startup** — Connects to the Datris MCP server via SSE, discovers tools via `tools/list`, loads the server's instructions and resources, pushes `FRED_API_KEY` from `.env` into Datris as a tap secret (`create_tap_secret`), and provisions four taps (`fred_tap`, `equities_tap`, `crypto_tap`, `sec_tap`), each wired to its own pipeline. Idempotent: re-running against an existing install is a no-op. Taps are created with scheduling disabled so the agent's own loop drives refreshes.
2. **User asks a question** — MacroAgent determines which data family is relevant
3. **Refresh** — The agent calls `run_tap`. The platform runs the tap's Python `fetch()` in a Docker sandbox and submits records into the pipeline
4. **Wait for ingestion** — Ingestion is asynchronous; the agent polls `get_pipeline_status` with the run's `publisherToken` until every row's state is `end` or `error`
5. **Query** — The agent queries the landed data via `query_postgres` and answers grounded in actual numbers
6. **Background refresh** — Every `REFRESH_INTERVAL_MINUTES` the scheduler calls `run_tap` for every pipeline that has been exercised at least once
7. **Tap health** — Ask about failures and the agent calls `get_tap_logs` to report the last runs and any errors

## Data Sources

| Source | Data | Auth |
| - | - | - |
| FRED | Macro series — yields, VIX, CPI, unemployment, credit spreads | [Free API key](https://fred.stlouisfed.org/docs/api/api_key.html) |
| yfinance | Equity / ETF OHLCV (SPY, QQQ, GLD, TLT, XLE, IWM) | None |
| CoinGecko | Crypto prices, market cap, volume (BTC, ETH, SOL) | None (30 req/min) |
| SEC EDGAR | 10-K / 10-Q filings | None (10 req/sec) |

## Quick Start

```bash theme={null}
cd examples/market-macro-agent
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env → add ANTHROPIC_API_KEY (and optionally FRED_API_KEY)

# Make sure Datris is running:
cd ../../ && docker compose up -d && cd examples/market-macro-agent

uvicorn main:app --reload --port 8001
```

Open **[http://localhost:8001](http://localhost:8001)**

## Demo Queries

* `"What's the current macro picture?"` — runs the FRED + equities taps, then queries
* `"Refresh all taps"` — triggers `run_tap` across all four data families
* `"Is crypto confirming the risk-on trade in equities?"` — cross-pipeline analysis
* `"Show me the last few tap runs"` — exercises `get_tap_logs` live

## Environment Variables

| Variable | Default | Description |
| - | - | - |
| `ANTHROPIC_API_KEY` | — | **Required** |
| `MODEL` | `claude-sonnet-4-6` | Claude model |
| `PORT` | `8001` | uvicorn port |
| `MCP_SERVER_URL` | `http://localhost:3000/sse` | Datris MCP server SSE endpoint |
| `FRED_API_KEY` | — | [Get one free](https://fred.stlouisfed.org/docs/api/api_key.html) |
| `REFRESH_INTERVAL_MINUTES` | `15` | Background refresh cycle |

## Requirements

* Python 3.11+
* [Datris Platform](https://github.com/datris/datris-platform-oss) running via Docker
* Anthropic API key


# Preprocessor
Source: https://docs.datris.ai/examples/preprocessor

Preprocessor example

**Location:** `examples/preprocessor/`

A Flask REST API that demonstrates the pipeline's preprocessor capability. The preprocessor is called before data quality and transformation, allowing you to modify, enrich, or filter data before the pipeline processes it. Supports both synchronous and asynchronous modes.

## Endpoints

* `POST http://localhost:5500/preprocess/sync` — Process data synchronously and return the result
* `POST http://localhost:5500/preprocess/async` — Process data asynchronously and POST results back via callback

## Setup

```bash theme={null}
cd examples/preprocessor
pip install flask requests
```

## Run

```bash theme={null}
python app.py    # runs on port 5500
```

## Usage

Configure a pipeline's `source.preprocessor` with the endpoint URL. The pipeline sends data to your preprocessor before ingestion, allowing custom transformations.

See [Preprocessor](/preprocessor) for the full preprocessor protocol and configuration reference.


# Topic Subscriber
Source: https://docs.datris.ai/examples/topic-subscriber

Topic subscriber example

**Location:** `examples/topic-subscriber/`

An ActiveMQ/STOMP consumer that subscribes to pipeline notification events. Use this to trigger downstream workflows when pipelines complete processing.

## Features

* Durable subscription with automatic reconnection
* Supports Virtual Topic and Durable Topic subscription styles
* Graceful shutdown on SIGINT/SIGTERM
* Logs to both stdout and file (`notification_consumer.log`)

## Configuration (`.env`)

```bash theme={null}
ACTIVEMQ_HOST=localhost
ACTIVEMQ_PORT=61613
ACTIVEMQ_USER=admin
ACTIVEMQ_PASSWORD=admin
ACTIVEMQ_TOPIC=oss-pipeline-notification
USE_VIRTUAL_TOPIC=true    # true = Virtual Topic queue, false = durable topic subscription
ACTIVEMQ_CLIENT_ID=my-subscriber
```

## Setup

```bash theme={null}
cd examples/topic-subscriber
pip install stomp.py python-dotenv
```

## Run

```bash theme={null}
python app.py
```

The subscriber prints pipeline processing notifications as they arrive, including pipeline name, destination, and pipeline token.

See [Notifications](/notifications) for how pipeline notifications work.


# Incidents & Recovery Agent
Source: https://docs.datris.ai/incidents

The platform opens an incident when a data flow breaks, diagnoses it, and — within limits you set — repairs and verifies it.

With the recovery agent on, Datris stops waiting for you to notice a broken data flow. When a scheduled tap finally fails (after the platform's own retries), a pipeline job errors, a tap goes stale, or a pipeline's volume swings hard, the platform opens an **incident**, diagnoses it with the same judgment as the Ops chat, proposes a fix, passes every action through the [agent policy](/agent-policy), executes what it may, **verifies the fix with a real run — reverting it if verification fails** — and records the whole story.

The loop is: observe → diagnose → propose → gate → execute → verify → record. The AI proposes; deterministic platform code decides. Limits, verification and the revert are code the model cannot talk its way past.

## Enabling it

Requires the [agent policy](/agent-policy) (`USE_AGENT_POLICY=true`) for its approval flow.

1. Add to your `.env`:

   ```bash theme={null}
   RECOVERY_AGENT_ENABLED=true
   # optional: JSON POST on incident open / awaiting_approval / resolved / failed / abandoned
   # (also receives doctor_error / doctor_recovered when DOCTOR_INTERVAL_MINUTES is set)
   INCIDENT_WEBHOOK_URL=
   ```

2. Recreate the Datris container:

   ```bash theme={null}
   docker compose up -d --force-recreate datris
   ```

3. Set the **recovery mode** in Configuration → Agent Policy. It ships as `off`, so nothing changes until you choose:

| Mode | Behavior |
| - | - |
| `off` | Today's behavior: failures get an AI suggestion in Run History, nothing more |
| `propose` | Incidents open and are diagnosed; **every** proposed action queues under Activity → Approvals — nothing executes without a person |
| `autopilot` | Actions follow your per-action policy matrix (`auto` runs, `approve` queues, `deny` refuses) |

A per-tap or per-pipeline override can put one resource on a different mode — one critical tap on `propose` while everything else runs `autopilot`, or the reverse.

## What opens an incident

| Signal | When |
| - | - |
| Tap failure | A scheduled run fails **after** the retry ladder is exhausted |
| Pipeline failure | A pipeline job ends in error |
| Stale tap | A scheduled tap hasn't run within 2× its cadence |
| Volume anomaly | A pipeline's records in the current 24-hour window are ±60 % vs the previous 24-hour window (with at least 3 runs in that prior window) |

Guards, all enforced in code: one open incident per resource; a resource whose latest run is healthy never gets one; a cooldown (default 6 h) after a failed or abandoned attempt; a cap on simultaneous open incidents. Volume incidents are **diagnosis-only** — the value is the triage, not an action.

## What the agent may do

During diagnosis it holds **read-only tools** — it cannot change anything while investigating. Among them is the [doctor](/doctor), so when a failure is really the platform (a missing secret, a model or embedding service not answering, a full disk) it can say so and hand the incident to a human with the fix, instead of re-running a tap that will fail the same way. Its proposal may only contain: re-run the tap, fix the tap script (then test it), or "this needs a human" with an explanation. Deletes, schema migrations and secret changes are structurally impossible for it. Script fixes land as a new version, so history is never lost.

**Verification is mandatory**: after acting, the tap must complete a real run healthily, and when it feeds a pipeline the load must finish without error. If verification fails, the platform reverts the script to the pre-incident version, closes the incident as failed, and starts the cooldown.

Every call it makes runs as its own `recovery-agent` API key and is tagged with the incident id, so with the audit log on, the [audit log](/audit-log) is the incident's complete ledger ([strict mode](/audit-log#strict-mode) guarantees no entry is dropped) — and revoking that key stops the agent entirely, independent of any flag.

## Limits

Set under the agent policy's `recovery` section; enforced by the runner, not the prompt:

| Limit | Default |
| - | - |
| AI calls per incident | 12 |
| Actions per incident | 3 |
| Runtime per incident | 15 minutes |
| Simultaneous open incidents | 10 |
| Cooldown after a failure | 6 hours |

## Watching it

* **Activity → Incidents**: every incident with its state, classification, step-by-step narrative, and any approvals it waits on; a person can abandon one at any time.
* **Ops chat**: incidents the platform is working appear in the assistant's context — ask "what happened to the prices tap?" and it explains the incident record instead of re-diagnosing.
* **Agents**: `list_incidents` / `get_incident` MCP tools (read-only — only the platform opens incidents).
* **Metrics**: `datris_incidents_total{kind,outcome}` and `datris_incident_duration_seconds` on the Prometheus endpoint.

## Related

* [Agent Policy](/agent-policy) — the gate every recovery action passes through
* [Audit Log](/audit-log) — the incident's ledger, joined on the incident id


# Datris
Source: https://docs.datris.ai/index

The data control plane for AI agents

[datris.ai](https://datris.ai)

Agents ask Datris for data. Datris finds it, acquires it, validates it, lands it in the stores you already run, and returns it with provenance — over MCP, without ever holding your keys. Developers get the same platform through JSON configuration, the UI, the CLI, and the REST API.

**New here?** [Install in minutes](/installation) → [Quick Start](/quick-start) → [Connect your agent](/configuring-claude) → [Give it the Datris skill](/agent-skill).
**Going to production?** [Security architecture](/production/security-architecture) → [Upgrades & supported versions](/production/upgrades) → [Licensing & support](/production/licensing-support).
**Building?** [Taps](/taps) → [Pipelines](/pipeline-configuration) → [Destinations](/destinations/postgres) → [API reference](/api-reference/pipeline-api).

Datris sits beside your warehouse and lake; it doesn't replace them. It runs on-prem, in any cloud, or on your laptop — anywhere Docker runs — on open-source infrastructure with no proprietary services and no vendor lock-in. There is no managed service; your deployment lives inside your perimeter.

### One Control Plane, Every Agent

Your agents already acquire, validate, and load data. Without a control plane, they do it badly: credentials scattered across configs and prompts, the chat log as the only record, generated scripts that lose state when regenerated. Datris puts that work behind one governed surface:

* **One MCP door** — 73 capabilities behind a single [MCP server](/mcp-server) (stdio and SSE). Claude, Cursor, OpenClaw, and any MCP-compatible agent learn one interface instead of 73 integrations
* **Vault-brokered credentials** — the agent references a secret by name; it never holds a key. Agent-written code runs in an [isolated container](/tap-execution-isolation) with no keys inside
* **Every run recorded** — job state and row counts for every run, provenance on its rows with stamping on (see [strict mode](/provenance#strict-mode)); every generated script [versioned in git](/tap-github-storage)
* **Durable state** — pipelines and [sync bookmarks](/taps) live in the platform, not in the agent's context, so a regenerated script picks up where the old one left off
* **Explain and repair** — when a run fails, [AI Error Explanation](/ai-error-explanation) returns the root cause in plain English instead of a stack trace

### The Operating Loop

Same loop every run, and with the audit log on the same audit trail every time ([strict mode](/audit-log#strict-mode) makes it complete) — so the agent's job is reasoning about the data, not improvising infrastructure.

| Step | What Datris does |
| - | - |
| **Acquire** | [Taps](/taps) — AI-generated scripts pull from APIs, websites, files, or databases, on demand or on a schedule |
| **Validate** | Plain-English [AI rules](/data-quality/ai-rules) plus JSON/XML schema validation; failures abort the run or are logged as warnings — your call per rule |
| **Land** | Multi-destination [pipelines](/pipeline-configuration) write to warehouses, databases, object stores, queues, and vector stores in parallel |
| **Observe** | Job state, provenance, and the [Data Catalog](/data-catalog) — queryable by agents and humans over MCP and REST |
| **Explain & Repair** | [AI Error Explanation](/ai-error-explanation) diagnoses failures; the Assistant proposes and applies the fix |

## AI-Powered Features

Intelligence at every stage — from ingestion to delivery, Datris makes data engineering accessible through natural language.

* **MCP server (AI agent integration)** - Built-in [MCP](https://modelcontextprotocol.io) server lets AI agents (Claude, Cursor, OpenClaw, custom frameworks) natively interact with the pipeline — register pipelines, upload files, trigger jobs, profile data, run semantic searches, and query databases. Supports stdio and SSE transports
* **AI-powered data quality** - Validate with plain English rules via `aiRule`. Examples:
  * *"Validate that all email addresses are properly formatted and all phone numbers contain 7–15 digits"*
  * *"Ensure price is positive, quantity is a whole number, and discount never exceeds price"*
  * *"Check that start\_date is before end\_date and both are in YYYY-MM-DD format"*
  * *"Verify that country codes are valid ISO 3166-1 alpha-2 codes"*
  * *"Flag any row where revenue minus cost does not equal profit within a 0.01 tolerance"*
* **AI transformations** - Describe transformations in natural language. Examples:
  * *"Convert all dates to YYYY-MM-DD format and normalize phone numbers to E.164"*
  * *"Categorize transactions as 'small' under $100, 'medium' $100–$1000, or 'large' over $1000"*
  * *"Extract city and state from the address column into separate columns"*
  * *"Standardize company names — remove Inc, LLC, Corp suffixes and trim whitespace"*
  * *"Convert all currency amounts from EUR to USD using a rate of 1.08"*
* **Assistant** - Conversational tap and pipeline creation. Describe a source ("yfinance daily prices for the S\&P 500") and Datris generates the tap script, builds the pipeline, and runs it — grouped into a [Data Catalog](/data-catalog) for organization. The fastest way to onboard a new external data source
* **Taps** - AI-generated Python scripts that fetch data from external sources (APIs, web scraping, databases) and push it into pipelines. Describe what data you want in plain English, and Datris generates the script. Includes AI diagnosis when scripts fail, CRON scheduling, and credentials management via Vault secrets
* **Datris CLI** - Command-line interface for ingesting data, running queries, and managing pipelines. `datris ingest data.csv --ai-validate "prices > 0" --ai-transform "convert dates to YYYY/MM/DD"`
* **AI schema generation** - Upload any data and receive a complete, ready-to-register pipeline configuration — field names and types inferred automatically
* **AI data profiling** - Upload a data and get summary statistics, quality issues, and suggested validation rules — all powered by AI analysis
* **AI error explanation** - When jobs fail, AI analyzes the error chain and explains the root cause in plain English. No more digging through stack traces
* **AI providers** - Anthropic Claude (choose any model), OpenAI (chose any model, plus embeddings models), Azure OpenAI (the same OpenAI models served from your Azure resource), Amazon Bedrock (Claude through your AWS account with IAM auth), Grok (xAI's models through their OpenAI-compatible API), or local models via Ollama (Llama, Mistral, Phi). No vendor lock-in — switch providers without changing your pipeline config

## RAG Pipeline

Full RAG pipeline built in. Extract, chunk, embed, and upsert documents into any major vector database — build retrieval-augmented generation workflows without leaving your pipeline.

* **5 vector databases** - Qdrant, Weaviate, Milvus, Chroma, pgvector (PostgreSQL)
* **Chunking strategies** - Fixed-size, sentence, paragraph, recursive
* **Embedding providers** - OpenAI (the recommended default when you have an OpenAI key), Azure OpenAI (an embedding deployment on your Azure resource), bundled TEI (`bge-m3` — runs locally, no API key), or Ollama (local models)
* **Document extraction** - PDF, Word, PowerPoint, Excel, HTML, email, EPUB, plain text

## Key Features

* **Configuration-driven** - Define pipelines entirely via MCP, the Datris UI, or directly with JSON. Extend it with AI instructions for data quality and transformations. You even have the option of defining your own preprocessor via a REST endpoint
* **Multiple ingestion methods** - MCP, data upload API, MinIO bucket events, database polling, Kafka streaming
* **Data quality** - AI rules (LLM-generated), and/or JSON/XML schema validation
* **Transformations** - AI transformations (LLM-generated), destination schema (drop/rename/retype columns) for structure data
* **Multiple destinations** - Write to PostgreSQL, MongoDB, Snowflake, Databricks, Kafka, ActiveMQ, REST endpoints, Qdrant, Weaviate, Milvus, Chroma, pgvector, or MinIO/S3 (Parquet, ORC, or Apache Iceberg tables) in parallel
* **Event notifications** - Subscribe to pipeline processing events via ActiveMQ topics

## Architecture

The plane Datris controls: every source, every destination, every AI call, every job — one MCP surface, one audit trail, one place credentials live. Datris owns acquisition, validation, storage, and observability so your agents can focus on reasoning, not integration. Humans get the same plane through the UI, CLI, and REST API.

Self-hosted on proven open-source infrastructure — no proprietary services, no vendor lock-in, no surprise bills:

| Service | Purpose |
| - | - |
| **MinIO** | S3-compatible object store for file staging and data output |
| **MongoDB** | Configuration store, job status tracking, metadata |
| **ActiveMQ** | File notification queue, pipeline event notifications |
| **HashiCorp Vault** | Secrets management (database credentials, API keys) |
| **Apache Kafka** | Optional streaming source and destination |
| **Apache Spark** | Local Spark for writing Parquet, ORC, and Iceberg tables to MinIO |

## Processing Flow

```
Source (Data Upload / MinIO Event / Database Pull / Kafka)
  |
  v
Preprocessor (optional REST endpoint)
  |
  v
Data Quality (AI rules, header validation, schema validation)
  |
  v
Transformation (AI transformations)
  |
  v
Destinations (executed in parallel)
  ├── Object Store (MinIO or AWS S3 - Parquet, ORC)
  ├── PostgreSQL (COPY bulk insert)
  ├── MongoDB (document upsert)
  ├── Snowflake (PUT + COPY INTO / MERGE into the user's account)
  ├── Databricks (Unity Catalog managed Delta tables via SQL warehouse)
  ├── Kafka (topic producer)
  ├── ActiveMQ (queue)
  ├── REST Endpoint (HTTP POST)
  ├── Qdrant (vector database - chunking, embeddings, RAG)
  ├── Weaviate (vector database - chunking, embeddings, RAG)
  ├── Milvus (vector database - chunking, embeddings, RAG)
  ├── Chroma (vector database - chunking, embeddings, RAG)
  └── pgvector (PostgreSQL vector database - chunking, embeddings, RAG)
  |
  v
Notifications (published to ActiveMQ topic)
```

## Retrieval Flow

```
Client (AI Agent via MCP / Developer via REST API)
  |
  v
Interface
  ├── MCP Server (stdio or SSE)
  └── REST API (POST /api/v1/query/* and /api/v1/search/*)
  |
  v
Query Source
  ├── PostgreSQL (read-only SQL SELECT queries)
  ├── MongoDB (document queries with filters and projections)
  ├── Snowflake (pipeline-scoped read-only SQL: SELECT, SHOW, DESCRIBE)
  ├── Databricks (pipeline-scoped read-only SQL: SELECT, SHOW, DESCRIBE)
  ├── Qdrant (semantic search)
  ├── Weaviate (semantic search)
  ├── Milvus (semantic search)
  ├── Chroma (semantic search)
  ├── pgvector (semantic search via PostgreSQL)
  ├── Pipeline Configurations (list/get)
  ├── Job Status (by pipeline token or pipeline name)
  ├── AI Schema Generation (from uploaded files)
  └── AI Data Profiling (statistics, quality issues, suggested rules)
```

## Supported Data Formats

| Format | Input | Output |
| - | - | - |
| CSV | Configurable delimiter, header, encoding | Parquet, ORC, database, Kafka, ActiveMQ |
| JSON | Single object or NDJSON (one per line) | MongoDB, Kafka, REST |
| XML | Single document or one per line | Database, Kafka, REST |
| Excel (XLS) | Worksheet selection, auto-CSV conversion | Same as CSV |
| Unstructured | PDF, Word, PowerPoint, Excel, HTML, email, EPUB, text | Object store, Qdrant, Weaviate, Milvus, Chroma, pgvector |
| Archives | .zip, .tar, .gz, .jar | Extracted and processed individually |

## Quick Links

* [Installation](/installation) - Get running with Docker Compose
* [Quick Start](/quick-start) - End-to-end walkthrough
* [Data Catalog](/data-catalog) - Organize related taps and pipelines into named groups
* [Datris CLI](/cli) - Command-line interface for ingesting, querying, and managing pipelines
* [Pipeline Configuration](/pipeline-configuration) - Full JSON configuration reference
* [Preprocessor](/preprocessor) - External preprocessing via REST endpoints
* [API Reference](/api-reference/pipeline-api) - REST API documentation
* [AI Schema Generation](/api-reference/schema-generation-api) - Generate pipeline configs from files using AI
* [AI Configuration](/ai-configuration) - Configure AI providers (Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, Grok, Ollama)
* [AI Data Quality](/data-quality/ai-rules) - Natural language validation
* [AI Transformation](/transformation/ai-transformation) - Natural language transformation
* [AI Data Profiling](/ai-data-profiling) - Profile data files and get recommended rules
* [AI Error Explanation](/ai-error-explanation) - Automatic plain-English error analysis
* [Qdrant Destination](/destinations/qdrant) - Vector database for RAG with chunking, embeddings, and metadata
* [Weaviate Destination](/destinations/weaviate) - Vector database for RAG with chunking, embeddings, and metadata
* [Milvus Destination](/destinations/milvus) - Scalable vector database for RAG
* [Chroma Destination](/destinations/chroma) - Lightweight vector database for RAG — single container
* [pgvector Destination](/destinations/pgvector) - PostgreSQL vector database for RAG — no separate server required
* [Query API](/api-reference/query-api) - Query PostgreSQL and MongoDB via REST API
* [Search API](/api-reference/search-api) - Semantic search across vector databases via REST API
* [MCP Server](/mcp-server) - AI agent integration via Model Context Protocol
* [Example Applications](/examples) - Vector store chat, Kafka loader, preprocessor, and more


# Supported Data Types
Source: https://docs.datris.ai/ingestion/data-types

Supported data types for ingestion

This reference lists every data type the pipeline supports, along with mappings to PostgreSQL and Spark types used during storage and transformation. The external warehouse destinations have their own DDL mappings from the same pipeline types — see the type-mapping tables on the [Snowflake](/destinations/snowflake#type-mapping) and [Databricks](/destinations/databricks#type-mapping) destination pages.

## Type Reference

The **PostgreSQL Type** column shows the literal type Datris emits in the generated `CREATE TABLE` DDL. Datris remaps only a few types to their canonical Postgres aliases (`tinyint`/`smallint` → `int2`, `float` → `float4`, `double` → `float8`, `string` → `text`); every other type is emitted verbatim.

| Pipeline Type | Description | PostgreSQL Type | Spark Type |
| - | - | - | - |
| `boolean` | True or false | `boolean` | `BooleanType` |
| `tinyint` | 8-bit signed integer (-128 to 127) | `int2` | `ByteType` |
| `smallint` | 16-bit signed integer (-32768 to 32767) | `int2` | `ShortType` |
| `int` | 32-bit signed integer | `int` | `IntegerType` |
| `bigint` | 64-bit signed integer | `bigint` | `LongType` |
| `float` | 32-bit IEEE 754 floating point | `float4` | `FloatType` |
| `double` | 64-bit IEEE 754 floating point | `float8` | `DoubleType` |
| `decimal(p,s)` | Fixed-precision number with `p` total digits and `s` fractional digits | `decimal(p,s)` | `DecimalType(p,s)` |
| `string` | Variable-length text, unbounded | `text` | `StringType` |
| `varchar(n)` | Variable-length text up to `n` characters | `varchar(n)` | `StringType` |
| `char(n)` | Fixed-length text of exactly `n` characters | `char(n)` | `StringType` |
| `date` | Calendar date without time | `date` | `DateType` |
| `timestamp` | Date and time with microsecond precision | `timestamp` | `TimestampType` |

## Precision and Scale for Decimal

The `decimal(p,s)` type requires two parameters:

* **p (precision)**: total number of digits, range 1 to 38.
* **s (scale)**: number of digits to the right of the decimal point, range 0 to p.

Examples:

| Declaration | Stores | Max Value |
| - | - | - |
| `decimal(5,2)` | Up to 5 digits, 2 after the decimal | 999.99 |
| `decimal(10,0)` | Up to 10 integer digits, no fractional part | 9999999999 |
| `decimal(18,6)` | 18 total digits, 6 fractional | 999999999999.999999 |

## Integer Type Selection

Choose the narrowest integer type that fits your data to reduce storage and improve Spark performance:

| Type | Byte Size | Range |
| - | - | - |
| `tinyint` | 1 | -128 to 127 |
| `smallint` | 2 | -32,768 to 32,767 |
| `int` | 4 | -2,147,483,648 to 2,147,483,647 |
| `bigint` | 8 | -9.2 x 10^18 to 9.2 x 10^18 |

## String Type Selection

| Type | Use When |
| - | - |
| `string` | Maximum length is unknown or varies widely |
| `varchar(n)` | A known upper bound exists and you want the database to enforce it |
| `char(n)` | Values are always exactly `n` characters (e.g., ISO country codes, fixed-format identifiers) |

All three string types map to `StringType` in Spark. The length constraint is enforced only at the PostgreSQL layer.

## JSON and XML Special Types

JSON and XML pipelines are semi-structured: the source schema must contain **exactly one field**, named literally `_json` or `_xml` (matching the source file type), with type `string`. The validator rejects any schema with a different field count or a different field name.

| Field name | Behavior |
| - | - |
| `_json` | The whole record is treated as a nested JSON document. Stored as a document in MongoDB (default destination for JSON), or as a native `json` column in PostgreSQL. No schema validation is applied to the nested content. |
| `_xml` | The whole record is treated as an XML document. Stored as a native `xml` column in PostgreSQL, or as a string in MongoDB. |

Declare the single field with type `string` in the schema:

```json theme={null}
{
  "fields": [
    { "name": "_json", "type": "string" }
  ]
}
```

For XML, use the same shape with a single `_xml` field. During ingestion, the pipeline preserves the raw content without attempting to parse it into individual columns. This is useful for semi-structured data that should be queried with JSON or XML functions downstream.

## Type Coercion

During ingestion, the pipeline converts field values from strings to the declared type:

1. **Null or empty string** — passed through unchanged (no type conversion is applied). The Postgres `COPY` layer then converts an empty field to `NULL`.
2. **Numeric types** (`int`, `bigint`, `float`, `double`, `decimal`, etc.) — the value is parsed using Scala's built-in conversion. `decimal` is parsed via `toDouble` in the row-map path. If parsing fails (e.g., `"abc"` in an `int` column), the job fails with an error.
3. **String types** (`string`, `varchar`, `char`) — stored as-is with no conversion.
4. **Date and timestamp** — stored as the raw string value. The destination (PostgreSQL, Spark) handles date/timestamp parsing according to its own rules.


# Database Source Pulling
Source: https://docs.datris.ai/ingestion/database-pull

Pull data from database sources

The pipeline can pull data directly from relational databases on a schedule. Configure a database source, set a cron expression, and the pipeline handles connection management, incremental extraction, and secret retrieval from Vault.

## Supported Databases

| Database | JDBC Driver |
| - | - |
| PostgreSQL | `org.postgresql.Driver` |
| MySQL | `com.mysql.cj.jdbc.Driver` |
| MSSQL | `com.microsoft.sqlserver.jdbc.SQLServerDriver` |

<Note>
  The PostgreSQL and MySQL drivers ship with the server. The Microsoft SQL Server JDBC driver is not bundled — to use an MSSQL source, add the `mssql-jdbc` jar to the server classpath (for example by adding it to `build.sbt` and rebuilding the image).
</Note>

## Configuration

Database sources are configured in the `databaseAttributes` section of a pipeline configuration. Connection credentials (JDBC URL, username, password) are stored in Vault — not in the pipeline config itself.

```json theme={null}
{
  "name": "customer_sync",
  "source": {
    "databaseAttributes": {
      "type": "postgres",
      "postgresSecretsName": "oss/postgres",
      "database": "production",
      "schema": "public",
      "table": "customers",
      "cronExpression": "0 */15 * * * ?",
      "timestampFieldName": "updated_at",
      "includeFields": ["id", "email", "name", "updated_at", "status"]
    },
    "schemaProperties": {
      "fields": [
        { "name": "id", "type": "bigint" },
        { "name": "email", "type": "varchar(255)" },
        { "name": "name", "type": "varchar(200)" },
        { "name": "updated_at", "type": "timestamp" },
        { "name": "status", "type": "varchar(20)" }
      ]
    }
  },
  "destination": {
    "database": {
      "dbName": "datris",
      "schema": "public",
      "table": "customers",
      "usePostgres": true
    }
  }
}
```

## Configuration Reference

| Property | Required | Description |
| - | - | - |
| `type` | Yes | One of `postgres`, `mysql`, `mssql` |
| `postgresSecretsName` | Conditional | Vault secret name for PostgreSQL credentials |
| `mysqlSecretsName` | Conditional | Vault secret name for MySQL credentials |
| `mssqlSecretsName` | Conditional | Vault secret name for MSSQL credentials |
| `cronExpression` | Yes | Quartz-format cron expression controlling the pull schedule |
| `database` | No | Database name; when set, the generated query references the table as `database.schema.table` |
| `schema` | No | Schema within the database |
| `table` | Yes\* | Table to pull data from (\*unless `sqlOverride` is set) |
| `timestampFieldName` | Yes\* | Column used for incremental pulls. Effectively required — the generated query always appends this column and orders by it. |
| `includeFields` | No | Array of column names to select; when omitted, the column list is taken from the pipeline's `source.schemaProperties.fields` (not `SELECT *`) |
| `sqlOverride` | No | Custom SQL query that replaces the generated SELECT statement |
| `outputDelimiter` | No | Delimiter for CSV output (default `,`) |

## Secrets in Vault

Database credentials are never stored in the pipeline configuration. Instead, the pipeline reads them from HashiCorp Vault using the secret name configured above.

The Vault secret must contain `username`, `password`, and `jdbcUrl` keys:

```json theme={null}
{
  "username": "pipeline_reader",
  "password": "s3cureP@ss",
  "jdbcUrl": "jdbc:postgresql://db.internal.example.com:5432/production"
}
```

Store the credentials in Vault using the CLI:

```bash theme={null}
vault kv put secret/oss/postgres \
  username="pipeline_reader" \
  password="s3cureP@ss" \
  jdbcUrl="jdbc:postgresql://db.internal.example.com:5432/production"
```

## Cron-Based Scheduling

The `cronExpression` field accepts a Quartz cron string with six fields (seconds, minutes, hours, day-of-month, month, day-of-week):

| Expression | Schedule |
| - | - |
| `0 */15 * * * ?` | Every 15 minutes |
| `0 0 * * * ?` | Every hour on the hour |
| `0 0 2 * * ?` | Daily at 02:00 |
| `0 0 0 ? * MON` | Every Monday at midnight |

## Incremental Pulls

Incremental extraction relies on `timestampFieldName`, which is effectively required: the generated `SELECT` always appends that column to the projection and orders the result by it (`ORDER BY {timestampFieldName}`). After each pull, the pipeline records the value of that column from the **last (highest-ordered) row returned** — this is the high-water mark. It is not computed with a `MAX()` aggregate; it is simply the timestamp of the final row in the ordered result set.

On the next execution, the pipeline adds a `WHERE {timestampFieldName} > {lastValue}` clause to fetch only new or updated rows. The high-water mark is stored in MongoDB in the `{environment}-data-pull` collection.

## Custom SQL

Set `sqlOverride` to run an arbitrary query instead of a simple `SELECT ... FROM table`:

```json theme={null}
{
  "databaseAttributes": {
    "type": "postgres",
    "postgresSecretsName": "oss/postgres",
    "sqlOverride": "SELECT o.id, o.total, c.email FROM orders o JOIN customers c ON o.customer_id = c.id",
    "cronExpression": "0 */30 * * * ?"
  }
}
```

## Field Filtering

Use `includeFields` to select a subset of columns from the source table:

```json theme={null}
{
  "databaseAttributes": {
    "table": "users",
    "includeFields": ["id", "username", "email", "created_at"]
  }
}
```

When `sqlOverride` is set, `includeFields` is ignored because the SQL query already defines the column list.

## Troubleshooting

| Symptom | Check |
| - | - |
| Connection refused | Verify the `jdbcUrl` in Vault is correct and the database allows connections from the pipeline host. |
| Authentication failed | Confirm the Vault secret name is correct and contains valid `username`/`password`/`jdbcUrl` keys. |
| Zero rows returned on incremental pull | The high-water mark in MongoDB may already be ahead of the data. Delete the entry in the `{environment}-data-pull` collection to reset it. |
| SQL syntax error | When using `sqlOverride`, test the query directly against the database first. |


# API File Upload
Source: https://docs.datris.ai/ingestion/file-upload

Ingest data via API file upload

Upload data files directly to the pipeline through the REST API. The pipeline accepts any file type (CSV, JSON, XML, Excel, PDF, Word, PowerPoint, HTML, email, EPUB, plain text, and archives), handles decompression automatically, and returns a tracking token.

## Endpoint

```
POST /api/v1/pipeline/upload
```

## Request

The request uses `multipart/form-data` encoding with the following parts:

| Part | Required | Description |
| - | - | - |
| `file` | Yes | The data file to upload |
| `pipeline` | Yes | Name of the target pipeline configuration |
| `publishertoken` | No | Opaque token identifying the data publisher; used for lineage tracking |

### Example

```bash theme={null}
curl -X POST "http://localhost:8080/api/v1/pipeline/upload" \
  -F "file=@transactions_2025.csv" \
  -F "pipeline=transactions" \
  -F "publishertoken=finance-team-a"
```

## Response

A successful upload returns HTTP 200 with a **bare string** body — the pipeline token itself (a unique UUID generated for the job), not a JSON object:

```
d4f8e1a2-7b3c-4e9f-a5d6-1c2b3e4f5a6b
```

Use this token to track the job's processing status, view errors, or cancel the job.

When a compressed archive is processed on the per-file path (non-CSV pipelines, or a single-file archive), the response is instead the literal string:

```
3 file(s) submitted
```

## Compressed Files

When the uploaded file has a compressed extension, the archive is extracted **in memory** and processed **synchronously** — nothing is staged to MinIO:

| Extension | Handling |
| - | - |
| `.zip` | Extracted in memory |
| `.gz` | Decompressed in memory (gunzip only — a `.tar.gz` is not untarred; the inner `.tar` is submitted as a single file) |
| `.tar` | Extracted in memory |
| `.jar` | Extracted in memory |

Compressed archives may contain multiple data files. Handling depends on the pipeline type:

* **CSV pipelines** — when an archive contains multiple inner files, they are concatenated into a single job. The header is kept from the first file and stripped from the 2nd and later files. The response is the single pipeline token.
* **Non-CSV pipelines, or single-file archives** — each inner file is submitted as its own job. The response is always the `"N file(s) submitted"` string on this path, even for a single file (`1 file(s) submitted`) — the pipeline token is not returned.

## Uncompressed Files

Files without a recognized compressed extension (e.g., `.csv`, `.json`, `.xml`, `.xls`, `.pdf`, `.docx`, `.txt`, `etc`) are read into memory and submitted directly. They are not staged to MinIO.

## Processing Flow

1. The client sends the multipart request.
2. The endpoint validates that the named `pipeline` is registered.
3. The file bytes are read into memory and the extension is inspected.
4. **Compressed path**: the archive is decompressed in memory. For CSV pipelines with multiple inner files, the files are concatenated (headers stripped after the first) and submitted as one job; otherwise each inner file is submitted individually.
5. **Uncompressed path**: the file contents are passed directly to the ingestion pipeline.
6. The endpoint returns the pipeline token (or the `"N file(s) submitted"` string).

## Error Responses

| HTTP Status | Cause |
| - | - |
| 400 | Missing `pipeline` request parameter (Spring binding error) |
| 500 | Pipeline not registered, empty file, internal error, or processing failure |

The 500 response covers all errors handled by the controller, including an empty file and an unregistered pipeline. There is no explicit `413` handler; exceeding the configured multipart size limit produces Spring's default multipart-size error.

## Size Limits

Two limits apply, and both are disk-bound — an upload streams to the staging area and never sits in the server's heap, so a multi-GB file lands on the default 2 GB heap.

* **Upload ceiling** — `DATRIS_MAX_UPLOAD_MB` (default `1024`, i.e. 1 GB) sets `spring.servlet.multipart.max-file-size` / `max-request-size`. It is checked while the file is received, before the API key is validated, which is why it stays small and separate from the run budget. Set it in `.env` to accept larger files:

  ```bash theme={null}
  DATRIS_MAX_UPLOAD_MB=4096
  ```

* **Per-run disk budget** — `PIPELINE_MAX_PAYLOAD_MB` (default `4096`, `0` = unlimited) is the most one run may stage on disk. An upload — or the combined contents of a batch archive — above it fails with a message naming the variable, and nothing is left on disk. See the [configuration reference](/configuration-reference#container-environment-variables).

Exceeding the upload ceiling produces Spring's default multipart-size error; exceeding the run budget produces the 500 response described above.

## End-to-end example: CSV to PostgreSQL

Register a pipeline, upload a CSV, and query the result — the whole loop over REST, no agent involved.

### 1. Start the stack

If you haven't installed yet, follow [Installation](/installation) — the
one-command installer gets the stack running in a `./datris` directory. Already
installed? Start it from your install directory:

```bash theme={null}
docker compose up -d
```

Wait for the `datris` container to show `Started Application`.

### 2. Register a pipeline

Create a pipeline configuration that defines the source schema and PostgreSQL destination:

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "name": "employees",
    "source": {
      "schemaProperties": {
        "fields": [
          {"name": "id", "type": "int"},
          {"name": "first_name", "type": "string"},
          {"name": "last_name", "type": "string"},
          {"name": "email", "type": "string"},
          {"name": "salary", "type": "double"}
        ]
      },
      "fileAttributes": {
        "csvAttributes": {
          "delimiter": ",",
          "header": true,
          "encoding": "UTF-8"
        }
      }
    },
    "destination": {
      "database": {
        "dbName": "datris",
        "schema": "public",
        "table": "employees",
        "usePostgres": true
      }
    }
  }'
```

### 3. Upload a CSV file

Create a sample CSV file:

```bash theme={null}
cat > /tmp/employees.csv << 'EOF'
id,first_name,last_name,email,salary
1,John,Doe,john@example.com,75000.00
2,Jane,Smith,jane@example.com,82000.00
3,Bob,Johnson,bob@example.com,68000.00
EOF
```

Upload it:

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/upload \
  -F "file=@/tmp/employees.csv" \
  -F "pipeline=employees"
```

The response body is the `pipelineToken` for tracking — a plain UUID:

```
5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e
```

### 4. Check status

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/status?pipelinetoken=5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e"
```

The status shows processing stages: `begin`, `processing`, `end`.

### 5. Verify the data

Query the data via the REST API:

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/postgres \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT * FROM public.employees"}'
```

The default database is `datris` — this is where all PostgreSQL pipelines write unless you specify a different `dbName` in the pipeline's destination config.


# Kafka Topic Consumption
Source: https://docs.datris.ai/ingestion/kafka

Consume data from Kafka topics

The pipeline can consume data from Kafka topics in real time. Configure the Kafka consumer in `application.yaml`, and the pipeline subscribes to topics based on registered pipeline configurations with `streamAttributes.type: "kafka"`.

<Note>
  **Kafka is opt-in.** The bundled `kafka`, `zookeeper`, and `kafka-ui` services are gated behind the `kafka` Compose profile. To use the bundled broker, set `COMPOSE_PROFILES=kafka` in `.env` and run `docker compose up -d` — the `kafka-data`, `zookeeper-data`, and `zookeeper-log` volumes come along automatically. The `kafka-clients` library is included in the assembly JAR, so pipelines that point at any external Kafka broker work without enabling the bundled one.
</Note>

## Enabling the Kafka Consumer

Set the `kafkaConsumer` block in `application.yaml`:

```yaml theme={null}
kafkaConsumer:
  enabled: true
  bootstrapServers: "kafka:9092"
  groupId: "oss-group"
  topicPollingInterval: 500
  topicPrefix: "oss"
```

## Configuration Reference

| Property | Required | Description | Default |
| - | - | - | - |
| `enabled` | No | Enable or disable the Kafka consumer | `false` |
| `bootstrapServers` | Yes | Comma-separated list of Kafka broker addresses | -- |
| `groupId` | Yes | Consumer group ID; controls offset tracking and partition assignment | -- |
| `topicPollingInterval` | No | Milliseconds between poll cycles | `500` |
| `topicPrefix` | No | Prefix prepended to pipeline names to form topic names | `oss` |

## How Topics Are Resolved

At startup, the consumer reads all pipeline configurations from the database and subscribes to topics for pipelines that have `streamAttributes.type: "kafka"`. The topic name is constructed as `{topicPrefix}.{pipelineName}`.

For example, with `topicPrefix: "oss"` (defined in `kafkaConsumer.topicPrefix` in `application.yaml`) and a pipeline named `orders`, the consumer subscribes to topic `oss.orders`.

When a Kafka message arrives on `oss.orders`, the consumer strips the prefix and looks up the pipeline named `orders` to determine how to process the data.

## Data Formats

The format of incoming records is determined by the pipeline configuration — not auto-detected. Configure the pipeline's `source.fileAttributes` to match the data format:

* **CSV** — use `csvAttributes` with the appropriate delimiter. Records are parsed by position against the pipeline schema.
* **JSON** — use `jsonAttributes`. Field names in the payload are matched against the pipeline schema by name.
* **XML** — use `xmlAttributes`. Element names are matched against the pipeline schema by name.

Any plain text data can be consumed as long as the pipeline schema and file attributes match the format.

## Full Example

A complete setup with a pipeline configuration and the Kafka consumer enabled:

**application.yaml**

```yaml theme={null}
kafkaConsumer:
  enabled: true
  bootstrapServers: "kafka:9092"
  groupId: "oss-group"
  topicPollingInterval: 500
  topicPrefix: "oss"
```

**Pipeline configuration (orders)**

```json theme={null}
{
  "name": "orders",
  "source": {
    "streamAttributes": {
      "type": "kafka"
    },
    "fileAttributes": {
      "jsonAttributes": {
        "encoding": "UTF-8"
      }
    },
    "schemaProperties": {
      "fields": [
        { "name": "id", "type": "bigint" },
        { "name": "email", "type": "string" },
        { "name": "order_date", "type": "date" },
        { "name": "total", "type": "double" }
      ]
    }
  },
  "destination": {
    "database": {
      "dbName": "datris",
      "schema": "public",
      "table": "orders",
      "usePostgres": true
    }
  }
}
```

Produce a test message and verify ingestion:

```bash theme={null}
# Produce a JSON record to the topic
echo '{"id":1001,"email":"jane@example.com","order_date":"2025-06-15","total":249.99}' | \
  docker exec -i kafka kafka-console-producer --bootstrap-server kafka:9092 --topic oss.orders

# Check pipeline status
curl -s "http://localhost:8080/api/v1/pipeline/status?pipelinename=orders" | jq .
```

## Offset Management

The pipeline uses Kafka's auto-commit to track offsets using the configured `groupId`. If the consumer restarts, it resumes from the last committed offset. New consumer groups start from the earliest offset.

To reprocess a topic from the beginning, reset the consumer group offsets:

```bash theme={null}
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --group oss-group \
  --topic oss.orders \
  --reset-offsets --to-earliest --execute
```

## Troubleshooting

| Symptom | Check |
| - | - |
| Consumer not subscribing to a topic | Verify the pipeline has `streamAttributes.type: "kafka"` and was registered before the server started. |
| Records not parsing | Confirm the pipeline's `fileAttributes` (csvAttributes, jsonAttributes, or xmlAttributes) matches the format of your Kafka records. |
| Lag increasing | Check consumer group lag with `kafka-consumer-groups.sh --describe`. Consider scaling partitions or adding consumer instances with the same `groupId`. |
| Connection failures | Verify `bootstrapServers` addresses are reachable from the pipeline host. Check Kafka broker logs for authentication or authorization errors. |


# MinIO Bucket Ingestion
Source: https://docs.datris.ai/ingestion/object-store

Ingest from MinIO object store

The data platform monitors the `{env}-raw` MinIO bucket for new files. When a file is dropped into this bucket, the data platform picks it up and processes it automatically. There are three patterns: single data file, metadata file with data file, and bulk upload.

## File Naming Convention

### Single Data File

Place a file in the MinIO raw bucket following this naming pattern:

```
{pipeline}.{publishertoken}.{anything}.pipeline.{ext}
```

| Segment | Description |
| - | - |
| `pipeline` | Name of the registered pipeline configuration |
| `publishertoken` | Identifier for the data publisher — must be a valid UUID. If the second segment is not a UUID, it is ignored and a random UUID is generated |
| `anything` | Any additional text (timestamp, version, etc.) |
| `pipeline` (literal) | Fixed literal sentinel string |
| `ext` | File extension: `csv`, `json`, `xml` |

**Example:**

```
stock_price.28c39a6f-de32-4625-9bee-6af9ee547798.2026-03-15.1.pipeline.csv
```

### Metadata File + Data File

For more control, upload the data file first, then upload a metadata file (`.metadata.json`):

**Data file:** `stock_price.2026-03-15.pipeline.csv`

**Metadata file:** `stock_price.2026-03-15.metadata.json`

```json theme={null}
{
  "dataFileName": "stock_price.2026-03-15.pipeline.csv",
  "pipeline": "stock_price",
  "publisherToken": "28c39a6f-de32-4625-9bee-6af9ee547798"
}
```

The data file must be uploaded **before** the metadata file.

### Bulk Upload

Upload multiple files to a directory, then trigger processing with a metadata file:

```json theme={null}
{
  "dataFilePath": "s3://oss-raw/bulk/stock_price/batch-001/",
  "pipeline": "stock_price",
  "bulkUpload": true,
  "publisherToken": "28c39a6f-de32-4625-9bee-6af9ee547798"
}
```

**Important:** Use a unique `dataFilePath` directory for each bulk upload. Do not reuse directories.

### Compressed Files

Compressed archives (`.zip`, `.gz`, `.tar`, `.jar`) are automatically decompressed. The pipeline name and publisher token are parsed from the archive filename:

```
stock_price.28c39a6f-de32-4625-9bee-6af9ee547798.2026-03-15.pipeline.zip
```

Note that the bundled MinIO setup (`docker/minio-init.sh`) only registers bucket events for the `.metadata.json`, `.pipeline.csv`, `.pipeline.json`, and `.pipeline.xml` suffixes. To have archives picked up automatically, add a matching `mc event add ... --suffix .pipeline.zip` rule (see below).

## How It Works

1. A file is placed in a MinIO bucket (via `mc cp`, API, or another system)
2. MinIO sends an S3-compatible event notification to the ActiveMQ file-notifier queue
3. The pipeline polls the queue on a configurable schedule (default: every 5 seconds)
4. The pipeline parses the event, resolves the pipeline, and processes the file

## Configuration

The queue name and polling interval are configured in `application.yaml`:

```yaml theme={null}
# Environment prefix - queue name becomes {environment}-file-notifier
environment: oss

# Polling interval in milliseconds
schedule:
  checkFileNotifierQueue: "5000"

# Days to retain processed message IDs for deduplication
ttlFileNotifierQueueMessages: "60"
```

## Message Deduplication

The pipeline tracks processed message IDs in MongoDB to prevent duplicate processing. Each processed message ID is stored with a TTL (default: 60 days). If the same event arrives again within the TTL window, it is acknowledged and discarded.

## MinIO Webhook Setup

Configure MinIO to send bucket notifications. The Docker setup (`docker/minio-init.sh`) points a webhook target at the `/minio-events` endpoint and registers per-suffix events on the raw bucket:

```bash theme={null}
# Using mc to configure notifications (auth_token must match MINIO_WEBHOOK_TOKEN on the datris service)
mc admin config set myminio notify_webhook:1 \
  endpoint="http://datris:8080/minio-events" \
  auth_token="${MINIO_WEBHOOK_TOKEN}"

mc admin service restart myminio
mc event add myminio/oss-raw arn:minio:sqs::1:webhook --suffix .metadata.json --event put
mc event add myminio/oss-raw arn:minio:sqs::1:webhook --suffix .pipeline.csv --event put
mc event add myminio/oss-raw arn:minio:sqs::1:webhook --suffix .pipeline.json --event put
mc event add myminio/oss-raw arn:minio:sqs::1:webhook --suffix .pipeline.xml --event put
```

Alternatively, ActiveMQ-based notifications can be configured (see MinIO documentation for AMQP notification targets).


# Install Datris from an agent or CI job (no terminal)
Source: https://docs.datris.ai/install-for-agents

Install the self-hosted Datris stack with nobody at a keyboard, check that it is up, connect the MCP server and add the datris-platform skill. Written for coding agents and CI jobs.

This page takes an agent, or a CI job, from an empty machine to a connected MCP client without anyone answering a prompt. It covers the installer's behaviour when no terminal is attached, the single-file Compose alternative, how to tell the stack is up, how to connect the MCP server, and how to add the `datris-platform` skill. For the interactive install a person runs, see [Installation](/installation).

## Prerequisites

* **Docker** with the **Compose v2** plugin (`docker compose version` answers), and a **running daemon** (`docker info` succeeds). The installer stops with an error if any of these is missing.
* **curl**, plus **openssl** (or `/dev/urandom` and `xxd`). The installer uses them to fetch files and to generate the tap runner token.
* **A POSIX shell on macOS or Linux.** The installer is a `sh` script. On Windows, use the [single-file Compose](#single-file-compose-alternative) path instead.
* **Memory:** 4 GB available to Docker for a minimal install, about 8 GB for the full stack with the bundled embedding server. See [Installation → Minimal install](/installation#minimal-install).
* **At least one AI provider key** in the environment: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `XAI_API_KEY`, the Azure OpenAI trio, or `AI_PROVIDER=bedrock`. See [Without an AI key](#without-an-ai-key) for why.
* **Free host ports.** The stack publishes 8080 (API), 4200 (UI), 3000 (MCP server), 61616 and 8161 (ActiveMQ), 9000 and 9001 (MinIO), 27017 (MongoDB) and 8200 (Vault); also 5432 (Postgres) unless `DATRIS_POSTGRES` is `external` or `none`, and 11434 (bundled embedding server) unless `DATRIS_EMBEDDING` is `openai` or `none`. Opt-in profiles publish more. See [Installation → Services](/installation#services).

## Install without a terminal

The installer reads its prompts only from the controlling terminal (`/dev/tty`), never from stdin. When it cannot open a terminal it skips every prompt, prints `Non-interactive — using the detected key(s).` when it finds a key, and takes its answers from environment variables. Redirecting stdin is not enough: a process that still has a controlling terminal is prompted, and waits.

If your agent's shell tool already runs without a controlling terminal, the plain piped command is prompt-free. To make sure, start the shell in a new session, which detaches it from any terminal.

<CodeGroup>
  ```bash macOS theme={null}
  # At least one provider key (ANTHROPIC_API_KEY, OPENAI_API_KEY, ...) must already be exported; this stops early if none is.
  [ -n "${ANTHROPIC_API_KEY:-}${OPENAI_API_KEY:-}${XAI_API_KEY:-}${AZURE_OPENAI_API_KEY:-}${AI_PROVIDER:-}" ] \
    || { echo "export a provider key first, such as ANTHROPIC_API_KEY or OPENAI_API_KEY" >&2; exit 1; }
  curl -fsSL https://get.datris.ai/install.sh \
    | DATRIS_DIR="$PWD/datris" python3 -c 'import os,sys
  pid = os.fork()
  if pid == 0:
      os.setsid(); os.execvp("sh", ["sh"])
  sys.exit(os.waitstatus_to_exitcode(os.waitpid(pid, 0)[1]))'
  ```

  ```bash Linux (not tested) theme={null}
  # At least one provider key (ANTHROPIC_API_KEY, OPENAI_API_KEY, ...) must already be exported; this stops early if none is.
  [ -n "${ANTHROPIC_API_KEY:-}${OPENAI_API_KEY:-}${XAI_API_KEY:-}${AZURE_OPENAI_API_KEY:-}${AI_PROVIDER:-}" ] \
    || { echo "export a provider key first, such as ANTHROPIC_API_KEY or OPENAI_API_KEY" >&2; exit 1; }
  curl -fsSL https://get.datris.ai/install.sh | DATRIS_DIR="$PWD/datris" setsid -w sh
  ```
</CodeGroup>

macOS does not ship `setsid`, so the macOS form uses a short Python wrapper that makes the same system call, waits for the installer and returns its exit code. That form was run, launched from inside a terminal, and asked nothing. The Linux form uses `setsid` from util-linux (`-w` waits for the installer and returns its exit code); it was not run for this page.

To keep the installer's output for later, add `> install.log 2>&1` at the end of either command.

### Environment variables

The installer reads these on a fresh install. Values you pass are written to `<install dir>/.env`, which is kept at permissions `600`.

| Variable | What it does |
| - | - |
| `DATRIS_DIR` | Install directory. Default `./datris` under the current directory. Its `.env` holds the provider key, so run the installer from a directory outside your project or set `DATRIS_DIR`, and never commit that `.env`. |
| `DATRIS_REF` | Git ref of the repository to fetch the runtime files from. Default `main`. |
| `ANTHROPIC_API_KEY` | Anthropic key for chat and CodeGen. |
| `OPENAI_API_KEY` | OpenAI key for chat and CodeGen; also enables OpenAI embeddings. |
| `XAI_API_KEY` | Grok (xAI) key for chat and CodeGen. `GROK_MODEL` optionally overrides the model. |
| `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_MODEL` | Azure OpenAI. All three are required together. When `AZURE_OPENAI_API_KEY` is set without the endpoint or model, the installer prints a warning and writes none of them; an endpoint or model without the key is ignored silently. The endpoint is the resource base URL, the model is the chat deployment name. For keyless Entra ID auth, set `AI_PROVIDER=azure` with the endpoint and model and no key, plus either all three of `AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and `AZURE_CLIENT_SECRET` or none of them to use the identity assigned to the Azure host. |
| `AI_PROVIDER` | Which provider handles chat and CodeGen: `anthropic`, `openai`, `azure`, `grok` or `bedrock`. Always written to `.env`. Unset, the installer chooses as described in [Which chat provider the install uses](#which-chat-provider-the-install-uses). A value that names a provider with no key in the environment, or an unknown value, stops the install with an error. |
| `AI_PROVIDER=bedrock` | Use Claude through Amazon Bedrock. Optionally with `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (leave both unset to use the host's IAM role or default credential chain), `AWS_REGION` and `BEDROCK_MODEL`. AWS keys alone never select Bedrock. |
| `DATRIS_POSTGRES` | `bundled` (default), `external` or `none`. `external` also reads `POSTGRES_JDBC_URL`, `POSTGRES_USER` and `POSTGRES_PASSWORD`, and stops with an error when `POSTGRES_JDBC_URL` is missing. |
| `DATRIS_EMBEDDING` | `openai`, `tei` or `none`. Default `openai` when `OPENAI_API_KEY` is set, otherwise `tei`. `openai` without `OPENAI_API_KEY` stops with an error. |
| `DATRIS_PROFILES` | Comma-separated opt-in bundled services (`qdrant`, `weaviate`, `chroma`, `kafka`). Written to `.env` as `COMPOSE_PROFILES`, together with the in-network host and port each store needs (for example `QDRANT_HOST=qdrant`, `QDRANT_PORT=6334`; `KAFKA_BOOTSTRAP_SERVERS=kafka:9092` for `kafka`). |
| `QDRANT_HOST`, `WEAVIATE_HOST`, `CHROMA_HOST`, `MILVUS_HOST` | External vector stores, each with an optional port and API key in the same pattern (`QDRANT_PORT`, `QDRANT_API_KEY`; Chroma takes no key). Written to `.env`. A host set here wins over the same store named in `DATRIS_PROFILES`, and no local container is started for it. |
| `KAFKA_BOOTSTRAP_SERVERS` | An external Kafka broker list. Written to `.env`. |
| `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_USER`, `SNOWFLAKE_PRIVATE_KEY`, `SNOWFLAKE_PASSWORD` | Snowflake destination credentials, written to `.env` when `SNOWFLAKE_ACCOUNT` is set. |
| `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`, `DATABRICKS_TOKEN` | Databricks destination credentials, written to `.env` when `DATABRICKS_HOST` is set. |
| `DATRIS_GOVERNED=1` | Switch on the governance controls for production on a fresh install: writes `USE_USER_AUTH`, `USE_API_KEYS`, `USE_AUDIT_LOG` and `USE_AGENT_POLICY` as `true`. `1`, `true`, `yes` or `on` switch them on; `0`, `false`, `no` or `off` leave them off without asking; any other value stops the install before anything is written. Unset, an interactive run asks (default No) and a run without a terminal leaves them off. Ignored on an upgrade: the existing `.env` is not edited, and the installer prints the four lines to add. With it on, the first UI login is `admin` with the bootstrap password the server prints once on its first boot (`docker compose logs datris \| grep "Bootstrap login"`; save it, since recreating the container discards that log), and every CLI, MCP and API client needs a key. The same hint is printed under `DATRIS_NO_START=1`. See [Switch on the governance controls](/quick-start#2-switch-on-the-governance-controls). |
| `DATRIS_NO_START=1` | Write the files and `.env`, then stop before pulling images or starting anything. Useful to inspect what an install would do. |
| `DATRIS_SKIP_DOCTOR=1` | On an upgrade, skip the pre-upgrade `datris doctor` check. |
| `DATRIS_UPDATE_CHECK=0` | On a fresh install, write `DATRIS_UPDATE_CHECK=0` to `.env` so `datris doctor` never asks datris.ai whether a newer version exists. Without it, a fresh install prints a two-line notice saying what that check sends (server version, OS, CPU architecture) and how to switch it off. Ignored on an upgrade: the existing `.env` is not edited, so add the line yourself. See [Doctor](/doctor#update-check). |

### What a run without a terminal chooses

When a variable above is unset, the installer takes these defaults:

| Setting | Default |
| - | - |
| Postgres | Bundled container (also provides pgvector) |
| Embeddings | OpenAI `text-embedding-3-small` when `OPENAI_API_KEY` is set (no local embedding container). Otherwise the bundled TEI server, which downloads a 2.2 GB model on first boot. |
| Extra vector stores | None (pgvector inside Postgres only) |
| Kafka | None |
| Snowflake, Databricks | Not configured |

It also generates a random `TAP_RUNNER_TOKEN` in `.env`.

### Which chat provider the install uses

One provider handles chat and CodeGen. The installer writes it to `.env` as `AI_PROVIDER`, next to the keys:

| Keys in the environment | Written to `.env` |
| - | - |
| Anthropic only | `ANTHROPIC_API_KEY`, `AI_PROVIDER=anthropic`, `EMBEDDING_PROVIDER=tei` |
| OpenAI only | `OPENAI_API_KEY`, `AI_PROVIDER=openai`, `EMBEDDING_PROVIDER=openai`, `TEI_ENABLED=0` |
| Grok only | `XAI_API_KEY`, `AI_PROVIDER=grok`, `EMBEDDING_PROVIDER=tei` |
| Anthropic and OpenAI, with `AI_PROVIDER=openai` | `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `AI_PROVIDER=openai`, `EMBEDDING_PROVIDER=openai`, `TEI_ENABLED=0` |
| Anthropic and OpenAI, with `AI_PROVIDER=anthropic` | `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `AI_PROVIDER=anthropic`, `EMBEDDING_PROVIDER=openai`, `TEI_ENABLED=0` |

When several provider keys are set, say which one handles chat and CodeGen with `AI_PROVIDER`. If you leave it unset, a run without a terminal takes the first configured provider in the order `anthropic`, `openai`, `azure`, `grok`, and prints `Several AI providers configured` with the provider it chose. That order is a tie-break kept so existing automated installs behave as before. A run in a terminal asks instead.

The AI settings are seeded into Vault on the first start only; after that, change the chat provider from the **Configuration** tab in the UI.

Anthropic does not offer an embeddings API, and neither does xAI. `DATRIS_EMBEDDING` accepts `openai`, `tei` (the bundled local server) and `none`.

### Without an AI key

The stack cannot start without an AI provider, so a fresh install with none stops before it pulls anything, with `error: No AI provider key set, and Datris cannot start without one`. Set one of the provider variables above and run it again.

With `DATRIS_NO_START=1` the installer still writes the files and exits 0, and warns that Datris will not start until a provider key is added to `.env`.

### Running it again

If `<install dir>/.env` already exists, the installer runs in **upgrade mode**: no prompts, `.env` is left exactly as it is, and provider keys and the selection variables above are ignored. It refreshes the runtime files and, without `DATRIS_NO_START=1`, pulls the images and recreates the containers. When the `datris` CLI is installed it first runs `datris doctor --pre-upgrade` and stops on an error-level finding (`DATRIS_SKIP_DOCTOR=1` bypasses it). See [Upgrading](/production/upgrades).

A fresh run that stops with an error before it finishes writing `.env` removes that partial file and prints `Install stopped before finishing — removed the partial`, so the next run is a fresh install again and reads the corrected variables. A `.env` that existed before the run is never removed.

### A second install on the same Docker host

Every Datris container has a fixed name (`datris`, `mcp-server`, `vault`, `postgres` and so on). Before it pulls anything, the installer checks for containers with those names that belong to a different Compose project, including stopped ones. If it finds any, it lists them and stops with `container name conflict — resolve the above and re-run`. Either install into the existing directory (an upgrade), or remove the old installation first with `docker compose --profile "*" down` in its directory. Plain `down` keeps named data volumes; `down -v` deletes them. This check is skipped under `DATRIS_NO_START=1`.

An agent must ask the user before removing an existing installation. Volume names come from the install directory's name, so reinstalling into a directory with the same name (after a plain `down`) reuses the earlier Vault volume: the earlier AI settings are kept and newly exported provider keys are not seeded. Change them from the **Configuration** tab in the UI.

### Adding vector stores

pgvector comes with the bundled Postgres and needs nothing else. For a bundled Qdrant, Weaviate or Chroma, name it in `DATRIS_PROFILES`; for a store you already run, set its `_HOST` variable. Both are written to `.env` on a run without a terminal. Milvus is external only. To add a store after the install, see [Installation](/installation).

## Single-file Compose alternative

A single self-contained Compose file with the init scripts and config inlined. It needs Docker Compose 2.23 or later and has no prompts at all, so it needs no detaching:

```bash theme={null}
mkdir datris && cd datris
curl -fsSLO https://get.datris.ai/docker-compose.standalone.yml
# At least one provider key (ANTHROPIC_API_KEY, OPENAI_API_KEY, ...) must already be exported, or be in .env.
docker compose -f docker-compose.standalone.yml up -d
```

You can put the variables in a `.env` next to the file instead; Compose reads it automatically. This path does not write an `AI_PROVIDER` pin. When several provider keys are present and `AI_PROVIDER` is unset, first-boot seeding picks OpenAI, so set `AI_PROVIDER=anthropic` or `AI_PROVIDER=openai` to choose. With `OPENAI_API_KEY` set, the file uses OpenAI embeddings; add `TEI_ENABLED=0` to skip the bundled embedding server and its model download. See [Installation](/installation) for the full description.

## Check that it is up

The installer's exit code does not prove the stack is up. After starting the containers it polls for up to five minutes, prints a warning if the server has not answered, and exits 0 either way. Poll the two public endpoints yourself. Neither needs an API key.

```bash theme={null}
# Wait up to 10 minutes for the API server
i=0
until curl -fsS --max-time 5 http://localhost:8080/api/v1/version >/dev/null; do
  i=$((i+1)); [ "$i" -ge 120 ] && { echo "server did not come up" >&2; exit 1; }
  sleep 5
done

# Server version, and which features are on
curl -fsS http://localhost:8080/api/v1/version

# Per-store status: "up", "down" or "not_configured"
# (with DATRIS_GOVERNED=1 or USE_API_KEYS=true add: -H "x-api-key: <key>")
curl -fsS http://localhost:8080/api/v1/health/services
```

* [`GET /api/v1/version`](/api-reference/version-api) returns the server version and settings such as `useApiKeys`.
* [`GET /api/v1/health/services`](/api-reference/health-api) returns a `status` for each store. A store you chose not to install reports `not_configured`; one that is configured but unreachable reports `down` with a message. With `DATRIS_GOVERNED=1` (or `USE_API_KEYS=true`) this call needs a key, `-H "x-api-key: <key>"` with a key from Configuration → API Keys; without one it returns `401 {"error":"Authentication required"}`, which `curl -f` turns into exit 22. `/api/v1/version` stays keyless.
* `docker compose ps` in the install directory (add `-f docker-compose.standalone.yml` for the single file) shows each container's state and health.

For a deeper check, install the `datris` CLI (`pip install datris-mcp-server` or `brew install datris/tap/datris`) and run [`datris doctor`](/doctor) from the install directory. It checks Vault, the AI slots, disk, data volumes, the `.env` and whether the MCP server answers. Exit codes: `0` ok, `1` warnings, `2` errors, `3` server unreachable. When API keys are on, set `DATRIS_API_KEY` first.

## Connect the MCP server

The stack runs the MCP server in the `mcp-server` container, serving SSE at `http://localhost:3000/sse`. Most MCP clients connect through the `mcp-remote` bridge (needs Node.js):

```json theme={null}
{
  "mcpServers": {
    "datris": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "http://localhost:3000/sse", "--transport", "sse-only"]
    }
  }
}
```

API keys are off by default (`USE_API_KEYS=false`), so no key is needed. When they are on, append `"--header", "x-api-key:<your-key>"` to `args`. Issue keys from **Configuration → API-Keys** in the UI.

To run the server over stdio instead, use the PyPI package `datris-mcp-server` (needs `uv`):

```json theme={null}
{
  "mcpServers": {
    "datris": {
      "command": "uvx",
      "args": ["datris-mcp-server"],
      "env": {
        "DATRIS_API_URL": "http://localhost:8080"
      }
    }
  }
}
```

Stdio has no request headers, so when API keys are on, add `"DATRIS_API_KEY": "<your-key>"` to `env`.

The `mcp-server` container runs the Docker image `datrisai/datris-mcp-server`. To run that image on its own against a Datris server, set `DATRIS_API_URL`; the image starts the SSE transport on port 3000.

Client-specific steps are in [Configuring Claude](/configuring-claude) and [Configuring OpenClaw](/configuring-openclaw). Transports, environment variables and authentication are in [MCP Server](/mcp-server).

## Install the skill

The `datris-platform` skill tells a coding agent when and how to use Datris during project work. Copy it into the project's skills folder:

```bash theme={null}
git clone --depth 1 https://github.com/datris/datris-platform-oss.git /tmp/datris-oss
mkdir -p .claude/skills && cp -r /tmp/datris-oss/skills/datris-platform .claude/skills/
```

For other agents and locations, see [Agent Skill](/agent-skill).

## Troubleshooting

| Output | Cause and fix |
| - | - |
| `error: DATRIS_EMBEDDING=openai requires OPENAI_API_KEY` | Set `OPENAI_API_KEY`, or use `DATRIS_EMBEDDING=tei` or `none`. Then run it again; the failed run leaves no `.env` behind. |
| `error: DATRIS_POSTGRES=external requires POSTGRES_JDBC_URL` | Set `POSTGRES_JDBC_URL` (base URL, no database, for example `jdbc:postgresql://host:5432`), `POSTGRES_USER` and `POSTGRES_PASSWORD`. Then run it again. |
| `Existing .env found — leaving it untouched (upgrade mode, no prompts).` on what should be a fresh install | A `.env` from an earlier install is in the install directory. Delete it, or choose a new `DATRIS_DIR`. |
| `error: No AI provider key set, and Datris cannot start without one` | Export a provider key, or set `AI_PROVIDER=bedrock`, and run it again. |
| `error: AI_PROVIDER=openai needs OPENAI_API_KEY, which is not set` (or the same for another provider) | `AI_PROVIDER` names a provider that has no key in the environment. Set the key, or change or unset `AI_PROVIDER`. |
| `Azure OpenAI needs AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_MODEL too — skipping.` | Set all three Azure variables together. |
| The installer prints a key prompt and waits | The process still has a controlling terminal. Use the detached form from [Install without a terminal](#install-without-a-terminal); redirecting stdin does not help. |


# Installation
Source: https://docs.datris.ai/installation

How to install and set up the Datris Data Platform

## Prerequisites

* [Docker](https://docs.docker.com/get-docker/)
* **Memory:** 4 GB available to Docker for the [minimal install](#minimal-install),
  about 8 GB for the full stack with the bundled embedding server. On Docker
  Desktop this is the VM allocation under Settings → Resources, not your
  host's total RAM.

<Note>
  **License.** Datris is open source under AGPL-3.0. Running an unmodified copy inside your own organization needs nothing more; a commercial license is available for hosted, embedded, and supported deployments — see [Licensing & Support](/production/licensing-support).

  **Telemetry.** The platform sends no usage data, crash reports, or license checks anywhere. The two outbound requests it can make are an on-demand model-catalog refresh from the Configuration tab, which falls back to a built-in list when it fails, and the `datris doctor` update check, which sends only the server version, operating system and CPU architecture to datris.ai when you run `datris doctor` (add `DATRIS_UPDATE_CHECK=0` to `.env` to switch it off) — see [Security Architecture](/production/security-architecture#no-telemetry) for how to verify.
</Note>

## Quick Start

All you need is Docker and one AI provider key (Anthropic, OpenAI, Grok,
Azure OpenAI, or Amazon Bedrock) — no git checkout, no build tools. The installer pulls
the pre-built images from Docker Hub, fetches the few runtime files Compose
needs into a `./datris` directory, seeds a `.env`, and starts the full stack:

```bash theme={null}
curl -fsSL https://get.datris.ai/install.sh | sh
```

Installing from a coding agent or a CI job with no terminal attached? Export the key first (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or another provider's): the installer asks nothing and reads it from the environment. Full steps: [Install for Agents](/install-for-agents).

Installing for production? Add `DATRIS_GOVERNED=1` (`curl -fsSL https://get.datris.ai/install.sh | DATRIS_GOVERNED=1 sh`) to switch on user login, API keys, the audit log and the agent policy in the new `.env`; for an existing install see [Switch on the governance controls](/quick-start#2-switch-on-the-governance-controls).

### Minimal install

Trying Datris on a laptop? The default stack runs about ten containers and
the bundled embedding server (TEI) downloads a 2.2 GB model on first boot.
None of that is required. Three settings cut it to eight small containers,
no model download, and roughly 3.5 GB of memory:

```bash theme={null}
# .env — or export before `docker compose up`
TEI_ENABLED=0               # skip the local embedding server and its 2.2 GB download
EMBEDDING_PROVIDER=openai   # semantic search via OpenAI instead (needs OPENAI_API_KEY);
                            # drop this line if you don't have an OpenAI key — search is simply off
POSTGRES_ENABLED=0          # optional: skip the bundled Postgres; MongoDB stays as the destination
```

The installer applies the first two for you when you pick OpenAI embeddings,
and the single-file Compose path below defaults to OpenAI embeddings whenever
an `OPENAI_API_KEY` is set. Everything here is reversible: flip a value back
to `1`, run `docker compose up -d`, and the service returns with its data
volume intact. The datris server itself runs on a 2 GB heap by default; see
[JVM Heap Sizing](#jvm-heap-sizing).

The installer walks you through a short setup — press Enter at every prompt
for a standard install:

1. **AI keys.** Datris can use Anthropic Claude, OpenAI, or Grok (xAI) —
   the installer prompts for each key in turn; enter any you have (one is
   enough, each prompt is one Enter to skip). If you prefer Claude through
   Amazon Bedrock or OpenAI through Azure OpenAI, skip the key prompts and
   the installer offers those routes next — Azure asks for your key,
   resource endpoint, and chat deployment name; Bedrock asks for AWS
   credentials (or Enter to use the host's IAM role) and a region.
   When you enter more than one key, the installer asks which provider
   handles chat and CodeGen; you can change it later in the Configuration tab.
   The stack needs one AI provider to start, so the installer stops if none
   is given.
2. **Databases and stores.** Pick what to run — each can be **bundled** (a
   local container), **external** (point at a service you already have), or
   **skipped**:
   * **Postgres** — structured destination + pgvector (default: bundled)
   * **Semantic search embeddings** — OpenAI `text-embedding-3-small`
     (recommended: no 2.2 GB model download, no resident container) or the
     bundled local TEI server for data that can't leave the machine
   * **Vector stores** — pgvector by default (included with Postgres, no
     extra container); optionally add qdrant, weaviate, or chroma
   * **Kafka** — local test broker or external (default: none)
   * **Snowflake / Databricks** — optionally store destination credentials
     now so pipelines can use them on day one

It prints an install summary, starts the stack, and then checks every store
you configured — a mistyped external hostname or credential shows up right
there by name, not later mid-pipeline. On first start, `vault-init` seeds
your keys and store credentials into Vault.

**Changing your mind later** is one line in `./datris/.env` — for example
`POSTGRES_ENABLED=0`, `COMPOSE_PROFILES=qdrant,kafka`, or
`EMBEDDING_PROVIDER=openai` — followed by `docker compose up -d`. Disabling a
store removes its container but keeps its data volume, so re-enabling
restores the data. The `.env.example` comments document every knob.

<Note>
  The `install.sh` installer is a POSIX shell script, so it runs on **macOS and
  Linux**. On **Windows**, run it from a POSIX shell — **WSL2** (recommended) or
  **Git Bash** — or skip the installer and use the single-file Compose option
  below, which works natively in PowerShell.
</Note>

Or, for a fully self-contained single file (no installer) — the init scripts
and config are inlined, so nothing else is needed (requires Docker Compose
≥ 2.23). This is the simplest path on **Windows**:

<CodeGroup>
  ```bash macOS / Linux theme={null}
  curl -O https://get.datris.ai/docker-compose.standalone.yml
  ANTHROPIC_API_KEY=sk-ant-... docker compose -f docker-compose.standalone.yml up -d
  ```

  ```powershell Windows (PowerShell) theme={null}
  curl.exe -O https://get.datris.ai/docker-compose.standalone.yml
  $env:ANTHROPIC_API_KEY="sk-ant-..."
  docker compose -f docker-compose.standalone.yml up -d
  ```
</CodeGroup>

The bundled embedding server (TEI) is optional and starts by default, pulling
a 2.2 GB model on first boot. To skip it, add `TEI_ENABLED=0` — and, if you
have an OpenAI key, `OPENAI_API_KEY=sk-...` — to the environment or a `.env`
next to the file. With an OpenAI key present the standalone file uses OpenAI
embeddings automatically; without one, semantic search is off until you
enable an embedding provider in the Configuration tab.

In PowerShell, use `curl.exe` (the bundled `curl` alias maps to
`Invoke-WebRequest` and takes different flags), and set the key with `$env:`
since the inline `KEY=value command` syntax is bash-only.

### Verify

```bash theme={null}
curl http://localhost:8080/api/v1/version
```

## Alternative: install from a git clone

If you'd rather work from a checked-out repository — for example to track the
source, customize `docker-compose.yml`, or contribute — clone the repo and start
from the checked-out compose file:

```bash theme={null}
git clone https://github.com/datris/datris-platform-oss.git
cd datris-platform-oss
cp .env.example .env       # add ANTHROPIC_API_KEY and/or OPENAI_API_KEY (or the AZURE_OPENAI_* trio, XAI_API_KEY, or AI_PROVIDER=bedrock)
docker compose up -d
```

This still pulls the pre-built images from Docker Hub — no build tools required.
To build the images yourself, see [Building from Source](#building-from-source)
below.

## Upgrading

Upgrading pulls the latest images **and** refreshes the Compose file, and your data
survives in Docker volumes. The procedure for each install method, the pre-upgrade
self-check, and what to do about stale secrets live on
[Upgrades & Supported Versions](/production/upgrades).

## Reset everything (destroys all data)

If you don't care about anything on this machine and want a completely fresh install — same data layout as a brand-new clone — wipe all volumes:

```bash theme={null}
cd datris            # your install dir: ./datris (installer), the standalone file's dir, or your clone
docker compose down -v
docker compose up -d
```

**Warning:** `docker compose down -v` is destructive. The `-v` flag removes every Docker volume the project owns (both named volumes and the anonymous volumes the data services create automatically). You will lose:

* **All your pipelines, runs, taps, and metadata** (MongoDB internal `oss` database) and **all data landed in Postgres** (`datris` database)
* **All Vault secrets** — API keys, database credentials, AI configuration (will be re-seeded from your `.env` on next start, but only the defaults — any UI-edited overrides are gone)
* **All MinIO object storage** — raw uploads, configs, temp files, processed outputs
* **MongoDB destination data**
* **Queued messages and offsets** in Kafka, Zookeeper, ActiveMQ
* **The bundled `bge-m3` embedding model** in the `tei-data` volume (will re-download \~2.2 GB on first start)
* **Cached pip wheels** in the `pip-cache` volume (taps that need extras like `yfinance` will re-download on first run)

Only run `down -v` if you are certain none of the above matters, e.g. on a brand-new dev machine, after exporting anything you needed, or on a CI runner. **Never run `down -v` on a production or shared instance.**

After `docker compose up -d`, `vault-init.sh` re-seeds the AI configuration secrets from your `.env`, MinIO buckets are recreated, Postgres starts empty, and the bundled embedding service re-downloads `bge-m3` (a few minutes one-time).

> **Note for production deployments:** the `deploy/docker-compose.prod.yml` file used by dedicated production installs uses bind mounts to host directories under `/data/*` instead of Docker volumes, so `docker compose -f docker-compose.prod.yml down -v` does **not** wipe the data — the host directories survive. To reset a prod install, you'd need to also delete the relevant `/data/*` directories on the host, which is a much riskier operation and not recommended outside of disaster recovery.

## Volumes

All stateful services use named Docker volumes, created automatically by `docker compose up` — no user action required:

| Volume | Mounted at | Purpose |
| - | - | - |
| `postgres-data` | postgres `/var/lib/postgresql/data` | Pipeline data landed in Postgres + pgvector collections |
| `mongodb-data` / `mongodb-configdb` | mongodb `/data/db`, `/data/configdb` | All pipeline, tap, and platform configuration |
| `minio-data` | minio `/data` | Object storage — uploads, configs, pipeline output |
| `vault-data` | vault `/vault/file` | Secrets (persist across restarts) |
| `tei-data` | tei `/data` | The bundled embedding model (\~2.2 GB) |
| `pip-cache` | datris `/root/.cache/pip` | Caches downloaded pip wheel files so tap-installed packages don't have to be re-downloaded |

Because these are **named** volumes, they re-attach whenever a container is recreated or a store is disabled (`POSTGRES_ENABLED=0`) and later re-enabled — disable → re-enable is lossless. If you upgraded from a version where the data services used anonymous volumes, the first `up -d` after this change starts the stores on fresh named volumes; the installer-era data lives on in the old anonymous volumes (visible via `docker volume ls -f dangling=true`) and can be copied across with `docker run --rm -v <old>:/src:ro -v <new>:/dst alpine cp -a /src/. /dst/`.

Commonly used packages (`requests`, `beautifulsoup4`, `pandas`, `lxml`, `feedparser`, `boto3`, `google-cloud-storage`, `azure-storage-blob`, `openpyxl`, `pyyaml`, `python-dateutil`, `pytz`) are baked into the image. When a tap needs something extra (e.g. `yfinance`), pip downloads it on first run (\~30 seconds) and caches the wheel in `pip-cache`. Subsequent container restarts re-run `pip install` for those extras, but the install is near-instant because the wheel is already cached locally.

## Services

| Service | Port | Purpose |
| - | - | - |
| Pipeline Server | 8080 | REST API and data processing |
| Pipeline UI | 4200 | Web dashboard |
| MCP Server | 3000 | AI agent integration (MCP protocol) |
| MinIO | 9000 (API), 9001 (Console) | Object storage |
| MongoDB | 27017 | Configuration and status store |
| ActiveMQ | 61616 (broker), 8161 (console) | Message queue and notifications |
| Vault | 8200 | Secrets management |
| PostgreSQL | 5432 | Database destination + pgvector — **optional** (`POSTGRES_ENABLED=0` to disable, or point at an external Postgres) |
| TEI | 11434 | Bundled embedding server — **optional** (`TEI_ENABLED=0` to disable; recommended alternative: OpenAI embeddings) |
| Qdrant | 6333/6334 | Vector store — **opt-in** (`COMPOSE_PROFILES=qdrant`) |
| Weaviate | 8079, 50051 | Vector store — **opt-in** (`COMPOSE_PROFILES=weaviate`) |
| Chroma | 8000 | Vector store — **opt-in** (`COMPOSE_PROFILES=chroma`) |
| Kafka | 9092 | Streaming — **opt-in** (`COMPOSE_PROFILES=kafka`; includes Zookeeper on 2181 and Kafka UI on 8085) |

<Note>
  **Optional vs opt-in.** Services that ship enabled (Postgres, TEI) are disabled with an explicit `*_ENABLED=0` in `.env` — an `.env` without these lines keeps them running, so upgrades never silently drop a service you were using. Brand-new services (vector stores, Kafka) activate via `COMPOSE_PROFILES`. Datris bundles `kafka-clients`, so pipelines that talk to an **external** Kafka don't need the local broker at all — set `KAFKA_BOOTSTRAP_SERVERS` in `.env` instead. One teardown nuance: `docker compose down` ignores inactive profiles, so for a full teardown use `docker compose --profile "*" down`.
</Note>

## Web UIs

| UI | URL | Credentials |
| - | - | - |
| **Datris Platform UI** | [http://localhost:4200](http://localhost:4200) | none |
| **Datris Platform API** | [http://localhost:8080](http://localhost:8080) | none |
| **MCP Server (SSE)** | [http://localhost:3000/sse](http://localhost:3000/sse) | none |
| **MinIO Console** | [http://localhost:9001](http://localhost:9001) | `minioadmin` / `minioadmin` |
| **ActiveMQ Console** | [http://localhost:8161](http://localhost:8161) | `admin` / `admin` |
| **Kafka UI** (opt-in) | [http://localhost:8085](http://localhost:8085) | none — only with `COMPOSE_PROFILES=kafka` |
| **Vault UI** | [http://localhost:8200](http://localhost:8200) | Token: run `docker compose exec -T datris cat /vault-token/token` (random per install) |

## API Keys and AI Providers

Datris supports six AI providers. Set your keys in `.env`:

| Provider | Environment Variable | Used For |
| - | - | - |
| **Anthropic Claude** | `ANTHROPIC_API_KEY` | AI data quality, transformations, error explanation, schema generation, profiling |
| **OpenAI** | `OPENAI_API_KEY` | Same as above, plus embeddings for vector database / RAG |
| **Azure OpenAI** | `AZURE_OPENAI_API_KEY` + `AZURE_OPENAI_ENDPOINT` + `AZURE_OPENAI_MODEL` | Same as OpenAI, served from your Azure resource (models addressed by deployment name) |
| **Amazon Bedrock** | `AI_PROVIDER=bedrock` + `AWS_REGION` (AWS keys optional — IAM role works) | Claude served through your AWS account with IAM auth and AWS billing |
| **Grok (xAI)** | `XAI_API_KEY` | Chat and CodeGen on Grok via xAI's OpenAI-compatible API (no embeddings) |
| **Ollama** (local) | `OLLAMA_MODEL` (requires uncommenting the `ollama` service in `docker-compose.yml`) | Same as above — no API key needed, runs locally |

At least one AI provider key is required for AI features. The embedding provider for RAG is configured via Vault secrets — see [AI Configuration](/ai-configuration) for details.

## Infrastructure Details

### MinIO

The `minio-init` container automatically creates the required buckets:

* `{env}-raw` - File upload staging
* `{env}-temp` - Temporary processing files
* `{env}-data` - Pipeline output (object store destination)
* `{env}-config` - Configuration files (validation schemas)

Where `{env}` is the environment name (default: `oss`). See [Configuration Reference](/configuration-reference) for the `environment` setting.

### Vault

On first boot, the `vault-init` container seeds Vault with default secrets for the bundled services (MinIO, ActiveMQ, MongoDB, PostgreSQL) plus your AI provider API keys and any external-store credentials (Kafka, vector stores, Snowflake, Databricks) from `.env`. Vault uses durable file storage on the `vault-data` volume, so secrets — including any you add later in the Configuration tab or via taps — **persist across restarts and rebuilds**. `.env` is the first-boot seed only; after that the Configuration tab is the source of truth. See [How Configuration Persists](/ai-configuration#how-configuration-persists) for details and the clean-reset path.

### Vector Databases

pgvector is included whenever Postgres is enabled (bundled or external). The others are one line in `.env`:

* **Qdrant** — high-performance vector database: `COMPOSE_PROFILES=qdrant`
* **Weaviate** — open-source vector database: `COMPOSE_PROFILES=weaviate`
* **Chroma** — lightweight, single container: `COMPOSE_PROFILES=chroma`
* **Milvus** — scalable vector database; runs external-only (it needs its own etcd/minio stack — see the Milvus standalone install), then set `MILVUS_HOST`/`MILVUS_PORT` in `.env`

Combine profiles with commas: `COMPOSE_PROFILES=qdrant,kafka`. To use a managed/cloud vector store instead of a bundled one, skip the profile and set the store's `*_HOST` / `*_PORT` / `*_API_KEY` variables in `.env` (first boot seeds the credentials into Vault; afterwards manage them in Configuration → Secrets).

### External databases and stores

Every optional store can point at infrastructure you already run instead of a bundled container — the installer asks, or set the variables in `.env` before first boot: `POSTGRES_JDBC_URL`/`POSTGRES_USER`/`POSTGRES_PASSWORD` (with `POSTGRES_ENABLED=0`), `KAFKA_BOOTSTRAP_SERVERS`, vector-store `*_HOST` variables, and `SNOWFLAKE_*` / `DATABRICKS_*` destination credentials. These are **first-boot seeds**: vault-init writes them into Vault create-if-absent, and from then on Configuration → Secrets is the source of truth. The platform's availability probes treat external stores exactly like bundled ones — the pipeline wizard and the Assistant offer whatever is actually reachable.

## Configuration

The pipeline server reads configuration from `application.yaml`, mounted from `docker/config/application.yaml`.

See [Configuration Reference](/configuration-reference) for the full list of properties.

## JVM Heap Sizing

The `datris` service runs a Spring Boot JVM. Its heap is governed by the `JAVA_OPTS` environment variable, passed in via `docker-compose.yml`. The default is a **2 GB maximum heap**:

```
JAVA_OPTS=-Xms512m -Xmx2g   # default
```

That default is the same for every install size. What changes the memory footprint is which optional services run alongside it, measured on a live stack:

| Install | Running services | Memory (approx.) |
| - | - | - |
| [Minimal](#minimal-install) (`TEI_ENABLED=0`, `POSTGRES_ENABLED=0`) | datris, MongoDB, MinIO, ActiveMQ, Vault, UI, MCP server, tap runner | 3.5 GB |
| Default | minimal + Postgres | 4 GB |
| Default + bundled TEI embedder | the full stack; TEI alone uses about 3 GB | 7 GB |

So plan on **4 GB of Docker memory** for a minimal or default install without TEI, and **8 GB** when the bundled embedding server runs. Override `JAVA_OPTS` in your `.env` file on larger hosts if you hit `OutOfMemoryError` during heavy ingestion, embedding batches, or large query results. Suggested sizings:

| Host RAM | Suggested `-Xmx` |
| - | - |
| 16 GB | `-Xmx4g` |
| 24 GB | `-Xmx8g` |
| 32 GB+ | `-Xmx12g` |

Example for a 24 GB host:

```bash theme={null}
# .env
JAVA_OPTS=-Xms1g -Xmx8g
```

Then restart the service:

```bash theme={null}
docker compose up -d datris
```

**Docker Desktop note (macOS / Windows):** Docker Desktop runs containers inside a Linux VM with its own RAM allocation, which may be much lower than your host's total RAM. Check the VM ceiling with:

```bash theme={null}
docker info --format '{{.MemTotal}}' | awk '{printf "%.1f GB\n", $1/1024/1024/1024}'
```

If the VM is capped below what you need, raise it in Docker Desktop → Settings → Resources → Memory. The JVM `-Xmx` must be smaller than the Docker VM's allocation, otherwise the kernel inside the VM will OOM-kill containers under load.

**Runner limits.** The two script runners carry their own ceilings so a burst of taps or AI rules cannot exhaust the host: `datris-tap-runner` at 1 CPU / 2 GB / 256 processes and `datris-codegen-runner` at 1 CPU / 1 GB / 128 processes, each running at most 4 scripts at once (further runs wait their turn). These are limits, not reservations, so the memory figures above do not grow. On a larger host that runs many taps on overlapping schedules, raise them together in `.env`:

| Variable | Default | Raise when |
| - | - | - |
| `TAP_RUNNER_MAX_CONCURRENT` | `4` | Tap runs show `queued` in their history, or `datris doctor` warns on `runner.concurrency` |
| `TAP_RUNNER_CPUS` | `1` | The host has 2 or more CPUs to spare: `2` is a good value for most hosts. Never set it above the host's CPU count; Docker then refuses to create the container |
| `TAP_RUNNER_MEMORY` | `2g` | You raise `TAP_RUNNER_MAX_CONCURRENT` (about 512 MB per extra concurrent script) |
| `CODEGEN_RUNNER_MAX_CONCURRENT` / `CODEGEN_RUNNER_MEMORY` | `4` / `1g` | Many pipelines run AI rules or transformations at the same time |

See [Overload protection](/tap-execution-isolation#overload-protection) for every setting.

## Building from Source

For development or contributing:

### Prerequisites

* Java 17+
* [SBT](https://www.scala-sbt.org/download.html)

### Build and run

```bash theme={null}
# Build the server JAR
sbt clean assembly

# Start with local builds (edit docker-compose.yml to uncomment build: lines)
docker compose up --build
```

In `docker-compose.yml`, uncomment the `build:` lines and comment out the `image:` lines for the services you want to build locally:

```yaml theme={null}
datris:
  # image: datrisai/datris-server:latest
  build: .  # Build from source
```


# Airflow provider for Datris: trigger and monitor taps from a DAG
Source: https://docs.datris.ai/integrations/airflow

Install airflow-provider-datris from PyPI, add a Datris connection, and run Datris taps as Airflow 3 tasks with streamed logs and XCom metrics.

<Note>
  **Beta.** The Airflow provider is new. Core functionality is stable, but field
  names and operator arguments may evolve based on feedback.
</Note>

Datris fits the modern data stack as **execution**: Airflow is the orchestrator,
Datris runs the work. The `airflow-provider-datris` package ships an operator
that triggers a Datris tap, waits for the resulting pipeline to reach a terminal
state, streams Datris logs into the Airflow task log, and surfaces run tokens and
metrics as XComs.

No execution moves into Airflow — it only triggers and observes. If you don't run
Airflow, nothing changes: Datris's built-in scheduler keeps working as before.

## Install

```bash theme={null}
pip install airflow-provider-datris
```

The provider targets Airflow 3.x.

## Configure a connection

Add a connection of type **Datris** in the Airflow UI (or via env var / CLI):

| Field | Maps to | Required |
| - | - | - |
| Host | Datris base URL | yes |
| Password | API key (`x-api-key`) | only when the Datris install requires keys |
| Extra | `{"verify": true, "timeout": 30}` | no |

**When is the API key required?**

* **Local / single-tenant with `useApiKeys=false` (default):** leave the password
  blank — the header is ignored.
* **Single-tenant with `useApiKeys=true`, or a multi-tenant install:**
  set the password to an API key that holds the `tap:run` capability for the
  taps you trigger.

<Warning>
  API keys are capability-scoped. A key scoped `tap:run:owner=self` can only run
  taps it created. For an Airflow connection, use a full-access key, or a key with
  an unscoped `tap:run` capability that also created the taps it triggers. See
  [API Keys](/api-keys).
</Warning>

## Trigger a tap

```python theme={null}
from datris_provider.operators import DatrisRunTapOperator

ingest_customers = DatrisRunTapOperator(
    task_id="ingest_customers",
    tap_name="customers_pg_to_snowflake",
    datris_conn_id="datris_default",
    wait_for_completion=True,
    poll_interval=15,
)
```

The operator:

* Calls `POST /tap/run` with `mode=run`.
* Polls `GET /pipeline/status?publishertoken=...&withrollup=true` to a terminal
  state.
* Streams Datris log lines into the Airflow task log on each poll.
* Pushes `publisher_token`, `pipeline_tokens`, and final metrics
  (`row_count`, `duration_ms`) as XComs.
* On Airflow task timeout or DAG cancellation, calls `POST /job/kill` so the
  Datris job transitions to `CANCELLED`.

### Date-windowed backfills

Pass per-run params via `tap_params` (named to avoid Airflow's reserved
`params` attribute). Each value is exposed to the tap script as a
`DATRIS_TAP_PARAM_<key>` environment variable, so the script can read a run-specific window. Values are
Jinja-templated by Airflow:

```python theme={null}
DatrisRunTapOperator(
    task_id="ingest_daily",
    tap_name="orders_api_to_postgres",
    datris_conn_id="datris_default",
    wait_for_completion=True,
    tap_params={"since": "{{ ds }}", "until": "{{ next_ds }}"},
)
```

Inside the tap script, read them as env vars (e.g. `os.environ["DATRIS_TAP_PARAM_since"]`).

## One scheduler at a time

A tap should be scheduled by **either** Datris **or** Airflow — never both, or it
fires twice. The rule is simple and based on the tap's cron:

* **Tap has a cron schedule** → Datris's built-in scheduler owns it. The operator
  **refuses to trigger** it and fails the task, so you can't accidentally
  double-fire.
* **Tap has no cron (manual-only)** → Airflow owns scheduling. The operator
  triggers it normally.

So to drive a tap from Airflow, **leave its cron empty** in Datris (don't enable
a schedule on the tap). If you see the operator fail with a "tap is
Datris-scheduled" error, remove the tap's cron — or remove the Airflow DAG and
let Datris run it on its cron.

## Retries and idempotency

Datris debounces rapid duplicate `mode=run` calls for the same tap. If a retry
lands inside the debounce window, `/tap/run` returns
`status="skipped", persistedReason="debounced"`. The operator treats this as
**success with a warning** — the in-flight run is the real one, and failing the
task would mask it.

## Troubleshooting

| Symptom | Cause / fix |
| - | - |
| `401`/`403` from `/tap/run` | Missing or wrong API key, or the key lacks `tap:run` for that tap. Check the connection password and key capabilities. |
| Task succeeds but no data loaded | The tap returned zero records, or has no target pipeline. Check the tap's Run History in Datris. |
| Operator fails: "tap is Datris-scheduled" | The tap has a cron, so Datris already schedules it. Remove the tap's cron to let Airflow drive it (or remove the DAG). |
| `params` not visible in the script | Read them as `DATRIS_TAP_PARAM_<key>` environment variables, not function arguments. Nested values arrive JSON-encoded. |

## What's not covered yet

Deferrable/async operators, a standalone sensor, file-upload pipeline operators,
and Airflow Datasets / OpenLineage lineage are planned for a future release.


# MCP server for data ingestion and ETL
Source: https://docs.datris.ai/mcp-server

Give Claude, Cursor, or any MCP client a governed way to create pipelines, run taps, load data into Postgres, Snowflake, Databricks, S3, or a vector store, and verify it landed.

The pipeline includes a built-in [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server that lets AI agents interact with the platform natively. Any MCP-compatible agent — Claude Desktop, Claude Code, Cursor, or custom agentic frameworks — can discover database metadata, create and manage pipelines, upload files, monitor jobs, profile data, search vector databases, query structured data, answer questions with AI-powered RAG, and upload configuration files — all without custom integration code.

<Note>
  **Governed when the controls are on.** With API keys, the agent policy and the audit log switched on (see [Switch on the governance controls](/quick-start#2-switch-on-the-governance-controls)), the tool catalog an agent sees is filtered to what its API key's capabilities allow, so a `read-only` key never sees `create_tap`. Every call that changes state is checked against the [agent policy](/agent-policy) — run, wait for a person, or refuse — and recorded in the [audit log](/audit-log) under the key's label (best effort by default; [strict mode](/audit-log#strict-mode) refuses a call rather than lose its entry). See [Authentication](#authentication) below for issuing per-agent keys.
</Note>

The MCP server is a lightweight Python service that routes all operations through the pipeline's REST API. It runs alongside the pipeline in Docker or locally for development.

## Resources

The MCP server exposes resources that agents can read on demand for detailed documentation.

| Resource URI | Description |
| - | - |
| `datris://pipeline-config-reference` | Complete reference for building pipeline configurations — all source types, data quality rules, transformations, and destination types with JSON examples |
| `datris://tap-workflow-reference` | Canonical reference for tap work: creation (instruction vs. script), per-run `params` with the script-side env-var contract, incremental sync via persistent state (`DATRIS_TAP_STATE` bookmarks — what to track, monotonic-cursor and backfill rules), the scheduling rule with a Quartz CRON cookbook, the test-before-run validation discipline, the params-vs-secrets table, run flow + `persistedReason` handling, document-tap ledger, and outcome verification via `publisherToken` + `get_tap_logs`. Re-readable on demand mid-session whenever an agent needs to verify its mental model. |

## Available Tools

### Pipeline Management

| Tool | Description |
| - | - |
| `list_pipelines` | List all registered pipeline configurations |
| `get_pipeline` | Get a specific pipeline configuration by name |
| `create_pipeline` | Create a pipeline from sample data (base64-encoded). Schema is auto-detected. Supports three destination categories: structured (postgres, mongodb, snowflake, databricks), object store (Parquet files, ORC files, or an Iceberg table in MinIO / AWS S3), and vector (pgvector, qdrant, weaviate, milvus, chroma). For S3, set `provider: "s3"` and reference a Platform-tab `credentialsSecret`; Snowflake and Databricks likewise take a Platform-tab `credentialsSecret` plus a warehouse and database/catalog. Optional `catalog` to group related pipelines. |
| `delete_pipeline` | Delete a pipeline and its destination data |
| `list_pipeline_versions` | List a pipeline's definition-change history (newest first): `version`, `createdAt`, `createdBy`, `changeNote`. An **empty** result means it hasn't been edited since versioning was enabled — not that it has no version; the current number is the `version` field on `list_pipelines` / `get_pipeline` (≥ 1 for every pipeline) |
| `get_pipeline_version` | View one historical snapshot of a pipeline's definition — the full config as it was at that version. Read-only |
| `diff_pipeline_versions` | Compare two pipeline versions — server-computed field-by-field config diff. `version` is the selected/newer snapshot, `against` the baseline |
| `restore_pipeline_version` | Roll a pipeline back (or forward) to a prior version. **Append-only**: writes the chosen snapshot as a new latest version; nothing is overwritten. "Rolling forward" is the same call with a higher version |
| `upload_data` | Upload data (base64-encoded) to a pipeline for processing. New CSV columns are auto-added to the schema ([schema evolution](/schemas#schema-evolution)). Returns pipeline token. |
| `get_job_status` | Get job status. Pass `pipeline_token` for a `{rollup, events}` response — poll `rollup.allDone`, then read `rollup.status` (`success` / `warning` / `error`) and per-job `lastError`. Pass `pipeline_name` for a paginated summary across recent jobs. |
| `set_catalog` | Set or clear the catalog grouping label on an existing pipeline or tap. Pass exactly one of `pipeline` or `tap`. |
| `rename_catalog` | Rename a whole catalog: every tap and pipeline in it moves to `new_name` in one call. The new name must not already exist (a rename never merges; move items with `set_catalog` to combine catalogs). Returns per-item `failed` entries and `affectedKeys` (API keys still scoped to the old name). Uncataloged cannot be renamed |
| `delete_catalog` | Delete a catalog by moving every tap and pipeline in it to Uncataloged. No tap, pipeline or data is deleted; per-item failures come back under `failed`. Uncataloged cannot be deleted |
| `export_catalog` | Back up a catalog as one zip archive: every tap and pipeline in it, each tap's script and incremental state, and referenced validation schemas. Read-only. Returns `manifest` (catalog, taps, pipelines, `requiredSecrets`, warnings), then `archiveBase64` and `sizeBytes`. Secret values are never included. Uncataloged cannot be exported |
| `restore_catalog` | Restore an `export_catalog` archive (`archive_base64`) as `catalog`, under the same or a new name. Never merges: refused with the clashing names when the catalog or any archived tap or pipeline name already exists. `mode=overwrite` replaces items already in the catalog and needs `confirm` equal to the name; `restore_state=false` skips tap incremental state. Restored taps are disabled; missing secrets come back under `missingSecrets`, unrestored items under `failed` |
| `catalog_enable_order` | The order to enable a catalog's taps in: levels, taps that read no other tap's tables first. Per tap: `hasPassingTest`, `lastRealRunLanded`, `blockedBy` (sources that have not landed data yet) and `ready`; taps on a dependency cycle under `cycles`. Read-only; needs `tap:read`. `create_tap` and `update_tap` take an optional `depends_on` list to declare dependencies instead of the inferred ones |
| `enable_catalog_ordered` | Start one job that enables a catalog's taps in their enable order: per tap, test if needed, enable, one real run, then wait for the data to land before the taps that read it. Taps whose sources did not land are skipped with the reason. Returns `jobId` and `plan`; progress is the `job` block of `catalog_enable_order`. Needs `tap:run` (and update + run on every tap in the catalog); agent policy action `tap:run:enable-ordered`. Cancel is in the UI or REST only |
| `kill_job` | Kill a running job by pipeline token |
| `profile_data` | AI-profile data (base64-encoded) with summary stats and suggested DQ rules |
| `get_dest_types` | Propose real column types for a pipeline whose destination columns are all stored as text: samples up to 1000 landed rows, infers types deterministically and returns per-column evidence. Postgres, Snowflake and Databricks destinations only; nothing is changed by this call |
| `apply_dest_types` | Apply destination column types to an all-string pipeline (requires explicit user approval). `fields` must list every destination column with its intended type (`string`, `boolean`, `int`, `bigint`, `float`, `double`, `date`, `timestamp`); landed data is migrated first and any value that will not cast fails the whole apply with nothing changed |
| `suggest_field_protection` | Suggest per-field protection (`hmac`, `mask` with optional `preserve`, `redact`, `drop`, or none, each with a reason) from field names and types only — no values are sent to the model and nothing is saved. Pass `pipeline` or `fields`; the agent shows the suggestions and asks before passing any as `protect` to `create_pipeline`. See [Field protection](/transformation/field-protection#suggestions). |
| `get_version` | Get pipeline server version and the tap budgets in force on this install (`pipelineMaxPayloadMB`, `tapScriptTimeoutSeconds`, `tapRunTimeoutSeconds`, each with a `...Source` of `env`, `deprecated-alias`, `derived` or `default`). Call it before sizing a large source and compare estimates with these values, not with the documented defaults |
| `check_service_health` | Check which backend services are up, down, or not configured (slow — use for diagnostics only) |
| `run_doctor` | Operational self-check: Vault token expiry, AI slot secrets, embedding model, disk, version skew, optional AI probe — each finding with its fix (diagnostics only; see [Doctor](/doctor)) |

### Vector Database Search

Semantic search across any of the pipeline's supported vector databases. Each tool takes a natural language query and returns the most similar document chunks with scores and metadata.

| Tool | Description |
| - | - |
| `search_qdrant` | Search a Qdrant collection |
| `search_weaviate` | Search a Weaviate class |
| `search_milvus` | Search a Milvus collection |
| `search_chroma` | Search a Chroma collection |
| `search_pgvector` | Search a pgvector PostgreSQL table |

### Database Queries

Read-only queries against the pipeline's backend databases.

| Tool | Description |
| - | - |
| `query_postgres` | Execute a read-only SQL SELECT query against PostgreSQL |
| `query_mongodb` | Query a MongoDB collection with filter and projection |
| `query_objectstore` | Read rows from a pipeline's Object Store destination (Parquet files, ORC files, or an Iceberg table in MinIO or AWS S3). For an Iceberg table the response also carries the snapshot id (as a string) and timestamp that were read. Pass the pipeline name; the server resolves bucket, prefix, format, and credentials from the pipeline config. |
| `query_snowflake` | Run a read-only query (SELECT/WITH, SHOW, DESCRIBE) against the Snowflake account a pipeline loads into. Pass the pipeline name; the server resolves the account, credentials, warehouse, and role from the pipeline config — credentials never leave the server. Omit `sql` to preview the pipeline's destination table. Metadata discovery uses the same tool via `SHOW DATABASES` / `SHOW SCHEMAS` / `SHOW TABLES` / `DESCRIBE TABLE`. |
| `query_databricks` | Run a read-only query (SELECT/WITH, SHOW, DESCRIBE) against the Databricks workspace a pipeline loads into. Pass the pipeline name; the server resolves the workspace, credentials, SQL warehouse, and catalog from the pipeline config — credentials never leave the server. Omit `sql` to preview the pipeline's destination table. If the SQL warehouse is stopped, the first query auto-starts it and may take longer. |
| `query_natural` | Ask a question in natural language — AI generates and executes SQL |

### Metadata Discovery

Explore the structure of PostgreSQL, MongoDB, and vector databases managed by the platform. Use these tools to understand what data is available before writing queries or running searches.

| Tool | Description |
| - | - |
| `list_postgres_databases` | List all PostgreSQL databases |
| `list_postgres_schemas` | List schemas in a PostgreSQL database |
| `list_postgres_tables` | List tables in a schema (supports vector-only filter) |
| `list_postgres_columns` | List columns and types for a specific table |
| `list_mongodb_databases` | List all MongoDB databases |
| `list_mongodb_collections` | List collections (optionally filtered by database) |
| `list_qdrant_collections` | List all collections in Qdrant |
| `list_weaviate_classes` | List all classes in Weaviate |
| `list_milvus_collections` | List all collections in Milvus |
| `list_chroma_collections` | List all collections in Chroma |
| `list_pgvector_collections` | List all pgvector tables in PostgreSQL |

### AI

| Tool | Description |
| - | - |
| `ai_answer` | Answer a question using AI based on provided context (RAG). Accepts an optional `sources` array of provenance handles, echoed back on the response so the answer and its provenance travel together |

### Taps

| Tool | Description |
| - | - |
| `create_tap` | Create a tap from an instruction (AI generates script), a user-provided script, or config only. Optionally set target pipeline, CRON schedule, secret name, and `tap_type` (`structured` or `document`) |
| `list_taps` | List all taps with status, `tapType`, target pipeline, schedule, and last run info |
| `get_tap` | Get full details of a single tap including its Python script source |
| `run_tap` | Execute a tap and push to the target pipeline. Accepts an optional `params` object — per-call values (date windows, ticker lists, page cursors, etc.) get injected as `DATRIS_TAP_PARAM_<key>` env vars the script can read for that one run only. Response carries `persisted` + `persistedReason` (`success`, `no_records`, `run_error`, `no_target_pipeline`, `test_mode`, `debounced`), `recordCount`, `publisherToken`, and `pipelineTokens` for watching async loads. Records themselves are not returned — use `test_tap` to preview what a script produces. A run whose staged payload exceeds the per-run disk budget `PIPELINE_MAX_PAYLOAD_MB` (default 4096 MB, 0 = unlimited; the value in force is in `get_version`) stops with an actionable error |
| `test_tap` | Test-run a tap without pushing data to the pipeline. Previews the first 20 records by default — pass `limit` to change the cap (`limit: 0` = no cap, streams the whole source), and read the returned `recordCount` as the preview size, not what `run_tap` will produce. **Call this before the first `run_tap` of any newly-created or just-updated script**, and before setting a `cron_expression` on a never-validated script — setting a cron on an untested script ships a guaranteed-bad nightly run |
| `update_tap` | Update a tap's CONFIG (enabled, schedule, pipeline, description) without touching the script. To replace the SCRIPT itself, call `create_tap` again with the same name and a new `script` or `instruction` — `create_tap` upserts by name |
| `get_pipeline_status` | Watch ingestion progress after `run_tap`. Pass the response's `publisherToken` (covers every job the run submitted — structured = 1, document = N) or a single `pipelineToken`. Returns a `rollup` (with `allDone`, `status`, per-job `lastError`) plus the raw `events`; poll until `rollup.allDone` is `true` |
| `get_pipeline_result` | Read the rows a [Live Read](/destinations/live-read) pipeline (the `scratch` destination) produced. Only Live Read pipelines have a result — anything else is a 404. The first rows already ride on the `get_pipeline_status` rollup as `resultPreview`, so call this only when `resultTruncated` is `true`. Pass `publisher_token` or `pipeline_token` plus optional `offset` / `limit`; `limit` is clamped server-side, so advance by `offset + returnedCount`. Paging never re-runs the source. Results expire after the retention window — a 410 means run the pipeline again |
| `wait_seconds` | Sleep 1–120 seconds, then return. Use to pace polling of long-running pipeline work without burning tool calls — typical pattern is `run_tap` → poll → `wait_seconds(5)` → poll → `wait_seconds(10)` → poll → ... with exponential backoff capped at 60s (120s only if progress is genuinely glacial). Reset to a short wait whenever a poll shows new jobs flipped to a terminal state |
| `get_tap_logs` | Get run history for a tap (last 50 entries with status, duration, errors, and `publisherToken` for each run that submitted records). Pivot from a scheduled-run entry to `get_pipeline_status(publisher_token=...)` to confirm the run actually landed in the destination |
| `get_tap_ledger` | Document taps only: read the ledger of discovered documents (URI, filename, status, hashes, timestamps). Pass `clear_uri` to force-reprocess one file, or `clear_all=true` to force a full re-scan |
| `get_tap_state` | Read a tap's incremental-sync state — the bookmark/cursor its script committed after the last successful run (also included in `get_tap`). `state` is `null` for non-incremental taps or before the first successful run |
| `set_tap_state` | Overwrite a tap's state (e.g. rewind a cursor so a window is re-fetched), or pass `reset=true` to clear it so the next run does a full first-run fetch. The state shape is defined by the tap's own script |
| `create_tap_secret` | Create or update a Vault secret for a tap to read as env vars (tagged `_type=tap`). Fails on collision unless `overwrite=true`. Reserved AI-provider names are blocked |
| `delete_tap_secret` | Delete a tap secret. Only `_type=tap` secrets can be removed — human-owned secrets must be deleted from Configuration → Secrets in the UI |
| `list_tap_secrets` | List the names of existing tap secrets (secrets tagged `_type=tap`). Call before `create_tap_secret` to reuse a suitable existing secret rather than asking the user for credentials again |
| `get_tap_secret_fields` | Return the field NAMES (keys only — never values) of an existing tap secret. Use after `list_tap_secrets` to verify a candidate secret has the keys the tap script will need |
| `delete_tap` | Delete a tap and its stored script (also clears the ledger and staged MinIO objects for document taps) |
| `list_tap_versions` | List a tap's definition-change history (newest first): `version`, `createdAt`, `createdBy`, `changeNote`. The platform snapshots a tap's config + script on every create/update. An **empty** result means the tap hasn't been edited since versioning was enabled — not that it has no version; the current number is the `version` field on `list_taps` / `get_tap` (≥ 1 for every tap) |
| `get_tap_version` | View one historical snapshot of a tap's definition — the full config and the pinned Python script as they were at that version. Read-only |
| `diff_tap_versions` | Compare two tap versions — server-computed field-by-field config diff plus a line-level script diff. `version` is the selected/newer snapshot, `against` the baseline |
| `restore_tap_version` | Roll a tap back (or forward) to a prior version. **Append-only**: writes the chosen snapshot as a new latest version (config + that version's script); nothing is overwritten. "Rolling forward" is the same call with a higher version. Does **not** run the tap |

For a document tap, `create_tap` **requires** the `target_pipeline` to have an `unstructuredAttributes` source and a vector-store destination (qdrant, pgvector, weaviate, milvus, or chroma). The server rejects mismatched pairings with HTTP 400.

### Configuration

| Tool | Description |
| - | - |
| `upload_config` | Upload a JSON Schema config file (base64-encoded content) |
| `update_secret` | Update an AI provider secret (anthropic, openai, azure, grok, ollama, embedding) to configure API keys |
| `list_platform_secrets` | List the names of human-owned platform secrets (the Platform tab in Configuration → Secrets). These hold destination credentials and infrastructure connections. Read-only — agents cannot create, modify, or delete them. |
| `get_platform_secret_fields` | Return the field NAMES (no values) of a platform secret. Use after `list_platform_secrets` to verify a candidate has the required keys for a destination — e.g. an S3 `credentialsSecret` must contain `accessKey`, `secretKey`, `region`. |

### Agent policy and approvals

| Tool | Description |
| - | - |
| `get_agent_policy` | Read this instance's [agent policy](/agent-policy): for each action whether an agent may do it on its own, must wait for a person to approve it, or is refused. Call before a delete or destination-type migration to know whether it will run or queue |
| `list_pending_approvals` | The actions this agent queued for approval, with their state |
| `get_approval` | Poll one queued approval by id until it is executed, rejected or expired |

Every tool that changes platform state also accepts an optional `reason` — one line the agent gives for the change. It is recorded in the [audit log](/audit-log) and shown on the approval card. When the policy requires approval, the mutating tool returns `status: pending_approval` with an `approvalId` instead of performing the action.

### Incidents (recovery agent)

| Tool | Description |
| - | - |
| `list_incidents` | The platform's [recovery-agent incidents](/incidents), newest first — kind, resource, state, classification and narrative. Read-only: only the platform opens incidents |
| `get_incident` | One incident by id, with its full step-by-step story and any approvals it waits on. When the operator asks about a failure the platform is already working, explain this record instead of re-diagnosing |

### Discovery and provenance

| Tool | Description |
| - | - |
| `find_data` | Find datasets by meaning: ranks the pipelines the key can read against a natural-language query and returns location, [freshness](/provenance), provenance handles, lineage and a pre-filled `howToQuery` hint naming the existing query/search tool. Discovery only — the query call stays the agent's own. With `include_unity_catalog`, also searches the Unity Catalog tables visible to the Databricks secrets the key can read. See [Unity Catalog](/destinations/unity-catalog) |
| `get_provenance` | Resolve a stamped `_datris_run_id` back to its origin: pipeline run, tap run, script commit, config version and declared source. See [Provenance](/provenance) |
| `get_lineage` | Traverse the lineage graph from one node — upstream, downstream or both, to a bounded depth — with freshness, optionally the most recent recorded runs (what each read and wrote, per destination), and optionally column-level lineage for a pipeline or dataset. Use it for impact analysis before changing or deleting something. See [Lineage](/provenance#lineage--freshness) |
| `browse_unity_catalog` | Browse what a Databricks Platform secret can see in Unity Catalog, one level per call: catalogs, schemas, tables (marking tables a Datris pipeline loaded via the `datris_pipeline` tag), then columns and tags. No pipeline needed. See [Unity Catalog](/destinations/unity-catalog) |

## Monitoring Active Agents

The [Datris UI](http://localhost:4200) includes an **Agent Monitor** tab that shows a live view of every agent currently connected to the MCP server and a streaming log of the tool calls each one is making. Agents are labeled using their MCP `clientInfo.name` — so Claude Desktop appears as `claude-ai`, Claude Code as `claude-code`, Cursor as `cursor`, and so on — with graceful fallbacks to tenant name, API-key name, or session id for clients that don't supply one. See [Monitoring → Agent Monitor](/monitoring#agent-monitor) for details.

## Setup

### Docker (automatic)

The MCP server starts automatically with `docker-compose up` in SSE mode on port 3000. No additional setup required.

### Local (for Claude Desktop / Claude Code)

The MCP server is published on PyPI. Use `uvx` to run it directly:

```bash theme={null}
uvx datris-mcp-server
```

## Transport Modes

| Mode | Use Case | Command |
| - | - | - |
| **stdio** | Claude Desktop, Claude Code, local agents | `python server.py` |
| **SSE** | Docker, remote agents, web clients | `python server.py --sse --port 3000` |
| **Streamable HTTP** | Remote agents using the streamable HTTP transport | `python server.py --streamable-http --port 3000` |

## Configuring Claude

Client-side setup for Claude Desktop and Claude Code — including recommended and alternative transports, and example first prompts — lives on its own page: [Configuring Claude](/configuring-claude).

## Environment Variables

| Variable | Default | Description |
| - | - | - |
| `DATRIS_API_URL` | `http://localhost:8080` | Datris REST API server URL |
| `REQUIRE_API_KEY` | `false` | Reject SSE/HTTP sessions that connect without `x-api-key` (and stdio sessions started without `DATRIS_API_KEY`) |
| `DATRIS_API_KEY` | — | Server side, stdio transport only: the API key forwarded as `x-api-key` on every tool call (stdio has no request headers). The `datris` CLI also reads this variable and sends it as the `x-api-key` header on its SSE connection. |

All database connections, vector search, and embedding are handled by the pipeline server. The MCP server only needs the pipeline URL.

## Authentication

The MCP server has no API key of its own. Each connecting agent sends an `x-api-key` header per session and the MCP server forwards it as-is to the Datris REST API on every tool call. Issue and manage agent keys from **Configuration → API-Keys** in the UI.

When the Datris platform has [`USE_API_KEYS=true`](/api-keys), set `REQUIRE_API_KEY=true` on the MCP server too — otherwise the MCP layer accepts unauthenticated connections that then fail with 401 at the REST tier. The pairing is intentional: turn both on, or neither.

Each agent should connect with its own labeled key (e.g. `claude-desktop`, `cursor`, `prod-agent`) so its traffic appears under a distinct identity in the request logs and can be revoked independently. Issue from the UI:

1. **Configuration → API-Keys → Issue new key**
2. Label the key for that agent (`claude-desktop`, `cursor`, etc.)
3. Pick a capability template (`read-only`, `rag-builder`, `ops`) or build a custom scope
4. Copy the value once at issue time and paste it into the agent's MCP client config

The platform's [capability framework](/api-keys#the-capability-model) enforces what each key is allowed to do — a `read-only` agent can't `create_tap` even if its system prompt tells it to try. See [API Keys](/api-keys#issuing-keys-for-cli-mcp-and-other-clients) for the full per-client pattern.

## Example Agent Workflows

<Note>
  The platform's canonical workflow is delivered to every connected agent through the MCP `instructions` field — agents don't need to memorize it. These examples show the same flow applied to common tasks.

  Core rules the agent receives on connect: **check what exists before creating** (`list_pipelines`, `list_taps`); **keep pipeline configs simple** (source + destination only — never pass `profile_data` output into a pipeline config); **after `run_tap`, read `persisted` and `persistedReason` before doing anything else**; **verify real completion via `get_pipeline_status(publisher_token=...)`, not by the response body**; **set `cron_expression` on the tap when the user describes a recurrence — never propose external schedulers or shell snippets**; **call `test_tap` before `run_tap` and before setting a cron on any newly-created or newly-updated script**; and **only narrate work that actually completed in the current turn** — if the tool calls didn't fire, say so honestly rather than confabulating success; a repeated request ("run it again") is a new operation requiring a fresh tool call, never a result forecast from the previous run.

  For long-form tap workflow guidance — params, scheduling, validation, error handling, the Quartz CRON cookbook — agents can re-read the `datris://tap-workflow-reference` resource on demand at any point in a session.
</Note>

For project-level guidance that goes beyond the connect-time instructions, install the [datris-platform skill](/agent-skill) in the project. It tells a coding agent when to reach for Datris at all, and to use these MCP tools first and the REST API only for code it writes into the project.

### Ingest a CSV file directly

1. **Check for an existing pipeline** — `list_pipelines`. If one already fits the data, skip to step 3.
2. **Create a pipeline** — `create_pipeline` with the CSV as sample data. Schema is auto-detected. Keep the config simple; only pass `codegen_rule` / `codegen_transform` if the user explicitly asks for validation or transformation.
3. **Upload** — `upload_data` with the CSV content (base64).
4. **Verify it landed** — `get_job_status` with the `pipelineToken` returned from `upload_data`. Poll until `rollup.allDone` is `true`, then read `rollup.status` (`success`, `warning`, `error`). On failure, `rollup.jobs[].lastError` carries the failing process and message.

### Onboard a new external data source via a tap

1. **Check existing taps** — `list_taps`. If a tap already covers this source, run or test it directly.
2. **Create credentials** — if the source needs an API key, call `create_tap_secret` with the credential fields (they become env vars inside the script).
3. **Create the tap** — `create_tap` with an instruction (AI generates the script) or with your own Python `fetch()` function (faster and more reliable). Pass `secret_name` to bind the credentials from step 2. For PDFs/Word/HTML into a vector-store pipeline, pass `tap_type="document"`.
4. **Test** — `test_tap`. If it fails, read the error, fix the script, and call `create_tap` again. Repeat until the test passes.
5. **Run** — `run_tap`. Read the response's `persisted` field first:
   * `persisted: true` → capture `publisherToken` and continue to step 6.
   * `persisted: false` → read `persistedReason` (`no_target_pipeline`, `test_mode`, `run_error`, `no_records`, `debounced`), tell the user exactly why, and stop. (`run_tap` does not return records — call `test_tap` if you need to preview what the script produces.) `debounced` means the same tap was triggered within the last 5 seconds; the earlier run is still executing — do not retry, look it up via `get_tap_logs` and poll `get_pipeline_status` instead.
6. **Verify it landed** — `get_pipeline_status(publisher_token=...)` and poll until `rollup.allDone` is `true`. Then `rollup.status` is the outcome (`success`, `warning`, `error`) and `rollup.jobs[].lastError` carries any failure detail. The tap log only tells you the script ran; the publisher token is how you confirm records actually reached the destination.
7. **Schedule (optional)** — `update_tap` with a `cron_expression`. For each scheduled run, pick the corresponding entry from `get_tap_logs` to get its `publisherToken`, then verify with step 6.

### Build and query a RAG knowledge base from external documents

1. **Create a vector-store pipeline** — `create_pipeline` with a Qdrant / Weaviate / Milvus / pgvector / Chroma destination and an `unstructuredAttributes` source.
2. **Create a document tap** — `create_tap` with `tap_type="document"` and an instruction describing the document source (e.g., *"ingest every PDF under legal-contracts/2026/ in S3"*). If the source needs credentials, call `create_tap_secret` first.
3. **Test and run** — `test_tap`, then `run_tap`. Read `persisted` / `persistedReason` exactly as above.
4. **Verify every document landed** — `get_pipeline_status(publisher_token=...)`. Document taps fan out to N jobs (one per file) under one publisher token; poll until `rollup.allDone` is `true`. `rollup.jobs[]` has one entry per file with its own `status` and `lastError`.
5. **Inspect the ledger (optional)** — `get_tap_ledger` shows which files were discovered and processed. On subsequent runs, unchanged files are skipped automatically; pass `clear_uri` or `clear_all=true` to force a re-scan.
6. **Search** — `search_qdrant` (or the matching tool for your destination) to retrieve relevant chunks.
7. **Answer** — `ai_answer` with the retrieved chunks as context and the user's question.

### Discover and query existing data

1. **List databases** — `list_postgres_databases`.
2. **List schemas** — `list_postgres_schemas` for the target database.
3. **List tables** — `list_postgres_tables` (supports a vector-only filter).
4. **Inspect columns** — `list_postgres_columns` to understand structure.
5. **Query** — `query_postgres` with a SELECT, or `query_natural` to ask a plain-English question and have the AI generate + run the SQL.

### Cross-modal analysis (structured + vector)

1. **Vector search** — `search_pgvector` (or another vector tool) for relevant document chunks.
2. **Structured query** — `query_postgres` or `query_natural` for related metrics.
3. **Combine** — merge the two result sets in the response.

### Automated quality monitoring of scheduled taps

1. **List taps** — `list_taps` to find the tap of interest.
2. **Read recent runs** — `get_tap_logs` returns the last 50 entries with status, record count, duration, errors, and `publisherToken` for each run that submitted records.
3. **Verify completion** — for any run of interest, call `get_pipeline_status(publisher_token=...)` and read `rollup.status` to confirm records actually landed in the destination. The tap log tells you the script ran; the publisher token tells you the load finished.
4. **Diagnose failures** — on an errored job, read `rollup.jobs[].lastError` (`processName` + `description`) and either fix the tap (`create_tap` again with a corrected script) or retarget the pipeline (`update_tap`).

## CLI Examples

The Datris CLI connects to the MCP server and provides the same capabilities from the terminal. See [CLI](/cli) for the full reference.

```bash theme={null}
# Install
brew tap datris/tap
brew install datris

# Ingest a CSV into PostgreSQL
datris ingest sales-data.csv --dest postgres

# Ingest with AI validation and transformation
datris ingest trades.csv --dest postgres \
  --ai-validate "all prices must be positive and dates must be YYYY-MM-DD" \
  --ai-transform "convert dates to YYYY/MM/DD and uppercase all ticker symbols"

# Ingest into a vector store for RAG
datris ingest manual.pdf --dest pgvector

# Ingest + analyze in one command
datris ingest trades.csv --dest postgres --ai-analyze "What are the top 5 stocks by volume?"
datris ingest report.pdf --dest pgvector --ai-analyze "What was the company's revenue?"

# Analyze existing data (PostgreSQL, MongoDB, or vector stores)
datris analyze "top 5 stocks by volume" --table trades
datris analyze "What is the return policy?" --table support_docs --dest pgvector

# Query PostgreSQL directly
datris query "SELECT * FROM public.sales LIMIT 10"

# Semantic search (raw results)
datris search "quarterly revenue" --store pgvector --collection financial_docs

# List pipelines, check health, get status
datris pipelines
datris health
datris status my_pipeline
```


# Monitoring
Source: https://docs.datris.ai/monitoring

Monitor your pipelines

## Pipeline Tokens

Every pipeline processing job is assigned a unique **pipeline token** — a UUID that identifies the job throughout its lifecycle. You can monitor jobs via the REST API, the MCP server, the CLI, or the [Datris UI](http://localhost:4200).

Pipeline tokens are returned:

* In the response body of `POST /api/v1/pipeline/upload` (for uncompressed files)
* In the job status when querying by pipeline name

## Job Status

### Query by Pipeline Token

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/status?pipelinetoken=pt-abc12345-..."
```

Returns an array of status entries for the job, one per processing stage:

```json theme={null}
[
  {
    "id": 1,
    "dateTime": "2026-03-15T10:00:00Z",
    "pipeline": "sales_data",
    "processName": "StreamNotifier",
    "publisherToken": null,
    "pipelineToken": "pt-abc12345-...",
    "filename": "sales_data",
    "state": "begin",
    "code": "begin",
    "description": "Process started",
    "epoch": 1710500400000
  },
  {
    "id": 2,
    "dateTime": "2026-03-15T10:00:01Z",
    "pipeline": "sales_data",
    "processName": "DataQuality",
    "publisherToken": null,
    "pipelineToken": "pt-abc12345-...",
    "filename": "sales_data",
    "state": "processing",
    "code": "processing",
    "description": "Running CodeGen data quality rule",
    "epoch": 1710500401000
  },
  {
    "id": 3,
    "dateTime": "2026-03-15T10:00:03Z",
    "pipeline": "sales_data",
    "processName": "PostgresLoader",
    "publisherToken": null,
    "pipelineToken": "pt-abc12345-...",
    "filename": "sales_data",
    "state": "end",
    "code": "end",
    "description": "Process completed",
    "epoch": 1710500403000
  }
]
```

When a job fails, its `Process completed, error:` entry's `description` gives the error and its causes, one per line, after that prefix; the full stack trace is in that entry's optional `detail` field (shown under **Stack trace** in the Ops activity view, and omitted by the MCP status tools) and in the server log under the job's pipeline token.

### Query by Publisher Token

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/status?publishertoken=pub-abc12345-..."
```

Returns every status row whose `publisherToken` matches — covers all ingestion jobs a single caller submitted. Tap runs set a publisherToken on every job they spawn, so one query covers a structured tap (1 job) or a document tap (N jobs, one per file) in a single call. Use `publisherToken` when you need to watch "this entire run," `pipelineToken` when you need the detail of one specific job.

Add `&withrollup=true` to wrap the response in a `{rollup, events}` object that classifies each job (`success`, `warning`, `error`, `processing`, `timed_out`) and exposes `rollup.allDone` for a single boolean to poll on. See the [status API reference](/api-reference/status-api#rollup-response) for the full shape.

Agents call the same query via the MCP `get_pipeline_status` tool — pass `publisher_token` from a `run_tap` response and poll until `rollup.allDone` is `true`. The MCP tool sets `withrollup=true` automatically. For an `upload_data` flow, `get_job_status` does the same with the `pipelineToken` returned from the upload.

### Query by Pipeline Name

```bash theme={null}
curl "http://localhost:8080/api/v1/pipeline/status?pipelinename=sales_data&page=1"
```

Returns an array of job summaries for the pipeline (20 per page):

```json theme={null}
[
  {
    "createdAtTimestamp": "2026-03-15T10:00:00Z",
    "createdAt": 1710500400000,
    "updatedAt": 1710500403000,
    "pipeline": "sales_data",
    "pipelineToken": "pt-abc12345-...",
    "process": "PostgresLoader",
    "startTime": "2026-03-15T10:00:00Z",
    "endTime": "2026-03-15T10:00:03Z",
    "totalTime": "3s",
    "status": "end"
  }
]
```

## Job Lifecycle

Jobs progress through these states:

| State | Description |
| - | - |
| `INITIALIZED` | Job created, queued for processing |
| `PROCESSING` | Running in a dedicated thread |
| `COMPLETED` | Finished (check status messages for success or error) |
| `CANCELLED` | Job was killed via the `kill_job` API or MCP tool |

## Processing Stages

Each job logs status messages as it progresses through stages:

1. **FileNotifier** / **StreamNotifier** - Initial file or stream intake
2. **DataQuality** - Validation (if configured)
3. **Transformation** - Data transformation (if configured)
4. **JobRunner** - Orchestration of destination loaders
5. **\[LoaderName]** - Each destination loader (e.g., `PostgresLoader`, `SparkObjectStoreLoader`)

Each stage logs `begin`, `processing` (with details), and `end` messages.

## Status Storage

Job statuses are stored in MongoDB in the `{environment}-pipeline-status` collection. Each entry contains the pipeline token, process name, status, message, and timestamp.

## Concurrent Job Handling

* All destination loaders for a single job execute in parallel on a 20-thread pool
* Jobs targeting the same database table are serialized (only one runs at a time)
* Multiple jobs for different pipelines run concurrently

## Datris UI

The [Datris UI](http://localhost:4200) provides a visual interface for managing your entire Datris platform. Six top-level tabs — **Assistant** (in-product agent), **MCP** (server status, tools, and a pop-out Agent Monitor for live AI agent activity), **Catalog** (taps and pipelines grouped by catalog), **Search** (semantic search across vector destinations), **Ops** (ingestion activity dashboard, job history, file-upload ingestion), and **Configuration** (AI providers, secrets, data sources, code repository, users, API keys, audit log, agent policy) — cover everything you can do through the API.

## Ops Activity Dashboard

The **Ops → Activity** tab is the at-a-glance health view for your platform. It pulls every tap run and pipeline job in a rolling window (24h / 7d / 30d, switchable) and surfaces what landed, what's broken, and what hasn't refreshed when it should.

**KPI tiles** across the top show total runs, successes, failures, and items ingested for the window. A failing tile turns red and links to the Failures pane below.

**Runs over time** and **Items ingested over time** are stacked time-series charts. Runs are colored by outcome (success green, warning amber, error red) so a spike of failures is immediately visible against the success baseline.

**Failures pane** lists every tap or pipeline that had a failure in the window, deduped down to one row per unique item with an `(N attempts)` count next to the reason. Items whose most-recent run is now healthy are marked **✓ recovered** and dimmed at the bottom of the list. Each unrecovered row has a **Re-run** button:

* Tap failure → Re-run triggers the tap directly.
* Pipeline failure with an upstream tap → Re-run triggers that tap to retry the load. Direct-upload pipelines with no associated tap stay un-rerunnable from here (the user has to re-upload the source file).

While a re-run is in flight, the button swaps to an amber spinner with "Running…" and disables itself; on completion the dashboard reloads and the row either disappears (full recovery) or shows the updated count.

**Per-pipeline volume** is a 7-day rolling table at the bottom — each row a pipeline, with today's count, 7-day average, a sparkline, and a vs-avg percentage. Useful for spotting a pipeline that suddenly stopped ingesting or a backfill spike.

**Tap status semantics:** `no_records` (script ran cleanly but returned zero records — legitimate for polling/incremental taps on a quiet day) is treated as healthy, not as a failure. It does not count in the Failures tile and does not appear in the Failures pane. See [Taps → Run Status Outcomes](/taps#run-status-outcomes).

## Automatic Retry of Scheduled Runs

When a scheduled (cron) tap run fails on a transient error — the source briefly unreachable, a rate limit, a network blip — Datris automatically re-runs it a few times with a short delay before flagging it. Only runs that had not yet delivered any data downstream are retried, so a retry can never duplicate records. Manual runs are never auto-retried: running a tap yourself pauses the retry ladder until its next scheduled run.

Defaults: up to 3 retries with 5- and 15-minute backoff. Tune or disable with `cron.retry.enabled`, `cron.retry.cap`, and `cron.retry.backoffMinutes`.

## AI Fix Suggestions

When a failure persists — a scheduled run that is still failing after its retries, or any pipeline job failure — Datris diagnoses it and proposes a fix:

* **Ops → Ingestion**: failed runs show a one-line 💡 suggestion directly in the list; clicking through to the run opens a **Suggested fix** card with the full diagnosis and the concrete change to make.
* **Tap Run History**: failed runs show the same one-line suggestion, with the diagnosis and suggested fix in the expanded entry.

Suggestions are advisory only — nothing is changed automatically. Apply the fix through the tap or pipeline edit dialogs, or ask the Assistant to make the change.

## Agent Monitor

The **Agent Monitor** lives on the MCP tab and shows a live view of every AI agent currently connected to the platform's MCP server, along with a streaming log of the tool calls each agent is making. Both panes (Connections and Activity Log) are sized to fit the viewport on initial load and reflow when you resize the window — the visualization stays roughly the upper third and the log fills the rest, with its own internal scroll. A pop-out icon next to the title opens the same view in a separate browser window so you can park it on a second monitor while you work elsewhere in the UI.

The visualization pane draws one icon per active MCP session on the right of the MCP server icon, connected by a line that pulses whenever a call is in flight. Idle sessions fade out automatically once they disconnect.

Each agent label uses the most descriptive identifier available, in this order:

1. The MCP `clientInfo.name` supplied during the client's handshake (e.g. `claude-ai`, `claude-code`, `cursor`)
2. The tenant name (multi-tenant deployments)
3. The API-key name from the `api-keys` secret (single-tenant deployments with named keys)
4. The API-key prefix
5. The session short-id (last-resort fallback)

The activity log below the visualization lists every tool call as it happens — timestamped per the platform's configured [date format and timezone](/configuration-reference), with the calling agent, tool name, argument preview, record count, response size, status, and latency. Clicking a row expands it to reveal the full arguments and response bodies as pretty-printed JSON. A header toolbar lets you copy the full log (with per-row detail) to the clipboard or clear the on-screen history.

Activity is held in an in-memory ring buffer on the MCP server (the most recent 200 calls) and served via the UI's internal `/api/v1/mcp/activity` proxy. It is not persisted — restarts clear the history. For a durable, queryable record of what each agent (and each human) actually changed, enable the [Audit Log](/audit-log); its entries carry the same key label the monitor shows.

Session liveness is tracked from the MCP connection itself: agents appear when their SSE/HTTP session opens and disappear when it closes. A long-idle agent stays visible as long as the connection is open — useful when an agent is between tool calls and you don't want it disappearing from the Connections pane every few seconds.

The Activity Log re-syncs when you return to the tab. Whether you navigated to another route, switched to a different browser tab, or stepped away for a long enough idle that the browser throttled the polling, the log refreshes to the current server state on return — no stale empty view to confuse the trail.

## CLI

Check job status from the terminal:

```bash theme={null}
datris status my_pipeline
datris health
```


# Pipeline Notifications
Source: https://docs.datris.ai/notifications

Set up pipeline notifications

The pipeline publishes event notifications after each destination loader completes. Notifications are sent to an ActiveMQ **Virtual Topic** and can be consumed by any number of independent subscribers.

## How It Works

The pipeline uses ActiveMQ's [Virtual Destinations](https://activemq.apache.org/virtual-destinations) pattern:

1. Each destination loader (PostgreSQL, MongoDB, Snowflake, Databricks, Kafka, ActiveMQ, Object Store, Qdrant, Weaviate, Milvus, Chroma, pgvector) publishes a notification to `VirtualTopic.{environment}-pipeline-notification`
2. Consumers subscribe by creating a queue named `Consumer.{name}.VirtualTopic.{environment}-pipeline-notification`
3. ActiveMQ automatically copies topic messages to all matching consumer queues
4. Each consumer gets its own independent queue with its own message cursor

No subscription API is needed — consumers simply connect to their named queue.

## Consuming Notifications

### Queue Naming Convention

```
Consumer.{consumer-name}.VirtualTopic.{environment}-pipeline-notification
```

For example, with environment `oss`:

* `Consumer.analytics.VirtualTopic.oss-pipeline-notification`
* `Consumer.audit-log.VirtualTopic.oss-pipeline-notification`
* `Consumer.dashboard.VirtualTopic.oss-pipeline-notification`

Each consumer queue receives a copy of every notification independently.

### Quick Start: Python Consumer (Minimal)

> The bundled ActiveMQ image (`apache/activemq-classic`) enables its STOMP connector on port 61613 by default, but `docker-compose.yml` only publishes 61616 (OpenWire) and 8161 (web console). To connect over STOMP from outside the Docker network, add `"61613:61613"` to the `activemq` service's `ports` list and recreate the container.

```python theme={null}
import stomp
import json

class NotificationListener(stomp.ConnectionListener):
    def on_message(self, frame):
        notification = json.loads(frame.body)
        print(f"Pipeline: {notification['pipeline']}, Destination: {notification['destination']}")

conn = stomp.Connection([('localhost', 61613)])
conn.set_listener('', NotificationListener())
conn.connect('admin', 'admin', wait=True)
conn.subscribe(
    destination='/queue/Consumer.myapp.VirtualTopic.oss-pipeline-notification',
    id=1,
    ack='auto'
)
```

> **Note:** This is a minimal snippet. For a production-ready consumer with automatic reconnect, heartbeats, graceful shutdown, and configurable virtual topic support, see [Topic Subscriber](/examples/topic-subscriber).

### Example: Java/JMS Consumer

```java theme={null}
ConnectionFactory factory = new ActiveMQConnectionFactory("tcp://localhost:61616");
Connection connection = factory.createConnection("admin", "admin");
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("Consumer.myapp.VirtualTopic.oss-pipeline-notification");
MessageConsumer consumer = session.createConsumer(queue);
consumer.setMessageListener(message -> {
    TextMessage text = (TextMessage) message;
    System.out.println("Notification: " + text.getText());
});
```

## Message Filtering

Notifications include JMS message properties that can be used with **JMS selectors** for filtering:

| Property | Description |
| - | - |
| `pipeline` | Pipeline name |
| `destination` | Destination type (`postgres`, `mongodb`, `snowflake`, `databricks`, `objectStore`, `kafka`, `activemq`, `qdrant`, `weaviate`, `milvus`, `chroma`, `pgvector`) |
| `schema` | Database schema (database destinations) |
| `database` | Database name (database destinations) |
| `table` | Table name (database destinations) |
| `topic` | Kafka topic (Kafka destinations) |
| `queueName` | Queue name (ActiveMQ destinations) |
| `prefixKey` | Object store prefix (object store destinations) |

To filter, use a JMS selector when creating the consumer:

```java theme={null}
MessageConsumer consumer = session.createConsumer(queue, "pipeline = 'sales_data' AND destination = 'postgres'");
```

## Notification Payload

```json theme={null}
{
  "pipeline": "sales_data",
  "publisherToken": "publisher-123",
  "pipelineToken": "pt-abc12345-...",
  "destination": "postgres",
  "prefixKey": null,
  "objectStoreUrl": null,
  "objectStoreTemporaryUrl": null,
  "schema": "public",
  "database": "analytics",
  "table": "sales",
  "topic": null,
  "queueName": null
}
```

Fields are populated based on the destination type. Unused fields are `null`.

## Enabling Notifications

Notifications are enabled by default in `application.yaml`:

```yaml theme={null}
sendPipelineNotifications: "true"
```

Set to `"false"` to disable all notifications.


# Pipeline Configuration
Source: https://docs.datris.ai/pipeline-configuration

Configure data pipelines

Pipelines are configured entirely through JSON. Each pipeline defines a source, optional processing steps (preprocessing, data quality, transformation), and one or more destinations.

You don't need to write JSON by hand — once you have the Datris Data Platform up and running, the [Datris UI](http://localhost:4200) provides a step-by-step pipeline wizard that builds the full configuration for you. Just pick your source, destination, and any processing options, and the platform generates the JSON behind the scenes.

## Full Configuration Example

```json theme={null}
{
  "name": "sales_data",
  "source": {
    "schemaProperties": {
      "fields": [
        {"name": "order_id", "type": "int"},
        {"name": "customer_name", "type": "string"},
        {"name": "amount", "type": "double"},
        {"name": "order_date", "type": "date"},
        {"name": "region", "type": "string"}
      ]
    },
    "fileAttributes": {
      "csvAttributes": {
        "delimiter": ",",
        "header": true,
        "encoding": "UTF-8"
      }
    }
  },
  "preprocessor": {
    "endpoint": "http://my-service:8080/preprocess",
    "async": false,
    "bearerToken": "my-token",
    "timeoutMs": 300000
  },
  "dataQuality": {
    "validateFileHeader": true,
    "aiRule": {
      "instruction": "Amount must be a positive number, all dates must be in YYYY-MM-DD format",
      "onFailureIsError": true
    },
  },
  "transformation": {
    "aiTransformation": {
      "instruction": "convert all date values to YYYY-MM-DD format. Trim whitespace from all columns. Remove duplicate rows."
    }
  },
  "destination": {
    "objectStore": {
      "prefixKey": "sales/daily",
      "fileFormat": "parquet",
      "partitionBy": ["region"],
      "deleteBeforeWrite": false,
      "writeMode": "append"
    },
    "database": {
      "dbName": "analytics",
      "schema": "public",
      "table": "sales",
      "keyFields": ["order_id"],
      "usePostgres": true,
      "truncateBeforeWrite": false,
      "useTransaction": true
    },
    "kafka": {
      "topic": "sales-events",
      "keyField": "order_id"
    }
  }
}
```

## Configuration Fields

### Top Level

| Field | Type | Required | Description |
| - | - | - | - |
| `name` | string | Yes | Pipeline name (max 80 characters) |
| `source` | object | Yes | Source configuration |
| `preprocessor` | object | No | Optional REST endpoint called before processing |
| `dataQuality` | object | No | Data validation rules |
| `transformation` | object | No | Data transformation settings |
| `destination` | object | Yes | One or more output destinations |
| `catalog` | string | No | Free-form label that groups related pipelines and taps in the [Data Catalog](/data-catalog). Empty or null = Uncataloged. |
| `tags` | string\[] | No | Free-form discovery labels — `find_data` and the catalog search rank over them. See [Provenance, Lineage & Discovery](/provenance). |
| `provenance` | object | No | `{ "stamp": true }` appends `_datris_*` provenance fields to every row this pipeline lands when stamping succeeds (a row that cannot be stamped loads unstamped unless [strict mode](/provenance#strict-mode) is on; XML is never stamped). Off by default. See [Provenance, Lineage & Discovery](/provenance). |
| `authoritative` | boolean | No | Source of authority. Unset or `true`: this pipeline's dataset may be the system of record (a single destination is, by default). `false`: everything it lands is a derived copy — a rollup, a replica, an index built from another dataset. See [Provenance, Lineage & Discovery](/provenance#source-of-authority). |
| `protection.purgeSource` | boolean | No | Only read when a source field carries `protect`. Default `true`: once the protected copy exists, the ingest object (bucket drop, Kafka temp object, or archive drop) is deleted along with the raw staged file. `false` keeps the ingest object. See [Field Protection](/transformation/field-protection#source-retention). |
| `protection.preset` | string | No | `hipaa-safe-harbor` recognises HIPAA Safe Harbor identifiers from field names: saving is refused while a recognised field has no `protect` (unless exempted), and a recognised column added later by schema evolution is protected in the same run. Delimited (CSV) pipelines only; refused on JSON, XML and unstructured sources. An aid, not a certification. See [HIPAA Safe Harbor preset](/transformation/field-protection#hipaa-safe-harbor-preset). |
| `protection.presetExempt` | array | No | Field names the preset recognises that you keep unprotected on purpose (case-insensitive). Only read when `protection.preset` is set. |
| `destination.authoritative` | string | No | For pipelines with several destinations only: which destination kind (`postgres`, `mongodb`, `snowflake`, …) is the golden copy. Without it, all of them show as *undeclared* in lineage until you decide. |

### Source

| Field | Type | Required | Description |
| - | - | - | - |
| `schemaProperties` | object | Yes\* | Schema definition with field names and types. \*Required for structured/semi-structured data |
| `fileAttributes` | object | No | File format configuration (CSV, JSON, XML, XLS, unstructured) |
| `databaseAttributes` | object | No | Database pull configuration |
| `streamAttributes` | object | No | Stream source configuration (Kafka) |

### Source > Schema Properties

| Field | Type | Description |
| - | - | - |
| `dbName` | string | Database name associated with this schema (used by some destinations) |
| `fields` | array | List of `{"name": "...", "type": "..."}` objects |
| `fields[].protect` | object | Optional. `{"method": "hmac" \| "mask" \| "redact" \| "drop"}`; with `mask`, optional `"preserve": "last4" \| "domain" \| "year" \| "first3"`. Applied before data quality, transformation, and every destination. See [Field Protection](/transformation/field-protection). |
| `schemaVersion` | int | Schema version (default `1`) |

Supported types: `boolean`, `int`, `tinyint`, `smallint`, `bigint`, `float`, `double`, `decimal(p,s)`, `string`, `varchar(n)`, `char(n)`, `date`, `timestamp`

### Source > File Attributes

**CSV:**

```json theme={null}
"csvAttributes": {
  "delimiter": ",",
  "header": true,
  "encoding": "UTF-8"
}
```

**JSON:**

```json theme={null}
"jsonAttributes": {
  "everyRowContainsObject": true,
  "encoding": "UTF-8"
}
```

**XML:**

```json theme={null}
"xmlAttributes": {
  "everyRowContainsObject": true,
  "encoding": "UTF-8"
}
```

**Excel:**

```json theme={null}
"xlsAttributes": {
  "worksheet": 0,
  "tempCsvFileDelimiter": ","
}
```

**Unstructured:**

```json theme={null}
"unstructuredAttributes": {
  "fileExtension": "pdf",
  "preserveFilename": true
}
```

### Source > Database Attributes

| Field | Type | Required | Description |
| - | - | - | - |
| `type` | string | Yes | `postgres`, `mysql`, or `mssql` |
| `postgresSecretsName` | string | Conditional | Vault secret name for Postgres credentials |
| `mssqlSecretsName` | string | Conditional | Vault secret name for MSSQL credentials |
| `mysqlSecretsName` | string | Conditional | Vault secret name for MySQL credentials |
| `cronExpression` | string | Yes | Cron schedule for polling (e.g., `0 */5 * * * ?`) |
| `database` | string | No | Database name |
| `schema` | string | No | Schema name |
| `table` | string | Yes\* | Table to query (\*unless `sqlOverride` is set) |
| `includeFields` | array | No | Column whitelist |
| `timestampFieldName` | string | Yes\* | Column for incremental pulls (\*unless `sqlOverride` is set) |
| `sqlOverride` | string | No | Custom SELECT query (replaces auto-generated query) |
| `outputDelimiter` | string | No | Delimiter for CSV output (default `,`) |

### Preprocessor

| Field | Type | Default | Description |
| - | - | - | - |
| `endpoint` | string | Required | URL of the preprocessing service |
| `async` | boolean | `false` | If true, returns immediately |
| `bearerToken` | string | null | Authorization bearer token |
| `timeoutMs` | int | 300000 | Request timeout (milliseconds) |
| `batchSize` | int | 0 | Rows per call for row-based payloads; `0` sends every row in one call. When `> 0`, each request also carries `batch` and `ofBatches`, and every response's `data.rows` are stitched in call order. See [Preprocessor](/preprocessor#batching) |

### Data Quality

See [Data Quality](/data-quality/ai-rules) for detailed documentation.

| Field | Type | Description |
| - | - | - |
| `aiRule` | object | AI rule — plain-English validation instruction |
| `validateFileHeader` | boolean | Validate CSV header matches schema field order |
| `validationSchema` | string | Path to JSON Schema file for JSON/XML validation |

### Transformation

See [AI Transformation](/transformation/ai-transformation) for detailed documentation.

| Field | Type | Description |
| - | - | - |
| `aiTransformation` | object | AI transformation — plain-English transformation instruction |

### Destination > Object Store

The object store destination writes Parquet files, ORC files, or an [Apache Iceberg table](/destinations/iceberg) to either the built-in MinIO (default) or to AWS S3. The provider is selected by the `provider` field; everything else is shared across both.

| Field | Type | Default | Description |
| - | - | - | - |
| `prefixKey` | string | Required | Path prefix for output files under the bucket |
| `fileFormat` | string | `parquet` | Output format: `parquet`, `orc`, or `iceberg` (a table at the prefix with atomic commits, `merge` upserts, and schema evolution) |
| `partitionBy` | array | null | Column names for partitioning |
| `destinationBucketOverride` | string | null | Custom bucket (default: `{environment}-data` for MinIO; **required** for S3) |
| `deleteBeforeWrite` | boolean | false | Delete existing data at path before writing |
| `writeToTemporaryLocation` | boolean | false | Write to temp location first |
| `writeMode` | string | `append` | `append`, `overwrite`, `ignore`, `errorifexists`, or `merge` (`iceberg` only; requires `keyFields`) |
| `keyFields` | array | null | Natural-key columns for `writeMode: merge` on an Iceberg table. Each must be a destination schema column; rejected with any other write mode |
| `provider` | string | `minio` | `minio` (built-in, default) or `s3` (AWS S3) |
| `endpoint` | string | null | S3 endpoint URL override. Must use `https://`. Leave unset for the AWS regional default. Ignored for `minio`. |
| `credentialsSecret` | string | null | For `provider=s3`: name of a Platform-tab Vault secret containing `accessKey`, `secretKey`, `region` (and optionally `sessionToken`). Leave unset only when Datris runs on an AWS instance role. |

For S3-specific behavior — credentials secret format, per-bucket credential isolation, multiple S3 destinations in one deployment — see the [S3 Destination](/destinations/s3) page.

### Destination > Database

| Field | Type | Default | Description |
| - | - | - | - |
| `dbName` | string | Required | Database name |
| `schema` | string | Required | Schema name |
| `table` | string | Required | Table name |
| `keyFields` | array | null | Natural-key columns (enables upsert on PostgreSQL, MongoDB, Snowflake, and Databricks). For MongoDB the keys are matched against fields *inside* the stored JSON document, so they don't need to appear as schema columns; use lowercase field names |
| `usePostgres` | boolean | false | Write to PostgreSQL |
| `useMongoDB` | boolean | false | Write to MongoDB |
| `useSnowflake` | boolean | false | Write to the user's Snowflake account — see [Snowflake Destination](/destinations/snowflake) |
| `useDatabricks` | boolean | false | Write a Unity Catalog managed Delta table in the user's Databricks workspace — see [Databricks Destination](/destinations/databricks) |
| `warehouse` | string | null | Snowflake virtual warehouse name, or Databricks SQL warehouse ID (required for Snowflake; for Databricks, optional when the secret has a `warehouse` field) |
| `role` | string | null | Snowflake role to assume (optional, Snowflake only) |
| `credentialsSecret` | string | null | Platform secret holding destination credentials (required for Snowflake and Databricks) |
| `manageTableManually` | boolean | false | If false, auto-creates tables |
| `truncateBeforeWrite` | boolean | false | Truncate table before loading (atomic replace on Databricks) |
| `useTransaction` | boolean | true | Wrap in a transaction (Postgres/Snowflake; no effect on Databricks — each load is one atomic Delta commit) |
| `options` | array | null | Custom COPY options (e.g., `["FORMAT csv", "DELIMITER ','"]`) |

Exactly one of `usePostgres`, `useMongoDB`, `useSnowflake`, or `useDatabricks` must be `true`. For Databricks, `dbName` names the Unity Catalog **catalog**.

### Destination > Kafka

| Field | Type | Default | Description |
| - | - | - | - |
| `topic` | string | Required | Kafka topic name |
| `keyField` | string | null | Column to use as message key |
| `overrideBootstrapServers` | string | null | Custom bootstrap servers |
| `timeoutMs` | int | 10000 | Producer timeout |

### Destination > ActiveMQ

| Field | Type | Description |
| - | - | - |
| `queueName` | string | ActiveMQ queue name |

### Destination > REST Endpoint

| Field | Type | Default | Description |
| - | - | - | - |
| `endpoint` | string | Required | URL to POST data to |
| `async` | boolean | false | If true, doesn't wait for response |
| `bearerToken` | string | null | Authorization token |
| `apiKey` | string | null | API key for authentication |
| `timeoutSeconds` | int | 0 | Request timeout (seconds); 0 means unset |
| `timeoutMs` | int | 300000 | Request timeout (milliseconds) |
| `batchSize` | int | 0 | Rows per call for row-based payloads; `0` sends every row in one call. When `> 0`, each request also carries `batch` and `ofBatches`. See [REST Endpoint](/destinations/rest-endpoint#batching) |

### Destination > Qdrant

| Field | Type | Description |
| - | - | - |
| `collectionName` | string | Qdrant collection name |
| `chunking` | object | Chunking config (see below) |
| `metadata` | object | Key-value metadata to attach to each chunk |
| `embeddingSecretName` | string | Optional override of the embedding Vault secret. Defaults to the server-level `ai.embedding.secretName`. |
| `qdrantSecretName` | string | Vault secret for Qdrant connection |

### Destination > Weaviate

| Field | Type | Description |
| - | - | - |
| `className` | string | Weaviate class name |
| `chunking` | object | Chunking config (see below) |
| `metadata` | object | Key-value metadata to attach to each chunk |
| `embeddingSecretName` | string | Optional override of the embedding Vault secret. Defaults to the server-level `ai.embedding.secretName`. |
| `weaviateSecretName` | string | Vault secret for Weaviate connection |

### Destination > pgvector

| Field | Type | Description |
| - | - | - |
| `tableName` | string | PostgreSQL table name |
| `schemaName` | string | PostgreSQL schema name |
| `chunking` | object | Chunking config (see below) |
| `metadata` | object | Key-value metadata to attach to each chunk |
| `embeddingSecretName` | string | Optional override of the embedding Vault secret. Defaults to the server-level `ai.embedding.secretName`. |
| `postgresSecretName` | string | Vault secret for PostgreSQL connection |

### Destination > Milvus

| Field | Type | Description |
| - | - | - |
| `collectionName` | string | Milvus collection name |
| `chunking` | object | Chunking config (see below) |
| `metadata` | object | Key-value metadata to attach to each chunk |
| `embeddingSecretName` | string | Optional override of the embedding Vault secret. Defaults to the server-level `ai.embedding.secretName`. |
| `milvusSecretName` | string | Vault secret for Milvus connection |

### Destination > Chroma

| Field | Type | Description |
| - | - | - |
| `collectionName` | string | Chroma collection name |
| `chunking` | object | Chunking config (see below) |
| `metadata` | object | Key-value metadata to attach to each chunk |
| `embeddingSecretName` | string | Optional override of the embedding Vault secret. Defaults to the server-level `ai.embedding.secretName`. |
| `chromaSecretName` | string | Vault secret for Chroma connection |

### Chunking Config

Used by all vector store destinations.

| Field | Type | Default | Description |
| - | - | - | - |
| `strategy` | string | `recursive` | Chunking strategy: `fixed`, `sentence`, `paragraph`, `recursive` |
| `chunkSize` | int | 500 | Maximum characters per chunk |
| `chunkOverlap` | int | 50 | Overlap between consecutive chunks |
| `maxChunkTokens` | int | 0 (off) | Optional token-count cap. When set, the chunker refuses to emit any chunk over this estimate, splitting oversized output before it ever reaches the embedding API. Recommended: \~80% of the embedding model's input cap (e.g. 6500 for `text-embedding-3-small`). Without it, the server-side embedding guard catches oversized chunks as a fallback. |
| `tokensPerCharRatio` | float | 2.0 | Chars-per-token ratio for the chunker's heuristic counter. Lower is more conservative (more splits); raise to \~3.5 for predictable Latin prose. |

### Destination > Live Read (`scratch`)

Nothing is landed: the run's rows are held briefly and handed back to the caller, then expire. Use it for an answer now — a validation result or a transformed view — never for data you want to keep. See [Live Read](/destinations/live-read).

```json theme={null}
"destination": {
  "scratch": {}
}
```

| Rule | |
| - | - |
| Fields | None — the object is empty. |
| Sources | Structured only (CSV, JSON, XML); a sample is required at create time so the typed header exists. |
| Exclusive | Cannot be combined with any other destination in the same pipeline. |
| Reading rows | The status rollup carries `resultPreview`; page the rest with `GET /api/v1/pipeline/result`, `datris pipeline result`, or the `get_pipeline_result` MCP tool. |

### Destination > Schema Properties (Optional)

Define a separate destination schema if column mapping differs from the source:

```json theme={null}
"destination": {
  "schemaProperties": {
    "dbName": "analytics",
    "fields": [
      {"name": "order_id", "type": "int"},
      {"name": "customer", "type": "string"}
    ]
  }
}
```

## Multiple Destinations

A single pipeline can write to multiple destinations simultaneously. All destinations execute in parallel:

```json theme={null}
"destination": {
  "objectStore": { ... },
  "database": { ... },
  "kafka": { ... },
  "activeMQ": { ... },
  "restEndpoint": { ... }
}
```

`scratch` is the exception: a Live Read pipeline lands nothing, so it cannot be combined with another destination.


# Preprocessor
Source: https://docs.datris.ai/preprocessor

Data preprocessing configuration

A preprocessor is an optional REST endpoint that is called **before** data quality and transformation. It receives the ingested data and can modify, enrich, filter, or replace it before the pipeline continues processing.

Use a preprocessor when you need to call out to an external system for data enrichment, format conversion, or custom logic that runs before AI-powered data quality and transformation.

## Processing Flow

```
Source Data Ingested
  |
  v
Preprocessor (optional) <--- you are here
  |
  v
Data Quality
  |
  v
Transformation
  |
  v
Destinations
```

## Configuration

Add a `preprocessor` section to the pipeline configuration:

```json theme={null}
{
  "name": "stock_price",
  "source": { ... },
  "preprocessor": {
    "endpoint": "http://my-service:5500/preprocess/sync",
    "async": false,
    "bearerToken": null,
    "timeoutMs": 300000
  },
  "destination": { ... }
}
```

### Fields

| Field | Type | Default | Description |
| - | - | - | - |
| `endpoint` | string | Required | URL of the preprocessing service |
| `async` | boolean | `false` | Synchronous or asynchronous mode |
| `bearerToken` | string | `null` | Optional `Authorization: Bearer` token |
| `apiKey` | string | `null` | Optional API key sent to the endpoint as an `x-api-key` header |
| `timeoutMs` | int | `300000` | Request timeout in milliseconds |
| `timeoutSeconds` | int | `0` | Legacy timeout in seconds; used only when `timeoutMs` is `0` or omitted. Prefer `timeoutMs` |
| `batchSize` | int | `0` | Rows per call for row-based (CSV) payloads. `0` makes one call carrying every row; `> 0` sends batches of that many rows, each request also carrying `batch` and `ofBatches`, and stitches every response's `data.rows` in call order. See [Batching](#batching) |

## Synchronous Mode

In synchronous mode (`async: false`), the pipeline POSTs the data to the endpoint and waits for the response. The response replaces the pipeline data (with `batchSize > 0`, the stitched responses do — see [Batching](#batching)).

### Request Payload

```json theme={null}
{
  "pipelineToken": "5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e",
  "pipelineName": "stock_price",
  "data": {
    "size": 1024,
    "header": ["symbol", "price", "date"],
    "rows": [
      "AAPL,150.25,2026-03-15",
      "GOOG,2800.00,2026-03-15"
    ]
  }
}
```

For JSON/XML sources, `header` and `rows` are omitted and `rawData` contains the raw content (null fields are left out of the payload rather than sent as `null`).

### Expected Response

Return the (optionally modified) data in the same format:

```json theme={null}
{
  "data": {
    "size": 1024,
    "header": ["symbol", "price", "date"],
    "rows": [
      "AAPL,150.25,2026-03-15",
      "GOOG,2800.00,2026-03-15"
    ]
  }
}
```

Any of `header`, `rows`, or `rawData` you leave out of the response keeps its original value.

You can modify, add, or remove rows. The pipeline continues with whatever data is returned. If you return an `error` field, processing is aborted:

```json theme={null}
{
  "error": "Data validation failed: missing required field"
}
```

## Batching

By default (`batchSize` = `0`) the pipeline makes exactly one call carrying every row, and that call has to fit in memory (it is subject to `PIPELINE_MATERIALIZE_MAX_MB`). Set `batchSize` to a positive number to send row-based (CSV) payloads in batches of that many rows, one call per batch, in source order, streamed from disk. Each batched request carries the same envelope plus two extra top-level fields, `batch` (1-based) and `ofBatches`, and its `data.rows` holds only that batch. Every response is read exactly like the single-call response — `data.rows` replaces that batch's rows, and a response that leaves `rows` out keeps them — and the results are stitched, in call order, into the data the pipeline continues with. The `header` from the first response that carries one applies to the whole payload. JSON/XML payloads and `async` callbacks are unaffected by `batchSize` beyond one callback being awaited per batch.

## Asynchronous Mode

In asynchronous mode (`async: true`), the pipeline POSTs the data and then **waits for a callback** rather than using the response directly. This is useful for long-running preprocessing tasks.

### Flow

1. Pipeline POSTs data to the preprocessor endpoint
2. Preprocessor returns immediately (e.g., `200 OK`)
3. Preprocessor processes data in the background
4. Preprocessor POSTs the result back to the pipeline's callback endpoint
5. Pipeline resumes with the returned data

### Callback Endpoint

The preprocessor sends the result to:

```
POST http://{pipeline-host}:8080/api/v1/restendpoint/callback
```

Callback payload:

```json theme={null}
{
  "pipelineToken": "5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e",
  "pipelineName": "stock_price",
  "data": {
    "size": 1024,
    "header": ["symbol", "price", "date"],
    "rows": ["AAPL,150.25,2026-03-15"]
  }
}
```

The `pipelineToken` must match the token from the original request so the pipeline can correlate the callback with the waiting job.

If the callback is not received within `timeoutMs` milliseconds, the pipeline aborts with a timeout error.

## Example: Preprocessor Service

A complete working example is provided in [`examples/preprocessor/app.py`](https://github.com/datris/datris-platform-oss/blob/main/examples/preprocessor/app.py). This Python Flask application implements both synchronous and asynchronous preprocessing:

```python theme={null}
# Synchronous - process and return immediately
@app.route('/preprocess/sync', methods=['POST'])
def preprocess_sync():
    payload = request.get_json()
    data = payload.get('data')

    # Modify data here if needed

    return jsonify({
        'pipelineToken': payload.get('pipelineToken'),
        'pipelineName': payload.get('pipelineName'),
        'data': data
    })

# Asynchronous - return immediately, send callback later
@app.route('/preprocess/async', methods=['POST'])
def preprocess_async():
    payload = request.get_json()
    threading.Thread(
        target=send_callback,
        args=(payload.get('pipelineToken'),
              payload.get('pipelineName'),
              payload.get('data'))
    ).start()
    return jsonify({'status': 'accepted'}), 200

def send_callback(pipeline_token, pipeline_name, data):
    callback_url = 'http://localhost:8080/api/v1/restendpoint/callback'
    requests.post(callback_url, json={
        'pipelineToken': pipeline_token,
        'pipelineName': pipeline_name,
        'data': data
    })
```

To run:

```bash theme={null}
cd examples/preprocessor
python3 -m venv my-env
source my-env/bin/activate
pip install flask requests
python app.py  # Starts on port 5500
```

Then configure the pipeline:

```json theme={null}
"preprocessor": {
  "endpoint": "http://localhost:5500/preprocess/sync",
  "async": false,
  "timeoutMs": 300000
}
```


# Data Flows
Source: https://docs.datris.ai/production/data-flows

For a security reviewer: every place each kind of data Datris handles can go, which feature sends it there, and which setting changes that

Datris runs inside your perimeter, and your data goes only where you send it. Every destination outside the platform's own containers is one you configure: the AI provider in each AI slot, the agents you connect with an API key, the destinations your pipelines write to, and, if you turn them on, web search and a GitHub repository for scripts. No customer data reaches Datris.ai the company. This page lists, for each kind of data, where it can go and which setting changes that.

Destinations are not in the table: a pipeline destination (your Postgres, warehouse, object store or vector store) is where you send data on purpose, and the rows land there as you configured them.

## The matrix

The two model columns are alternatives. Each AI slot (AI Primary, CodeGen, embedding) points either at a local model you run or at a cloud provider, and a flow goes to whichever one that slot names. "Local model" means an Ollama endpoint you run, or the bundled `bge-m3` embedding service. "Cloud AI provider" means Anthropic, OpenAI, Amazon Bedrock, Azure OpenAI or Grok. "Connected agent" means an MCP client, the CLI or a script holding one of your API keys. Each number links to the note for that row.

| Data | Local model | Cloud AI provider | Connected agent | Datris.ai |
| - | - | - | - | - |
| Row values and samples | When an AI helper, the answer endpoint or an embedding runs [1](#1-row-values-and-samples) | When an AI helper, the answer endpoint, a chat assistant or an embedding runs [1](#1-row-values-and-samples) | Only through the query, search, profile, tap and Live Read tools its key allows [1](#1-row-values-and-samples) | Never |
| Column names and schema | When an AI helper runs [2](#2-column-names-and-schema) | When an AI helper or a chat assistant runs [2](#2-column-names-and-schema) | Only through the pipeline, catalog and lineage tools its key allows [2](#2-column-names-and-schema) | Never |
| Source credentials | Never; field names only [3](#3-source-credentials) | Never; field names only [3](#3-source-credentials) | Only a value the agent itself stores; never returned [3](#3-source-credentials) | Never |
| AI provider keys | Only its own key, if you set one [4](#4-ai-provider-keys) | Only its own key, to authenticate [4](#4-ai-provider-keys) | Write only: it can set a key, never read one [4](#4-ai-provider-keys) | Never |
| Logs and error text | When a fix suggestion or tap diagnosis runs [5](#5-logs-and-error-text) | When a fix suggestion, tap diagnosis, the recovery agent or a chat assistant runs [5](#5-logs-and-error-text) | Only through the status, log and incident tools its key allows [5](#5-logs-and-error-text) | Never |
| Generated scripts | When CodeGen writes, fixes, reviews or optimizes a script [6](#6-generated-scripts) | When CodeGen writes, fixes, reviews or optimizes a script, or a chat assistant reads one [6](#6-generated-scripts) | Only through the tap and pipeline tools its key allows [6](#6-generated-scripts) | Never |

The chat assistants cannot run on a local model. The Assistant, Search chat, Catalog chat, the Configuration assistant, Ops chat and the recovery agent run on the AI Primary provider and need Anthropic, OpenAI, Amazon Bedrock, Azure OpenAI or Grok; with AI Primary on Ollama they return an error instead of answering. One-shot helpers on AI Primary (profiling, header validation, fix suggestions, tap diagnosis, the answer endpoint) and everything on the CodeGen slot do run on a local model.

A connected agent is a program you run. What it receives goes wherever that program sends it, usually to the model provider behind it, so choose its key's capabilities with that in mind.

## Notes

### 1. Row values and samples

* **AI helpers that write configuration or code** send sample rows: schema generation from a file, data profiling, JSON Schema and XSD generation, files attached to an Assistant chat, and the sample rows in the prompts that generate an [AI data-quality rule](/data-quality/ai-rules) or an [AI transformation](/transformation/ai-transformation). Profiling uses AI Primary, schema, rule and transformation generation use CodeGen, and an attachment goes to the chat assistant it is attached to. Switch: `DATRIS_AI_SAMPLE_VALUES=false` makes each of them work from structure only. The generated rule or transformation script runs on your data inside the platform; the rows themselves are not sent at run time.
* **Features that answer questions over your data** send that data by design: Search chat, the answer endpoint (`POST /api/v1/ai/answer`, the `ai_answer` tool), and the Assistant reading query results or a Live Read preview. Search chat and the Assistant are chat assistants, so they need a cloud provider; the answer endpoint uses AI Primary and can be local.
* **Embeddings.** A pipeline with a vector destination sends each document chunk to the embedding endpoint, and a vector search sends the query text. Field protection does not apply to documents. Switch: point the embedding slot at the bundled `bge-m3` service or Ollama and nothing leaves; a cloud embedding endpoint is OpenAI or Azure OpenAI, because Anthropic offers no embedding model.
* **Connected agent.** The `query_*` tools, `search_*` tools, `ai_answer`, `profile_data`, `test_tap`, `run_tap` and `get_pipeline_result` (Live Read) return rows. Switch: API key capabilities decide which of these a key may call; Agent Policy can hold a tap run for approval but never gates reads.
* **Field protection** is the control across all of these: a column marked `protect` is already a pseudonym, mask, ciphertext or gone before AI rules, transformations, Live Read and destinations see it, so every later reader gets the protected value. The preprocessor and the file-level helpers above run before protection; see [What still sees raw data](/transformation/field-protection#what-still-sees-raw-data).

What `DATRIS_AI_SAMPLE_VALUES=false` does not cover:

* Search chat and the answer endpoint;
* the Assistant reading query results or a Live Read preview;
* the Assistant or the recovery agent reading a run's status, logs or job details with their tools;
* tap script generation, tap script review and diagnosis, and fix suggestions for failed tap runs;
* embeddings, which still send every document chunk and search query to the embedding endpoint.

The first four match the list under [What still sees raw data](/transformation/field-protection#what-still-sees-raw-data). For these, use field protection, a local model where one is supported, or an AI provider you have an agreement with for that data.

### 2. Column names and schema

* **AI helpers** send field names and types: protection suggestions, natural-language to SQL (the table's column names), column-level lineage inference (field names, the transformation instruction and its script), the catalog search ranking (pipeline names, descriptions and tags), and header validation, which compares a file's header line with the pipeline schema.
* With `DATRIS_AI_SAMPLE_VALUES=false`, a first line that looks like data is sent as `column_1`, `column_2`, … and a JSON object keyed by data is collapsed; the rules are in [Field Protection](/transformation/field-protection#what-still-sees-raw-data). Header validation (`validateFileHeader`) is not covered by the switch: it sends the header line as it is. Stored schema field names are always sent.
* **Chat assistants** read pipeline definitions, schemas and the catalog through their tools.
* **Connected agent.** `get_pipeline`, `list_pipelines`, `find_data`, `get_lineage`, `get_provenance` and the `list_postgres_*` style tools return names and schemas. Switch: API key capabilities.

### 3. Source credentials

* Credentials live in Vault. At run time a tap's secret is injected as environment variables into the isolated tap runner and reaches only the tap's own process and its source. Destination credentials are read by the server's writer. Neither is put in a model prompt.
* Tap script generation receives the field names of the tap's secret, never the values. Tap output is masked before it is stored, shown or sent to tap diagnosis, review or fix: every secret value of four or more characters is replaced. Masking matches the exact value, so a script that prints an encoded or partial form of a secret defeats it; do not log credentials.
* The in-platform Assistant and the Configuration assistant collect a credential through a form, never in chat, so it does not enter the model's context. Text you paste into any chat does.
* **Connected agent.** `list_tap_secrets`, `get_tap_secret_fields` and `get_platform_secret_fields` return names only. `create_tap_secret` takes a value the agent supplies and never returns it; an agent can change or delete only tap secrets, with one exception: it may set (never read) the six AI provider slots through `update_secret`, see note 4. Switches: API key capabilities (`secret:read`, `secret:write`), Agent Policy on `secret:write`, and `DATRIS_TAP_SECRET_SCOPE`, which keeps a tap (and so anyone who can write and run one) from reading a platform secret unless set to `any`.

### 4. AI provider keys

* Each provider's key is sent only to that provider, in the request's authentication header. Amazon Bedrock is signed with your AWS credentials instead, and Azure can use Microsoft Entra ID in place of a key.
* The secrets API masks key fields, and logged configuration is redacted.
* **Connected agent.** `update_secret` can set an AI slot's key; nothing returns one. Switches: API key capabilities (`secret:write`) and Agent Policy.

### 5. Logs and error text

* **Fix suggestions** for a failed pipeline run and the **recovery agent's** first incident prompt include the error text, which can quote row values. Switch: with `DATRIS_AI_SAMPLE_VALUES=false` that text is reduced to exception classes and the server's own messages. Anything the recovery agent or the Assistant then reads through its tools is not reduced.
* **Tap diagnosis, review, fix and optimization** send the tap's captured output and error (with secrets masked, see note 3). The switch does not cover them.
* **Web search.** If you configure the optional web-search slot (`ai.webSearch.secretName`), tap brainstorming, fixing and diagnosis can send a search query built from the tap's description and error to Anthropic or OpenAI web search, which searches the public web. This is a fifth destination, not in the table; it is off until you configure it.
* **Connected agent.** `get_job_status`, `get_pipeline_status`, `get_tap_logs`, `get_incident` and `list_incidents` return run messages and logs. Switch: API key capabilities.

### 6. Generated scripts

* Tap scripts and the scripts behind AI rules and transformations are generated, fixed, reviewed and optimized by the CodeGen slot, so their text goes to that model. Tap diagnosis uses AI Primary.
* Scripts are stored in the platform's object store. If you connect a [code repository](/tap-github-storage), every save is also a commit to that GitHub repository; this is a destination you choose, not in the table.
* **Connected agent.** `get_tap`, `get_tap_version`, `get_pipeline` and the version tools return script text. Switches: API key capabilities, and Agent Policy for creating, changing and restoring taps and pipelines.

## Switches

| Switch | Default | What it moves |
| - | - | - |
| AI Primary, CodeGen and embedding provider ([AI Configuration](/ai-configuration)) | Chosen at install; embeddings default to the bundled `bge-m3` service | Which model column a flow goes to. Anthropic and OpenAI are equal choices for AI Primary and CodeGen; the chat assistants need a cloud provider |
| `DATRIS_AI_SAMPLE_VALUES` (datris service) | `true` | `false`: AI helpers that write configuration or code send structure, not row values, and pipeline error text is reduced. See note 1 for what it does not cover |
| `protect` on a field ([Field Protection](/transformation/field-protection)) | none | The field is a pseudonym, mask, ciphertext or dropped before AI stages, Live Read, agents' queries and destinations see it |
| `DATRIS_TAP_SECRET_SCOPE` (datris service) | `tap` | Taps may use only tap secrets. `any` lets a tap read platform secrets |
| API key capabilities ([API Keys](/api-keys)) | Off until `USE_API_KEYS=true` | Which tools, and so which data, a connected agent can reach at all |
| Agent Policy ([Agent Policy](/agent-policy)) | Off until `USE_AGENT_POLICY=true` | Whether an agent's run, write or secret change happens now, waits for approval, or is refused. Reads are never gated |
| Web search (`ai.webSearch.secretName`) | Not configured | Lets tap brainstorming, fixing and diagnosis send a search query to Anthropic or OpenAI web search |
| Code repository ([GitHub Script Storage](/tap-github-storage)) | Off; scripts stay in the object store | Every script save becomes a commit to your GitHub repository |

## No cloud exit

To run with nothing leaving your perimeter for a model:

1. Point AI Primary and CodeGen at Ollama and the embedding slot at the bundled `bge-m3` service (or Ollama); see [Fully local with Ollama](/ai-configuration#fully-local-with-ollama).
2. Leave web search unconfigured.
3. Connect only agents that run on a model you trust with what their key can read, or none.

What stops working: the Assistant, Search chat, Catalog chat, the Configuration assistant, Ops chat and the recovery agent, because the chat assistants need a cloud provider today. Profiling, schema generation, AI rules and transformations, tap generation, fix suggestions, tap diagnosis, the answer endpoint and vector search keep working, with the quality of the local model you choose.

The only request the platform server makes to datris.ai is the model catalog fetch, which sends nothing beyond the request itself; it and the `datris doctor` update check are described on [Security Architecture](/production/security-architecture#no-telemetry).


# Dev, QA and production
Source: https://docs.datris.ai/production/environments

Run Datris as three installs, promote a catalog from dev to QA to production with a catalog archive and one GitHub repository, and configure, upgrade and target each environment

This page describes how to run separate development, QA and production environments with the current release. Developers build taps and pipelines in dev, a QA team checks them in qa, and prod runs them on a schedule against real destinations. Code lives in GitHub. Everything here works without code changes. Steps that still need a person are listed under [Limits today](#limits-today).

In short:

| Decision | Recommendation |
| - | - |
| Topology | One Datris install per environment, each on its own host or VM |
| Promotion unit | The catalog: `datris catalog export` on the source, `datris catalog restore` on the target |
| Secrets | The same secret names in every environment, with different values |
| Code | One GitHub repository, one branch per environment (`dev`, `qa`, `main`), the same path prefix on every install |
| Governance | dev open; qa and prod governed, with qa configured the same as prod |
| Agents and CLI | One MCP server entry and one scoped key per environment |
| Upgrades | The same release on all three, rolled out dev, then qa, then prod |

## Why three installs

Install Datris three times, once per environment, each in its own `DATRIS_DIR` on its own host:

```
dev   datris-dev.internal    installer, DATRIS_GOVERNED unset (or 0)
qa    datris-qa.internal     installer, DATRIS_GOVERNED=1
prod  datris.internal        installer, DATRIS_GOVERNED=1, DATRIS_ENV=production, TLS reverse proxy
```

* Each install has its own MongoDB, MinIO, Vault, Postgres, users, agent policy and audit log. The installs share nothing, so the isolation needs no configuration.
* Compose publishes fixed ports (8080, 4200, 3000, 9000 and others), so running two stacks on one host is not a supported layout.
* The internal environment prefix (as in `oss-data` and `oss/postgres`) is `oss` by default. Leave it at the default on every install so that secret paths and bucket names in a pipeline carry over from one install to another.
* Give qa the same instance size as prod so that QA runs are representative. Dev can be smaller and can use the bundled Postgres and MongoDB as destinations.

<Note>
  **Why not three catalogs on one install?** Catalogs such as `dev_sales`, `qa_sales` and `prod_sales` on one server do not work as environments:

  * Tap and pipeline names are unique across an install, so the same pipeline cannot exist in two catalogs. Promoting would mean renaming, and renaming breaks every reference to the item.
  * Secrets belong to the install, not to a catalog. Dev taps could read production credentials.
  * One install has one agent policy, one audit log, one user list and one upgrade. An experiment in dev can take production down.

  Multi-tenant mode (`MULTI_TENANT=true`) isolates the stores per tenant, but it is meant for separate customers. Its tenants share one server process and one release, and the installer does not set it up.
</Note>

## Make definitions portable

A [catalog archive](/data-catalog#what-an-archive-contains) restores cleanly on the next environment only if the definitions in dev were written for it. Follow these rules in dev.

**Keep everything that differs per environment in a secret.** Pipelines and taps refer to secrets by name only. Create the same names on qa and prod and give them each environment's values in Configuration → Secrets. An archive never carries secret values.

| What differs per environment | Put it in | Referenced as (see [Pipeline Configuration](/pipeline-configuration)) |
| - | - | - |
| Source database host, port, user, password | Platform secret | `postgresSecretsName`, `mysqlSecretsName`, `mssqlSecretsName` |
| Snowflake account, Databricks workspace | Platform secret | `destination.database.credentialsSecret` |
| S3 keys and region | Platform secret | `destination.objectStore.credentialsSecret` |
| Vector store connection | Platform secret | `qdrantSecretName`, `weaviateSecretName`, `milvusSecretName`, `chromaSecretName`, `postgresSecretName` (pgvector) |
| Source API tokens for taps | Tap secret (`_type=tap`) | the tap's `secretName` |

The archive lists the secret names the catalog needs. The restore result and `datris catalog restore` name any of them that do not exist on the target yet.

**Avoid the four fields that do not carry over.** These fields are specific to an environment and cannot come from a secret. If a pipeline sets one of them, the archive and the restore result carry a warning that the value `may not be valid on another install`:

* `destination.objectStore.endpoint`
* `destination.objectStore.destinationBucketOverride`
* `destination.database.warehouse` (Snowflake warehouse name or Databricks SQL warehouse id)
* `dataQuality.validationSchema` when it is an absolute `s3://` path

In dev, use the default bucket and a relative validation schema path, and use a warehouse name that exists under the same name in every environment. If a pipeline has to set one of these fields, edit it on the target after each restore.

**Use one catalog per team or data domain.** Do not use one catalog per environment. A catalog such as `sales` has the same name on all three installs and is promoted as a whole. Two teams that release on different schedules need two catalogs. **Uncataloged** cannot be exported, so create anything meant for production in a catalog. Keys scoped with `pipeline:create:catalog=<name>` enforce this for agents. See [Data Catalog](/data-catalog).

## GitHub layout

Use one repository with one branch per environment:

```
github.com/<org>/datris-taps
  branches: dev, qa, main
  taps/<tap>.py
  taps/pipelines/<pipeline>/data-quality.py
  taps/pipelines/<pipeline>/transformation.py
  catalogs/<catalog>/<catalog>-<yyyyMMdd-HHmm>.zip     (optional: archives committed by CI)
```

On each install, set Configuration → Code Repository to the same repository and the same path prefix (`taps/`), and to that environment's branch: `dev`, `qa` or `main`. Give each install a fine-grained token scoped to that one repository with **Contents: Read and write**. Setup is described in [Tap and pipeline scripts in GitHub](/tap-github-storage).

Scripts move from one environment to the next through a pull request:

1. A developer saves a tap in dev. Datris commits `taps/<tap>.py` on `dev` and pins the tap to that commit.
2. Open a pull request from `dev` to `qa`. Reviewers see the script diff in GitHub. Merge it.
3. Export the catalog from dev and restore it on qa (see [Promote a catalog](#promote-a-catalog)).
4. Test, enable and run the taps on qa.
5. Repeat with a pull request from `qa` to `main` and a restore on prod.

When you restore, the target compares each archived tap script with the file at the head of its branch:

| File at the branch head | What the restore does |
| - | - |
| Same bytes as the archive | Pins the tap to the existing commit and makes no commit. This is the normal case after the pull request is merged. |
| Missing | Commits the archived script on the branch. |
| Different, and the tap is not yet in the target catalog | Refuses the whole restore with 409, names the file, and writes nothing. |
| Different, and the tap is already in the target catalog (`--overwrite`) | Commits the archived script over it. If the tap on the target is GitHub-backed and the file was changed in the repository since that tap's recorded commit, the tap is not restored and the result lists it with the reason. |

Merge the pull request before you restore. Then the archive matches the branch and the restore leaves the repository unchanged. This comparison covers tap scripts only: AI data-quality and transformation scripts are regenerated on the target, as described under [Limits today](#limits-today).

**Branch protection on `main`.** Saving a tap or an AI script on a GitHub-backed install commits directly to its branch. You can set this up in one of two ways:

* **Prod is GitHub-backed.** The prod token must be allowed to push to `main`, so branch protection on `main` cannot block that token. Prod reads its scripts from the branch, so the restore comparison above applies to prod as well.
* **Prod uses built-in storage** (the default, MinIO). `main` can be fully protected and only dev and qa use GitHub storage. Prod keeps its scripts in its own object store, and a restore on prod does not compare against `main`.

Tap and pipeline definitions (schedules, parameters, destinations, rules), validation schemas and tap prompts are not stored in GitHub. They move inside the catalog archive. Committing the archive under `catalogs/` gives them a coarse history.

## Configure each environment

Set these in each install's `.env` (see [Configuration Reference](/configuration-reference)):

| Setting | dev | qa | prod |
| - | - | - | - |
| `DATRIS_GOVERNED` (install time) | 0 | 1 | 1 |
| `USE_USER_AUTH` / `USE_API_KEYS` / `USE_AUDIT_LOG` / `USE_AGENT_POLICY` | optional | all true | all true |
| `OIDC_ENABLED` | off | on, same identity provider app or a separate one | on |
| `DATRIS_ENV` | unset | unset (or `production` if external Postgres has TLS) | `production` |
| `DATRIS_TAP_SECRET_SCOPE` | `tap` (default) | `tap` | `tap` |
| `DATRIS_ALLOW_PRIVATE_EGRESS` | as needed | same as prod | `false` unless the endpoints are internal |
| `DATRIS_OBJECTSTORE_BUCKET_ALLOWLIST` | unset | set | set |
| `CAPABILITY_ENFORCEMENT` | `enforce` | `enforce` | `enforce` (`log-only` only to try out a new key scope) |
| `AUDIT_LOG_RETENTION_DAYS` | short | default | long, and shipped to your log aggregator |
| Reverse proxy with TLS, 8080 not exposed | optional | yes | yes |
| AI provider keys | team key, cheaper models allowed for chat | same models as prod | prod key |
| CodeGen model | **same as prod** | same as prod | reference |

Configure qa the same way as prod. Permission and policy problems then show up in QA, before the prod restore. `DATRIS_ENV` changes behaviour only when it is set to `production`: loads to external Postgres then require TLS.

The CodeGen model must be the same on all three installs. A restore regenerates AI data-quality and transformation scripts on the target. The script prod runs can only match the one QA tested if both were generated by the same provider and model. Set `ai.codegen.*` to the same values on every install.

Use these roles and keys (see [API Keys](/api-keys), [Agent Policy](/agent-policy) and [Single sign-on (OIDC)](/user-auth#single-sign-on-oidc)):

| Actor | dev | qa | prod |
| - | - | - | - |
| Developers (UI, SSO) | editor | viewer (or editor for QA engineers) | viewer |
| Release or platform team | admin | admin | admin |
| Developer agents (Claude Code, Cursor) | `rag-builder`-style key scoped to their catalog | none, or read-only | none |
| Ops agent | read and run | read and run | read-only key; runs and updates go to Approve under the agent policy |
| CI promotion job | export only (`pipeline:read` and `tap:read` on the catalog) | export and restore | restore (see [Promote a catalog](#promote-a-catalog)) |

Create keys from templates and rotate them on a schedule. Issue one key per agent per environment so that the audit log attributes every action to the right agent. On prod, start the agent policy from **Use recommended** and set every mutating action a prod agent can reach to **Approve**.

## Promote a catalog

These steps promote the `sales` catalog from dev to qa. Repeat them from qa to prod.

1. Merge the pull request from `dev` to `qa` (see [GitHub layout](#github-layout)).

2. Export from the source environment:

   ```bash theme={null}
   datris catalog export sales --out sales.zip
   ```

3. Restore on the target environment. Use `--overwrite` every time; it also works the first time, when the catalog does not exist on the target yet. Use `--no-state`, because a tap's incremental cursor records where dev got to, not where qa should start:

   ```bash theme={null}
   datris catalog restore sales.zip --overwrite --no-state
   ```

   A restore never merges. Overwrite replaces the catalog's taps and pipelines that are in the archive, and never touches another catalog. Items in the target catalog that are not in the archive are kept. If the restore names missing secrets, create them on the target with that environment's values before you test the taps.

4. Restored taps start disabled, so a restore never starts a schedule on its own. Test each tap (`datris tap test <name>`), then enable it (`datris tap update <name> --enabled`). When restored taps read each other's tables, use **Enable in order** on the catalog card, or have an agent call `enable_catalog_ordered`. Either one tests, enables and runs the taps in [dependency order](/taps#dependencies-between-taps).

5. Run the pipelines against the target's destinations.

Restore refuses the whole archive if any tap or pipeline name in it already exists on the target in a different catalog. The full rules are in [Data Catalog](/data-catalog) and in the [CLI](/cli) reference.

**A CI promotion job.** A GitHub Actions workflow can run steps 2 and 3:

```yaml theme={null}
name: promote-catalog
on:
  workflow_dispatch:
    inputs:
      catalog:
        required: true
      to:
        type: choice
        options: [qa, prod]
jobs:
  promote:
    runs-on: ubuntu-latest
    environment: ${{ inputs.to }}
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: "3.x"
      - run: pip install datris-mcp-server
      - name: export from the source environment
        env:
          MCP_SERVER_URL: ${{ inputs.to == 'qa' && secrets.DEV_MCP_URL || secrets.QA_MCP_URL }}
          DATRIS_API_KEY: ${{ inputs.to == 'qa' && secrets.DEV_EXPORT_KEY || secrets.QA_EXPORT_KEY }}
        run: datris catalog export ${{ inputs.catalog }} --out archive.zip
      - name: restore into the target environment
        env:
          MCP_SERVER_URL: ${{ inputs.to == 'qa' && secrets.QA_MCP_URL || secrets.PROD_MCP_URL }}
          DATRIS_API_KEY: ${{ inputs.to == 'qa' && secrets.QA_RESTORE_KEY || secrets.PROD_RESTORE_KEY }}
        run: datris catalog restore archive.zip --overwrite --yes --no-state
      - uses: actions/upload-artifact@v4
        with:
          name: ${{ inputs.catalog }}-${{ github.run_id }}
          path: archive.zip
```

* The runner must be able to reach each environment's MCP server. Use a self-hosted runner, or put the MCP port behind the TLS reverse proxy with `REQUIRE_API_KEY=true`.
* `datris catalog restore` exits 1 on a refusal or when an item could not be restored, so the job fails visibly.
* The job cannot finish the promotion: the restored taps are disabled, and a person tests and enables them (step 4).
* Keep every archive, either as a build artifact (as above) or committed under `catalogs/`, so that every prod state can be reproduced.

**The approval gate for prod.** Under the agent policy, a CI key counts as an agent ([What counts as an agent](/agent-policy#what-counts-as-an-agent)). A restore is checked as `pipeline:create`. If that action is on **Approve** in prod, the CI job's restore waits as a pending approval that nobody polls. [Per-resource overrides](/agent-policy#per-resource-overrides) can only tighten the policy, so you cannot exempt the CI key. Choose one of these:

* **A person restores on prod with their own key.** CI promotes to qa only. The prod policy stays strict.
* **The GitHub environment is the gate.** Put the prod job behind a GitHub environment with required reviewers, and leave `pipeline:create` on **Auto** in the prod policy. Agents on prod can then also create pipelines without approval, so keep the prod agent keys read-only.

## Upgrade order

Run the same release on all three installs. Upgrade dev and let it run for a while, then upgrade qa and run one promotion end to end, then upgrade prod. Run `datris doctor --pre-upgrade` on each install before you upgrade it (see [Doctor](/doctor#before-an-upgrade) and the [upgrade procedure](/production/upgrades#upgrade-procedure)).

Never restore an archive from a newer Datris into an older one. An install refuses an archive whose format is newer than it supports. The archive records the source release but the restore does not check it, so keep the target on a release at least as new as the source.

## Point agents and the CLI at an environment

Give each environment its own MCP client entry and its own key. With `mcp-remote` (see [Configuring Claude](/configuring-claude)):

```json theme={null}
{
    "mcpServers": {
        "datris-dev": {
            "command": "npx",
            "args": ["-y", "mcp-remote", "https://mcp.datris-dev.internal/sse", "--transport", "sse-only", "--header", "x-api-key:<dev key>"]
        },
        "datris-qa": {
            "command": "npx",
            "args": ["-y", "mcp-remote", "https://mcp.datris-qa.internal/sse", "--transport", "sse-only", "--header", "x-api-key:<qa key>"]
        },
        "datris-prod": {
            "command": "npx",
            "args": ["-y", "mcp-remote", "https://mcp.datris.internal/sse", "--transport", "sse-only", "--header", "x-api-key:<prod read-only key>"]
        }
    }
}
```

The CLI chooses the install from `MCP_SERVER_URL` and `DATRIS_API_KEY` alone (see [CLI](/cli) and [MCP Server](/mcp-server)). Set both per shell, for example with `direnv` or one shell function per environment:

```bash theme={null}
datris-qa() { MCP_SERVER_URL=https://mcp.datris-qa.internal/sse DATRIS_API_KEY="$DATRIS_QA_KEY" datris "$@"; }
```

There is no CLI profile file. The three UIs and the three MCP servers look and respond the same, and nothing shows which environment you are connected to. Name the entries and functions clearly, and give prod keys the fewest capabilities.

## Limits today

* AI data-quality and transformation scripts regenerate on restore with the target's CodeGen model, and the archived `codegen/*.py` files are reference copies only, so pin the same CodeGen model on all environments.
* There is no dry run: restore is all-or-nothing, with no preview of what `--overwrite` would change.
* Restored taps start disabled and there is no catalog-level CLI command to test and enable them, so a CI promotion ends with a person (UI **Enable in order**) or an agent calling `enable_catalog_ordered`.
* The agent policy cannot exempt one key: a CI key under Approve queues `restore_catalog`, so prod CI uses GitHub environment reviewers as its gate.


# Licensing & Support
Source: https://docs.datris.ai/production/licensing-support

What AGPL-3.0 means for your deployment, when a commercial license applies, and where to get help

## The open-source license

Datris is licensed under the [GNU Affero General Public License v3](https://github.com/datris/datris-platform-oss/blob/main/LICENSE). In plain terms:

* **Running an unmodified Datris inside your own organization** — the common case — needs no commercial license. You are not distributing it and you have not modified it.
* **If you modify Datris and distribute the modified version**, you must release your modifications under AGPL-3.0.
* **If you modify Datris and let users interact with the modified version over a network**, you must offer those users the corresponding source. This is the clause that separates the AGPL from the GPL and the one license scanners flag.
* **Your own code stays yours.** Agents that talk to Datris over MCP or REST, and taps and preprocessor scripts you write and run inside Datris, are not modifications of the platform.

This is a summary, not legal advice; have your counsel read it against your deployment. The longer treatment, including the questions legal teams usually ask, is at [datris.ai/licensing](https://datris.ai/licensing).

## Commercial licenses

The bits do not change under a commercial license; the terms do. Four license types are offered, each scoped to how you use Datris:

| License | What it lets you do |
| - | - |
| **Internal Use** | Run Datris inside your organization with support and indemnity, and no copyleft question to answer. |
| **Hosted / SaaS** | Operate Datris as part of a service you offer to third parties over a network. |
| **OEM / Redistribution** | Embed Datris inside your product and sublicense it to your customers. |
| **Enterprise Source License** | A perpetual license to the source: modify it, build on it, keep your changes private. No right to redistribute. |

Every commercial license includes a named technical contact with a response SLA on production incidents, security advisories ahead of public disclosure, an LTS branch with a stated patch window, an SBOM and signed builds, an IP warranty and indemnification, version and migration assistance, and a seat at roadmap discussions. Licenses are annual subscriptions scoped to your deployment; request a proposal at [datris.ai/licensing](https://datris.ai/licensing).

**Continuity.** Every version delivered during your term is licensed perpetually and irrevocably, the license converts to perpetual at no cost if commercial support ever ceases, and the complete source is public under the AGPL today. A Datris deployment never stops running because a vendor did.

## Community support

* **Bugs and feature requests:** [GitHub issues](https://github.com/datris/datris-platform-oss/issues)
* **Questions and design discussions:** [GitHub discussions](https://github.com/datris/datris-platform-oss/discussions)
* **Security reports:** privately, per [Vulnerability Disclosure](/production/vulnerability-disclosure)
* **Advisories and release notes ahead of disclosure:** [register your deployment](https://datris.ai/register) — Datris never calls home, so this is the only way we know you exist.


# Security Architecture
Source: https://docs.datris.ai/production/security-architecture

One page for a security reviewer: trust boundaries, how credentials are brokered, how agent actions are gated and recorded, what runs where, and what Datris deliberately does not do

This page is written for the person who has to sign off on running Datris. It states the model once and links to the detailed page for each control. Nothing here is aspirational: every control named is in the current release, and the [last section](#what-datris-does-not-do) lists what is not.

## Deployment model

Datris is fully self-hosted. It runs as a set of containers on infrastructure you operate — on-prem, in your cloud account, or on a laptop. There is no managed service, no hosted control plane, no support tunnel, and no subprocessor: no customer data reaches Datris.ai the company.

## Trust boundaries

```
                      ┌──────────────────────── your perimeter ────────────────────────┐
  Agent               │  MCP server ──► Datris server ──► Vault (credentials)          │
  (Claude, Cursor,    │      ▲               │   │                                     │
   CLI, script)  ─────┼──────┘               │   └─► Config store · object store ·    │
  holds a scoped      │  x-api-key           │       destinations (your Postgres,       │
  API key only        │                      │       warehouse, vector store, ...)      │
                      │                      ▼                                          │
                      │              tap runner (isolated container)                    │
                      │              · no platform credentials inside                   │
                      │              · no route to internal services                    │
                      │              · reaches its source + the Datris API only         │
                      └────────────────────────────────────────────────────────────────┘
```

The diagram does not draw two exits you configure yourself: the AI provider in each AI slot, and the agent you connect with an API key.

Every request follows the same path: the key identifies the caller, the capability check decides whether the route is allowed at all, the [agent policy](/agent-policy) decides whether it runs now, waits for a person, or is refused, and only then does the server consult Vault for the credential the run needs. Policy is evaluated before Vault is touched.

## Where data can go

Your data goes only where you send it. Besides the destinations your pipelines write to, data leaves the perimeter only to the AI provider you choose for each AI slot, to the agents you connect, and, if you turn them on, to web search and a GitHub repository for scripts. The chat assistants need a cloud AI provider, while code generation and embeddings can run on local models. [Data Flows](/production/data-flows) lists, for row values, schemas, credentials, provider keys, logs and scripts, every place each can go and the setting that changes it.

## Identity and authorization

* **Humans** log in to the UI with a username and password (BCrypt, cost 12). The first boot creates an `admin` account with a random bootstrap password, printed once to the server log (`docker compose logs datris | grep "Bootstrap login"`); change it at first login. Sessions are opaque random tokens in an `HttpOnly`, `SameSite=Strict` cookie with an 8-hour TTL. Three roles — admin, editor, viewer. With `OIDC_ENABLED=true`, people can instead **sign in with SSO** through your identity provider (OpenID Connect authorization code flow with PKCE; ID tokens checked for signature, issuer, audience, expiry and nonce, RS256/ES256 only); they become ordinary Datris users with one of the three roles, and the built-in `admin` account can never be reached through SSO. See [User Authentication](/user-auth) and [Single sign-on (OIDC)](/user-auth#single-sign-on-oidc).
* **Programmatic clients** — the CLI, MCP agents, scripts — each hold their own labeled API key, sent as `x-api-key` and validated against Vault. Every key carries an explicit list of capabilities (`resource:action[:scope]`); a call outside that list returns 403, and the MCP tool catalog an agent sees is filtered to match. Templates (`read-only`, `rag-builder`, `reporting`, `ops`) cover the common shapes. Keys are stored hashed, shown once, and revocable individually. See [API Keys](/api-keys).
* **Tap scripts** never receive a platform key. Each run gets a short-lived, read-only token for the platform API that the tap wrapper attaches automatically.

User login, API keys, the audit log and the agent policy ship in the open-source build and are off in the shipped defaults so that existing deployments are unchanged; they are switched on for production. A new install does it in one step with the installer option `DATRIS_GOVERNED=1`; an existing install adds `USE_USER_AUTH=true`, `USE_API_KEYS=true`, `USE_AUDIT_LOG=true` and `USE_AGENT_POLICY=true` to `.env` (see [Switch on the governance controls](/quick-start#2-switch-on-the-governance-controls)). The `governance.controls` [doctor](/doctor) check names any of the four that is off.

## Credentials

* **HashiCorp Vault holds every secret** — source and destination credentials, AI provider keys, the platform's own service passwords. The platform never persists a secret to disk outside Vault, and API responses mask sensitive fields.
* **Agents never hold a key.** An agent references a secret by name (`create_tap_secret`, `list_tap_secrets`, `get_tap_secret_fields` returns key names only). The value is delivered to the tap at run time as environment variables inside the runner and to the destination writer inside the server. It never appears in an agent's config, prompt, or tool result.
* **Agents can only touch their own secrets.** Secrets created through MCP are tagged as tap secrets; an agent cannot modify or delete a secret a person created in the UI.
* **Secrets are masked in logs.** Tap stderr is scrubbed before it is logged.

## Agent governance

* **Agent Policy** decides, per action, whether an agent runs it unattended, waits for a person to approve it, or is refused. It is enforced by the platform on every client — the in-platform Assistant, Claude, Cursor, the CLI, any script — and an agent cannot edit the policy or decide an approval, whatever its key holds. An approval executes exactly what was proposed; if the resource changed in between, it is refused as stale. See [Agent Policy](/agent-policy).
* **Audit log** (with the audit log on, `USE_AUDIT_LOG=true`) records every platform write, best effort by default and without loss in [strict mode](/audit-log#strict-mode) — create, change, run, delete, login, key rotation, denied request — attributed to a login, an API key, or the Assistant acting on a user's behalf. Agents can attach a one-line `reason` to any change and it is stored with the entry. Entries are queryable in the UI and over REST, exportable as CSV, and mirrored line-by-line to the server log (JSON under the `production` profile) so an existing log pipeline or SIEM receives the trail with no extra integration. See [Audit Log](/audit-log).
* **Provenance**, with stamping on for a pipeline, stamps every landed row (XML sources excepted; a run that cannot be stamped fails in [strict mode](/provenance#strict-mode)) with the run, config version, tap run, script commit, and declared source; `get_provenance` resolves a stamp back to its origin, and vector-search results carry the same fields per chunk so a RAG answer can cite where its context came from. See [Provenance, Lineage & Discovery](/provenance).
* **Versioning.** Every pipeline and tap definition change is snapshotted; restores are append-only. Tap scripts can be stored in a git repository you own. See [Tap GitHub Storage](/tap-github-storage).
* **Recovery agent** actions pass through the same policy gate; under `propose` mode everything it wants to do queues for approval. See [Incidents](/incidents).

## Isolation of agent-written code

Taps are Python that agents (or people) write and Datris runs. On Docker Compose the isolated runner is on by default: tap code executes in a separate `datris-tap-runner` container that holds no platform credentials and sits on a network with no route to the config store, the object store, or Vault. It can reach its data source, install the packages it declares into a throwaway environment, call back into the Datris API with its per-run read-only token, and return records. The server refuses to start with isolation on and a missing or default runner token. See [Tap Execution & Isolation](/tap-execution-isolation).

## Transport and storage

* **TLS** terminates at a reverse proxy you place in front of the server (nginx, Caddy, Traefik). Do not expose port 8080 directly. Container-to-container traffic inside the Compose network is plaintext; the host network is the trust boundary.
* **External databases.** With `DATRIS_ENV=production` the server refuses to start if a JDBC URL points at an external Postgres host without `sslmode=require` or stricter.
* **At rest.** Datris does not enforce encryption at rest on Postgres, MongoDB, MinIO, or Kafka; enable it per store for production.
* **Multi-tenancy**, when enabled, isolates tenants at the database level — separate Postgres databases and separate object-store buckets.

## Air-gapped operation

Every store Datris depends on is bundled or pointable at one you already run. For the AI features, local models through Ollama cover code generation and the one-shot helpers (profiling, fix suggestions, tap diagnosis), and the bundled `bge-m3` embedding server covers embeddings. The chat assistants (the Assistant, Search chat, Catalog chat, the Configuration assistant, Ops chat and the recovery agent) need one of the supported cloud providers today, so an air-gapped deployment runs without them. See [No cloud exit](/production/data-flows#no-cloud-exit). The bundled embedding model downloads once on first start; pre-pull the images and the model on a connected machine and move them across. Add `DATRIS_UPDATE_CHECK=0` to `.env` so `datris doctor` does not try to reach datris.ai (without a route out the attempt only times out after 3 s and is reported as skipped).

## No telemetry

The platform sends no usage data, crash reports, or license checks. There is no beacon in the server, the UI, the MCP server, or the installer, and no install id or other identifier is created. Verify it yourself:

```bash theme={null}
git clone https://github.com/datris/datris-platform-oss.git && cd datris-platform-oss

# No analytics, error-reporting or beacon library or vendor anywhere in the code.
# Prints nothing.
grep -rniE "telemetry|posthog|sentry|segment\.(io|com)|mixpanel|amplitude|google-analytics|googletagmanager|datadoghq|newrelic|bugsnag|rollbar\.com|hotjar" \
  --exclude-dir=node_modules --exclude-dir=.venv --include='*.scala' --include='*.ts' --include='*.py' --include='*.sh' --include='*.html' . \
  | grep -vE '/tests?/|\.spec\.ts'

# Every datris.ai address the code calls. Prints exactly two lines.
grep -rnE "[\"']https://([a-z0-9-]+\.)?datris\.ai/[^\"' ]*[\"']" \
  --exclude-dir=node_modules --exclude-dir=.venv --include='*.scala' --include='*.ts' --include='*.py' --include='*.sh' . \
  | grep -vE '/tests?/|\.spec\.ts|docs\.datris\.ai'
```

The second command prints two lines, one for each outbound request Datris can make. These are the only two; links to the documentation site (`docs.datris.ai`) are shown to you, never fetched.

| Request | When | What is sent | Timeout and failure | How to switch it off |
| - | - | - | - | - |
| Model catalog, `GET https://datris.ai/models.json` (`ModelCatalogAPIController.scala`) | Only when a person opens the AI configuration screen; cached for five minutes | Nothing beyond the request itself | 3 s; the model pickers fall back to a list built into the UI | No setting; block the address at your firewall and the built-in list is used |
| Update check, `GET https://get.datris.ai/version` (`mcp-server/doctor.py`) | Only when a person (or a script) runs `datris doctor` on the host. Never from the server, never on a timer, never from `datris doctor --pre-upgrade` or the installer | The server version, the operating system and the CPU architecture as query parameters (`?version=1.44.0&os=linux&arch=x86_64`), and a `User-Agent: datris-doctor/<cli version>` header. No API key, cookie, body, hostname or identifier | 3 s; the `version.update` row becomes a skip ("could not reach datris.ai; nothing received") and never changes the exit code | `DATRIS_UPDATE_CHECK=0` in the project `.env` (run doctor from the install directory or pass `--project-dir`, or set it in the environment); the row then reads "off" and nothing is sent |

Datris uses the update check to count the versions in use by distinct source address. Nothing is stored on your machine. The documentation site you are reading measures page views; the platform does not.

## Vulnerability handling

Private reporting, response targets, advisories, SBOMs, and dependency scanning are on [Vulnerability Disclosure](/production/vulnerability-disclosure).

## What Datris does not do

Stated so you can plan around it rather than discover it:

* No built-in MFA. MFA comes from your identity provider through OIDC single sign-on (`OIDC_ENABLED=true`, see [Single sign-on (OIDC)](/user-auth#single-sign-on-oidc)): people who sign in with SSO get whatever MFA, password rules and lockout the provider enforces. Local accounts, including the `admin` recovery account, still have a minimum password length only and no complexity rules or lockout.
* No SAML, and no group-to-role mapping from the identity provider yet: an SSO user's role is set on the Users screen (or `OIDC_DEFAULT_ROLE` for new people).
* No built-in rate limiting or WAF — put one in front of production deployments.
* No enforced encryption at rest — enable it per store.
* CORS allowed origins default to `*` for development — set `cors.allowedOrigins` to your real origins in production.
* No SOC 2 report. In a self-hosted deployment no customer data reaches Datris.ai, so most trust-services criteria fall on your environment; the controls that do apply to the vendor — secure development and vulnerability handling — are described above.

## Operator checklist for production

1. Run Vault in non-dev mode with sealed root tokens.
2. Put a TLS-terminating reverse proxy in front of the server; do not expose 8080.
3. Switch on the governance controls: `DATRIS_GOVERNED=1` on a new install, or `USE_USER_AUTH=true` and `USE_API_KEYS=true` (with the agent policy and audit log in step 5) in `.env`; change the admin bootstrap password immediately; issue one scoped key per agent.
4. Turn on SSO (`OIDC_ENABLED=true` with your identity provider's issuer and client) so people sign in with their company account and its MFA; keep the `admin` password for recovery. See [Single sign-on (OIDC)](/user-auth#single-sign-on-oidc).
5. Turn on `USE_AGENT_POLICY=true` and start from **Use recommended**; turn on `USE_AUDIT_LOG=true` and ship the log line to your aggregator. `datris doctor` shows `governance.controls` as ok once all four are on.
6. Keep the isolated tap runner on (the Compose default).
7. Set `DATRIS_ENV=production`, `sslmode=require` on external Postgres, and `cors.allowedOrigins`.
8. Enable encryption at rest on the stores; rotate API keys periodically.
9. Run the [doctor](/doctor) self-check before every upgrade.

If you run separate dev, QA and production installs, give QA the same governance settings as production (`DATRIS_GOVERNED=1`, SSO, the agent policy and the audit log), so that permission and policy problems show up in QA before a catalog reaches production. See [Dev, QA and production](/production/environments).


# Upgrades & Supported Versions
Source: https://docs.datris.ai/production/upgrades

Release cadence, which versions receive fixes, and how to upgrade each kind of install without losing data

## Release cadence and supported versions

Datris ships small releases often — typically several per month — and every release is tagged, published to Docker Hub as multi-architecture images, and described in the [changelog](/changelog). There is no separate stable and unstable line: `latest` is the current release.

| | |
| - | - |
| **Security fixes** | Only the latest minor version receives security updates. Upgrade to the current release to pick them up. |
| **Migration steps** | When a release needs one — a renamed service, a moved secret, a changed volume — it is in the changelog entry for that release. Read the entries between your version and the target before upgrading. |
| **Data compatibility** | Data lives in Docker volumes and is preserved across upgrades. Releases that change a Vault secret layout keep reading the old layout until you clean it up (see [Stale secrets](#stale-secrets-after-an-upgrade)). |
| **Long-term support** | An LTS branch with a stated patch window, and help moving between releases on your schedule, are part of a [commercial license](/production/licensing-support). |

Run the pre-upgrade self-check before every upgrade — it catches the things that turn an upgrade into an outage and prints the fix for each.

## Upgrade procedure

Upgrading means two things: pulling the latest pre-built images **and**
refreshing the Compose file itself, since a new release can rename, add, or drop
services. How you do it depends on how you installed. In every case your data is
preserved — it lives in Docker volumes that survive an upgrade.

**Step 0 — check the deployment first.** From the directory holding your
`docker-compose.yml` and `.env`, run the pre-upgrade self-check. It catches the
things an upgrade turns into an outage: data on an anonymous volume that
`--remove-orphans` would drop, a missing AI slot secret the new server would
crash-loop on, a disk too full to pull images, a container still running with a
stale `.env`. Nothing is changed; it prints the fix for each finding and the
upgrade command when the deployment is clean. See [Doctor](/doctor).

```bash theme={null}
datris doctor --pre-upgrade
```

**Installer (the `curl … | sh` Quick Start):** just re-run the install command.
It re-downloads the latest compose file and runtime scripts, leaves your `.env`
and data untouched, pulls the new images, and restarts. When the `datris` CLI is
on the machine it runs step 0 for you first: a warning continues, an error stops
the upgrade with the fix on screen (`DATRIS_SKIP_DOCTOR=1` bypasses it).

```bash theme={null}
curl -fsSL https://get.datris.ai/install.sh | sh
```

**Single-file (`docker-compose.standalone.yml`):** re-download the file (it
carries the latest topology and inlined scripts), then pull and restart.

```bash theme={null}
curl -O https://get.datris.ai/docker-compose.standalone.yml
docker compose -f docker-compose.standalone.yml pull
docker compose -f docker-compose.standalone.yml up -d --remove-orphans
```

**From source (cloned repo):** pull the latest compose file from git, then pull
images and restart.

```bash theme={null}
cd datris-platform-oss
git pull origin main
docker compose pull
docker compose up -d --remove-orphans
```

No build tools required for any of these — the images come pre-built from Docker Hub.

<Note>
  **Always pass `--remove-orphans` when upgrading manually.** (The installer does this for you.) Releases occasionally rename, remove, or repurpose service blocks in `docker-compose.yml` (e.g. v1.6.15 replaced the bundled Ollama service with TEI on the same host port `11434`). Without `--remove-orphans`, the previous version's container keeps running and holding the port, causing the new container to fail with `Bind for 0.0.0.0:<port> failed: port is already allocated`. The flag is safe — it only removes containers that are no longer defined in your current compose file. Volumes (and your data) are untouched.
</Note>

Your data is preserved across upgrades — Postgres pipelines, Vault secrets, MinIO buckets, MongoDB collections, and Kafka state all live in Docker volumes that survive `docker compose pull` and `docker compose up -d`.

### Stale secrets after an upgrade

Datris occasionally deprecates secret paths between versions (for example, v1.5.6 split AI configuration into three new Vault slots and stopped reading the old single-slot path). Because Vault data persists across upgrades, deprecated entries stay in Vault and continue to appear in the Configuration → Secrets list even though the server ignores them. To clean them up:

* **Targeted (preserves your data):** delete the stale entries one at a time from Configuration → Secrets in the UI, or run `docker compose exec -e VAULT_TOKEN="$(docker compose exec -T datris cat /vault-token/token)" vault vault kv delete secret/oss/<name>` for each deprecated path.
* **Total reset (destroys all data):** see the next section.

## Roll back

Every release is also published under its version tag, so rolling back is pinning the four `image:` lines in your Compose file to the version you were on (for example `datrisai/datris-server:v1.28.2`, and likewise `datris-ui`, `datris-mcp-server`, and `datris-tap-runner`), then `docker compose up -d`. Data written by the newer version stays in place; whether the older version reads it depends on what changed — check the changelog entry for the release you are backing out of before rolling back across one that migrated a store.

## Related

* [Doctor](/doctor) — the operational self-check, including `--pre-upgrade`
* [Installation](/installation) — volumes, services, and the full-reset path
* [Changelog](/changelog)


# Vulnerability Disclosure
Source: https://docs.datris.ai/production/vulnerability-disclosure

How to report a security issue in Datris, what to expect back, and how fixes and advisories reach you

## Reporting

Report suspected vulnerabilities **privately** through GitHub's security advisory process:

> [github.com/datris/datris-platform-oss/security/advisories/new](https://github.com/datris/datris-platform-oss/security/advisories/new)

Please do not open a public issue for a security report. If you cannot use GitHub, email `info@datris.ai` with the same detail you would put in an advisory: affected version, component, reproduction steps, and impact.

## Response targets

| | |
| - | - |
| Acknowledgement | within 3 business days |
| Remediation timeline communicated | within 10 business days |
| Fix delivery | in the next release; security fixes go to the latest minor version only — see [Upgrades & Supported Versions](/production/upgrades) |

## Advisories

Fixed vulnerabilities are published as GitHub Security Advisories on the repository and summarized in the [changelog](/changelog) with impact described in general terms. Deployments that [register with Datris.ai](https://datris.ai/register) and commercial licensees receive advisories ahead of public disclosure.

## SBOM and scanning

* **SBOM.** Every release publishes a CycloneDX software bill of materials for each of the four container images (server, UI, MCP server, tap runner), generated by Syft and attached to the release's `docker-publish` workflow run on GitHub.
* **Dependency scanning.** Trivy runs on every pull request and push to `main` and fails the build on new HIGH or CRITICAL findings with an available fix. Published images are re-scanned weekly for OS-package CVEs. Dependabot security updates cover GitHub Actions, npm, pip, and Docker base images; JVM dependencies are covered through the GitHub dependency graph.
* **Secret scanning and push protection** are enabled on the repository.

## Related

* [Security Architecture](/production/security-architecture) — the model these controls sit in
* The repository's `SECURITY.md` is the canonical statement of this policy.


# Data lineage for AI agents
Source: https://docs.datris.ai/provenance

Trace any stamped row back to the run, config version, script commit, and source that produced it. Agents query the same graph to check impact before changing anything and to find datasets by meaning.

Every answer an agent gives from Datris data should be traceable. Three features make that real:

* **Provenance stamping** (opt-in per pipeline) writes run identity onto every row, document, message and vector chunk a pipeline lands, when stamping succeeds (XML sources are never stamped; see [Strict mode](#strict-mode) for what happens when stamping fails).
* **Lineage** shows the chain *Source → Tap → Pipeline → Dataset → Catalog*, derived deterministically from your configuration — no AI, no inference.
* **`find_data`** lets an agent discover a dataset by meaning, learn where it lives, how fresh it is, and exactly how to query it — without Datris executing anything on its behalf.

## Provenance stamping

Turn it on per pipeline: **Catalog → pipeline → Edit → Destination step → "Stamp provenance on landed data"**, or set it in the config:

```json theme={null}
{
  "name": "orders",
  "provenance": { "stamp": true }
}
```

From the next run on, when stamping succeeds, every row the pipeline lands carries six extra fields (see [When stamping fails](#when-stamping-fails)):

| Field | Meaning |
| - | - |
| `_datris_run_id` | The pipeline run that landed this row |
| `_datris_ingested_at` | When it landed (UTC, ISO-8601) |
| `_datris_config_version` | The pipeline definition version at run time |
| `_datris_tap_run` | The tap run that fed it (`tapName\|runTime`; empty for direct uploads) |
| `_datris_script_sha` | The tap script's commit, for repository-backed taps |
| `_datris_source` | Where the data really comes from — the tap's declared `source`, else its endpoint host, else the host its script calls most; never credentials |

How the fields land depends on the destination:

* **PostgreSQL, Snowflake, Databricks** — columns, auto-added as text to existing tables on the first stamped run (the same additive schema evolution new CSV columns already get).
* **MongoDB and JSON data** — keys on each document. XML data is not stamped: an XML source always loads unstamped, in strict mode too.
* **Object store (CSV/Parquet)** — columns in the written files, from the first stamped run on.
* **Kafka / ActiveMQ** — fields in each message; consumers that ignore unknown fields are unaffected.
* **Vector stores** — payload metadata on every chunk, so search results cite the run and source per chunk.

Rows are **never rewritten retroactively**: provenance starts at the first run after the toggle is turned on, and older rows read as null/absent. Stamped columns are always stored as text and are never proposed for retyping by the destination-typing flow.

A subset of fields can be selected with `"provenance": { "stamp": true, "fields": ["_datris_run_id", "_datris_ingested_at"] }`.

### When stamping fails

By default stamping is best effort: a run whose data cannot be stamped still loads, unstamped. When stamping throws an error, the run's status records `Provenance stamping skipped (error): <reason>`; the other cases load unstamped with no status line. Known cases:

* a JSON payload that is not newline-delimited JSON, or has a line that does not parse;
* unstructured data with no vector destination (there is no metadata map to carry the stamp);
* for JSON and vector data, a `fields` selection that yields no value for the run (for example only `_datris_tap_run` on a direct upload; delimited data still gets the columns, empty);
* an XML source, which is never stamped.

### Strict mode

If every landed row must carry its stamp, set `PROVENANCE_STRICT=true` in `.env` and recreate the Datris container. Then a pipeline with `provenance.stamp` on fails its run with `Provenance could not be stamped (<reason>) and PROVENANCE_STRICT is on; nothing was loaded` instead of loading unstamped rows. The failure happens before any destination is written. Delimited, JSON and vector data that can be stamped are stamped exactly as in the default mode.

* **XML sources are exempt.** XML cannot be stamped, so under strict mode an XML pipeline with stamping on still saves and still loads unstamped; its run status says `Provenance stamping skipped (XML source is exempt from PROVENANCE_STRICT)`. Those rows are outside the strict guarantee. `datris doctor` lists such pipelines under `provenance.strict`.
* **Field names are checked.** A pipeline whose `provenance.fields` names a field that does not exist is refused when it is saved, even with `stamp` off; a pipeline saved earlier with such a name must have it removed before it can be saved again.
* **Stamping stays opt-in.** Strict mode does not turn stamping on; pipelines without `provenance.stamp` are untouched.

## Resolving a stamp

`GET /api/v1/provenance?runId=<_datris_run_id>` (or the `get_provenance` MCP tool) walks the chain and returns one document: the run's job status and record count, the tap run that fed it (with logs metadata and script commit), the pipeline config version snapshot, and the declared source. Pass the row's `_datris_tap_run` and `_datris_config_version` values when you have them for an exact, lookup-only resolution.

For runs recorded since run lineage was introduced, the same document also carries `input` (what the run read) and `outputs` (each destination the run wrote, with its own status), and the config version resolves from the recorded run when the row's stamp is not available.

## Lineage & freshness

`GET /api/v1/lineage` returns the whole graph; `GET /api/v1/lineage/{type}/{name}` returns one node's neighborhood — everything upstream and downstream of a source, tap, pipeline, dataset, or catalog. The graph is built from configuration, on request, plus what recorded runs actually wrote (below).

Freshness per pipeline combines the last successful landing, its record count, the incremental sync cursor (when the feeding tap keeps one), and the same stale classification the Ops Activity dashboard uses — the catalog and the dashboard never disagree.

In the UI, a pipeline's detail page shows its upstream tap and source, the datasets it lands in, and the freshness line. **Catalog → Lineage** opens the interactive graph: the whole chain laid out left to right, filterable by catalog, tag or name. Click any node for what feeds it, what depends on it, its freshness, and its recent runs, with links to the tap or pipeline.

### Run-level lineage

Every pipeline run is recorded as it completes: what it read (the tap run or uploaded file), which destinations it wrote — each with its own success or failure — how many records, and the definition version it ran under. Recording is automatic for every pipeline, independent of provenance stamping, and never fails or delays a run.

Two things follow from that record:

* **History, not just current config.** If a pipeline used to land into one table and now lands into another, the old dataset stays in the graph marked `historical` (dashed in the UI) with the runs that wrote it, instead of silently disappearing when the config changed.
* **Recent runs per node.** Ask for `?runs=10` on any neighborhood, or use the **Recent runs** section of the graph's side panel, to see what actually happened — including a multi-destination run where one destination failed and the others landed.

Runs are recorded from the first run after upgrading; earlier runs still resolve through provenance as before, but have no recorded outputs.

### Evidence on every hop

Each edge in the graph carries what actually traversed it in the last 90 days: runs, records landed, failed runs, and the time and status of the last run — aggregated from the recorded runs (pipeline hops) and the tap run log (source → tap). An edge with no recorded run behind it is a configuration claim, not evidence, and draws dotted and faint; edge width grows with records landed. The side panel shows the same numbers on each upstream and downstream chip ("8 runs · 12.4k rows · 8/31"), and `find_data` returns `evidence` per location. Bytes are not tracked.

### Source of authority

Provenance answers where a row came from; authority answers which copy you may cite. Every dataset node carries one of three labels, **declared, never inferred**:

* **system of record** (`authoritative`) — the copy to cite. A pipeline with a single destination is authoritative by default, so existing configurations need no change.
* **derived** — a copy: a rollup, a replica, an index built from another dataset, or a historical dataset nothing lands into anymore. Set `"authoritative": false` on the pipeline (wizard: *This pipeline lands a derived copy*; `create_pipeline`: `authoritative: false`).
* **undeclared** — more than one candidate and no decision: a pipeline with several destinations and no `destination.authoritative`, or two pipelines that both claim the same dataset. The graph shows an amber ring; saving a second authoritative writer for the same dataset is rejected.

`find_data` lists the authoritative location first and labels the rest; `get_provenance` reports the authority of each dataset a run wrote; the graph's **Canonical only** filter shows just the systems of record and their chains.

### Column-level lineage

For any pipeline, `GET /api/v1/lineage/columns/{pipeline}` (or the **Columns** section of a pipeline or dataset node in the graph) shows which destination columns come from which source fields. Two tiers, clearly labeled:

* **Exact** — derived from the pipeline's field schemas alone, no AI: name-identical fields pass through; with no transformation, source-only fields are marked dropped. Provenance stamp columns show as **system**.
* **Inferred** (opt-in) — for pipelines with an AI transformation, click **Infer from transformation** (or pass `infer=true`) and the code-generation model reads the instruction, the field lists and the stored CodeGen transformation script, returning only mappings it can point to evidence for: renames, derived columns, drops. Anything it cannot evidence stays **unresolved** rather than guessed. Results are cached per definition version and never recomputed unless the definition changes; inference runs on request, never during a run.

A dataset node shows the mapping of the pipeline that lands into it. A historical dataset shows the mapping as of the definition version of the last run that wrote it. Column edges are not drawn on the canvas — they live in the panel.

### Traversal for agents

The `get_lineage` MCP tool (or the neighborhood endpoint's `direction`, `depth`, `runs` and `columns` parameters) lets an agent ask "what is downstream of this tap?" before changing or deleting it, or "what did this pipeline actually write lately?" before trusting a dataset. Direction is `up`, `down` or `both`; depth bounds the hops; runs appends the recent recorded runs; columns embeds the column-level lineage for a pipeline or dataset node.

## Sources

Lineage and provenance name the *real* origin of data, not the tap. Each tap resolves to one source identity, deterministically:

1. The tap's declared **`source`** (set it in the tap wizard, or pass `source` to `create_tap` / `update_tap`) — e.g. `SEC EDGAR`.
2. For HTTP taps, the endpoint host.
3. For script taps, the host the script references most, normalized so `www.sec.gov` and `data.sec.gov` both read `sec.gov`.
4. `tap:<name>` when nothing better is known.

Several taps pulling from the same provider share one source node, so the graph shows the provider feeding many taps rather than one placeholder per tap. A script edit that switches provider moves the tap under its new source on the next graph build; rows already landed keep the `_datris_source` they were stamped with.

## Tags

Taps and pipelines take free-form `tags` (edit them in the wizards, or via the API). They render in the catalog tables and feed discovery ranking. That is the whole "business concept" layer: explicit, reviewable, boring.

## `find_data` — discovery without execution

The `find_data` MCP tool (or `GET /api/v1/catalog/find?query=…`) ranks the pipelines the calling key can read against a natural-language query — over names, descriptions, tags, catalogs, destination field names and source hosts. Each hit returns:

```json theme={null}
{
  "name": "orders",
  "description": "Daily order export",
  "tags": ["sales"],
  "catalog": "commerce",
  "freshness": { "state": "fresh", "lastLandedAt": "…", "recordCount": 14223 },
  "location": { "kind": "postgres", "database": "datris", "schema": "public", "table": "orders" },
  "howToQuery": { "tool": "query_postgres", "args": { "sql": "SELECT * FROM \"public\".\"orders\" LIMIT 100" } },
  "provenance": { "latestRunId": "…", "configVersion": 12, "scriptSha": "a1b2c3d" },
  "lineage": { "upstream": ["tap:orders-export"], "downstream": ["dataset:postgres:datris.public.orders"] }
}
```

`howToQuery` names an **existing** tool with pre-filled arguments; the agent still makes that call itself, under its own capabilities. That is deliberate:

1. Capabilities are enforced at the query call — a viewer key can find but not read.
2. Provenance attaches to the dataset actually queried, not a synthesized result.
3. The agent's loop stays visible — a wrong hint is a tool error the agent sees and corrects, not a silently wrong answer.

Ranking is deterministic by default; pass `ai=true` for an optional rerank of the top candidates by the primary AI model.

To carry provenance through to an answer, pass the handles to `ai_answer` as `sources` — they are echoed back on the response, alongside the answer text, and never fed to the model.


# Quick Start
Source: https://docs.datris.ai/quick-start

Fifteen minutes from install to a governed pipeline: an agent builds a tap, a person approves the run, and with provenance stamping on every landed row cites where it came from

This walkthrough ends where Datris is different: an agent lands data from a public source, but only after a person approves the run, and with provenance stamping on every row it lands can be traced back to the run, the script, and the source. You will need Docker, one AI provider key (Anthropic, OpenAI, or Grok), and an MCP client — Claude Desktop, Claude Code, or Cursor.

<Tip>
  Want the plain REST version first — register a pipeline, upload a CSV, query it? That walkthrough is on [File Upload](/ingestion/file-upload#end-to-end-example-csv-to-postgresql).
</Tip>

## 1. Install and start

```bash theme={null}
curl -fsSL https://get.datris.ai/install.sh | sh
```

Press Enter at every prompt for a standard install, paste your AI key when asked. Wait for the `datris` container to log `Started Application`, then confirm:

```bash theme={null}
curl http://localhost:8080/api/v1/version
```

Other install paths (Windows, single-file Compose, git clone) are on [Installation](/installation).

## 2. Switch on the governance controls

User login, API keys, the audit log and the agent policy ship in the open-source build and are switched on for production. They are off in the shipped defaults so that an upgrade with an existing `.env` behaves exactly as before.

**New install:** use this in place of the step 1 command, or answer yes when the installer asks:

```bash theme={null}
curl -fsSL https://get.datris.ai/install.sh | DATRIS_GOVERNED=1 sh
```

**Existing install:** first read and save the `admin` bootstrap password. The server prints it once, in the log of the container that seeded `admin`, and recreating the container (the next command) discards that log:

```bash theme={null}
cd datris && docker compose logs datris | grep "Bootstrap login"
```

If nothing is printed, the line is already gone (the container was upgraded or recreated since `admin` was seeded); follow [Recovering admin access](/user-auth#recovering-admin-access) to get a new one. If you already set your own `admin` password, use that.

Then add these four lines to `./datris/.env`:

```bash theme={null}
USE_USER_AUTH=true
USE_API_KEYS=true
USE_AGENT_POLICY=true
USE_AUDIT_LOG=true
```

Recreate the server and the MCP server so they pick them up (the MCP server reads `USE_API_KEYS` and starts requiring a key):

```bash theme={null}
cd datris && docker compose up -d --force-recreate datris mcp-server
```

`datris doctor` reports any of the four that is still off as a `governance.controls` warning.

Open [http://localhost:4200](http://localhost:4200). The first login is `admin` with the bootstrap password. On a new install it is in the log of the first boot (`cd datris && docker compose logs datris | grep "Bootstrap login"`); on an existing install use the one you saved above. Change it after you log in (your username in the top-right → **Change password**).

## 3. Set one policy rule

**Configuration → Agent Policy → Use recommended.** That starting point pauses deletes, job kills, and destination-type migrations for approval and refuses secret and configuration writes. Add one rule for this walkthrough: set **`tap:run`** to **Approve**. From now on, any agent that tries to run a tap waits for a person.

## 4. Issue a scoped key for your agent

**Configuration → API-Keys → Issue new key.** Label it after the client (`claude-desktop`, `cursor`), pick the **rag-builder** template — it can create pipelines and taps and run the ones it created, and nothing else — and copy the value; it is shown once.

## 5. Connect your agent

Add the Datris MCP server to your client with the key from step 4. Claude Desktop (`claude_desktop_config.json`), Claude Code (`.mcp.json` in your project root), and Cursor all take the same snippet — see [Configuring Claude](/configuring-claude) for the file locations and the no-Docker alternative.

```json theme={null}
{
  "mcpServers": {
    "datris": {
      "command": "npx",
      "args": [
        "-y", "mcp-remote", "http://localhost:3000/sse",
        "--transport", "sse-only",
        "--header", "x-api-key:<paste the key from step 4>"
      ]
    }
  }
}
```

Restart the client. In the Datris UI, the **Agent Monitor** tab now shows the agent connected under its key label, and streams every tool call it makes from here on.

## 6. Ask the agent to build a tap against a public source

The USGS earthquake feed needs no credentials, so nothing has to be stored before the first run. Ask your agent:

> "Create a Postgres pipeline called `earthquakes` and a tap called `usgs-earthquakes` that fetches the USGS 'all earthquakes, past day' GeoJSON feed at [https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all\_day.geojson](https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson) and produces one row per event with its id, time, place, magnitude, longitude, latitude, and depth. Test the tap so I can see a sample, but don't run it yet."

Watch the Agent Monitor: `create_pipeline`, `create_tap`, `test_tap`. The agent shows you a handful of sample rows. Nothing has been landed.

## 7. Turn on provenance stamping

In the UI: **Catalog → `earthquakes` → Edit → Destination step → "Stamp provenance on landed data"** → save. From the next run on, with stamping on, every row this pipeline lands carries six extra columns naming the run, the time, the config version, the tap run, the script commit, and the source. (Do this before the run rather than after — an approval executes exactly what was proposed, and a pipeline changed underneath a pending approval is refused as stale.)

## 8. Run it — and watch it wait for you

> "Run the `usgs-earthquakes` tap."

The agent calls `run_tap` and gets back `pending_approval` with an approval id instead of a result, because of the rule from step 3. It tells you so, and can poll `get_approval` while it waits. Nothing has run.

Open **Activity → Approvals**. The card shows exactly what will execute — the method, path, and body of the agent's request, with the reason the agent gave, if any. Click **Approve**. The platform performs the original request on your behalf; the agent's next poll sees it executed and it follows the load to completion with `get_pipeline_status`. Ask it to confirm:

> "Did the run land? How many rows?"

## 9. Ask where a row came from

> "Pick one row from `earthquakes` and tell me where it came from — the run, the tap run, the script commit, and the source."

The agent reads the row's `_datris_run_id` and calls `get_provenance`, which walks the chain and returns the pipeline run with its status and record count, the tap run that fed it, the pipeline definition version in force, and the declared source. You can see the raw stamp yourself:

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/query/postgres \
  -H "x-api-key: <your key>" -H "Content-Type: application/json" \
  -d '{"sql": "SELECT _datris_run_id, _datris_ingested_at, _datris_tap_run, _datris_source FROM public.earthquakes LIMIT 3"}'
```

(`_datris_script_sha` is populated when the tap's script is stored in a git repository — see [Tap GitHub Storage](/tap-github-storage).)

## 10. Read the record

**Configuration → Audit Log.** Two linked entries tell the whole story: the agent's queued `tap:run` request under its key label, and your approval executing it under your login. The pipeline and tap creations from step 6 are there too, attributed to the same key. Everything you just watched happen is now something you can answer questions about next month.

## What you just saw

* An agent with a **scoped key** could build and test, but its run was **gated by policy** — enforced by the platform, not by a prompt.
* A **person approved** exactly what was proposed, and the platform executed it on their behalf.
* With stamping on, every landed row carries **provenance** ([strict mode](/provenance#strict-mode) fails a run that cannot be stamped), and the agent can resolve it back to the run, script, and source.
* The **audit log** holds the request, the decision, and the result.

## Next steps

* [Agent Policy](/agent-policy) — the full action list, per-resource overrides, and approval limits
* [Provenance, Lineage & Discovery](/provenance) — lineage graphs, freshness, and `find_data`
* [Taps](/taps) — schedules, incremental state, secrets for sources that need them
* [Security Architecture](/production/security-architecture) — the whole model on one page for your reviewer
* [Pipeline Configuration](/pipeline-configuration) — everything a pipeline can do


# Give Claude Code a safe way to load data into PostgreSQL
Source: https://docs.datris.ai/recipes/claude-code-postgres

Connect Claude Code to Datris over MCP, gate the destructive actions behind human approval, load a table, and see the audit and lineage records it leaves behind.

This recipe is for the moment you want an agent to load data into a database you care about, and you are not willing to hand it the database credentials. Claude Code never sees a PostgreSQL connection string here. It talks to Datris over MCP, Datris holds the credentials, validates the data against a schema, and, with the audit log on, records who did what (see [strict mode](/audit-log#strict-mode)).

You will end up with:

* Claude Code connected to Datris with an identifiable API key
* A policy that lets Claude create and run pipelines on its own but pauses deletes and table rewrites for a person to approve
* A PostgreSQL table loaded by Claude, with a lineage record and an audit trail you can read

Time: about 15 minutes on a running Datris stack. If you have not installed Datris yet, start with the [Quick Start](/quick-start).

## 1. Turn on the guardrails

Three features are off by default so existing installs are unchanged. Add them to your `.env`:

```bash theme={null}
USE_API_KEYS=true
USE_AGENT_POLICY=true
USE_AUDIT_LOG=true
```

Recreate the Datris container so it picks them up:

```bash theme={null}
docker compose up -d --force-recreate datris
```

## 2. Issue a key for Claude Code

Open **Configuration → API-Keys → Issue new key**. Label it `claude-code`. The label is how this agent will appear in the audit log, the Agent Monitor, and version history.

Pick the **rag-builder** template, or build a list that lets the key create and run pipelines and taps and read results. Avoid **full-access** for an agent. Copy the key value when the modal shows it. It is shown once.

What a key may do at all is set here. What it may do unattended is the next step. The two are explained side by side in [API Keys](/api-keys#capabilities-vs-the-agent-policy).

## 3. Set the policy

Open **Configuration → Agent Policy** and click **Use recommended**. That starting point:

| Action | Mode |
| - | - |
| Create and run pipelines and taps | Auto |
| Delete a pipeline or tap, kill a job, migrate destination column types | Approve |
| Write secrets, code repositories, or platform configuration | Deny |

Anything not set is Auto. The policy applies within a few seconds of saving. A person clicking in the Datris UI is never gated. They are the approver.

## 4. Connect Claude Code

Add a `.mcp.json` to your project root:

```json theme={null}
{
    "mcpServers": {
        "datris": {
            "command": "npx",
            "args": [
                "-y",
                "mcp-remote",
                "http://localhost:3000/sse",
                "--transport",
                "sse-only",
                "--header",
                "x-api-key:<the key from step 2>"
            ]
        }
    }
}
```

Other clients and transports are covered in [Configuring Claude](/configuring-claude). Restart Claude Code, then confirm the round-trip:

> List my Datris pipelines and tell me what's there.

Watch the **Agent Monitor** tab in the Datris UI. The tool call appears there in real time, attributed to `claude-code`.

## 5. Load a table

Ask for something concrete:

> Generate 50 rows of sample order data (order\_id, customer, amount, ordered\_at) and load it into PostgreSQL as a pipeline called `orders`.

Claude calls `create_pipeline` with the sample data. Datris detects the schema, creates the table, bulk-loads the rows with `COPY`, and reports the outcome. Claude is told to verify completion through `get_pipeline_status` rather than trusting the response body, so it will poll until the run is done and tell you the row count.

Then have it prove the data is there:

> Query the orders pipeline and show me the five largest orders.

## 6. Try something the policy pauses

> Delete the orders pipeline.

Claude does not get a result. It gets a `pending_approval` response with an approval id, and it will tell you plainly that the action is waiting. Open **Activity → Approvals**. The card shows the exact request that will run, with any secret-looking values masked, and the reason Claude gave if it gave one.

Reject it. On its next poll Claude is told so. If you had approved it, the platform would have executed the original request attributed to you, and the audit log would hold two linked entries: the agent's queued request and your approval executing it.

## 7. Read what was recorded

**Configuration → Audit Log** now shows the whole session newest first: the pipeline create, the run, the query, the parked delete, and your rejection, each with the actor `claude-code` or your login. Filter by actor to see only what the agent did. **Export CSV** if someone else needs the trail, and note that the export is itself recorded.

**Catalog → Lineage** shows the `orders` pipeline and the table it wrote, with freshness and the run that produced it. Turn on **Stamp provenance on landed data** in the pipeline's destination step if you want every row in PostgreSQL to carry the run id when stamping succeeds ([strict mode](/provenance#strict-mode) fails the run otherwise), so a row can be traced back to its run, config version, and source at any later date.

## What you have now

Claude Code can build and run pipelines on its own, which is where the time savings are. It cannot delete, rewrite a landed table, or touch secrets without a person clicking approve, and it cannot talk its way past that because the gate is in the platform rather than in a prompt. Every action it took is attributed and queryable.

The same setup works unchanged for Claude Desktop, Cursor, or any script holding an API key, and the same policy gates the in-platform Assistant and the [recovery agent](/incidents).

## Next

* Point the key at a real source with a [tap](/taps), and set a schedule on it. Scheduled runs are the platform's own work and are not gated.
* Swap PostgreSQL for [Snowflake](/destinations/snowflake) or [Databricks](/destinations/databricks). Both keep credentials per pipeline, so the agent never holds them either.
* Tighten one resource without touching the rest: a per-resource override can pause runs of a single tap while every other tap runs freely. See [Agent Policy](/agent-policy#per-resource-overrides).


# Schema Definition & Auto-Generation
Source: https://docs.datris.ai/schemas

Define and auto-generate schemas

Schemas define the structure of data flowing through the pipeline. Every pipeline requires a source schema that describes the incoming fields and their types. Optionally, a destination schema can override types or rename fields when writing to a target.

## Defining Schemas

Schemas are declared in the `source.schemaProperties.fields` array of a pipeline configuration. Each field entry specifies a name and a data type.

```json theme={null}
{
  "source": {
    "schemaProperties": {
      "fields": [
        { "name": "id", "type": "bigint" },
        { "name": "email", "type": "varchar(255)" },
        { "name": "signup_date", "type": "date" },
        { "name": "balance", "type": "decimal(12,2)" },
        { "name": "is_active", "type": "boolean" },
        { "name": "notes", "type": "string" }
      ]
    }
  }
}
```

## Supported Data Types

When the UI or AI schema generation infers types, it uses these common types:

| Type | Description |
| - | - |
| `boolean` | True/false value |
| `int` | 32-bit signed integer |
| `bigint` | 64-bit signed integer |
| `float` | 32-bit floating point |
| `double` | 64-bit floating point |
| `string` | Variable-length text |
| `date` | Calendar date (no time component) |
| `timestamp` | Date and time with microsecond precision |

### MCP Server (AI Agents)

When an AI agent creates a pipeline via the MCP server, all fields are set to `string`. This is intentional — it ensures data is always ingested successfully regardless of format inconsistencies. Type enforcement happens at the destination (e.g., PostgreSQL column types) rather than at ingestion time, avoiding failed jobs from unexpected values.

### Advanced Types

These types are supported by the server but are never auto-generated. Use them when writing pipeline JSON configuration manually:

| Type | Description |
| - | - |
| `tinyint` | 8-bit signed integer |
| `smallint` | 16-bit signed integer |
| `decimal(p,s)` | Fixed-precision decimal with `p` total digits and `s` scale digits |
| `varchar(n)` | Variable-length text with maximum length `n` |
| `char(n)` | Fixed-length text of exactly `n` characters |

Refer to [data-types](/ingestion/data-types) for type mappings to PostgreSQL and Spark.

## Column Naming Rules

Column names in pipeline `schemaProperties.fields` must match `[A-Za-z0-9_]+` — letters, digits, and underscores only. Spaces, parentheses, percent signs, hyphens, and other punctuation are rejected by the pipeline validator at registration time. Names are also case-insensitive for duplicate detection: `Foo` and `foo` collide.

Recommended convention: lowercase snake\_case (e.g. `customer_id`, `eps_estimate`).

### Tap auto-normalization

When a tap returns records whose keys contain disallowed characters (common with pandas DataFrames or APIs that preserve human-readable column titles), the platform automatically normalizes them in `TapScriptRunner` before they enter the pipeline.

The conversion runs in this order:

1. **Spell out semantically meaningful special characters** as `_word_` tokens (see table below)
2. **Replace any remaining non-alphanumeric character** with `_`
3. **Collapse runs of `_`** into a single `_`
4. **Trim** leading/trailing `_`
5. **Lowercase**

#### Special character spell-out table

| Character | Replacement | Example |
| - | - | - |
| `%` | `percent` | `Surprise(%)` → `surprise_percent` |
| `#` | `num` | `Order#` → `order_num` |
| `$` | `dollars` | `Price$` → `price_dollars` |
| `&` | `and` | `R&D Spending` → `r_and_d_spending` |
| `@` | `at` | `Email@Domain` → `email_at_domain` |
| `+` | `plus` | `Type A+` → `type_a_plus` |
| `=` | `equals` | `a=b` → `a_equals_b` |
| `<` | `lt` | `a<b` → `a_lt_b` |
| `>` | `gt` | `a>b` → `a_gt_b` |
| `/` | `per` | `miles/hour` → `miles_per_hour` |
| `*` | `star` | `count*` → `count_star` |
| `^` | `pow` | `x^2` → `x_pow_2` |
| `~` | `approx` | `~Population` → `approx_population` |
| `!` | `bang` | `Score!` → `score_bang` |

Other special characters (spaces, hyphens, parentheses, brackets, dots, commas, colons, quotes, etc.) are simply replaced with `_`. So `EPS Estimate` becomes `eps_estimate` and `Cost ($)` becomes `cost_dollars`.

This normalization only applies to tabular results (`dataType: csv`). JSON and XML results destined for MongoDB are passed through unchanged in the `_json` field.

See [Taps → Script Requirements](/taps#script-requirements) for the script-side perspective.

## Auto-Generating a Schema

If you have a representative CSV file, the pipeline can infer a schema automatically using AI. POST the file to the `/api/v1/pipeline/generate` endpoint:

```bash theme={null}
curl -X POST "http://localhost:8080/api/v1/pipeline/generate" \
  -F "file=@sample.csv" \
  -F "pipeline=my_pipeline" \
  -F "header=true"
```

The AI analyzes the file content (up to 100 lines for CSV, or 10,000 characters for JSON/XML) and returns a complete pipeline configuration with inferred field names and data types. You can edit the output before saving it to a pipeline configuration. If the AI model declines to analyse a CSV file, the response still lists every field, typed `string`, and carries `"aiDeclined": true`: with `header=true` the fields keep the header's names, and when `header` is false (the default) or line 1 looks like data they are numbered `column_1`, `column_2`, …; see [When the model declines](/api-reference/schema-generation-api#when-the-model-declines).

In the UI, this happens automatically in Step 1 of the pipeline creation wizard when you upload a sample file (analysis starts as soon as the file is selected).

## AI-Generated Validation Schemas

For JSON and XML pipelines, you can also generate validation schemas using AI:

* **JSON Schema (Draft 4)** — for validating JSON data against an Everit-compatible schema
* **W3C XSD** — for validating XML data against an XML Schema

In the UI (Step 5 — Data Quality), choose "Generate schema with AI", enter a schema name, and provide sample data (or load it from your uploaded file). The AI generates a compliant schema that is stored in MinIO and referenced automatically in the pipeline config.

Via API:

```bash theme={null}
curl -X POST "http://localhost:8080/api/v1/config/generate-schema" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "json-schema",
    "name": "stock_prices_schema",
    "sampleData": "{\"symbol\": \"AAPL\", \"price\": 150.25}"
  }'
```

Valid types: `json-schema` (generates Draft 4 JSON Schema) and `xsd` (generates W3C XSD). The generated schema is stored at `{environment}-config/validation-schema/{name}.json` or `{name}.xsd`.

## Schema Evolution

Datris supports **additive schema evolution** — when new columns appear in uploaded data, the pipeline schema automatically expands to include them. No manual intervention or pipeline recreation is needed.

### How it works

1. You create a pipeline with a 3-column CSV (e.g., `name`, `age`, `city`)
2. Later, you upload a CSV with a 4th column (e.g., `name`, `age`, `city`, `email`)
3. Datris detects the new `email` column, adds it to the pipeline schema as type `string`, and runs `ALTER TABLE` on the PostgreSQL destination to add the column
4. The data is processed with all 4 columns
5. Previous rows have `NULL` for the new column; new rows have the full data

Each schema change increments a `schemaVersion` counter in the pipeline config, so you can track how the schema has evolved over time.

### What triggers evolution

* **New columns in the CSV header** that are not in the stored schema → automatically added as `string` type

If the destination schema already declares the new column (for example `admit_date` as `date`), the column joins the source schema only; the destination keeps its single declared entry and type, and the run status notes that it was kept as declared. A destination schema that already lists a column twice is repaired on the next run (the first entry and its type are kept). Object store destinations read files by column position, so when such a declared column arrives, the source schema and the destination schema must list the same columns in the same order; otherwise the run fails before writing and names both lists, and you reorder the source or destination schema.

### New columns and field protection

A new column is added without [field protection](/transformation/field-protection), even when other columns of the pipeline are protected. If the pipeline sets the [HIPAA Safe Harbor preset](/transformation/field-protection#hipaa-safe-harbor-preset) (`"protection": {"preset": "hipaa-safe-harbor"}`), a new column whose name the preset recognises (for example `patient_phone`) is added with the preset's method instead and protected in the same run, before any destination sees it; the run status names the column and the method, and the stored schema keeps its `protect`. A new column the preset does not recognise is added unprotected with a warning in the run status. The preset applies to delimited (CSV) pipelines only; it is refused on JSON pipelines, whose keys do not evolve this way.

### What does NOT trigger evolution

* **Missing columns** — if a CSV is missing a non-key column from the schema, the missing values are filled with empty strings (existing behavior). Key fields must always be present.
* **Column renames** — treated as a removal + addition. The old column gets empty values, and the new column is added.
* **Type changes** — new columns default to `string`. Changing an existing column's type requires manual pipeline update.
* **JSON/XML data** — these use a single `_json` or `_xml` field, so column-level evolution does not apply.

### Example

```bash theme={null}
# Create pipeline with 3-column data
curl -X POST http://localhost:8080/api/v1/pipeline/upload \
  -F "file=@data_v1.csv" -F "pipeline=customers"
# data_v1.csv: name,age,city

# Later, upload data with a new column
curl -X POST http://localhost:8080/api/v1/pipeline/upload \
  -F "file=@data_v2.csv" -F "pipeline=customers"
# data_v2.csv: name,age,city,email
# → Schema evolves automatically, ALTER TABLE adds "email" column
```

Check the evolved schema:

```bash theme={null}
curl http://localhost:8080/api/v1/pipeline?pipeline=customers
# source.schemaProperties.schemaVersion: 2
# source.schemaProperties.fields: [name, age, city, email]
```

## Source vs Destination Schemas

A **source schema** describes the data as it arrives (columns, JSON keys, database columns). It is always required.

A **destination schema** describes the data as it should be written to the target system. It is optional. When omitted, the destination inherits the source schema unchanged.

Use a destination schema when you need to:

* Widen a type (e.g., `int` to `bigint`) for the target table.
* Drop fields that should not reach the destination.

Source and destination fields are matched by **name** (case-insensitive), not by position. A destination field that does not exist in the source is not "renamed" — it has no source value to carry, so it is written as `NULL`. To carry a value under a different column name, perform the rename in a transformation step before the destination is applied.

```json theme={null}
{
  "source": {
    "schemaProperties": {
      "fields": [
        { "name": "user_id", "type": "int" },
        { "name": "full_name", "type": "varchar(100)" }
      ]
    }
  },
  "destination": {
    "schemaProperties": {
      "fields": [
        { "name": "user_id", "type": "bigint" },
        { "name": "full_name", "type": "varchar(200)" }
      ]
    }
  }
}
```

When both schemas are present, the pipeline maps source fields to destination fields by **lowercased name**. Destination fields that have no matching source field are written as `NULL`. Use a transformation step to rename or reshape fields before the destination schema is applied.


# Source Code
Source: https://docs.datris.ai/source-code

Open-source repository and project structure

Datris is fully open source. The complete platform — server, MCP server, CLI, UI, Docker configuration, and examples — is available on GitHub.

**[github.com/datris/datris-platform-oss](https://github.com/datris/datris-platform-oss)**

## Repository Structure

| Directory | Description |
| - | - |
| `datrisserver/` | Scala/Spring Boot pipeline server — ingestion, data quality, transformation, destinations |
| `mcp-server/` | Python MCP server and CLI (`server.py`, `cli.py`) |
| `ui/` | Angular web dashboard — pipeline management, ingestion monitoring, search, secrets |
| `docker/` | Vault and MinIO init scripts, Vault config, and the Compose-time server `application.yaml` override |
| `docs/` | Mintlify documentation (what you're reading now) |
| `examples/` | Example applications — RAG chat, Kafka loader, preprocessor, topic subscriber, MCP test, Airflow DAGs, market/macro agent |
| `test-data/` | Test data files and scripts |

## Key Files

| File | Description |
| - | - |
| `docker-compose.yml` | Full platform stack — all services in one command |
| `build.sbt` | Scala build configuration for the pipeline server |
| `mcp-server/pyproject.toml` | Python package config for the MCP server and CLI |
| `server.json` | MCP Registry server definition |
| `.env.example` | Template for environment variables (API keys, model config) |

## Getting the Code

```bash theme={null}
git clone https://github.com/datris/datris-platform-oss.git
cd datris-platform-oss
```

## Building from Source

### Pipeline Server (Scala)

```bash theme={null}
sbt clean assembly
```

Produces a fat JAR in `datrisserver/target/scala-2.12/`.

### Docker Images

```bash theme={null}
docker compose build --no-cache
docker compose up -d
```

### MCP Server and CLI

The MCP server and CLI are published on [PyPI](https://pypi.org/project/datris-mcp-server/) and installable via Homebrew:

```bash theme={null}
brew tap datris/tap
brew install datris
```

## Contributing

Fork the repository, make your changes, and submit a pull request. All contributions are welcome.

## License

Datris is licensed under [AGPL-3.0](https://github.com/datris/datris-platform-oss/blob/main/LICENSE).


# Tap Execution & Isolation
Source: https://docs.datris.ai/tap-execution-isolation

How tap code runs, and how Datris isolates it from platform internals

A tap's `fetch()` is Python that Datris executes on your server. By default it runs
in-process inside the server. For stronger separation, Datris can instead run each tap in a
dedicated, isolated **runner** container, cut off from the platform's credentials and internal
services.

<Note>
  On **docker compose / prod**, the isolated runner is **on by default** (`USE_TAP_RUNNER=true`).
  `sbt` / IDE runs (no sidecar) stay in-process. Tap behavior is identical either way — the same
  `fetch()` contract, the same environment variables, the same packages. The only difference is
  *where* the code runs and what it can reach. `scripts/install.sh` and `docker/vault-init.sh` mint `TAP_RUNNER_TOKEN`;
  the server refuses to start with isolation on and a missing or `changeme` token.
</Note>

## Execution modes

| | In-process (`sbt` / IDE, or `USE_TAP_RUNNER=false`) | Isolated runner (compose/prod default) |
| - | - | - |
| Where tap code runs | Inside the datris server container | A separate `datris-tap-runner` container |
| Platform secrets reachable | Limited (see [always-on protections](#always-on-protections)) | **No** |
| Internal services (database, object store, secrets store) reachable | Yes | **No** |
| Internet + platform API reachable | Yes | Yes |
| Extra packages | Installed in an isolated, throwaway environment | Installed in an isolated, throwaway environment |

**In-process** mode runs tap code inside the server. Tap code in this mode
shares the server's filesystem and network, so it can reach internal services. This is
the `sbt` / IDE path (no sidecar). Compose/prod should keep the isolated runner on.

In **isolated runner** mode (`USE_TAP_RUNNER=true`), tap code runs in a container that holds
no platform credentials and sits on a network with no route to the platform's internal
services. Even a misbehaving tap has nothing to read and nowhere to pivot — it can fetch from
its source, install the packages it declares, call back into the Datris API, and return
records. That is all. Enable it when you want defense-in-depth beyond the
[always-on protections](#always-on-protections).

## The runner container

The isolated runner is a dedicated `datris-tap-runner` sidecar — a small, standalone container
built from its **own minimal Python image** (not the server image). It is hardened to hold and
grant as little as possible:

* **No platform secrets.** The image carries no secrets-store credentials, no `.env` file, and
  no AI/database credentials — there is nothing sensitive inside it to read.
* **Network-isolated.** It runs on a dedicated network with **no route** to the platform's
  internal services (database, object store, secrets store). It can reach the public internet
  and the Datris API — nothing else internal.
* **Non-root, read-only filesystem.** Tap code runs as an unprivileged user on a read-only root
  filesystem, with all Linux capabilities dropped and privilege escalation disabled.
* **Ephemeral scratch.** The only writable area is a small in-memory scratch space (**512 MB**
  by default, a `tmpfs` mount, so it comes out of RAM rather than disk) that is wiped after every
  run, so nothing a tap writes — temp files, installed packages — survives into the next run. The
  per-run venv and any pip-installed packages live there too, so a script must **stream the
  source, never download it**: a `fetch()` that saves a large file to disk (or lets a library
  spool one under `TMPDIR`) fails with `No space left on device`. Read remote files directly over
  HTTP in chunks or row groups and `yield` records as they arrive. The 512 MB figure is set on the
  runner's `tmpfs` entry in the compose file; raise it only on hosts with the RAM to spare.

This is **operating-system container isolation** — separate process, filesystem, and network
namespaces sharing the host kernel. It is the right boundary for a self-hosted, single-operator
deployment. It is *not* a virtual-machine boundary: a kernel-level container escape would cross
it, so running fully untrusted, multi-tenant tap code calls for stronger per-run isolation — a
user-space kernel such as [gVisor](https://gvisor.dev) (`runsc`), or a microVM (Firecracker /
Kata).

Datris does not ship or configure these. But because the runner is a standard OCI container,
you can run it under a hardened runtime yourself if your host provides one — for example, with
gVisor installed, run the `datris-tap-runner` container under `runsc` (Docker's
`--runtime=runsc`, or `runtime: runsc` on the service). Datris hasn't validated this path, so
treat it as an advanced, self-managed hardening step.

## Always-on protections

These apply in **both** modes, regardless of the runner setting:

* **Platform secrets are never placed in a tap's environment.** The credentials the platform
  uses for itself (secrets-store token, AI provider keys, database passwords) are stripped
  from the tap's environment. A tap receives only Datris-injected variables (`DATRIS_*`), any
  per-run parameters, and **its own configured secret** — nothing else.
* **Extra packages install in isolation.** Packages a tap declares are installed into a
  throwaway virtual environment, never into the system Python. The pre-installed common
  packages remain available; the per-tap environment is discarded after the run.
* **Private-network egress is blocked.** HTTP taps and REST endpoint destinations refuse to
  connect to loopback, link-local, private, or cloud-metadata addresses, so a tap cannot be
  pointed at internal services. If your endpoints legitimately live on an internal network,
  set `DATRIS_ALLOW_PRIVATE_EGRESS=true` in `.env` (see the
  [Configuration Reference](/configuration-reference#container-environment-variables)).

### Overload protection

When taps and generated scripts run in the runner containers (the compose default), a burst of
scheduled taps or AI rules cannot take the Docker host down:

* **Bounded concurrency.** Each runner runs at most `TAP_RUNNER_MAX_CONCURRENT` scripts at once
  (default 4; `CODEGEN_RUNNER_MAX_CONCURRENT` for `datris-codegen-runner`). A run that arrives when
  every slot is taken is answered "busy" at once, and nothing is started for it.
* **The server waits its turn.** A busy answer is retried with a short backoff (never more than
  30 seconds between attempts) for up to `TAP_RUNNER_QUEUE_TIMEOUT_SECONDS` (default 600). A tap
  run that waited shows the time in its run history, for example `queued 12.3s` next to the
  duration (the duration includes the wait). A run that still has no slot after the timeout fails
  with one sentence naming `TAP_RUNNER_MAX_CONCURRENT`: raise it, or stagger tap schedules.
* **Container ceilings.** `datris-tap-runner` is limited to 1 CPU, 2 GB of memory (its 512 MB
  `/tmp` scratch counts toward it) and 256 processes; `datris-codegen-runner` to 1 CPU, 1 GB and
  128 processes (threads count toward the process ceiling). These are ceilings, not
  reservations: an idle runner uses no more than before.
* **Per-script limits.** Each script gets a 1024 MB address-space limit
  (`TAP_RUNNER_SCRIPT_MEMORY_MB`, `0` = none). A script that allocates past the limit fails with
  `MemoryError` instead of pushing the runner out of memory. The address-space limit is not
  enforced on macOS when you run the runner outside Docker.
* **Native thread pools sized to the runner.** Libraries such as numpy (OpenBLAS), OpenMP and MKL
  start one thread per CPU of the *host*, whatever the container's CPU limit. Each script gets
  `OPENBLAS_NUM_THREADS`, `OMP_NUM_THREADS` and `MKL_NUM_THREADS` set to
  `TAP_RUNNER_SCRIPT_THREADS` (default 1), so several pandas taps at once stay well inside the
  process ceiling. A tap can set its own value through its environment.

[`datris doctor`](/doctor) reports the concurrency setting and slot use as `runner.concurrency`,
and warns when runs waited for a slot in the last 24 hours (error when a run was turned away).

## Configuration

Set these on the `datris` service (for example in your `.env`). Defaults work out of the box.

| Variable | Default | Purpose |
| - | - | - |
| `USE_TAP_RUNNER` | `true` (compose) / unset=`false` (sbt) | Isolated runner. Set `false` to run in-process. |
| `TAP_RUNNER_URL` | `http://datris-tap-runner:8090` | Address of the runner service. |
| `TAP_RUNNER_TOKEN` | minted on first boot | Shared token authenticating the server to runner call. install.sh / vault-init mint it; changeme or empty fails fast when isolation is on. |
| `TAP_RUNNER_CALLBACK_HOST` | `datris` | Host a tap uses to call back into the platform API from the runner. The datris service name on the shared network. |
| `TAP_RUNNER_QUEUE_TIMEOUT_SECONDS` | `600` | How long a tap run or generated script waits for a free runner slot when the runner is busy, before failing with a message naming `TAP_RUNNER_MAX_CONCURRENT`. |

The runner containers take these (set them in `.env`; compose passes them to the right service):

| Variable | Default | Purpose |
| - | - | - |
| `TAP_RUNNER_MAX_CONCURRENT` | `4` | Most tap scripts `datris-tap-runner` runs at once. Further runs get "busy" and wait. |
| `TAP_RUNNER_SCRIPT_MEMORY_MB` | `1024` | Address-space limit per tap script, in MB. `0` = no limit. |
| `TAP_RUNNER_CPUS` / `TAP_RUNNER_MEMORY` / `TAP_RUNNER_PIDS` | `1` / `2g` / `256` | CPU, memory and process ceilings of the `datris-tap-runner` container. Raise the memory with `TAP_RUNNER_MAX_CONCURRENT`; set `TAP_RUNNER_CPUS=2` (or more) on hosts with the cores. Docker refuses a CPU value above the host's CPU count. |
| `CODEGEN_RUNNER_MAX_CONCURRENT` | `4` | Most generated scripts `datris-codegen-runner` runs at once. |
| `CODEGEN_RUNNER_SCRIPT_MEMORY_MB` | `1024` | Address-space limit per generated script, in MB. `0` = no limit. |
| `CODEGEN_RUNNER_CPUS` / `CODEGEN_RUNNER_MEMORY` / `CODEGEN_RUNNER_PIDS` | `1` / `1g` / `128` | CPU, memory and process ceilings of the `datris-codegen-runner` container. |
| `TAP_RUNNER_SCRIPT_THREADS` | `1` | `OPENBLAS_NUM_THREADS` / `OMP_NUM_THREADS` / `MKL_NUM_THREADS` given to each script unless it sets its own. `0` = leave them unset (image default; set it on the runner service to change it). |
| `TAP_RUNNER_SCRIPT_MAX_PROCS` | `0` (off) | Optional `RLIMIT_NPROC` per script (image default). Off by default because the kernel counts it per user across both runners and every thread, not per script; the container process ceiling bounds runaway forks instead. |
| `TAP_RUNNER_RETRY_AFTER_SEC` | `5` | Seconds a busy runner tells the server to wait before trying again (image default). |

The runner is defined as the `datris-tap-runner` service in `docker-compose.yml`; a normal
`docker compose up` starts it alongside the server and routes taps there unless you set
`USE_TAP_RUNNER=false`. To force in-process (for example a no-sidecar `sbt` run):

```bash theme={null}
USE_TAP_RUNNER=false sbt -Dconfig.file=... run
```

## What changes for tap authors

Almost nothing. Taps fetch data and **return records**; the platform writes them to the
destination pipeline. Reading a source over the internet, installing declared packages, using
the tap's own secret, and calling the Datris API via `DATRIS_PLATFORM_HOST` / `DATRIS_PLATFORM_PORT`
all work the same in both modes.

The one thing to know: in **isolated runner** mode a tap cannot open a *direct* connection to
the platform's internal database / object store by hostname — those live on a network the
runner can't reach. This is intentional, and it's not how taps are meant to work anyway: a tap
returns records and lets the pipeline handle the destination. If you have a legacy tap that
connected directly to an internal service, run it in-process or rework it to return records.

## Reading platform data from a tap

A tap can read data already in Datris — for example a `stock_tickers` table — to drive its
fetch. Do this through the **query API**, using the injected platform host and database name.
The same code works in both execution modes, because `DATRIS_PLATFORM_HOST` is set correctly
for each:

```python theme={null}
import os, json, urllib.request

def fetch():
    host = os.environ["DATRIS_PLATFORM_HOST"]   # "localhost" in-process, "datris" in the runner
    port = os.environ["DATRIS_PLATFORM_PORT"]
    db   = os.environ["DATRIS_POSTGRES_DATABASE"]
    req = urllib.request.Request(
        f"http://{host}:{port}/api/v1/query/postgres",
        data=json.dumps({"sql": "SELECT ticker FROM stock_tickers", "database": db}).encode(),
        headers={"Content-Type": "application/json"},
    )
    tickers = json.loads(urllib.request.urlopen(req).read())["results"]
    # ...use tickers to drive the rest of fetch()...
    return tickers
```

Note there is no credential in that call. When the deployment requires API keys, the platform mints a per-run token (`DATRIS_PLATFORM_TOKEN`) and the tap wrapper attaches it to `requests` / `urllib` calls aimed at `DATRIS_PLATFORM_HOST` automatically — the script above works unchanged in both modes and with keys on or off. The token is read-only and scoped to the run; the audit log records the callback as the tap.

Use `/api/v1/query/mongodb` (or `/api/v1/query/natural`) the same way for Mongo-backed data.
In the **isolated runner**, a tap **cannot** instead open a direct database connection (e.g. to
`postgres:5432`) — internal services aren't reachable from the runner's network. Querying
through the API is the supported path and works regardless of mode.

## When to use which

* **Isolated runner (compose/prod default)** — tap code stays separated from platform credentials and internal services. Isolated taps cannot open a *direct* connection to internal DB / MinIO / Vault; they return records and the pipeline writes destinations.
* **In-process (`sbt` / IDE, or `USE_TAP_RUNNER=false`)** — tap `fetch()` runs inside the server JVM. Fine for local development without the sidecar, or a legacy tap that still opens a direct internal connection. The server logs a loud warning. Do not treat a laptop compose install as in-process — compose isolates by default.

## Generated data-quality and transformation scripts

[AI data-quality rules](/data-quality/ai-rules) and [AI transformations](/transformation/ai-transformation)
work by having a model write a short Python script and running it against the pipeline's data.
That script is shaped by the data it reads, so Datris treats it like tap code and runs it in an
isolated runner too.

On a compose install the scripts run in **`datris-codegen-runner`**, a sibling of the tap runner
built from the same `datrisai/datris-tap-runner` image, with the same token and the same hardening
(non-root, read-only root filesystem, all capabilities dropped, no privilege escalation). What
differs is its network: it sits only on `codegen-net`, a compose network declared
`internal: true` that only the `datris` server also joins.

| | In-process (`sbt` / IDE, or `USE_CODEGEN_RUNNER=false`) | `datris-codegen-runner` (compose default) |
| - | - | - |
| Platform secrets in the environment | No (scrubbed) | No (none in the container) |
| Server filesystem (config, secrets-store token, staged files) | Readable | **Not present** |
| Internal services (database, object store, secrets store) | Reachable | **No route** |
| Internet | Reachable | **No route** |
| Datris API port | Reachable | **Refused** (403 from the server, see below) |

**How data moves.** The server pushes the script and the pipeline's staged input file to the
runner; the script gets no token and has nothing to call. The runner writes the input to a
per-run directory, runs `python3 script.py <input> [<output>]`, and sends back the result and,
for a transformation, the output file. Both directions are copied in 64 KB chunks: neither side
holds the file in memory. The server log shows the transfer and run time of each script
(`CodeGen runner: sent ... in ... ms, script ran in ... ms, received ... in ... ms`).

**Scratch space.** Per-run files live on the `codegen-scratch` named volume, mounted at `/scratch`
in the runner. It is on disk, not in memory, so there is no new size limit: the bound is free disk
on the Docker host, as it is for in-process runs. A transformation needs room for its input and
its output at once, in addition to the server's own staged copy. Each run's directory is removed
when the run ends (success, script failure, timeout or a dropped connection). Anything else a
script wrote, on the volume outside its own directory or in the runner's small `/tmp`, is removed
at the same time, and everything left by a killed runner is removed when it starts, so nothing
carries from one run to the next. Runs that overlap in time share the volume, as concurrent tap
runs share theirs. Processes a script starts are stopped when the script ends (one that detaches into a new session is not). The runner image
treats its scratch directory as dedicated (`CODEGEN_SCRATCH_DEDICATED=1`) and empties all of it,
so if you run the image outside compose, point `CODEGEN_SCRATCH_DIR` only at an otherwise unused
directory (see the [Configuration Reference](/configuration-reference#container-environment-variables)). When the
disk fills, the rule or transformation fails with an error naming the `codegen-scratch` volume.
The script's printed output (for an AI rule, its failure list) comes back in one result message
capped at 64 MB; a rule that prints a failure for nearly every row of a very large file fails with
"result line longer than ..." rather than returning the list.

**When the runner is absent.**

* `USE_CODEGEN_RUNNER=true` (the compose default): the runner is **required**. If it is stopped,
  unreachable, an older image without the `/execute-file` endpoint, or rejects the token, the rule
  or transformation fails with a message naming the runner. Datris never falls back to running the
  script in-process.
* `USE_CODEGEN_RUNNER` unset or `false` (`sbt` / IDE, or an install still on an older compose
  file): scripts run in-process, with only the scrubbed environment, as before. The server logs a
  warning at startup and on every run, and `datris doctor` reports `codegen.isolation` as `warn`.

`datris doctor` shows which mode is in effect: `ok` when scripts run in the runner and it answers,
`warn` when they run in-process, `error` when the runner is required but cannot run a script.

| Variable | Default | Purpose |
| - | - | - |
| `USE_CODEGEN_RUNNER` | `true` (compose) / unset=`false` (sbt) | Run generated scripts in `datris-codegen-runner`. Set `false` to run them in-process. |
| `CODEGEN_RUNNER_URL` | `http://datris-codegen-runner:8090` | Address of the runner service. |
| `CODEGEN_RUNNER_API_BLOCK` | `true` | The server refuses requests from the runner (see below). Set `false` only if the guard refuses traffic it should not. |

The runner shares `TAP_RUNNER_TOKEN` with the tap runner; nothing new to configure.

**What changes for scripts.** Generated scripts are already told to use only the standard library
and read the one file they are given. A script that opened a network connection or read a server
path used to work in-process; in the runner it fails. That is intended.

**The Datris API port.** The server listens on every network it joins, so `datris:8080` is
reachable over `codegen-net`, but the runner never needs to call the server (the server pushes to
it). With `USE_CODEGEN_RUNNER=true` the server refuses, with `403` and before any authentication,
every request that arrives on its own `codegen-net` interface and every request whose source is
the runner's address. This covers every path on the port, including `/actuator`. At startup the
server resolves the host in `CODEGEN_RUNNER_URL`, finds its own interface in the same subnet, and
logs the decision (`CodeGen runner API block: ...`). It re-resolves the runner every 30 seconds,
so a runner that starts after the server or restarts with a new address is covered.

The guard is built not to refuse anyone else. It does not refuse traffic arriving on the runner's
network when that network is the server's only one, or when another service (UI, MCP server, tap
runner, Vault, databases, MinIO) is on it. In those cases it refuses only the runner's own address
and logs a warning. While the runner's name does not resolve, nothing is refused and the server
retries every 2 seconds. A runner that resolves to the server itself or to loopback turns the guard
off, with a warning. The guard relies on `codegen-net` staying `internal: true`: without it the
server can send and receive host traffic through that network, and every request from the Docker
host would get `403`. If you must remove it, set `CODEGEN_RUNNER_API_BLOCK=false`. Outside compose (for example a Kubernetes Service address) the runner's source
address may not match, so use a network policy there.

<Warning>
  `CODEGEN_RUNNER_API_BLOCK=false` turns the guard off, and the runner can call the Datris API again.
  Use it only if the guard refuses traffic it should not, and keep `USE_API_KEYS` on so the port
  requires a key the script does not have.
</Warning>

To check the boundary on a running stack, run `python3 scripts/check-codegen-isolation.py` from a
checkout: it sends a probe script to the runner and exits non-zero if the script can reach the
internet, Vault, Postgres, MongoDB or MinIO, call the Datris API (anything but the server's
`403`), read a server path, write outside its scratch directory, or see a secret-bearing variable.


# GitHub Script Storage
Source: https://docs.datris.ai/tap-github-storage

Store tap Python scripts in a GitHub repository — real commits, PR review, and editing from your own IDE

By default, tap scripts live in Datris's built-in object storage. You can instead connect a
GitHub repository per environment: every script save becomes a commit, runs are pinned to the
exact commit they were saved at, and your team can review, blame, and edit tap code with normal
git tooling.

<Note>
  Repository storage is **opt-in** and coexists with built-in storage. Existing taps stay where
  they are until you move them; you can run a mix of both indefinitely, and move any tap back to
  built-in storage at any time.
</Note>

## What you get

* **A commit per save** — from the tap wizard, the Assistant, or the API. Commit messages are
  templated (e.g. `tap(weather-forecast): update via Datris`).
* **Runs pinned to a commit** — a scheduled run executes exactly the version that was saved,
  served from a local cache, so cron runs make no GitHub API calls and keep working if GitHub
  is unreachable.
* **External editing** — commit changes from your IDE; Datris detects the drift when the tap is
  opened and offers to load the latest version.
* **Conflict protection** — if the script changed in the repository after you opened it, saving
  is rejected with a clear message instead of silently overwriting.

## Setup

### 1. Create (or pick) a repository

Any repository works — private is typical. Scripts are written to
`<path prefix><tap-name>.py` (default `taps/<tap-name>.py`) on the branch you configure.

### 2. Create a fine-grained personal access token

A PAT is created under a GitHub **user account** and scoped to repositories — you don't add it
to the repository itself. Datris needs **Contents: Read and write** on just the one repo:

1. On GitHub, click your avatar → **Settings** → **Developer settings** (bottom of the left
   sidebar) → **Personal access tokens** → **Fine-grained tokens** → **Generate new token**.
2. Set a name and an expiration.
3. **Resource owner**: pick the user or organization that owns the target repository.
4. **Repository access**: choose *Only select repositories* and pick the repo.
5. Under **Permissions → Repository permissions**, set **Contents: Read and write**
   (**Metadata: Read** is added automatically as a mandatory permission).
6. Generate the token and copy it immediately — GitHub shows it only once.

<Warning>
  **For organization-owned repositories**: the org must allow fine-grained PATs under org
  **Settings → Third-party Access → Personal access tokens**, and some orgs require each token to
  be approved. If the connection test fails with an access error on an org repo, this is the
  first thing to check.
</Warning>

<Note>
  Fine-grained PATs **always expire**. When the token lapses, saves and non-cached reads for
  repository-backed taps fail until you update the secret with a fresh token. Put the expiry
  date somewhere you'll see it.
</Note>

### 3. Connect it in Datris

In **Configuration → Code Repository**:

1. Enter the repository (`owner/repo`), branch, and path prefix.
2. Under **Token secret**, click **Create one**, name it (e.g. `github`), and paste the token.
   This stores it in the secrets store with the shape Datris expects — a `token` field, tagged
   `_type=repo_token`.
3. Click **Test connection** — it verifies the token can reach the repository, has push
   access, and that the branch exists.
4. Check **Enable repository storage** and **Save**. Saving re-validates the connection.

Once enabled, **new** tap scripts default to the repository — including taps the Assistant
builds. Existing taps are never moved automatically.

<Tip>
  Creating the token secret manually in the Secrets tab also works, but two details matter: the
  field holding the PAT must be named `token`, and the secret must be tagged `_type=repo_token`
  or it won't appear in the Token secret dropdown. The **Create one** flow handles both.
</Tip>

### GitHub Enterprise Server

Set **API base URL** to your instance's API endpoint (e.g. `https://ghes.example.com/api/v3`).
Leave it at `https://api.github.com` for github.com.

## Day-to-day behavior

| Action | What happens in the repository |
| - | - |
| Save a script (wizard, Assistant, or API) | A commit on the configured branch; the tap pins to that commit |
| Run a tap (manual or cron) | Nothing — the pinned commit is read from a local cache |
| Edit the file directly in GitHub | Opening the tap loads the repository's latest version into the editor, with a banner explaining that runs keep the previously saved version until you test and save |
| Save when the file changed externally | Rejected with a conflict message — pull the latest, reapply your edits, save again |
| Delete the tap | A delete commit removes the file from the branch head; **git history is preserved** |
| Save a pipeline with an AI rule or AI transformation | A commit per script under `pipelines/<pipeline>/`; runs pin to it (see [Pipeline scripts](#pipeline-scripts)) |

Run history records the commit SHA each run executed, so any run can be traced to the exact
script version in your repository.

## Moving taps between backends

In the **Taps** list, each row shows a **GitHub** badge when its script is repository-backed.
The cloud action on the row moves a tap's script in either direction:

* **Move to GitHub** — commits the current script to the repository and re-points the tap.
* **Move to built-in storage** — copies the script back to object storage. The repository file
  is left in place (remove it manually if you want it gone).

Both directions are recorded as a new tap version, so the move itself is auditable and
reversible.

## Pipeline scripts

The Python scripts behind a pipeline's [AI data-quality rule](/data-quality/ai-rules) and
[AI transformation](/transformation/ai-transformation) use the same repository. With repository
storage enabled, the script generated for a new pipeline (or for a pipeline that has no script
yet) is committed instead of going to built-in storage.

**Path layout.** Each pipeline gets a folder under the configured path prefix, on the configured
branch. With the default prefix `taps/`:

```
taps/pipelines/<pipeline>/data-quality.py
taps/pipelines/<pipeline>/transformation.py
```

Tap files (`taps/<tap>.py`) are untouched, and the token needs no new permission.

**Commit messages.** `{name}` is `<pipeline>/<kind>` (`orders/dataQuality`,
`orders/transformation`), `{action}` is `create`, `update` or `delete`, and `{user}` is the label
of the API key that saved the pipeline, or `datris` when there is none (the script was generated during a run, or saved from a UI session without an API key).
With the default template (or an empty one) pipeline scripts use
`pipeline({name}): {action} via Datris`, for example `pipeline(orders/transformation): update via Datris`.
A custom template is applied as written, so a template written for taps (for example one starting
`tap(`) also prefixes pipeline commits that way.

**Pinned runs.** The pipeline records the commit SHA of each script. Runs read that commit from
the local cache, so a run makes no repository call once the commit is cached, and the run's
status line names the short SHA ("Running stored CodeGen script generated at … (code repository
commit abc1234)"). `GET /api/v1/pipelines/{name}/codegen-scripts` returns `storage: "github"`,
`repoPath` and `commitSha` for each script.

**Editing in the repository.** Runs stay on the recorded commit after an edit in GitHub. The
read endpoint then reports `drift: true` with the branch-head `headSha`, and for the transformation script the Columns section
of the [lineage panel](/provenance#column-level-lineage) shows "Changed in the repository" with a
**Load latest** action. To adopt the edit, call
`POST /api/v1/pipelines/{name}/codegen-scripts/{kind}/pull` (needs `pipeline:create`); it records
the head commit and makes no commit of its own. Later saves keep the adopted edit while the
instruction and source schema are unchanged.

**Conflicts.** When the instruction or schema changes, the save regenerates the script and commits
it on top of the recorded commit. If the file was edited in the repository since then, the commit
is rejected: the pipeline is saved, the script and its recorded commit are unchanged, and the save
response carries a warning naming the two ways forward: pull the edit, or replace it with
`POST /api/v1/pipelines/{name}/codegen-scripts/{kind}/regenerate?overwrite=true`. A forced regenerate
without `overwrite=true` is rejected the same way (409). A hand edit is never overwritten silently;
an overwritten edit stays in the repository history.

Until the conflict is resolved, runs keep executing the recorded commit, without the new instruction
or schema and with no model call. The run's status line names the commit and says to pull or
regenerate with `overwrite=true`, and the read endpoint shows `conflict: true` with the
`conflictReason`. Saving the same configuration again makes no model call and repeats the warning.
Pull or a regenerate with `overwrite=true` clears the conflict. A run that meets the same conflict
(for example after a version restore) records it the same way and runs the recorded commit.

**Moving a script.** An existing script keeps its storage on every save. To move one, call
regenerate with `storage=github` or `storage=builtin`; the old copy is left in place. Scripts that
existed before repository storage was enabled stay in built-in storage until you move them.

Leftover copies are not tracked. Deleting the pipeline removes only the copies its scripts point at
now, so files and objects left behind by earlier moves stay until you remove them. Moving a script
back to the repository writes over a leftover file at the same path without a conflict check, and
the commit is labelled `create`.

**When the repository is unreachable or the token has expired:**

* At save, the pipeline is saved and the script is pending with the reason; the save response
  carries a warning. The first run after the repository is back generates and commits it.
* A run whose recorded commit is cached is unaffected.
* A run whose commit is not cached (the cache does not survive an image rebuild) fails with
  "Code repository is unreachable and no cached copy … exists locally". It does not fall back to
  generating a different script.
* A run that generates a pending script and cannot commit it runs the generated script, logs a
  warning, and the script stays pending.

If repository storage is disabled while scripts point at it, their runs fail with "This pipeline's
CodeGen script is stored in a code repository, but no enabled repository is configured"; re-enable
the repository, or regenerate with `storage=builtin` to move the script back to built-in storage.

**Delete.** Deleting the pipeline makes a delete commit per script file; git history is kept. A
repository failure is logged and does not block the delete.

## Troubleshooting

| Symptom | Likely cause |
| - | - |
| Token secret dropdown is empty | The secret isn't tagged `_type=repo_token` — recreate it via **Create one** |
| Test connection: token rejected | Token expired or revoked — generate a new PAT and update the secret |
| Test connection: cannot push | Token is missing **Contents: Read and write** on this repository |
| Test connection: access error on an org repo | Org policy blocks fine-grained PATs, the token awaits approval, or SSO authorization is required |
| New taps still land in built-in storage | **Enable repository storage** isn't checked and saved |
| Scheduled runs keep working while GitHub is down | Expected — runs read the pinned commit from the local cache |
| Pipeline save warns that a script "changed in the code repository" | The script file was edited in GitHub — pull the edit, or regenerate with `overwrite=true` (see [Pipeline scripts](#pipeline-scripts)) |


# HTTP Tap Contract
Source: https://docs.datris.ai/tap-http-contract

Implement a tap as an HTTP endpoint you host — in any language

An **HTTP tap** is a tap whose fetch logic runs as a service *you* operate instead of a Python
script the platform executes. On every run — manual, scheduled, or test — Datris POSTs the run
context to your endpoint, and your endpoint responds with the same JSON envelope a Python tap
script produces. Everything downstream is identical: envelope parsing, data types, incremental
state, output caps, run history, retries, and scheduling.

Use an HTTP tap when you want to write the tap in Rust, Go, TypeScript, or anything else — or
when the fetch logic already exists inside a service you run. Your endpoint holds its own
upstream credentials; Datris only ever sends it one optional auth token. No tap code runs on
the platform at all.

<Note>
  HTTP taps cannot use the [platform-data callback](/taps#querying-datris-data) — the
  `DATRIS_PLATFORM_*` query API is reachable only from scripts running on the platform. If your
  tap's fetch logic depends on data already stored in Datris, keep it as a Python tap.
</Note>

## The request

On each run, Datris sends:

```http theme={null}
POST {endpointUrl}
Content-Type: application/json
Authorization: Bearer <endpoint_token>     # only when the tap has a secret (see Auth)
X-Datris-Tap: my-tap
User-Agent: datris-tap/1
```

```json theme={null}
{
  "tap": "my-tap",
  "params": {"start_date": "2026-08-01"},
  "state": {"cursor": "abc123"},
  "testLimit": 100
}
```

| Field | Type | Meaning |
| - | - | - |
| `tap` | string | The tap's name — useful when one service implements several taps. |
| `params` | object | Per-run parameters from `run_tap(params={...})`. `{}` on scheduled runs, so apply sensible defaults. Keys match `[A-Za-z_][A-Za-z0-9_]*`; values are strings. |
| `state` | object or null | The state your endpoint returned on the last **successful** real run; `null` on the first run. |
| `testLimit` | number or null | Usually `null`. A positive integer when the caller requests a capped sample (e.g. an API test with a limit) — cap the records you fetch and return when present. Test runs never persist regardless. |

The request times out after the platform's tap timeout for the mode it was called in, the same
knobs as script taps: a **test** gets `TAP_SCRIPT_TIMEOUT_SECONDS` (default 300 seconds), a real or
scheduled **run** gets `TAP_RUN_TIMEOUT_SECONDS` (default 3600 seconds). Answer well within it —
see [Long fetches](#long-fetches-chunk-with-state).

## The response

Reply `200 OK` with the tap envelope:

```json theme={null}
{
  "type": "json",
  "data": [{"id": 1, "price": 42.5}, {"id": 2, "price": 43.1}],
  "state": {"cursor": "def456"},
  "logs": "fetched 2 records from source in 0.4s"
}
```

| Field | Required | Meaning |
| - | - | - |
| `type` | **yes** | One of `json`, `csv`, `xml`, `text`, `document`. There is no type-sniffing on this path — you declare it. A missing or unknown `type` fails the run. |
| `data` | **yes** | The records. For `json`/`csv`/`document`: an array (an **empty array is the correct way to report "no new data"**). For `xml`/`text`: a string. |
| `state` | no | JSON object bookmark for incremental sync, max 64 KB. Committed only after a successful real run; test runs never commit. Omit it to leave the previous bookmark in place. |
| `logs` | no | Free-text log string, shown in the tap's run history — your stderr equivalent. |

Anything else — a non-200 status, a timeout, a malformed envelope — records a failed run, with
your response body (truncated) as the error message. Failed scheduled runs enter the platform's
normal cron retry ladder.

`document`-type records use the same shape as document taps:
`{"uri": "...", "filename": "...", "content": "<base64>", "content_hash": "...", "metadata": {...}}`.

The response is streamed to the platform's staging disk and capped at the per-run payload disk budget (`PIPELINE_MAX_PAYLOAD_MB`, default 4096 MB). Bigger
fetches should be chunked — see below.

## Auth

If the tap references a secret, the secret's **`endpoint_token`** field is sent as
`Authorization: Bearer <value>`. That is the **only** secret field ever forwarded — upstream
source credentials (API keys for the systems your service fetches from) belong to your service's
own configuration, not to Datris. A tap that names a secret with no usable `endpoint_token`
fails loudly before the call.

Use `https://` for any non-local endpoint — the token travels in a header.

## Long fetches: chunk with state

Multi-minute HTTP requests are fragile through proxies and load balancers. Don't answer one run
with a giant slow response — return one page quickly plus a `state` cursor, and let the next
run continue where you left off:

1. Run 1: request has `"state": null` → fetch the first page, respond with
   `"data": [...page 1...], "state": {"offset": 1000}`.
2. Run 2: request has `"state": {"offset": 1000}` → fetch the next page, respond with
   `"data": [...page 2...], "state": {"offset": 2000}`.
3. Caught up: respond `"data": [], "state": {"offset": 2000}` — a clean `no_records` run that
   keeps the bookmark.

Because state commits only on success, a failed run leaves the old cursor and the retry
re-fetches the same window — design your pages to be safe to re-deliver (upsert destinations
absorb the overlap).

## A minimal endpoint in Rust

Any HTTP stack works; here is the whole contract in one axum handler:

```rust theme={null}
use axum::{routing::post, Json, Router};
use serde_json::{json, Value};

async fn tap(Json(req): Json<Value>) -> Json<Value> {
    let offset = req["state"]["offset"].as_u64().unwrap_or(0);
    let limit = req["testLimit"].as_u64().unwrap_or(1000);

    // Fetch from your source here — this service holds its own API keys.
    let records: Vec<Value> = (offset..offset + limit.min(1000))
        .map(|i| json!({"id": i, "value": format!("row-{i}")}))
        .collect();

    Json(json!({
        "type": "json",
        "data": records,
        "state": {"offset": offset + records.len() as u64},
        "logs": format!("returned {} records from offset {}", records.len(), offset)
    }))
}

#[tokio::main]
async fn main() {
    let app = Router::new().route("/tap", post(tap));
    let listener = tokio::net::TcpListener::bind("0.0.0.0:8000").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}
```

The same shape ports directly to Go (`net/http` + `encoding/json`), Node/TypeScript
(`express`/`fastify`), or a serverless function — read the JSON body, write the envelope.

## Creating an HTTP tap

* **UI**: Catalog → Create Tap → choose **HTTP Endpoint**, paste the URL, optionally attach a
  secret whose `endpoint_token` your service checks. Test Script POSTs to your endpoint (with
  `testLimit: null` — the wizard does not cap HTTP-tap samples) and previews the response without persisting.

<Note>
  The request comes from **inside the Datris container**. For an endpoint running on the same
  machine as Docker, use `http://host.docker.internal:<port>/...` — `localhost` would point at
  the container itself and the connection will be refused.
</Note>

* **CLI**: `datris tap create --name my-tap --kind http --endpoint-url https://taps.example.com/my-tap --pipeline my-pipeline --cron "0 0 6 * * ?"`
* **MCP**: `create_tap` with `kind: "http"` and `endpoint_url`.

Params, scheduling, run history, `run_tap`, `get_pipeline_status` polling, and the sync-state
viewer all work exactly as they do for Python taps. What doesn't apply: AI script
generation/fix/review/optimize (there is no script), pip packages, script storage, and the
platform-data callback.


# Tap Prompt Fragments
Source: https://docs.datris.ai/tap-prompts

Inject source-specific context into the LLM when you create a tap

<Note>
  **Beta.** Prompt fragments are a new feature and the API may evolve based on user feedback. The storage model and injection points are stable. There is no dedicated UI for managing fragments today — they're managed via the REST API (see [Managing fragments](#managing-fragments)).
</Note>

Prompt fragments let you teach the tap generator about a specific data source — its conventions, rate limits, preferred libraries, required env vars — without having to re-type the same hints into every tap description. Once a fragment is configured, its content is **automatically appended to the LLM system prompt** whenever the user's text mentions the fragment's key or any of its aliases (case-insensitive, word-boundary match).

Fragments are per-tenant, stored in MongoDB (`{env}-tap-prompt`), and cached in memory with write-invalidation. No new dependencies and no schema changes — they piggyback on the same infrastructure as tap configs.

## Where they apply

A matching fragment is injected into every AI flow that touches a tap:

| Flow | Matched against |
| - | - |
| Tap script generation (`/tap/generate`) | The plain-English description |
| Tap script fix (`/tap/fix`) | The diagnosis + error + current script |
| Post-run script review (`/tap/review`) | The current script |
| Auto-optimize (`/tap/optimize`) | The current script |
| Tap brainstorm chat (`/tap/brainstorm`) | The rolling description + full chat history |

If a key appears in any of those texts, the fragment's `content` is appended to a `## User-provided context` block at the end of the system prompt.

## Managing fragments

Fragments are managed via the `/api/v1/tap-prompts` REST endpoints (GET / POST / DELETE / `suggest`). The only UI surface is **Configuration → Data Sources**, which edits the reserved `data-sources` fragment; other fragments are managed through the API. Tap creation and editing live in the **Catalog** tab, and prompt fragments continue to apply transparently to tap creation, brainstorm, auto-fix, and optimize whenever a matching keyword appears.

To add a fragment from the CLI or a script:

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/tap-prompts \
  -H "Content-Type: application/json" \
  -H "x-api-key: <your key>" \
  -d '{
    "key": "AWS",
    "aliases": ["S3", "EC2", "Lambda", "boto3"],
    "content": "Use boto3 for AWS APIs. Read credentials from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars (don't hardcode). Default region is us-east-1.",
    "enabled": true
  }'
```

GET `/api/v1/tap-prompts` lists all fragments; DELETE `/api/v1/tap-prompts/{key}` removes one.

### Each fragment has

* **Key** (required, unique) — the primary match keyword, e.g. `AWS`, `Polygon`, `Stripe`. Matched case-insensitively with `\b` boundaries so `AI` won't accidentally match `email` or `Gmail`.
* **Aliases** (optional, comma-separated) — additional match keywords. A single `AWS` fragment can also match `S3`, `EC2`, `Lambda`, `boto3`, etc.
* **Content** (required) — the extra system-prompt text. Typically 2-6 sentences covering library choice, env-var names, rate limits, pagination style, and gotchas.
* **Enabled** (default on) — setting it `false` keeps the fragment in storage but stops it from matching.

### Writing good fragments

Keep them dense and actionable:

* Lead with the **preferred library** and the exact **env var names** the generator should use. Example: `Use the stripe Python SDK (pip install stripe). Set stripe.api_key = os.environ.get("STRIPE_API_KEY")`.
* State **rate limits** explicitly so the generator can add throttling up front. Example: `Polygon.io free tier is 5 requests per second. Throttle with time.sleep(0.25) or a ThreadPoolExecutor with max_workers=3.`
* Note **required headers** or auth quirks. Example: `SEC EDGAR requires a User-Agent header identifying the requester (e.g. "CompanyName contact@email"). Rate limit: 10 requests per second.`
* Never include secrets or long examples — multiple matching fragments can crowd the LLM context.

## Injection preview in the tap wizard

After **Generate Script** or a **Brainstorm** reply, any matching fragment keys appear as chips under the description:

> Extra context applied: `AWS` `Polygon`

This makes it easy to see what the LLM saw, and catch cases where a fragment matched when it shouldn't have (or failed to match when it should have).

## HTTP API

The same storage is exposed via REST for scripting and tenant migrations. See [Taps API](/api-reference/taps-api) for field-by-field request/response shapes.

| Method | Path | Purpose |
| - | - | - |
| `GET` | `/api/v1/tap-prompts` | List all fragments |
| `GET` | `/api/v1/tap-prompts/{key}` | Get a single fragment |
| `POST` | `/api/v1/tap-prompts` | Create or update a fragment (keyed on `key`) |
| `DELETE` | `/api/v1/tap-prompts/{key}` | Delete a fragment |
| `POST` | `/api/v1/tap-prompts/suggest` | Ask the LLM to draft a fragment content body from `{ key, aliases, content }` |

## Performance

The injector caches matched fragments in a per-tenant `ConcurrentHashMap` with a 60s TTL as a safety net. Writes through the API (create / update / delete) invalidate immediately, so changes take effect on the next LLM call without restart. With zero fragments configured the hot path returns in microseconds — the tap flows pay no measurable cost.

## Multi-tenant

Fragments are fully tenant-scoped. Each tenant's fragments live in their own `{env}-tap-prompt` MongoDB collection and their own injector cache entry. A fragment configured on one tenant never leaks into another's prompts.


# Taps
Source: https://docs.datris.ai/taps

AI-generated Python scripts that fetch data from external sources

<Note>
  **Beta.** Taps are a new feature and the API and UI may evolve based on user feedback. Core functionality is stable and production-ready, but we may add fields, rename properties, or adjust workflow steps in upcoming releases.
</Note>

A **Tap** is an AI-generated Python script that fetches data from external sources — APIs, web scraping, databases, document repositories — and pushes it into a Datris pipeline. Describe what data you want in plain English, and Datris generates the script for you.

Taps can run on a schedule using CRON expressions, or be triggered manually. They support credentials management via Vault secrets and include AI-powered diagnosis when scripts fail, return zero records, or emit deprecation/warning output.

Tap scripts live in Datris's built-in object storage by default, or in a GitHub repository you
connect — every save becomes a commit, runs pin to the exact commit they were saved at, and your
team can review and edit tap code with normal git tooling. See
[GitHub Script Storage](/tap-github-storage).

## Tap Types

The wizard offers two ways to implement a tap:

* **I Have My Own Code** — paste your own Python `fetch()` script directly. The script runs through the same test / auto-fix / post-run review flow and can use any of the pre-installed packages or bring its own. (AI-generated scripts are still available through the Assistant and MCP `create_tap` with an `instruction`. Generation reports the CodeGen provider and model that wrote the script (the primary model when no CodeGen model is set): `POST /api/v1/tap/generate` and its `/status` poll return `provider` and `model`, and `create_tap` returns them as `generatedWith`, so the Assistant can tell you which model generated the tap.)
* **HTTP Endpoint** — you host the tap as an HTTP service, in any language; no code runs on the platform. See [HTTP taps](#http-taps-bring-your-own-service-in-any-language) below.

Whichever you pick, the tap's **output shape** — structured records vs. documents — is determined by what it returns, not by a setting. A tap that returns `{uri, filename, content}` dicts is a document tap (files staged for a vector-store pipeline, deduplicated via a ledger); anything else is structured. The wizard detects the shape from your test run and applies the matching pipeline rules.

| | Structured/Semi-Structured | Document |
| - | - | - |
| **Returns** | A list of records (rows or semi-structured dicts) | A list of `{uri, filename, content, ...}` dicts — raw file bytes base64-encoded |
| **Target pipeline** | CSV / JSON / XML source → Postgres, MongoDB, object store, Kafka, vector store, etc. | `unstructuredAttributes` source → vector store (Qdrant, pgvector, Weaviate, Milvus, Chroma) |
| **What the pipeline does** | Validates schema, applies data-quality/transformation rules, writes records | Extracts text from each document, chunks it, embeds it, upserts vectors |
| **Deduplication** | Pipeline-level (truncate-before-write, upserts) | Per-document **ledger** — each `uri` + content-hash is processed once. See [Document Taps](#document-taps) |
| **Chunking / embedding** | N/A | Configured on the **pipeline**, not the tap |

### HTTP taps: bring your own service, in any language

Besides Python scripts, a tap can be an **HTTP endpoint you host** — written in Rust, Go,
TypeScript, a serverless function, or an existing internal service. Datris POSTs the run
context (params, sync state, test limit) to your URL on each run, and your endpoint responds
with the same envelope a script produces. Scheduling, per-run params, incremental state, run
history, and retries all work identically; no code runs on the platform at all, and your
service keeps its own upstream credentials.

Choose **HTTP Endpoint** in the wizard, or `--kind http --endpoint-url ...` in the CLI. AI
generation, fix/review/optimize, pip packages, and the [platform-data
callback](#querying-datris-data) don't apply to HTTP taps — a tap that reads data already in
Datris must stay a Python tap. Full wire spec and a worked Rust example:
[HTTP Tap Contract](/tap-http-contract).

## Creating a Tap

The Tap creation wizard walks you through 4 steps — or 5, if you link the tap to a pipeline (the optional 5th step lets you run the tap immediately and push data through the pipeline before leaving the wizard).

### Step 1: Describe

Step 1 is where everything except testing happens — you name the tap, describe what you want, attach credentials, and generate or paste the script.

**Tap Name** — A short identifier for the tap (e.g., `weather-data` or `contracts-sharepoint`). If the name matches an existing tap (and you're not in edit mode), an amber warning appears under the field letting you know that continuing will overwrite the existing tap's configuration and script. The warning is non-blocking — overwriting is allowed if that's what you intend.

**Tap Type** — Three radio options:

* **Structured/Semi-Structured** (default) — AI generates a script that returns records flowing into a structured pipeline destination.
* **Document Ingestion** — AI generates a script that returns `{uri, filename, content}` dicts for each file; the pipeline handles text extraction, chunking, and embedding.
* **I Have My Own Code** — skip the AI and paste your own Python `fetch()` script. The AI brainstorm, Instruction textarea, and Generate Script button are all hidden; you see a full-width code textarea and a **Use My Code** button that uploads the pasted script to MinIO. Once uploaded, Next is enabled and the rest of the wizard (Edit & Test, Schedule, Review & Save) behaves identically to an AI-authored tap. Editing the textarea after upload invalidates the upload — you'll see the button flip to **Re-upload My Code** and Next will be disabled until you re-click it.

Pick this first — the brainstorm prompts, the suggested env vars, and the Step 4 pipeline wizard all branch on this choice. The selector is locked in edit mode (change the type by creating a new tap).

**Brainstorm with AI (Optional)** — Not sure how to describe what you want? Use the chat panel above the instruction box. Type a rough idea and the AI will:

1. Ask one focused clarifying question at a time
2. Suggest specific data sources (e.g., yfinance for stocks, Open-Meteo for weather, Alpha Vantage for fundamentals)
3. Recognize when you reference a Datris table (it knows the platform's metadata and query endpoints)
4. **Auto-update the Instruction box below on every turn** as the conversation progresses
5. Suggest the environment variable names you'll need (e.g., `ALPHA_VANTAGE_API_KEY`) when an external API requires authentication

When the AI suggests environment variables, a panel appears below the chat with the variable names as chips and a **+ Create tap secret with these keys** button that pre-fills the credentials section with those keys — you just paste in the values.

You can stop chatting at any time and edit the Instruction box directly, or skip the brainstorm entirely and write the instruction yourself.

**Instruction** — The technical directive used to generate the Python script. Filled in by the brainstorm chat or written directly.

**Credentials (Optional)** — Select an existing tap secret, edit one, or create a new one. Secret keys are injected as environment variables at runtime — the AI generates code that reads them via `os.environ.get('KEY_NAME')`. **Never hardcode credentials in scripts.**

**Generate Script** — Click to have the AI produce a Python script with a `fetch()` function. The result appears below in a scrollable preview, along with any extra pip packages it needs.

The generated script can use these pre-installed packages:

* `requests`, `beautifulsoup4`, `pandas`, `lxml`, `feedparser`
* `boto3` (AWS S3), `google-cloud-storage` (GCS), `azure-storage-blob` (Azure Blob)
* `openpyxl` (Excel), `pyyaml` (YAML), `python-dateutil`, `pytz`
* Additional packages can be specified and are installed at runtime via pip into a throwaway per-run environment. In in-process mode the pip download cache lives in a Docker volume so reinstalls skip the download; in the isolated runner (compose default) packages are fetched fresh on every run.

**Example instructions:**

* *"Fetch current weather data for New York from the Open-Meteo API"*
* *"Retrieve daily stock prices for all S\&P 500 companies from yfinance"*
* *"Get the latest news headlines from the BBC RSS feed"*
* *"Query the consumer\_discretionary\_earnings table on Datris to get tickers, then fetch historical earnings from Alpha Vantage"*

Two optional fields on the tap describe it for discovery and lineage: **Tags** (free-form labels that `find_data` and the catalog search rank over) and **Source** — where the data really comes from, such as `SEC EDGAR` or `financialmodelingprep.com`. Leave Source blank and the platform derives it from the endpoint host (HTTP taps) or the host the script calls most; see [Provenance](/provenance#sources).

### Step 2: Edit & Test

Review and edit the generated script. You can:

* Modify the Python code directly
* Add or remove pip packages
* Edit the instruction and click **Regenerate** to produce a new script
* Copy the script to your clipboard with the **Copy** button next to the editor
* Click **Test Script** to execute and preview results. Preview renders as JSON (up to 100 records) in a scrollable pane
* If the script fails, an **AI Diagnosis** explains what went wrong and is **auto-applied** — the platform fixes the script and retests, up to 3 attempts. If all retries fail, the diagnosis stays visible for you to review and apply manually
* Click **Stop Test** to cancel an in-flight test (also halts the auto-fix chain)

**Limit test sample** (checkbox, default on) with an inline record-count input (default `20`, minimum `1`): when enabled, the runner injects `DATRIS_TAP_TEST_LIMIT=<n>` into the script process. Well-written tap scripts honor this env var to cap both their `/api/v1/query/*` reads and per-item iteration loops, making tests fast and low-cost. When `fetch()` is a generator (it `yield`s records), the platform also stops pulling after `n` records and closes the generator; a returned list is never truncated, so a script that returns a list must cap its own reads. Test runs started from the MCP `test_tap` tool, the CLI (`datris tap test`) and `POST /api/v1/tap/run` with `mode: "test"` default to 20 records as well — pass `limit` (MCP/CLI) or `testLimit` (REST) to raise the cap, or `0` to disable it and stream the whole source. Because of that, the `recordCount` a test reports is a preview size, not what a real run will produce. Cron runs and `mode=run` runs (the *Run Tap* button, `run_tap`) never set this env var, so production runs always read every row from every source.

#### Post-run script review

When a test passes, the platform first sends the working script and its captured stderr/stdout to the LLM for a **functional review** — not a performance pass. The reviewer looks for signals in the script's own output that the script should behave differently:

* **Rate-limit / throttle / burst warnings** — add `time.sleep`, lower `max_workers`, add adaptive backoff on HTTP 429
* **Deprecation / migration hints** — switch to the recommended replacement endpoint, method, or field
* **Pagination / partial-response hints** ("truncated", "page 1 of 5", "next\_cursor=…") — add pagination so the tap keeps fetching until the source says no more
* **Schema-drift / auth warnings** — update parsing for renamed fields; for auth errors, add a clear stderr note naming the suspect env var

If the reviewer finds a signal, it regenerates the script with the fix applied, the UI swaps it in, and the test auto-reruns. An amber banner then lists what the reviewer changed (e.g. *Added 0.25s sleep between API calls — source reported burst pattern*). A passing script is otherwise kept exactly as tested — there is no automatic performance rewrite.

This review step runs once per successful test, so the reviewer won't loop on its own output. If you edit the script yourself and click Test again, the reviewer gets a fresh look at the new output.

The test must return records successfully before you can proceed to step 3.

### Step 3: Schedule (Optional)

Set up automatic runs with a CRON expression. Datris uses Quartz CRON: 6 fields — seconds, minutes, hours, day-of-month, month, day-of-week (plus an optional year), e.g. `0 0 * * * ?` for hourly. A malformed expression — including the familiar 5-field Unix form such as `*/1 * * * *` — is refused with **HTTP 400** and the error quotes the 6-field equivalent to resubmit.

* **Presets**: Every Hour, Daily (Midnight), Weekdays (Midnight), Weekly (Monday)
* **Custom**: Describe your schedule in plain English (e.g., "Every weekday at 4pm ET") and click Generate to create the CRON expression

The **Schedule is active** checkbox controls whether the cron actually fires. Uncheck it to pause a tap without losing its schedule definition — useful when you want to temporarily stop a tap but keep its CRON expression intact for later.

<Note>
  **Test before you schedule.** A tap can only be saved with a schedule once the exact script (or HTTP endpoint) that will run on it has passed a test run. Saving a schedule on a never-tested or just-edited script is refused with HTTP 409. The schedule is applied only after a server-side test run of the saved tap: the wizard does this for you automatically on save (save unscheduled, test, then schedule — if the test fails the tap is saved without its schedule and the error says so); via MCP save without the cron, call `test_tap`, then `update_tap` with the cron; via REST save without `cronExpression`, call `POST /api/v1/tap/run` with `mode: "test"`, then save again with the cron. The wizard's own Test Script button previews an unsaved script and does not satisfy the gate by itself. Cadence changes on an already-tested tap need no re-test.
</Note>

### Step 4: Review & Save

Review all settings (including a human-readable description of the CRON schedule, e.g. `0 0 16 ? * MON-FRI — at 4:00 PM, on MON-FRI`).

From step 4 you can also link the tap to a pipeline so its output flows through validation, transformation, and into your destination automatically. Two options — both branches change shape based on the tap's type:

**Structured taps**

* **Attach to Pipeline** — pick an existing pipeline whose source columns match the tap's test output exactly. If columns don't match, you'll see exactly which columns are missing or extra so you can fix the mismatch before attaching.
* **Generate Pipeline** — create a new pipeline pre-wired to the tap's output. The new pipeline's destination is Postgres (`public.<tap_name>`) for tabular data, or MongoDB for nested/document data. **All columns are created as `string`** for safe ingestion — tap output shape can vary across runs, and string columns ingest reliably regardless of what the script returns. You can promote individual columns to richer types in the pipeline editor once you're confident about the data shape.

**Document taps**

* **Attach to Pipeline** — the picker is filtered to only compatible pipelines: those whose source is `unstructuredAttributes` and whose destination is a vector store (Qdrant, pgvector, Weaviate, Milvus, Chroma). Column matching doesn't apply — every document has the same `{uri, filename, content}` shape. The selection card shows the destination (e.g. `pgvector → public.contracts`).
* **Generate Pipeline** — creates an unstructured → vector-store pipeline. You pick:
  * **Vector store** — only stores whose secret is configured *and* the service is reachable are listed (server probes each on modal open via `/api/v1/vector-stores/available`). If exactly one is available it's preselected; if none are, the modal blocks with a link to Configuration.
  * **Destination name** — collection name (Qdrant / Milvus / Chroma), class name (Weaviate), or table + schema (pgvector).
  * **Chunking strategy** — recursive (default), fixed, sentence, paragraph, or none. Plus chunk size (default 500 chars) and overlap (default 50). These live on the pipeline, not the tap.
  * **Embedding** is inherited from the environment's default embedding secret — no picker this pass.

The server rejects any save attempt that points a document tap at an incompatible pipeline (HTTP 400), regardless of where the save came from (UI, CLI, MCP, or raw API).

If a pipeline is linked when you click save, the wizard advances to step 5; otherwise it returns to the Taps list.

### Step 5: Run the Tap (Optional)

Step 5 only appears when the tap is linked to a pipeline. It's a one-time launch screen with two buttons:

* **Run the Tap Now** — runs the tap script immediately and pushes the records it returns into the linked pipeline. The pipeline applies any data quality and transformation rules and writes to its destination. After the run completes you're returned to the Taps list.
* **Done** — skip the run and return to the Taps list. You can run the tap later from the Taps page or via the MCP `run_tap` tool.

## Querying Datris Data

Tap scripts can query data already stored in the Datris platform. The following environment variables are **always automatically injected** at runtime — your script does not need to fall back to defaults:

| Variable | Description |
| - | - |
| `DATRIS_POSTGRES_DATABASE` | PostgreSQL database name (configured via `postgres.database` in `application.yaml`, or the tenant name in multi-tenant mode) |
| `DATRIS_MONGODB_DATABASE` | MongoDB database name (configured via `mongodb.database` in `application.yaml`, or the tenant name in multi-tenant mode) |
| `DATRIS_PLATFORM_HOST` | Hostname of the Datris platform (`localhost` when the tap runs in-process; `datris` — the server's service name — when it runs in the isolated runner container) |
| `DATRIS_PLATFORM_PORT` | Port of the Datris platform (`8080` by default) |
| `DATRIS_PLATFORM_TOKEN` | A per-run credential for the platform API, minted when the run starts and revoked when it ends. You normally never touch it — see below |

Calls to the platform are authenticated for you. When [API keys](/api-keys) are required, the platform attaches `DATRIS_PLATFORM_TOKEN` as `x-api-key` to any `requests` or `urllib` call whose host is `DATRIS_PLATFORM_HOST` — and only that host, so it never reaches a third-party API. The token is read-only (queries, search, metadata), scoped to the one run, and shows up in the [audit log](/audit-log) as the tap itself. If your script uses a different HTTP client, send the header yourself: `{"x-api-key": os.environ["DATRIS_PLATFORM_TOKEN"]}`. A tap never needs a platform credential in its secret.

In single-tenant deployments the postgres and mongo database names may differ (postgres defaults to `datris`, mongo defaults to `oss`); in multi-tenant mode both resolve to the same tenant name. Always use the variable that matches the backend you are querying.

Scripts can call the Datris API directly using these:

**Discover tables (PostgreSQL):**

```python theme={null}
import os, requests
host = os.environ['DATRIS_PLATFORM_HOST']
port = os.environ['DATRIS_PLATFORM_PORT']
pg_db = os.environ['DATRIS_POSTGRES_DATABASE']
base_url = f'http://{host}:{port}/api/v1'

# List all tables
response = requests.get(f'{base_url}/metadata/postgres/tables',
                        params={'database': pg_db, 'schema': 'public'})
tables = response.json()
```

Column names returned by `/api/v1/metadata/postgres/columns` reflect the live table schema, which always satisfies the platform's `[A-Za-z0-9_]+` rule (see [Column Naming Rules](/schemas#column-naming-rules)). You can use them directly in SELECT clauses without quoting.

**Query PostgreSQL data:**

```python theme={null}
response = requests.post(f'{base_url}/query/postgres', json={
    'sql': 'SELECT * FROM public.my_table',
    'database': pg_db
})
results = response.json()['results']
```

**Query MongoDB data:**

```python theme={null}
mongo_db = os.environ['DATRIS_MONGODB_DATABASE']

# List collections
response = requests.get(f'{base_url}/metadata/mongodb/collections',
                        params={'database': mongo_db})
collections = response.json()

# Query a collection
response = requests.post(f'{base_url}/query/mongodb', json={
    'database': mongo_db,
    'collection': 'my_collection',
    'query': '{}'
})
results = response.json()['results']
```

The brainstorm AI knows about all of these endpoints, so if you mention "the X table on Datris" or "the Y collection on Datris" it will write the instruction to use the appropriate backend — no need to describe schema or column names; the script can introspect those at runtime via `/api/v1/metadata/postgres/columns` or `/api/v1/metadata/mongodb/collections`.

## Dependencies between taps

A tap that reads another tap's destination table (through `/api/v1/query/postgres`, `/api/v1/query/mongodb` or the metadata endpoints) can only test successfully once that other tap has landed data: until then the table does not exist. Datris records these dependencies so it can tell you the order to enable a catalog's taps in.

* **Inferred.** When a tap's script is saved, the platform reads the table names its queries use (`FROM` / `JOIN` in SQL, `collection` in a Mongo query) and matches them against the destination tables of the pipelines in the same catalog. The taps that feed those pipelines are stored on the tap as `inferredDependsOn`. Common table expressions, tables of other catalogs and the tap itself are ignored. A tap saved before dependency tracking existed is analysed the first time its catalog's enable order (or the tap itself) is read.
* **Declared.** Set `dependsOn` to a list of tap names in the same catalog to state the dependencies yourself (for example for a read inference cannot see, such as a warehouse or object-store read). A declared list replaces the inferred one; an empty list `[]` declares that the tap has none. Blank names are refused and repeated names are stored once. Every name must be a tap of the same catalog, not the tap itself, and the list may not close a cycle: a save that breaks either rule is refused (400 naming the tap, or 409 with the `cycle`). Leaving `dependsOn` out of an update keeps the stored list.
* **Order only.** Dependencies are never enforced: saving, testing, running and scheduling a tap behave exactly as without them.

**Enable order.** `GET /api/v1/catalog/{name}/enable-order` (MCP: `catalog_enable_order`) groups the catalog's taps into levels: level 1 holds taps that read no other tap's tables, level 2 the taps that read level 1, and so on. For each tap it reports `hasPassingTest` (the last test run succeeded on the tap's current script; a test stamped for an older script, for example after an overwrite restore, does not count), `lastRealRunLanded` (the last real run succeeded with at least one record), `blockedBy` (dependencies that have not landed data yet) and `ready` (tested and not blocked). Taps on a dependency cycle are listed under `cycles` instead of a level. It needs `tap:read`; taps the key may not read are left out and named in `warnings`.

On the **Catalog** tab, each disabled tap shows its level, or `needs: x, y` while its sources have not landed data. The **Disabled** badge menu says which sources to run first, and offers **Run now** once the tap's current script has a passing test (`hasPassingTest`; a green test from before a script edit or an overwrite restore does not count, and the menu offers **Test, then enable** instead for a scheduled tap): one real run that lands its data without enabling the tap or changing its schedule. Enabled taps with a passing test have the same **Run now** action next to **Test run**.

## Credentials / Secrets

Tap secrets are stored in HashiCorp Vault and injected as environment variables when the script runs. They are managed directly in the tap wizard (Step 1).

* **Select existing**: Choose from previously created tap secrets
* **Create new**: Define key-value pairs inline (e.g., `API_KEY=your-key`)
* **Edit existing**: Modify an existing secret's fields
* **Use suggested keys**: When the brainstorm AI suggests environment variables, click **+ Create tap secret with these keys** to jump straight into the create form with the keys pre-filled

Tap secrets are tagged with `_type=tap` and only appear in the tap dropdown — not mixed with system secrets like database credentials.

### Taps can only use tap secrets

A tap's secret must be a tap secret. Platform secrets (AI provider keys, warehouse and database credentials, the GitHub token, anything without `_type=tap`) hold values the Secrets API never shows in clear text, and every field of a tap's secret is handed to the script, so a tap on a platform secret would expose them. Saving a tap that names one returns HTTP 400, and a tap saved that way before this rule fails its next run (scheduled runs included) with the same message, before any value is injected:

```
Tap 'my-tap' uses secret 'ai-primary', which is a platform secret. Taps may only use tap secrets (Configuration → Secrets → Tap). Create a tap secret with the fields this tap needs and select it, or set DATRIS_TAP_SECRET_SCOPE=any to allow platform secrets.
```

To fix an affected tap:

* **Create a tap secret** under Configuration → Secrets → Tap with the fields the script reads, and select it on the tap.
* **Tag a hand-made tap secret.** A secret you created for a tap outside the wizard, without `_type=tap`, counts as a platform secret. Add the field `_type` with the value `tap` to it in the Secrets tab. Tagging a secret as a tap secret makes its values readable by any tap, so only an admin should do it, and only for credentials a tap is meant to use.

The server names every affected tap in one warning line at startup, and the `tap.secret_scope` [doctor](/doctor) check lists them.

**Opt-out.** `DATRIS_TAP_SECRET_SCOPE=any` lets taps use platform secrets again. It re-opens the exposure above: anyone who can create and run a tap (an editor, or an API key that can create taps) can read any platform secret's values. Use it only as a bridge while you move affected taps to tap secrets. The server-managed secrets (field protection keys, API-key stores) stay refused whatever the setting. See the [Configuration Reference](/configuration-reference).

**Never hardcode credentials in scripts.** Always use `os.environ.get('KEY_NAME')`.

## Running Taps

<Note>
  On docker compose, tap code runs in an isolated runner container by default (`USE_TAP_RUNNER=true`),
  separated from the platform's credentials and internal services; `sbt` / IDE runs stay in-process. See
  [Tap Execution & Isolation](/tap-execution-isolation).
</Note>

### Test Run (from UI)

Click the play button on the taps list to open the Run page:

* Shows the tap instruction
* **Send to pipeline** checkbox (only if a target pipeline is configured)
* Displays script output, results table, and errors
* **Run Again** button for re-execution

### CRON Schedule

Datris schedules are Quartz CRON expressions: 6 fields — seconds, minutes, hours, day-of-month, month, day-of-week — plus an optional year. Note the leading **seconds** field: the 5-field Unix form most cron references show (`*/1 * * * *`) is not a Datris schedule. `POST /api/v1/tap` refuses a malformed expression with **HTTP 400**; for a 5-field input the error quotes the 6-field equivalent (here `0 */1 * * * ?`) to resend. Use `?` in whichever of day-of-month / day-of-week you leave unspecified — Quartz requires exactly one of the two to be `?`.

A schedule can only be saved once the tap's current script (or HTTP endpoint) has passed a test run — `POST /api/v1/tap` returns **HTTP 409** otherwise, and the error names the tap and the remedy: save the tap without `cronExpression`, call `test_tap`, then `update_tap` with the cron. Editing the script of a scheduled tap is refused the same way until the new script is re-tested. Cadence-only changes, clearing the cron, and metadata edits (description, tags, target pipeline, enabled) on a tested tap are always accepted. Taps scheduled before this rule existed keep running untouched; their first cron or script change is accepted if the tap has ever tested or run successfully.

Taps with a `cronExpression` run automatically. The scheduler checks every 30 seconds for taps that are due. A tap won't fire if it's already running. For a tap that has never run before, the scheduler anchors the "next valid cron time" to `updatedAt` or `createdAt`, so the first scheduled slot after the tap was saved fires on its own — no manual bootstrap run needed.

### Sending Data to Pipelines

When "Send to pipeline" is checked:

1. The tap executes the script
2. Records are converted to the pipeline's expected format (CSV or JSON)
3. Data is fed through the pipeline's processing chain (data quality, transformation, destinations)

### Watching a Run

`run_tap` returns as soon as records are handed off — the pipeline load runs async after that. The response carries:

* `persisted: true/false` with `persistedReason` when false (`no_target_pipeline`, `test_mode`, `no_records`, `run_error`, `debounced`) so callers know whether the data actually landed. `debounced` means another run for this same tap was triggered within the last 5 seconds — that earlier run is still executing; do not retry, just track it via `get_tap_logs` / `get_pipeline_status`.
* `publisherToken` — a single ID covering every ingestion job this run submitted. Document taps fan out to many jobs per run; one `publisherToken` covers them all.
* `pipelineTokens` — the per-job IDs (length 1 for structured, N for document).
* `recordCount` — how many records the script produced.

`run_tap` does **not** return the records themselves. Use `test_tap` (or the **Test Script** button in the UI) when you need to preview what a script produces before pushing it — `test_tap` previews the first 20 records by default (`limit` changes the cap, `limit: 0` removes it) and returns a sample of up to 20 rows with a `recordsTruncated` flag when trimmed — its `recordCount` is the preview size, not the size of the source.

Poll `GET /api/v1/pipeline/status?publishertoken=...&withrollup=true` (or the MCP `get_pipeline_status` tool, which sets `withrollup=true` automatically) until `rollup.allDone` is `true`, then read `rollup.status` for the outcome and `rollup.jobs[].lastError` for any failure detail.

### Per-run Parameters

`run_tap` accepts an optional `params` object — a map of caller-supplied values that get injected into the script as environment variables for that one run only. Use this for values that vary per call (date windows, ticker lists, page cursors, batch sizes, geographic regions). Each key/value becomes an env var the script reads:

```python theme={null}
import os, json

start_date = os.environ.get("DATRIS_TAP_PARAM_start_date", "2026-01-01")   # cron-safe default
end_date   = os.environ.get("DATRIS_TAP_PARAM_end_date")
tickers    = json.loads(os.environ.get("DATRIS_TAP_PARAM_tickers", "[]"))  # JSON-encoded list
```

* **Key constraints:** must match `[A-Za-z_][A-Za-z0-9_]*` so the keys map cleanly onto env var names. Anything else is rejected with an actionable error.
* **Value handling:** strings pass through; numbers/booleans are stringified; nested objects/arrays are JSON-encoded so the script can `json.loads()` them back.
* **Scheduled runs supply no params** — cron-triggered runs have an empty params bag. Scripts MUST apply sensible defaults when the env var is absent.

**Use `params` for things that change between runs; use `secret_name` for credentials that don't.** Rewriting a tap secret on every run to smuggle per-call values through is an anti-pattern — it clobbers concurrent runs, pollutes audit history, and wastes Vault writes.

### Incremental Sync (State)

For taps that run on a schedule, the platform persists a small JSON **state** object (a bookmark) between runs, so each run fetches only what's new instead of re-fetching everything. A well-written incremental tap runs at constant size forever, no matter how large the source grows.

**The contract:**

1. Before each run, the last committed state is injected into the script as the `DATRIS_TAP_STATE` environment variable (absent on the very first run).
2. The script saves new state by assigning a dict to the module-global `DATRIS_STATE` inside `fetch()`.
3. The platform commits the new state **only after a successful run**. A failed run leaves the old bookmark in place, so the automatic retry re-fetches the same window — no data holes. With `keyFields` configured on the destination, overlapping re-fetches upsert safely.

```python theme={null}
import os, json

def fetch():
    global DATRIS_STATE
    state = json.loads(os.environ.get("DATRIS_TAP_STATE") or "{}")
    since = state.get("updated_after", "1970-01-01T00:00:00Z")

    rows = get_rows_updated_after(since)   # fetch only what changed

    if rows:
        newest = max(r["updated_at"] for r in rows)
        DATRIS_STATE = {"updated_after": max(since, newest)}
    return rows
```

**What to track** depends on what the source offers, in order of preference: a modified-since filter (state = newest timestamp seen), append-only ordering (state = highest ID seen), an opaque continuation token (store it verbatim), or — for full-fetch-only sources — a hash of the payload so unchanged runs return `[]` and record a clean `no_records`. When you create a tap from a plain-English instruction, the script generator applies this ladder automatically for scheduled taps whenever the source supports it.

**Rules:**

* State is a cursor, not a data store — a timestamp, an ID, a token, or a hash. The platform rejects state over 64 KB.
* Cursors must be monotonic: advance with `max(previous, newest_seen)` so re-processing old data never moves the bookmark backwards.
* Backfills don't move the bookmark: when an explicit window is passed via `run_tap` `params`, the script should leave `DATRIS_STATE` unset for that run.
* Never put credentials in state — it is stored and displayed unmasked.
* Test runs read state but never commit it.

**Inspecting and resetting:** the bookmark icon on the taps list shows a tap's current state, when it was committed, and a **Reset** button (the next run then does a full first-run fetch). Agents use `get_tap_state` / `set_tap_state`; the REST surface is `GET/POST/DELETE /api/v1/tap/state`. Deleting a tap deletes its state, so a recreated tap starts fresh.

### Run Status Outcomes

Each tap run produces one of three states on its `lastRunStatus` field and matching run log entry:

| Status | Meaning |
| - | - |
| `success` | Script ran, returned records, and the pipeline accepted them. The `recordCount` is non-zero. |
| `no_records` | Script ran cleanly but returned zero records. Legitimate for polling/incremental taps on a quiet day (no new data since last run, weekend with markets closed, dedup-already-loaded). NOT counted as a failure in the Failures pane on the Ops Activity dashboard. |
| `failure` | Script errored (Python exception, HTTP error from the source, timeout, output-size guard, etc.). The `error` field carries the message. A run that timed out also keeps the script's own logs and reports a partial record count — the records that reached the staging file before the kill (nothing landed in the destination; the tap's last-run record count stays 0). |

Cron-scheduled runs that legitimately produce zero records (incremental tap that's caught up, polling tap with no new entries) record `no_records` rather than `failure` and the Ops Activity dashboard surfaces them as healthy.

### Output Size Guard

A tap run's records never sit in the server's heap: the wrapper writes them to a staged file on the `datris-staging` volume as the script returns them, and the pipeline reads that file. What is capped is disk — `PIPELINE_MAX_PAYLOAD_MB` (property `pipelineMaxPayloadMB`) is the per-run payload disk budget in megabytes, default **4096**, `0` = unlimited. A run that stages more than the budget is stopped with an actionable error naming the variable (`The payload exceeded the configured disk budget for one run (PIPELINE_MAX_PAYLOAD_MB = 4096 MB, got ~4097 MB). Raise it, or split the source (for a tap, chunk the source range via run_tap params).`). A multi-GB fetch runs on the default 2 GB heap.

To size a run against the budget: records are staged as JSON with their keys repeated on every row, so a staged record costs roughly **26 bytes × column count** for numeric, timestamp and short-string data (long text or nested fields add their own length) — a 25-column record is about 650 bytes, so 20,000,000 rows stage about 13 GB. Reading only the columns the destination needs cuts that linearly.

`TAP_MAX_OUTPUT_MB` is the deprecated alias of `PIPELINE_MAX_PAYLOAD_MB`, honoured for one release when the new variable is unset; the server logs a deprecation warning at boot when the alias is set.

If a backfill blows the budget, raise it, or reduce the source range using `run_tap` `params` (shorter date window, smaller page, per-symbol chunks) and call again. Multiple smaller runs all land in the same destination pipeline; with `keyFields` configured on a Postgres or MongoDB pipeline, overlapping ranges upsert safely. For a scheduled tap that keeps growing toward the cap, the durable fix is making it incremental — see [Incremental Sync (State)](#incremental-sync-state) — so every run carries only the delta.

## Data Types

The system automatically detects the data type from what `fetch()` returns:

| Return Value | Detected Type |
| - | - |
| List of dicts with `uri` and `content` keys | **Document** (triggers the document-tap path) |
| List of flat dicts (all scalar values) | **JSON** (records — projected to CSV when the target pipeline is CSV, with the union of keys as the header) |
| List of dicts with nested objects | **JSON** |
| List of lists/tuples | **CSV** (rows — the first list is the header; a CSV pipeline gets it as the header row, a JSON pipeline gets the arrays as-is) |
| Generator / iterator (`fetch()` uses `yield`, or returns an iterator) | Detected from the **first record** exactly as above, then streamed to disk one record at a time |
| Generator yielding batches (`DataFrame` / `RecordBatch` / `Table` / list of dicts / list of lists) | Detected from the **first row of the first item**, then each batch is written as its **rows** — `count` is rows, `columns` is the union of the batches' column names |
| XML string | **XML** |
| Other string | **Text** |

The `uri` + `content` detection takes priority — if your script returns records with those keys, the runner routes every element through the document pipeline path even if the tap's type is `structured`. In practice the type radio and the return shape should always agree; this rule just makes misconfigurations fail loudly rather than silently corrupt a structured destination.

## Pipelines and Taps

The Pipelines page shows a **Tap** column for each pipeline, indicating which tap (if any) feeds it. Click the tap name to jump straight to that tap's edit page.

On the Pipeline creation wizard, select **From Tap** to auto-populate the pipeline's source type and schema from a tap's test results. The tap's data type and columns are used to configure the pipeline automatically.

### Fetch-only taps

Every tap has a pipeline. When you want a tap to fetch and hand the rows back — an answer now, a validation result, a transformed view — rather than land them, wire it to a [Live Read](/destinations/live-read) pipeline (the `scratch` destination). The run goes through data quality and transformation as usual; the rows ride on the run's status rollup (`resultPreview`) and can be paged with `get_pipeline_result`, `datris pipeline result`, or the **View result** panel on the run detail page. Live Read results are not catalogued or queryable later and expire, so read them promptly. If the rows turn out to be worth keeping, change the pipeline's destination — the tap, its schedule and its incremental state do not move.

## Run History

Click the history icon on the taps list to view a tap's run history. Each entry shows:

* Status (success/failure)
* Timestamp and duration (formatted using the configured `dateFormat` and `dateTimezone` — see [Configuration Reference](/configuration-reference))
* Record count
* Whether data was sent to a pipeline
* Expandable script output logs and error messages

## MCP Tools

AI agents can manage taps via MCP:

| Tool | Description |
| - | - |
| `create_tap` | Create a tap from a plain-English instruction (AI generates the script), a user-provided script, config only, or `kind: "http"` + `endpoint_url` for a [user-hosted HTTP endpoint](/tap-http-contract) |
| `create_tap_secret` | Create a tap secret the agent can then reference via `secret_name` on `create_tap`. Fails on name collision unless `overwrite=true`. Reserved AI-slot names are blocked. Agents can only overwrite secrets tagged `_type=tap` — human-owned secrets are refused. |
| `delete_tap_secret` | Delete a tap secret. Only `_type=tap` secrets can be deleted by an agent; human-owned secrets must be removed from Configuration → Secrets in the UI. |
| `list_tap_secrets` | List the names of existing tap secrets (tagged `_type=tap`). Call before `create_tap_secret` to reuse a suitable existing secret instead of asking the user for credentials again. |
| `get_tap_secret_fields` | Return the field names (keys only — never values) of an existing tap secret. Use after `list_tap_secrets` to verify a candidate secret has the keys the tap script will need. |
| `list_taps` | List all taps with status, schedule, and last run info |
| `get_tap` | Get full details of a single tap including its Python script |
| `run_tap` | Execute a tap and push fetched data to the target pipeline. Response carries `persisted`, `persistedReason`, `recordCount`, `publisherToken`, `pipelineTokens` so the agent can confirm what landed. Records themselves are not returned — use `test_tap` to preview a script. |
| `test_tap` | Test-run a tap without pushing data to the pipeline |
| `get_pipeline_status` | Watch ingestion progress after a `run_tap`. Pass the response's `publisherToken` to get status rows for every job this run submitted (one for structured taps, N for document taps). |
| `get_pipeline_result` | Read the rows a [Live Read](/destinations/live-read) pipeline produced. The first rows are already on the `get_pipeline_status` rollup (`resultPreview`); call this only when `resultTruncated` is true. Paging never re-runs the source; results expire. |
| `update_tap` | Update a tap's config (enabled, schedule, pipeline, description, `depends_on`) without regenerating the script |
| `get_tap_logs` | Get run history for a tap (last 50 entries). Each entry that submitted records includes its `publisherToken`, so an agent can pivot from a scheduled-run log entry to `get_pipeline_status(publisher_token=...)` and verify the run actually landed in the destination — not just that the script ran. |
| `get_tap_ledger` | Document taps only: read the ledger of discovered documents (URI, filename, status, hashes, first/last seen timestamps). Pass `clear_uri` to force one file to be re-processed on the next run, or `clear_all=true` to force a full re-scan. |
| `get_tap_state` | Read a tap's incremental-sync state (the bookmark committed by its last successful run). |
| `set_tap_state` | Overwrite a tap's state (e.g. rewind a cursor to re-fetch a window), or pass `reset=true` to clear it so the next run does a full first-run fetch. |
| `list_tap_versions` | List the change history of a tap's definition, newest first (`version`, `createdAt`, `createdBy`, `changeNote`). An empty result means the tap has not been edited since versioning was enabled; the current version number is the `version` field on the tap itself. |
| `get_tap_version` | View one historical snapshot of a tap: the full config and pinned script as they were at that version. Read-only. |
| `diff_tap_versions` | Compare two versions of a tap: a field-by-field config diff plus a line-level script diff. `version` is the selected snapshot, `against` the baseline. |
| `restore_tap_version` | Roll a tap back (or forward) to a prior version. Append-only: the chosen snapshot is written as a new latest version, so history is never overwritten. |
| `delete_tap` | Delete a tap and its stored script |

End-to-end agent flow when a tap needs credentials:

```
create_tap_secret(name="stripe-key", fields={"apiKey": "sk_..."})
create_tap(name="stripe-charges", script="...", secret_name="stripe-key")
```

## CLI Commands

```bash theme={null}
# List all taps
datris taps

# Create a tap with AI-generated script
datris tap create "Fetch weather data from Open-Meteo API" --pipeline weather --cron "0 0 * * * ?"

# Create a tap with your own script
datris tap create --script ./my_script.py --name my-tap --pipeline weather

# Create an HTTP tap — a service you host speaks the tap contract (see HTTP Tap Contract doc)
datris tap create --name my-tap --kind http --endpoint-url https://taps.example.com/my-tap --pipeline weather

# Create a tap config only (add script later)
datris tap create --name my-tap --pipeline weather

# Show tap details including script
datris tap show my-tap

# Test-run a tap (no data push)
datris tap test my-tap

# Run a tap manually
datris tap run my-tap

# Read the rows a fetch-only (Live Read) tap run produced
datris pipeline result <pipeline-token> --out rows.jsonl

# View run history
datris tap logs my-tap

# Update a tap's schedule or config
datris tap update my-tap --cron "0 0 * * * ?" --enabled

# Disable a tap
datris tap update my-tap --disabled

# Delete a tap
datris tap delete my-tap
```

## Configuration

| Setting | File | Default | Description |
| - | - | - | - |
| `tapScriptTimeoutSeconds` (`TAP_SCRIPT_TIMEOUT_SECONDS`) | `.env` | `300` | Wall-clock ceiling for a tap **test**, in seconds |
| `tapRunTimeoutSeconds` (`TAP_RUN_TIMEOUT_SECONDS`) | `.env` | `3600` | Wall-clock ceiling for a real or scheduled tap **run**, in seconds. Unset resolves to `max(3600, tapScriptTimeoutSeconds)` |
| `schedule.checkTapSchedules` | `application.yaml` | `30000` | CRON scheduler poll interval in milliseconds |
| `dateFormat` | `application.yaml` | `yyyy-MM-dd HH:mm:ss z` | SimpleDateFormat pattern used for tap run timestamps |
| `dateTimezone` | `application.yaml` | `UTC` | IANA timezone (e.g., `America/New_York`) for displayed timestamps |

## Document Taps

Document taps feed files (PDFs, DOCX, HTML, Markdown, etc.) into a vector-store pipeline. The pipeline owns text extraction, chunking, and embedding — the tap's only job is **discovery and retrieval**.

### Return shape

Each element of the list `fetch()` returns is a dict describing one document:

| Field | Required | Description |
| - | - | - |
| `uri` | ✓ | Unique source identifier (URL, S3 key, file path). Primary key for change detection. |
| `filename` | ✓ | Original filename **with extension** — the extension drives `TextExtractorUtil`'s parser choice (`.pdf` → PDFBox, `.docx` → POI, etc.). |
| `content` | ✓ | Raw file bytes, **base64-encoded**: `base64.b64encode(raw_bytes).decode('ascii')`. Must be the complete file, not a chunk. |
| `content_hash` | optional | Precomputed ETag or hash for change detection. If omitted, the platform computes SHA-256 of the decoded bytes. |
| `metadata` | optional | Arbitrary `dict[str, str]` attached to the ledger entry (e.g., `{"author": "ACME", "region": "EU"}`). |

**Rules the generator and users must follow:**

* Return one entry per source file. Never chunk, split, or segment. Never decode the bytes.
* Don't generate embeddings, pick embedding models, choose a vector store, or create tables — those are pipeline concerns.
* Don't invent metadata fields that describe pipeline behavior (`chunk_size`, `embedding_model`, `target_table`); those belong on the pipeline.

The AI brainstorm chat and script generator both enforce these rules: if you ask for chunking or embedding choices, the assistant redirects you to pipeline configuration.

### The Document Ledger

Every document the tap successfully stages is recorded in a **ledger** — a MongoDB collection keyed by `{tapName}|{uri}`. On each run, the tap:

1. Loads `uri → contentHash` for every existing ledger entry owned by this tap.
2. For each document returned by `fetch()`, computes (or reads) its content hash.
3. **Skips** the document if the ledger already has an entry with the same hash (just refreshes `lastSeenAt` for operator visibility).
4. Otherwise, stages the bytes to MinIO, writes a ledger entry (`status: staged`), submits to the pipeline, and marks the entry `processed` (or `failed` on error).

This is what makes document taps safe to run on a schedule: re-runs only process new or changed files.

**Ledger fields:** `uri`, `tapName`, `stagedPath` (MinIO key), `filename`, `contentHash`, `firstSeenAt`, `lastSeenAt`, `status` (`staged`/`processed`/`failed`), `metadata`.

**Managing the ledger:**

* **UI** — the document-ledger button (page icon) next to a document tap in the Taps list opens a modal listing every entry. Actions: **Clear All** (force full re-scan on next run), or delete a single entry (force that one file to re-process).
* **REST** — `GET /api/v1/tap/ledger?name={tap}`, `DELETE /api/v1/tap/ledger?name={tap}&uri={uri}` for a single entry, `DELETE /api/v1/tap/ledger?name={tap}` to clear all.
* **MCP** — `get_tap_ledger` with optional `clear_uri` / `clear_all`.

**Automatic cleanup:**

* Deleting a tap clears its ledger entries and deletes all staged MinIO objects.
* Deleting a pipeline's data (from the Pipelines page → Delete Data Only, or `DELETE /api/v1/pipeline?pipeline=X&deleteData=true`) clears the ledger for every document tap pointed at that pipeline — so the next run re-ingests every file into the now-empty destination. Without this, taps would skip docs they already "processed" and the pipeline would stay empty.

### Pipeline compatibility

The server validates at two points:

1. **On save** (`POST /tap`) — if `tapType == "document"` and `targetPipeline` is set, it checks the pipeline's shape. Rejects with HTTP 400 if the source isn't `unstructuredAttributes` or the destination isn't a vector store.
2. **On feed** (`TapRunner.feedDocumentPipeline`) — re-checks before submitting bytes, so a pipeline that was reshaped after the tap was saved fails the run with a clear error instead of silently corrupting a destination.

### Concurrency

Document taps feed many documents through `StreamNotifier.process` in rapid succession. Each becomes its own `JobRunner`, and all five vector-store loaders are now **race-safe** when multiple runners hit `ensureCollection` / `ensureTable` at the same time on a fresh pipeline:

* **pgvector** — transactional advisory lock plus `CREATE TABLE IF NOT EXISTS`; serializes concurrent sessions at the schema-qualified table name.
* **Qdrant / Milvus / Weaviate** — try-create; on error, re-check existence and swallow if a concurrent runner won the race.
* **Chroma** — sends `get_or_create: true` plus a re-GET fallback for older servers.

***

## Script Requirements

The `fetch()` function must:

* Take no arguments
* Return a list of dictionaries (records), a list of lists (rows, header first), or a string (JSON/XML/text) — or **`yield`** the records / rows one at a time (any generator or iterator works)
* **For large sources, `yield`.** The platform streams whatever `fetch()` produces to disk one record at a time, so a run is never limited by memory — but only if the script does not build the whole result first. A `fetch()` that appends millions of rows to a list (or loads a whole file into pandas and calls `to_dict("records")`) is killed for memory, and the run fails with an error naming memory and `yield`. Read the source in chunks / pages / batches (`pd.read_csv(..., chunksize=50000)`, `pyarrow.parquet.ParquetFile(...).iter_batches()`, one API page at a time) and `yield` each row before fetching the next chunk. Never cap the rows or split the source into several runs for size — the one exception is when the estimate below exceeds a budget and the user has chosen to chunk the range.
* **Yield batches for a large columnar or tabular source.** For a large columnar or tabular source, `yield` each batch (a pyarrow `RecordBatch`, a pandas `DataFrame`, or a list of dicts) instead of each row — the platform serialises a batch natively, several times faster — and read only the columns the pipeline needs (`columns=[...]`). Yielding one row at a time remains correct for small or API-paged sources. Each batch is written as its rows, so nothing downstream changes: the record count is rows, the columns are the union of the batches' column names, and the staged records are identical to yielding the batch's own rows (`df.to_dict("records")` / `batch.to_pylist()`) one at a time. Floats inside a batch are written with up to 15 significant digits rather than Python's shortest round-trip form — the type and the destination schema are unchanged. Two other cosmetic differences in a batch: `-0.0` lands as `0.0`, and a `/` inside a string is escaped as `\/`; both parse to the same value.
* **Progress is reported for you.** A `fetch()` that yields records makes the platform print `[wrapper] streamed N records (M s)` to the run log every 100,000 records or every 30 seconds, so a long or timed-out run shows how far it got. No script change is needed, and a `fetch()` that returns a list prints none.
* **Stream the source, never download it.** The isolated tap runner has no disk of its own: its only writable area is a 512 MB in-memory scratch space shared with the per-run venv and any pip-installed packages (see [Tap execution isolation](/tap-execution-isolation)). A `fetch()` that saves a large file locally before reading it — `urlretrieve`, `open(path, "wb")`, `requests` with `stream=True` written to a file, a library that spools under `TMPDIR` — fails with `No space left on device`. Read remote files directly: `pyarrow.parquet.ParquetFile(fsspec.open(url).open())` (or an `io.BytesIO` of a single small file), `pd.read_csv(url, chunksize=...)`, `requests.get(url, stream=True).iter_lines()`, and `yield` records as they arrive.
* **Check the time budget too.** A test is killed after `TAP_SCRIPT_TIMEOUT_SECONDS` (default 300 s) and a real or scheduled run after `TAP_RUN_TIMEOUT_SECONDS` (default 3600 s); both ceilings in force are reported by `get_version` / `GET /api/v1/version`. A test is also capped at 20 records, so a passing test says nothing about how long a real run takes — only an estimate does. A timed-out run says which mode it ran in and which variable to raise.
* **Check both budgets before you build.** Estimate the run from two measured numbers, and show them before creating anything:
  * **Disk** — records are staged as JSON with their keys repeated on every row, so a staged record costs roughly **26 bytes × column count** for numeric / timestamp / short-string data (long text or nested fields add their own length). `rows × that` is what the run stages, capped by `PIPELINE_MAX_PAYLOAD_MB` (default 4096 MB, `0` = unlimited). Read the value in force from `get_version` / `GET /api/v1/version` (`pipelineMaxPayloadMB`) — an install that raised the variable runs with a different ceiling, and the estimate is compared with that, not with the default.
  * **Time** — a `fetch()` that yields one record at a time moves about **15,000 rows per second**; one that yields batches moves about **90,000 rows per second**. Say which rate the estimate used, and prefer batches for any source over about a million rows. Compare the minutes with `TAP_RUN_TIMEOUT_SECONDS` (default 3600 s) — again, the value in force is `tapRunTimeoutSeconds` from `get_version` / `GET /api/v1/version`.
  * Show the estimate as a four-line table — rows, bytes/record, GB staged, minutes — so the row-count assumption can be corrected. For example, 20,000,000 rows × 25 columns → \~650 bytes/record → \~13 GB staged → \~4 minutes at the batch rate.
  * If either estimate is over its budget, choose before building: raise the variable (the staging volume needs matching free space; a longer timeout holds a runner slot that long), narrow the window, or chunk the range via `run_tap` `params` — several runs land in the same pipeline. Reading only the columns the destination needs cuts both numbers linearly. Finding out after several minutes of streaming is the expensive way.
* Let exceptions propagate (the wrapper captures the traceback for diagnosis); only catch what you can actually recover from
* Include timeouts on network requests (30 seconds recommended)
* Use `os.environ.get()` for any credentials
* Read `DATRIS_POSTGRES_DATABASE`, `DATRIS_MONGODB_DATABASE`, `DATRIS_PLATFORM_HOST`, and `DATRIS_PLATFORM_PORT` directly without fallback defaults — they are always injected by the platform
* **Column names are auto-normalized** for tabular results (list of dicts). The platform converts each key to lowercase snake\_case using only `[a-z0-9_]` so that downstream pipeline registration succeeds and SQL queries don't need quoting. Examples: `EPS Estimate` → `eps_estimate`, `Surprise(%)` → `surprise_percent`. You can return raw source column names — the platform will clean them — but for clarity in the test preview, prefer to emit clean keys directly. **JSON/XML results destined for MongoDB are not normalized** (they go through as raw blobs in the `_json` field). See [Schema Definition → Column Naming Rules](/schemas#column-naming-rules) for the underlying validator rule.


# AI Transformation
Source: https://docs.datris.ai/transformation/ai-transformation

AI-powered data transformation

AI transformations apply natural language instructions to transform data. Describe the transformation in plain English — Datris generates a Python script from your instruction and runs it against all data in an isolated runner container.

## Configuration

Add an `aiTransformation` block to the `transformation` section:

```json theme={null}
"transformation": {
    "aiTransformation": {
        "instruction": "convert all date values from YYYY-MM-DD format to MM/DD/YYYY format"
    }
}
```

| Field | Type | Description |
| - | - | - |
| `instruction` | string | A natural language description of the transformation to apply to every row. |

## Examples

**Date format conversion:**

```json theme={null}
"aiTransformation": {
    "instruction": "convert all date values from YYYY-MM-DD format to MM/DD/YYYY format"
}
```

**Phone number standardization:**

```json theme={null}
"aiTransformation": {
    "instruction": "standardize the phone_number column to +1-XXX-XXX-XXXX format"
}
```

**Data enrichment / categorization:**

```json theme={null}
"aiTransformation": {
    "instruction": "add a 'sector' column at the end based on the stock symbol, categorizing each as one of: Technology, Healthcare, Finance, Energy, Consumer, Industrial, Other"
}
```

**Combined operations:**

```json theme={null}
"aiTransformation": {
    "instruction": "convert dates to YYYY/MM/DD. Trim leading/trailing whitespace from all columns. Remove duplicate rows."
}
```

## How it works

1. When the pipeline is saved, Datris combines the source schema (delimiter, column names and types) with your instruction and sends a single prompt to the AI model. When a run has to generate the script instead (see [When the script is generated](#when-the-script-is-generated)), it uses the column names and sample rows from that run's data.
2. The AI generates a self-contained Python 3 script (stdlib only) that reads the input file, applies the transformation, and writes the output. The script is stored.
3. Every run executes the stored script locally via `python3`. Runs make no model call.
4. The transformed data replaces the original and continues through the pipeline to destinations.

AI transformations run after data quality checks and before data is written to destinations.

On a compose install the script runs in `datris-codegen-runner`, an isolated container with no platform secrets, no access to the server's files, and no route to the internal services or the internet. The server sends it the script and the input file and streams the output file back. If the runner is stopped or unreachable the transformation fails with an error naming it; it never falls back to running the script in the server. See [Generated data-quality and transformation scripts](/tap-execution-isolation#generated-data-quality-and-transformation-scripts).

An `aiTransformation` whose `instruction` is empty or only whitespace is ignored: no script is generated at save and the run skips the stage with a status line saying so.

## When the script is generated

The script is generated once and stored, not on every run:

* **At save** — Saving a pipeline with an AI transformation (`POST /api/v1/pipeline`, the MCP `create_pipeline` tool, or the UI wizard) generates the script from the source schema: the delimiter and the ordered field names and types. No sample rows are available when a pipeline is saved, so state any format the transformation depends on in the instruction (for example "dates are YYYY-MM-DD"). The script is stored in the built-in object store under `pipeline-scripts/<pipeline>/`, or committed to the code repository when one is enabled (see [Pipeline scripts](/tap-github-storage#pipeline-scripts)).
* **Every run** — Each run executes the stored script. A run makes no model call, gives the same result for the same data, and keeps working if the model is unreachable. The status shows "Running stored CodeGen script generated at ...".
* **Re-saving** — Saving again regenerates only when the instruction or the source schema (delimiter, field names, types or order) changed. Re-saving an unchanged pipeline makes no model call. A changed CodeGen model does not regenerate: the script keeps the model that wrote it.
* **Pending** — If the script cannot be generated at save (no AI key, model unreachable, AI disabled), the pipeline is still saved, the save response carries a warning and `codegenScripts` lists the script as `pending` with the reason. JSON and XML pipelines have no column schema to generate from, so they are always saved pending. The first run generates the script from its data, stores it, and later runs reuse it.
* **Header differs** — If the header that reaches the transformation differs from the one the script was written for (for example a column was removed by field protection or is missing from the file), that run generates a script for itself only and keeps the stored one. The status says why.
* **Failures** — A stored script that exits with an error or times out fails the run. It is not regenerated automatically: the error names when the script was generated and the regenerate endpoint.

To read the stored script, call `GET /api/v1/pipelines/{name}/codegen-scripts`. To replace it (for example to adopt a new CodeGen model, or after a wrong script), call `POST /api/v1/pipelines/{name}/codegen-scripts/transformation/regenerate`, which needs the `pipeline:create` capability. If generation fails there, the current script stays in place.

With a code repository configured under **Configuration > Code Repository**, the script for a new pipeline is committed to it as `<pathPrefix>pipelines/<pipeline>/transformation.py` and every run executes the recorded commit. You can review it, blame it and edit it in the repository; an edit is used only after you adopt it with `POST /api/v1/pipelines/{name}/codegen-scripts/transformation/pull`. See [Pipeline scripts](/tap-github-storage#pipeline-scripts) for commit messages, conflicts, and what happens when the repository is unreachable.

The **Columns** section of the [lineage panel](/provenance#column-level-lineage) shows when and by which model the stored transformation script was generated, or "Script pending" with the reason, and offers View and Regenerate.

Pipelines created before stored scripts generate their script on the first run after the upgrade and reuse it after that.

## Values withheld

When the server sets `DATRIS_AI_SAMPLE_VALUES=false`, the prompt carries the column names and schema types (or a value-free skeleton of a JSON or XML file) and the line "Sample rows withheld by configuration" instead of sample rows. The generated script still runs on the full data. If the transformation depends on what values look like, spell the format out in the instruction. See [Field Protection](/transformation/field-protection#what-still-sees-raw-data).

## Works with all file types

* **CSV/delimited files** — The script reads and writes CSV with the appropriate delimiter.
* **JSON files** — The script parses JSON, transforms records, and writes JSON.
* **XML files** — The script uses `xml.etree.ElementTree` to parse, transform, and write XML.

## CLI

```bash theme={null}
datris ingest data.csv --dest postgres --ai-transform "convert dates to YYYY/MM/DD and uppercase all names"
```

## Requirements

* `ai.enabled: true` must be set in `application.yaml`
* The Vault secret for the AI provider must be configured (see [AI Configuration](/ai-configuration))
* `python3` must be available on the pipeline server runtime


# Dropping Columns
Source: https://docs.datris.ai/transformation/dropping-columns

Remove columns from data

## Prefer `protect: {"method": "drop"}`

To keep a column out of every later stage and destination, mark it on the source schema with `"protect": {"method": "drop"}`. It is removed right after the preprocessor, before data quality, transformation, and any AI stage, and it works for every destination, including the object store, MongoDB, and JSON pipelines (top-level keys). See [Field Protection](/transformation/field-protection) for the other methods (`hmac`, `mask`, `redact`).

```json theme={null}
{"name": "internal_code", "type": "string", "protect": {"method": "drop"}}
```

## Destination schema projection

You can also exclude columns from the output by defining a **destination schema** that omits unwanted columns. No special configuration is needed — simply remove the columns from the destination schema that you don't want in the output.

## How It Works

For the PostgreSQL, Snowflake, and Databricks destinations, when a destination schema (`destination.schemaProperties`) is defined, only the columns listed in it are written to the destination. Columns present in the source but absent from the destination schema are dropped. These loaders project each row onto the destination columns by name.

## Example

Given source data with columns `id`, `name`, `email`, `internal_code`, `created_at`:

```json theme={null}
{
  "name": "employees",
  "source": {
    "schemaProperties": {
      "fields": [
        {"name": "id", "type": "int"},
        {"name": "name", "type": "string"},
        {"name": "email", "type": "string"},
        {"name": "internal_code", "type": "string"},
        {"name": "created_at", "type": "timestamp"}
      ]
    },
    "fileAttributes": {
      "csvAttributes": {"delimiter": ",", "header": true, "encoding": "UTF-8"}
    }
  },
  "destination": {
    "schemaProperties": {
      "fields": [
        {"name": "id", "type": "int"},
        {"name": "name", "type": "string"},
        {"name": "email", "type": "string"}
      ]
    },
    "database": {
      "dbName": "mydb",
      "schema": "public",
      "table": "employees",
      "usePostgres": true
    }
  }
}
```

In this example, `internal_code` and `created_at` are present in the source but excluded from the destination schema, so they are dropped from the output.

## Notes

* Column dropping applies to delimited/CSV row data loaded into PostgreSQL, Snowflake, or Databricks, where each row is projected to the destination columns by name. The object store (Parquet, ORC, or Iceberg) destination maps values to destination fields by position, so its destination schema must list the same columns in the same order as the source. MongoDB and the vector destinations do not use per-column projection. JSON and XML pipelines use a single raw `_json`/`_xml` document that passes through without per-column projection, so column dropping does not apply to them.
* The destination schema field names must match source schema field names (case-insensitive)
* Column order in the destination schema determines the output column order


# Field Protection (PII / PHI)
Source: https://docs.datris.ai/transformation/field-protection

Pseudonymize, mask, redact, encrypt, or drop sensitive fields before any AI stage or destination sees them

Field protection rewrites sensitive columns as the very first step after ingest, so every later stage only ever sees the protected value. You opt in per field with a `protect` object on the source schema.

## Why

Datris sends samples of your data to an AI provider when a pipeline uses an [AI rule](/data-quality/ai-rules), an [AI transformation](/transformation/ai-transformation), or error explanation. Field protection runs **before** all of them. A protected column reaches the AI provider, Live Read, and every destination only in its protected form, so the raw value of that column is never sent to a model provider. That means you do not need a business associate agreement (or a similar data-processing agreement) with the provider for the columns you protect. Columns you do not protect are sent as they are.

Protection is local: no value leaves the Datris server to be protected, and no status line, error message, or log line carries a raw value.

## Methods

| Method | Result | Example input | Example output |
| - | - | - | - |
| `hmac` | Keyed HMAC-SHA256 pseudonym (64 lowercase hex characters) under the environment's key. Equal inputs give equal outputs, so the column still joins and deduplicates. | `A-10442` | `9f2c…e41b` |
| `mask` | Every character replaced with `*`. | `secret` | `******` |
| `mask` + `"preserve": "last4"` | Every character but the last four replaced with `*`. A value of four characters or fewer is masked entirely. | `123456789` | `*****6789` |
| `mask` + `"preserve": "domain"` | `***` followed by `@` and the domain. A value without `@` is masked entirely. | `user@example.com` | `***@example.com` |
| `mask` + `"preserve": "year"` | A leading 4-digit year is kept; every letter and digit after it is masked and separators stay. A value without a leading year is masked entirely, so `03/15/1985` and `19850315` lose the year too (safe, but store dates as `YYYY-MM-DD` to keep it). | `1987-06-05` | `1987-**-**` |
| `mask` + `"preserve": "first3"` | The first three characters are kept and every other character is masked. A value of three characters or fewer is masked entirely. | `02139` | `021**` |
| `redact` | The value becomes `[REDACTED]`. | `anything` | `[REDACTED]` |
| `drop` | The column (or top-level JSON key) is removed and never reaches a later stage or destination. | | |
| `encrypt` | AES-256-GCM ciphertext under the environment's current encryption key, with a fresh random IV per value and bound to the pipeline and field name. Equal inputs give different outputs. The only reversible method: an authorized caller can [reveal](#revealing-a-value) the original. | `jane@example.com` | `enc:v1:Qm9…x3A` |

Empty values stay empty for every method.

An `encrypt` value is longer than its input: `enc:v<n>:` followed by the base64url of the input's UTF-8 bytes plus 28 bytes (a 12-byte IV and a 16-byte authentication tag). A 20-character email becomes 71 characters. If the destination column is a `varchar(n)`, size it for the ciphertext, not the original value.

## Config example

```json theme={null}
{
  "name": "patients",
  "source": {
    "fileAttributes": {
      "csvAttributes": {"delimiter": ",", "header": true, "encoding": "UTF-8"}
    },
    "schemaProperties": {
      "fields": [
        {"name": "mrn", "type": "string", "protect": {"method": "hmac"}},
        {"name": "email", "type": "string", "protect": {"method": "mask", "preserve": "domain"}},
        {"name": "dob", "type": "string", "protect": {"method": "mask", "preserve": "year"}},
        {"name": "notes", "type": "string", "protect": {"method": "redact"}},
        {"name": "phone", "type": "string", "protect": {"method": "encrypt"}},
        {"name": "ssn", "type": "string", "protect": {"method": "drop"}},
        {"name": "visit_date", "type": "string"},
        {"name": "amount", "type": "string"}
      ]
    }
  },
  "destination": {
    "database": {
      "dbName": "datris",
      "schema": "public",
      "table": "patients",
      "usePostgres": true
    }
  }
}
```

To keep the ingest object after protection, add `"protection": {"purgeSource": false}` at the top level (see [Source retention](#source-retention)).

From an agent, `create_pipeline` takes the same policies as a `protect` map keyed by field name, for example `{"mrn": {"method": "hmac"}, "ssn": {"method": "drop"}}`.

## Suggestions

Datris can propose a starting point. The configured CodeGen model receives the field **names and types only** (one `name:type` line per field) and returns, for each field, `hmac`, `mask` with an optional `preserve`, `redact`, `drop`, or no protection (it never suggests `encrypt`; choose that yourself), with a one-line reason. No row value is sent, and nothing is saved: you confirm the fields you want and then save them as `protect`.

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/protect/suggest \
  -H "x-api-key: <key>" -H "Content-Type: application/json" \
  -d '{"pipeline": "fp_demo"}'
```

Pass `{"pipeline": "<name>"}` to use a saved pipeline's source schema, or `{"fields": [{"name": "...", "type": "..."}]}` for a schema you have not saved yet. Each entry in the response has `name`, `type`, `suggested` (`{"method", "preserve"}`, or `null` for no protection), `reason`, and `current`: the field's existing `protect`, so running it again on a protected pipeline never reads as an overwrite. The `model` field names the model that answered.

Every suggestion follows the [type rules](#type-rules): a field that is not `string` is only ever suggested `drop` or nothing, and `preserve` only comes with `mask`. With `pipeline`, the saved destination is checked too: a `keyFields` column is only ever suggested `hmac` or nothing, and a field whose destination type is not `string` is only suggested `drop` or nothing. A JSON pipeline's `_json` field is not sent to the model; list its top-level keys in `fields` instead. One call takes at most 150 fields; for a wider table, pass subsets through `fields` (more returns HTTP 400 before the model is called). The call needs the same permission as reading a pipeline, is recorded as a `pipeline / protect-suggest` event in the [audit log](/audit-log) (field names only, failed model calls included), and returns HTTP 400 with `{"error": "..."}` when AI is not configured.

Names can mislead, so treat the answer as a proposal. From an agent, the `suggest_field_protection` tool returns one line per field; the agent shows them to you and asks which to accept before passing any of them as `protect` to `create_pipeline`.

## HIPAA Safe Harbor preset

For health data, set `"protection": {"preset": "hipaa-safe-harbor"}` at the top level of the pipeline instead of classifying every column by hand. The preset recognises the 18 identifier classes of the HIPAA Safe Harbor method (45 CFR 164.514(b)(2)) from field **names**, using a fixed table built into the server. No model is called, so the same names always give the same answer.

<Warning>
  The preset is an aid to applying Safe Harbor, not a certification of compliance. It protects the fields it recognises by name and tells you what it could not decide. Whether a data set is de-identified is still your determination. Expert Determination and business associate agreements are out of scope.
</Warning>

### Classes and default methods

Before a name is looked up it is lowercased, split on separators and camelCase, and a leading word such as `patient`, `pt`, `member`, `subscriber` or `guarantor` is dropped, so `patient_email`, `memberEmail` and `email` are the same field. The rest of the name must match one of the table's names exactly, so `name_of_drug`, `zip_file`, `phone_model` and `date_format` are not classified. A trailing number is ignored (`phone2` is a phone).

| Class | Safe Harbor identifier | Names recognised (examples) | Default method |
| - | - | - | - |
| `name` | Names | `name`, `first_name`, `last_name`, `full_name`, `middle_name`, `middle_initial`, `maiden_name`, `contact_name`, `emergency_contact` | `redact` |
| `geographic` | Geographic subdivisions smaller than a state | street, billing and shipping address, `addr`, address line, `apt`, `city`, `county`, `precinct`, `place_of_birth`, `birthplace` | `redact` |
| `geographic` | ZIP code | `zip`, `zip_code`, `postal_code` | `mask` keeping the first 3 (`first3`) |
| `geographic` | ZIP+4 extension | `zip4`, `zip_plus_4`, `zip_plus_four`, `plus4`, `zip_ext`, `zip_code_4`, `postal_code_4`, `zip_code_ext`, `zip_suffix` | `redact` (the extension alone would keep three of its four digits under `first3`) |
| `geographic` | Geocodes | `latitude`, `longitude`, `lat`, `lng`, `lon`, `geolocation`, `coordinates` | `drop` |
| `date` | All elements of dates (except year) directly related to an individual | `dob`, `date_of_birth`, `birth_date`, `dos`, admission, discharge, death (`admitted_at`, `discharged_at`, `died_on`), service, visit (`visit_dt`), appointment (`appt_date`), lab and collection dates | `mask` keeping the year |
| `phone` | Telephone numbers | `phone`, `phone_number`, `telephone`, `cell`, `mobile`, `mobile_no`, `home_phone`, `guardian_phone` | `redact` |
| `fax` | Fax numbers | `fax`, `fax_number` | `redact` |
| `email` | Email addresses | `email`, `email_address` | `redact` |
| `ssn` | Social security numbers | `ssn`, `social`, `social_security_number`, `ssn_last4`, `last4_ssn`, `ssn4` | `drop` |
| `mrn` | Medical record numbers | `mrn`, `medical_record_number`, `chart_number` | `hmac` |
| `health_plan` | Health plan beneficiary numbers | `health_plan_id`, `beneficiary_id`, `member_id`, `member_number`, `subscriber_id`, `subscriber_number`, `insurance_id`, `policy_number`, `group_number`, `medicare_id`, `medicaid_id` | `hmac` |
| `account` | Account numbers | `account`, `acct`, `account_number`, `account_id`, `bank_account`, `card_number`, `credit_card_number`, `claim_number`, `prescription_number`, `routing_number`, `account_number_last4` | `hmac` |
| `license` | Certificate/license numbers | `license`, `license_number`, `licence_number`, `certificate_number`, `drivers_license` | `hmac` |
| `vehicle` | Vehicle identifiers and serial numbers, including license plates | `vin`, `license_plate`, `plate`, `vehicle_serial_number` | `hmac` |
| `device` | Device identifiers and serial numbers | `device_id`, `device_serial_number`, `serial_number`, `serial_no`, `udi`, `imei` | `hmac` |
| `url` | Web URLs | `url`, `website`, `website_url`, `web_url` | `redact` |
| `ip` | IP addresses | `ip`, `ip_address`, `ipv4`, `ipv6` | `redact` |
| `biometric` | Biometric identifiers, including finger and voice prints | `fingerprint`, `biometric_data`, `voiceprint`, `retina`, `retina_scan`, `iris`, `face_id`, `signature` | `drop` |
| `photo` | Full-face photographs and comparable images | `photo`, `photograph`, `face_photo`, `headshot`, `picture`, `image`, `image_url`, `photo_url` | `drop` |
| `other_id` | Any other unique identifying number, characteristic or code | `national_id`, `unique_identifier`, `passport_number`, `tax_id`, `tin`, `ein` | `hmac` |

Stable identifiers use `hmac` so the column still joins. Phone and fax numbers are redacted rather than masked to the last four digits, because a partial number is not Safe Harbor. The name is looked up with its leading prefixes dropped one at a time, longest first, so `member_id`, `patient_member_id` and `subscriber_number_2` are health plan numbers. A bare `id` (including `patient_id` once the prefix is dropped) is **not** classified, so key columns are not swept up; `other_id` needs an explicit word such as `national_id` or `unique_identifier`. Ages are left unclassified, as are free-text columns. Person-name variants such as `username`, `display_name`, `doctor_name`, `provider_name`, `first` and `last` are not classified either; protect them yourself if they name a patient or a relative. Generic and event dates (`date`, `created_at`, `updated_at`, `timestamp`, `start_date`, `order_date`, `due_date`) are deliberately left unclassified so the preset does not destroy operational timestamps; the proposal lists them in a review note.

### What the preset does to common column names

The defaults are chosen for health data. If you opt in on another kind of data set, expect these effects on ordinary-looking columns:

* **Dropped** (the column is removed): `image`, `picture`, `image_url`, `photo`, `photo_url`, `profile_image`, `lat`, `lng`, `lon`, `latitude`, `longitude`, `signature`, `ssn` and its variants.
* **Redacted** (`[REDACTED]`): a bare `name`, `city`, `address`, `cell`, `mobile`, `phone`, `email`, `url`, `website`, `ip`.
* **Pseudonymised** (`hmac`, still joins): `account`, `acct`, `license`, `plate`, `serial_number`, `member_id`, `mrn`, `card_number`.
* **Year only**: `dob`, `dos`, admission, discharge, visit and appointment dates.

At save these show up as refusals you can answer with `protect` or `presetExempt`. Schema evolution is different: a **new** column in one of these classes is protected, or removed for a drop class, in the same run, and the run's status line is the only trace. Exempt such a name in advance if you need it in the clear.

### Proposal endpoint

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/pipeline/protect/preset \
  -H "x-api-key: <key>" -H "Content-Type: application/json" \
  -d '{"preset": "hipaa-safe-harbor", "fields": [{"name": "patient_name", "type": "string"}, {"name": "dob", "type": "string"}, {"name": "zip", "type": "string"}, {"name": "visit_count", "type": "int"}]}'
```

Pass `{"preset", "pipeline": "<name>"}` to use a saved pipeline's source schema, or `{"preset", "fields": [...]}`. The response has `fields` (each recognised field with `name`, `class`, `method`, `preserve`, `reason`, and `current`, the field's existing `protect` or `null`), `unclassified` (the names the table did not recognise, for a person to look at), and `review` (notes for rules a method cannot express). Each proposed method follows the [type rules](#type-rules) the same way [suggestions](#suggestions) do: a `keyFields` column is only proposed `hmac` or nothing, and a field whose source or destination type is not `string` is only proposed `drop` or nothing; `method` is `"none"` in that case and `reason` says why. `_json` and `_xml` are skipped, and one call takes at most 150 fields. With `pipeline`, a JSON, XML or unstructured pipeline is refused with HTTP 400 rather than answered with an empty proposal, because the preset cannot see its keys; pass the keys to check as `fields` instead. The call needs the same permission as reading a pipeline. It is stateless, saves nothing and is not recorded in the audit log, because it reads no data and sends nothing out. For fields the table does not recognise you can still ask the [suggestions](#suggestions) endpoint; the preset never calls it.

### Review notes

The proposal adds a note instead of pretending a method covers these rules:

* **Ages over 89.** Safe Harbor requires ages over 89 (and the dates or years that reveal them) to be aggregated into a single category of 90 or older. The preset does not aggregate ages; the note appears when an age or date field is present.
* **Small-population ZIP prefixes.** The first three ZIP digits must be replaced with `000` where the area they cover holds 20,000 people or fewer. `first3` keeps the first three digits and does not check population; the note appears when a ZIP field is present.
* **Free text.** Columns such as `notes`, `comments` or `description` may contain identifiers inside their values. The preset classifies by name only; the note names those columns.
* **Other dates.** Unclassified columns whose name ends in `date`, `dt`, `at`, `time` or `timestamp` (`created_at`, `start_date`, `event_time`) may be dates related to an individual. The preset did not mask them; the note names them so you can protect the ones that belong to a patient.

### Enforcement

When a pipeline sets the preset:

* **At save**, a field the table recognises that has no `protect` is refused, for example `Preset 'hipaa-safe-harbor': field 'phone' looks like phone and has no protection. Add protect to it or list it under protection.presetExempt`. This applies even when no field has `protect` yet. When the field's source or destination type is not `string`, only `drop` can protect it, and the message says so. An unknown preset name is refused too.
* **Delimited sources only.** The preset is refused on a JSON pipeline, because a JSON pipeline's keys are not schema fields the preset could check or evolve: list the keys to protect with `protect` explicitly (the [proposal endpoint](#proposal-endpoint) accepts any field list), and protect any key that appears later yourself. It is refused on XML and unstructured sources, like `protect`.
* **For new columns**, a column added later by [schema evolution](/schemas#schema-evolution) that the table recognises is added with its default method (clamped by the type rules) and protected in that same run, before any later stage or destination sees it. The run status says, for example, `Preset hipaa-safe-harbor: new column 'patient_phone' protected as phone (redact)`, and the stored pipeline shows the new field's `protect`. A new column the table does not recognise is added unprotected, as on any pipeline, with a warning: `new column '<name>' is not recognised by the preset and is not protected`. A new column listed under `presetExempt` is added unprotected with an info line. A recognised new column whose default method the type rules refuse (for example a `keyFields` column given anything but `hmac`) lands as is with a warning, and the next save of the pipeline is refused until you protect it or exempt it.

To keep a recognised field in the clear on purpose, list it under `presetExempt`:

```json theme={null}
"protection": {
  "preset": "hipaa-safe-harbor",
  "presetExempt": ["admission_date"]
}
```

Without the preset, a column added by schema evolution to a protected pipeline lands unprotected; that is the main reason to set it.

The pipeline wizard applies the preset without writing JSON; see [In the wizard](#in-the-wizard). An agent does the same through MCP with `suggest_field_protection` (`preset`) and `create_pipeline` (`protect_preset`, `protect_exempt`).

## In the wizard

In the pipeline wizard's **Source Schema** step, every field has a **Protect** select (None, `hmac`, `mask`, `redact`, `drop`, `encrypt`), and choosing `mask` adds a **Keep** select (mask all, last 4, email domain, year, first 3). A choice the [type rules](#type-rules) do not allow, such as `hmac` on an `int` field, switches back to None with the reason shown under the field. A column you chose as a key field is offered only None and `hmac` (a suggestion for any other method is declined with the reason), and a field protected with anything but `hmac` is left out of the destination's **Key Fields** list.

**Suggest protection** sends the field names and types only, never data, to the [suggestions](#suggestions) endpoint and fills each empty Protect select with its suggestion, marked with the reason until you **Keep** it (which applies the suggestion, even over a value you had already chosen) or **Clear** it; nothing is saved until you save the pipeline. JSON and XML pipelines show no Protect column, so set `protect` on a JSON pipeline's top-level keys in the config JSON instead. The pipeline's detail page lists its protected fields and their methods.

On a delimited (CSV) source, the **Preset** select beside **Suggest protection** offers **HIPAA Safe Harbor**. Choose it and click **Apply**: the field names and types go to the [proposal endpoint](#proposal-endpoint), every recognised field you have not set yet gets the preset's method, and each recognised field shows its class beside the Protect select. A field you had already set keeps your value and shows what the preset proposes, with **Keep** to take it or **Dismiss** to leave yours; a proposal the [type rules](#type-rules) refuse is shown as not applied with the reason. Below the fields, the wizard lists the fields the preset did not recognise and the [review notes](#review-notes); read both, because the preset is an aid, not a compliance determination. **Keep the preset enforced on this pipeline** is ticked by default and saves `protection.preset`, so later saves and new columns stay covered (see [Enforcement](#enforcement)); untick it to keep the methods without the preset. While the preset is enforced, a recognised field set to None shows an **Exempt** toggle, which adds it to `presetExempt`. The summary lists any recognised field that is None and not exempt, because the save would be refused, and an exemption for a field no longer in the schema is dropped on save. Applying the preset again leaves exempt fields at None. Reopening the pipeline restores the preset, the exemptions and the class labels without changing the saved methods, and the detail page shows `Preset: HIPAA Safe Harbor (enforced)` with any exempt fields.

## Where it runs

The stage runs once per job, **after the [preprocessor](/preprocessor)** and **before** data quality, transformation, Live Read, and every destination. The preprocessor still sees raw values; if your preprocessor endpoint must not receive them, protect the data upstream or do not use a preprocessor on that pipeline.

Each run that protects data adds one status line such as `Protected 3 fields: mrn=hmac, email=mask:domain, ssn=drop`. Column-level lineage shows protected fields as derived with the method as evidence, and dropped fields as dropped.

## Type rules

* `hmac`, `mask`, `redact`, and `encrypt` produce a string, so the field must be `string` in the source schema and, when a destination schema lists it, `string` there too.
* `drop` works on a field of any type.
* A column named in the destination's `keyFields` may only use `hmac`: `mask` and `redact` would give different rows the same key (so upserts would merge them), `encrypt` gives the same value a different ciphertext on every run (so upserts would never match), and `drop` would remove the key.
* The source must be delimited (CSV and similar) or JSON. For JSON pipelines the source schema lists `_json` plus one entry for each protected **top-level** key; nested keys are not supported yet.
* Unknown methods, `preserve` on anything but `mask`, and the reserved methods `fpe` and `tokenize` are rejected when the pipeline is saved. Like other validation errors, the save returns HTTP 400 with `{"error": "..."}` naming the field.

## The key

`hmac` uses one key per environment, stored in Vault as the secret `<environment>/field-protection`, field `key`. The server issues it the first time a protected run needs it and records an `key / issue / field-protection` event in the [audit log](/audit-log).

The key is read fail-closed: if Vault cannot be read, the run fails rather than writing data with a new key. Datris never rotates the key automatically. Rotating it yourself changes every pseudonym, so new values will no longer match or join with data already landed.

`encrypt` uses separate, versioned keys in the same secret: fields `enc.v1`, `enc.v2`, and so on, with `encCurrent` naming the version new values are encrypted with. The server issues `enc.v1` the first time an `encrypt` run needs it (audited as `key / issue / field-protection`). Keys never leave the server and no user ever holds one. See [Key rotation](#key-rotation).

The `field-protection` secret cannot be referenced by a tap or a pipeline.

## Revealing a value

`encrypt` is the one reversible method. To read an original value back, send the ciphertexts you read from the destination to the reveal endpoint:

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/protect/reveal \
  -H "x-api-key: <key with protect:reveal>" -H "Content-Type: application/json" \
  -d '{"pipeline": "patients", "field": "phone", "values": ["enc:v1:Qm9…x3A"]}'
```

The response has one slot per input value, in order:

```json theme={null}
{"pipeline": "patients", "field": "phone", "values": ["555-0142"], "revealed": 1, "failed": 0, "errors": []}
```

* **Who can call it:** an API key granted `protect:reveal`, or an admin session. No key template and neither the editor nor the viewer role carries it, so it must be granted explicitly in the API Keys tab (the `full-access` template and legacy unscoped keys hold it through `*:*`). There is no MCP tool, Assistant action, or CLI command for it (the only UI path is the Search screen below, which calls this same endpoint), and the recovery agent's key does not have it, so an agent can never reveal a value. The reveal and rotate endpoints enforce their capability themselves, so `CAPABILITY_ENFORCEMENT=log-only` does not open them.
* **What is checked:** the field must carry `protect` with method `encrypt` (otherwise HTTP 400). Each ciphertext is bound to its pipeline and field name, so a value copied to another pipeline or column does not reveal: that slot is `null` and `errors` gets `{"index", "message"}`, and the rest of the call still succeeds.
* **Limit:** at most 1000 values per call (more returns HTTP 400).
* **What is logged:** every call is a `protect / reveal` event in the [audit log](/audit-log) naming who called, the pipeline, the field, and how many values were requested, revealed, and failed (outcome `warning` when any failed). The values themselves and the ciphertexts are never logged or recorded. A call without the capability is refused with HTTP 403 and recorded as a `security` denied event.

Reveal is stateless: Datris never queries a destination to find the values, so you choose exactly which rows to reveal.

**From the Search screen.** In Search → Traditional, a cell holding a ciphertext shows a lock badge with its key version (`enc:v2`) instead of the base64. When the result set has encrypted cells, admins (or anyone, when the install uses API keys without user logins) see **Reveal encrypted values** in the results header; editors and viewers see the badges but no button. One click reveals every encrypted cell on screen through the endpoint above, one call per encrypted column (split at 1000 values), so the audit log gets one `protect / reveal` entry per column per click. Revealed cells show the plaintext with an open-lock badge; a value the server could not reveal keeps its lock and shows the server's reason on hover. Databricks, Snowflake, and object-store queries use the pipeline selected in the form. For PostgreSQL and MongoDB the pipeline is found from the table or collection you queried; if no pipeline or more than one writes it, pick the pipeline beside the button. Without the capability the screen shows "You do not have the protect:reveal capability" and nothing is retried. Nothing is stored: plaintext stays only in the page, and running the query again shows the ciphertext.

## Key rotation

To start encrypting with a new key, rotate it with a key that has `protect:admin` (or an admin session):

```bash theme={null}
curl -X POST http://localhost:8080/api/v1/protect/keys/rotate -H "x-api-key: <admin key>"
```

The response is `{"version": 2}`. Runs after this write `enc:v2:` values. The old versions stay in the secret, so rows already landed with `enc:v1:` still reveal; Datris does not re-encrypt landed rows. The rotation is recorded as a `key / rotate / field-protection` event. The `hmac` key is never rotated by this endpoint.

To retire an old version, save the `field-protection` secret (Secrets tab or `PUT /api/v1/secrets/field-protection`) without its `enc.v<n>` field, using a key that has `protect:admin`. Never remove the current version's field: `encCurrent` must keep naming a stored version. The highest version number cannot be removed, and at least one version must remain. After that, every value encrypted with that version can no longer be revealed; re-landing those rows from the original source is the only way back. The rotate endpoint never reuses a retired version number.

### Editing the key secret through the API

The secrets API guards `<environment>/field-protection`:

* **`key` (the hmac key)** cannot be changed or removed through the API by anyone. The request fails with `409` and nothing is written. Changing it changes every hmac pseudonym, so the only supported path is a deliberate edit of the secret in Vault.
* **Adding, changing or removing an `enc.v<n>`, or changing `encCurrent`** needs `protect:admin`. A key with only `secret:write` gets `403` and a `security / denied` audit entry; nothing is written. Each `enc.v<n>` value must be 64 hex characters (a 32-byte AES-256 key) and `encCurrent` must name a stored version; otherwise the request fails with `400`. The next run after an accepted edit encrypts with the new current version; no restart is needed.
* **Accepted edits are audited** with the changed field names only, never the values: `key / rotate / field-protection` when an `enc.v<n>` was added or changed, `key / retire / field-protection` when one was removed, and `key / set-current / field-protection` when only `encCurrent` moved.
* **A save that leaves every key field as it is** (fields sent back masked or blank) works with plain `secret:write`, so the Secrets tab can round-trip the secret.
* **Deleting the secret** (`DELETE /api/v1/secrets/field-protection`) is refused with the same `409` for everyone: the next protected run would issue a new hmac key and every landed ciphertext would be orphaned.
* **Creating the secret** through the API is refused with `409`: the server issues it on the first protected run.
* **Field names** are limited to `key`, `encCurrent`, `enc.v<n>` (with `n` from 1 to 1000000, no sign or leading zero) and `createdByKeyLabel`. Any other field, including look-alikes such as `Key` or `enc.v05` and a `_type` tag, fails with `400` for every caller. `encCurrent` follows the same number rule.
* **Version numbers are never reused.** Retiring removes older versions only: removing every `enc.v<n>`, or the highest version number, fails with `400`. A new `enc.v<n>` must be numbered above the highest stored version; re-adding a retired lower number fails with `400`.

## Source retention

Protecting the loaded data is not enough if the raw file stays on disk or in a bucket. As soon as the protected copy exists, Datris deletes:

* the raw staged file, including the payload a preprocessor received as input;
* the ingest object, when the run came from a bucket drop, a Kafka temp object, or an archive drop (the archive and its extracted files both go).

`POST /pipeline/upload` streams the file straight into staging and writes no bucket object, so for uploads only the staged file applies.

Each purge is recorded as a `pipeline / purge-source` event in the [audit log](/audit-log). A purge that fails is a warning on the run, never a failed run.

Because the raw source is gone, a later failure in the same run cannot be replayed from Datris: re-upload the file or re-run the tap.

To keep the ingest object, set `"protection": {"purgeSource": false}` at the top level of the pipeline. The staged file is still deleted. Pipelines without any `protect` field never purge anything.

## What still sees raw data

Field protection applies inside a pipeline run. A few helpers that write configuration or code read your data before protection applies, or read columns you have not protected:

* schema generation from an uploaded file (`POST /pipeline/generate`, the wizard's Generate Schema and sample-file analysis);
* data profiling (`POST /pipeline/profile` and the `profile_data` MCP tool);
* JSON Schema and XSD generation from a pasted sample (`POST /config/generate-schema`);
* files attached to an Assistant chat;
* the sample rows the CodeGen [data-quality rule](/data-quality/ai-rules) and [transformation](/transformation/ai-transformation) generators put in their prompt (protected columns are already protected there; the other columns are sent as they are);
* the AI fix suggestion and recovery-agent diagnosis for a failed pipeline run, whose error text can quote row values (data-quality failure reasons, a script's error output).

If you handle regulated data, set `DATRIS_AI_SAMPLE_VALUES=false` on the datris service and recreate it. With the switch off, none of these helpers sends a row value to a model. Each one works from structure only and says so in its response:

| Helper | With `DATRIS_AI_SAMPLE_VALUES=false` |
| - | - |
| Schema generation (CSV) | No model call. Every field is typed `string` and named from the header, or `column_1`, `column_2`, … when `header` is false or line 1 looks like data; the generated config then sets `csvAttributes.header` to `false` so a pipeline saved from it does not skip its first data row, and the pipeline wizard turns its **File has header row** checkbox off and says so. The response carries `valuesWithheld: true` and the wizard shows a note |
| Data profiling | The model gets per-column statistics (type, count, nulls, distinct count, value lengths) for CSV, or a value-free skeleton for JSON and XML. `sampleValues` are empty and the response carries `valuesWithheld: true`. If the model declines, errors or times out, the response holds the server's statistics only, with a `note` |
| JSON Schema / XSD generation | The model gets a skeleton of the sample (keys or element and attribute names, value types, no values). The response carries `valuesWithheld: true`. Formats, enums and patterns that could only be read from values are not inferred |
| Assistant attachments | The chat gets the filename, type, row count, numbered CSV columns (line 1 is never sent) and JSON top-level key names. The attached file itself is unchanged, so the Assistant can still upload it to a pipeline |
| CodeGen rules and transformations | The prompt carries the column names and schema types (or a skeleton for JSON and XML) and states that sample rows were withheld. The generated script still runs on the full data. An instruction that depends on what values look like should spell the format out |
| Failed-run fix suggestions and recovery diagnosis (pipelines) | The fix-suggestion prompt and the recovery agent's initial incident prompt get only exception class names, stack frames and the server's own fixed message text. Data-quality failure reasons and script output are replaced by `<details withheld by configuration>`. Anything the recovery agent or the Assistant then reads through its tools (run status, logs, job details) is **not** scrubbed. The run status and the stored incident keep the full text for people |

What is still sent is structure, and some structure can carry data, so these rules apply:

* **Column names from a file.** Schema generation, profiling and the CodeGen prompts take the `header` flag as a statement that line 1 is a header. With `header=true`, line 1 is sent as column names only when it looks like a header, cell by cell:

  * a cell is sent as a name only if it uses letters, digits, underscores and spaces (no dash, dot or other punctuation), starts with a letter or underscore, has at most 64 characters and at most two digits, is not all digits, and is not two or more capitalised words (a full name). Any other cell is sent as `column_N`. So `full_name`, `email`, `first name`, `addr1` and `zip_code` are sent; `First Name`, `e-mail` and `account123` become `column_N`.
  * if any cell has a dash next to a digit (`MRN-12345`, `2024-01-05`), an `@`, more than four digits in a row, more than two digits, or no letter at all (`42`, `555 1234`), the whole line is treated as data and every column is numbered.

  This check cannot recognise every value. A headerless first row made only of words still passes as names and is sent as they are: single words in any case (`John,Smith,Boston,MA`; `jane,doe,asthma,f`; `Diabetes,Type,Two`), all-caps or lower-case multi-word cells (`JOHN SMITH`, `jane doe`), and names the capitalised-words test misses (`John A Smith`, `McDonald Smith`, `O Brien`). Profile or generate a headerless file with `header=false` (the profiling endpoint defaults to `true`); then no cell of line 1 is sent. The run-time header check (`dataQuality.validateFileHeader`) is not covered by the switch: it sends line 1 to the model as it is.
* **Assistant attachments** have no `header` option, so line 1 of a CSV attachment is never sent: columns are listed as `column_1`, `column_2`, … and every line counts as a row.
* **Stored schema field names.** The field names saved in a pipeline's schema are treated as operator-entered and are always sent to the CodeGen generators. With the switch off, the `ai.sample_values` doctor check warns, naming up to five pipelines, when a stored source field name does not read as a column name.
* **JSON keys.** Key names are sent, except where an object reads as a map keyed by data: a key that has a dash next to a digit, an `@`, more than four digits in a row or more than two digits, that is two or more capitalised words, or that uses punctuation other than `_`, `.`, `-` and space; more than 50 keys; two or more keys whose values are all objects of the same shape (records keyed by a person's name or an id); or, below the top level, two or more keys whose values are all arrays (`{"patients": {"smith": [..], "jones": [..]}}`). Such an object is sent as `{"<key>": <value shape>}`. Otherwise keys that are plain words, including `john.smith` or `jane-doe`, are sent: a map keyed by names is collapsed only when its values are objects of one shape (or, below the top level, all arrays). A map keyed by plain names whose values are strings, numbers or booleans is sent with its keys, at the top level (`{"smith": "a", "jones": "b"}`) or nested (`{"scores": {"smith": 1, "jones": 2}}`), because a record's own fields (`"address": {"city": .., "zip": ..}`) have exactly that shape. Keys such as `zip-code` or `content.type` are kept.
* **XML.** Element and attribute names are sent. Text, attribute values and namespace URIs are not (namespace prefixes are kept).
* **Unparseable files** are described only by their type and size, never by their content.

The default is `true` (today's behaviour). Any value other than `false` counts as `true` and is logged once, so set `false` explicitly. The `ai.sample_values` [doctor](/doctor) check reports the effective value and warns when a pipeline protects fields while the switch is on.

The switch does not cover features whose purpose is to answer questions over your data. They send that data to the model by design:

* Search chat and the answer endpoint;
* the Assistant reading query results or a Live Read preview;
* the Assistant or the recovery agent reading a run's status, logs or job details with their tools (those records keep the full error text);
* tap script generation, review and diagnosis, and fix suggestions for failed tap runs, which see a tap's captured output or the source's documentation rather than pipeline rows.

For these, field protection is the control: protected columns are already ciphertext, tokens or masks by the time they are read. For anything else in those flows, use an AI provider you have an agreement with for that data.

For every place each kind of data can go, including embeddings and connected agents, see [Data Flows](/production/data-flows).

## Not yet supported

* Format-preserving encryption and tokenization.
* Nested JSON keys.
* XML sources.
* Vector destinations.


# User Authentication
Source: https://docs.datris.ai/user-auth

Enable username/password login, single sign-on (OIDC), admin gating, and per-user roles for the Datris UI.

By default, the Datris OSS install runs single-tenant with no login screen — anyone who reaches the UI URL has full access. For shared installs (a team running Datris on a server, a managed deployment, anything past your laptop), turn on user authentication so the UI requires a login and the Configuration tab is admin-only.

## What it does

When `USE_USER_AUTH=true`:

* The UI shows a **login screen** before any tab is reachable.
* Sensitive **Configuration** sub-tabs (Secrets, API-Keys, [Audit Log](/audit-log), [Agent Policy](/agent-policy)) are gated to **admin** users. The AI Providers, Data Sources, and Code Repository sub-tabs remain visible to all roles. The Users and API-Keys sub-tabs appear based on the active flags (see the [matrix](/api-keys#supported-configurations) for full visibility rules).
* A new **Users** sub-tab appears under Configuration for adding, removing, and changing roles (admin-only by virtue of being inside Configuration).
* Sessions are server-side, with a TTL index cleaning them up automatically.

When the flag is off, the toggle has zero effect on data — pipelines, taps, secrets, and run history all work the same.

## Enabling it

1. Read and save the seed account's bootstrap password. The server generates it at random when it seeds `admin` (on the first boot, whether or not `USE_USER_AUTH` is on) and prints it once, in the log of that container only. Recreating the container (step 3) discards that log, so read it first:

   ```bash theme={null}
   docker compose logs datris | grep "Bootstrap login"
   ```

   If nothing is printed, the container has been upgraded or recreated since `admin` was seeded and the line is gone; see [Recovering admin access](#recovering-admin-access). If you already set your own `admin` password, use that.

2. Add the flag to your `.env`:

   ```bash theme={null}
   USE_USER_AUTH=true
   ```

3. Recreate the Datris container:

   ```bash theme={null}
   docker compose up -d --force-recreate datris
   ```

4. Open the UI. You'll see the login screen.

5. Log in as `admin` with the password from step 1, then change it right away (your username in the top-right → **Change password**, minimum 5 characters). After that, `admin` is your administrator account.

<Tip>
  A new install can switch on login, API keys, the audit log and the agent policy in one step: run the installer with `DATRIS_GOVERNED=1`. See [Switch on the governance controls](/quick-start#2-switch-on-the-governance-controls).
</Tip>

<Note>
  The default `admin` row is seeded on **every startup** (idempotent — only when the user table is empty), regardless of whether `USE_USER_AUTH` is on, with a fresh bootstrap password printed once to that container's log. The line does not survive an upgrade or a recreate of the container, so save it when you first install. That means flipping the flag on later "just works" — there's no separate provisioning step.
</Note>

## Roles

| Role | Can read pipelines / data | Can edit pipelines / taps / secrets | Can manage users + Configuration |
| - | :-: | :-: | :-: |
| **admin** | ✓ | ✓ | ✓ |
| **editor** | ✓ | ✓ | |
| **viewer** | ✓ | | |

All roles see the **Configuration** tab in the top nav, but the sub-tabs differ: admins see Secrets, API-Keys, Audit Log, and Agent Policy; editors and viewers see only AI Providers, Data Sources, Code Repository, and Users. Everything outside Configuration (Pipelines, Taps, Search, etc.) is accessible to all roles per the read/write rules above.

## Adding users

As an admin: **Configuration → Users → Add user**. Fill in username + role and an initial password (type one, or click the generate button for a strong random one). Send the user their username and password out-of-band, and ask them to change it after their first login. Through the API, a `POST /api/v1/auth/users` body that omits `password` generates a random temporary password and returns it once as `temporaryPassword`.

You can change a user's role or delete them from the same screen. Datris won't let you remove the **last admin** — there's an `adminCount` guard on delete and role-change.

## Changing your own password

Click your username in the top-right → **Change password**. Enter your current password and the new one (min 5 chars).

If you've forgotten your password and there's another admin: have them delete and re-add your user with a new initial password and send it to you; log in with it and change it.

If you're the **only** admin and you've lost the password, see [Recovering admin access](#recovering-admin-access).

## Recovering admin access

Use this when you are the only admin and cannot log in: you lost the password, or the `admin` bootstrap password is no longer in the server log (it is printed once, and an upgrade or recreate of the container discards it).

At startup the server gives every account that has no password a new random bootstrap password and prints it once. So clear the account's password in MongoDB, then recreate the server. With the bundled MongoDB and the default environment `oss`, users live in the `oss-user` collection of the `oss` database (the server's internal database):

```bash theme={null}
docker compose exec mongodb mongosh oss --quiet \
  --eval 'db.getCollection("oss-user").updateOne({username: "admin"}, {$set: {passwordHash: null}})'
docker compose up -d --force-recreate datris
docker compose logs datris | grep "Assigned a bootstrap password"
```

The last line prints `User 'admin' had no password set ... Assigned a bootstrap password: <password>`. Log in as `admin` with it and change it right away. Other users are untouched. Until you recreate the server, the cleared account cannot log in at all.

With the [audit log](/audit-log) on, the rotation is also recorded as `user` / `rotate-password` on the account, with the `system` actor and the time, just before that boot's `system` / `start` entry. The password itself is never in the audit log: it was printed to the log of the server container that started at that time, so if you find a `rotate-password` row you did not expect, that container's log is where the password went. An admin clearing a password from **Configuration → Users** is recorded separately as `user` / `reset-password`, with the admin as the actor.

## Single sign-on (OIDC)

With user authentication on, Datris can also sign people in through your company identity provider (Okta, Microsoft Entra ID, Google Workspace, Keycloak, Auth0, or any OpenID Connect provider). The login screen gains a **Sign in with SSO** button next to the username and password form. Sign-in uses the OIDC authorization code flow with PKCE.

Someone who signs in this way becomes an ordinary Datris user with one of the three roles above, so sessions, role checks, the Users screen and the [audit log](/audit-log) work exactly as they do for password users. API keys, the CLI and MCP clients are unaffected: SSO is for people using the UI.

### MFA and password rules

Datris does not run its own multi-factor authentication. For people who sign in with SSO, MFA, password rules and lockout are whatever your identity provider enforces, so turn MFA on there. Local accounts (including `admin`) still use a password with a minimum length only and no lockout.

### Settings

| `.env` variable | `application.yaml` key | Default | Meaning |
| - | - | - | - |
| `OIDC_ENABLED` | `oidc.enabled` | `false` | Switches SSO on. Only takes effect with `USE_USER_AUTH=true`. |
| `OIDC_ISSUER` | `oidc.issuer` | (empty) | The provider's issuer URL. It must equal the `issuer` in the provider's `/.well-known/openid-configuration` document exactly, trailing slash included. |
| `OIDC_CLIENT_ID` | `oidc.clientId` | (empty) | The client (application) id registered at the provider. |
| `OIDC_REDIRECT_URI` | `oidc.redirectUri` | (empty) | Where the provider sends people back: `https://<your-ui-host>/api/v1/auth/oidc/callback`. Register exactly this value at the provider. |
| `OIDC_SCOPES` | `oidc.scopes` | `openid email profile` | Scopes requested. |
| `OIDC_USERNAME_CLAIM` | `oidc.usernameClaim` | `email` | The ID-token claim whose value becomes the Datris username. |
| `OIDC_DEFAULT_ROLE` | `oidc.defaultRole` | (empty) | Role for a person who has no Datris user yet: empty refuses them, `viewer` or `editor` creates them with that role. `admin` is not accepted and counts as empty. |
| (none) | `oidc.secretName` | `oss/oidc` | Vault secret holding the client secret. |

The client secret is never put in `.env` or the compose file. Store it in Vault under `oss/oidc`, key `clientSecret`:

```bash theme={null}
docker compose exec -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN="$(docker compose exec -T datris cat /vault-token/token)" vault \
  vault kv put secret/oss/oidc clientSecret=<client-secret>
```

No tap or pipeline can reference this secret.

At the provider, register a **confidential** client (one with a client secret) that uses the authorization code flow, with `/api/v1/auth/oidc/callback` on your UI host as its redirect URI, and PKCE method `S256` if the provider asks. The client authenticates to the token endpoint with HTTP Basic (`client_secret_basic`). Recreate the Datris container after changing any setting:

```bash theme={null}
docker compose up -d --force-recreate datris
```

The server checks the settings once at startup, without contacting the identity provider. If `OIDC_ISSUER`, `OIDC_CLIENT_ID` or `OIDC_REDIRECT_URI` is missing, the issuer or redirect URI is not an http(s) URL, or the Vault secret `oss/oidc` (key `clientSecret`) is not there, the server still starts and password login keeps working: SSO is switched off, the reason is logged once at ERROR, and the button is hidden. `OIDC_ENABLED=true` without `USE_USER_AUTH=true` also leaves SSO off (logged once at WARN).

An identity provider that is down or unreachable is different: the server does not contact it until someone presses the button, so it starts normally and the **Sign in with SSO** button stays visible. Each press then fails: the person lands back on the login screen with a "could not be completed" message (or, if the provider went down after a recent successful sign-in, the browser cannot reach the provider's sign-in page), and the server logs one ERROR line per failed attempt. Password login is unaffected. SSO works again as soon as the provider is reachable; no restart is needed.

### How a person maps to a user and role

* **Username.** The value of the `OIDC_USERNAME_CLAIM` claim, trimmed and lower-cased, is the Datris username. It must use letters, digits and `. _ @ -` only. When the claim is `email` and the provider says the address is not verified (`email_verified: false`), sign-in is refused.
* **Existing user.** Keeps the role set on **Configuration → Users**. To let someone in with a role you choose, add them there first (for example `alice@example.com` as editor); the password you set for them is not needed for SSO.
* **Unknown person.** Refused with "ask an administrator to add you" unless `OIDC_DEFAULT_ROLE` is `viewer` or `editor`, in which case the user is created with that role and a random password nobody is told. Promote people to admin on the Users screen.
* **The recovery account.** The built-in `admin` account can never be reached through SSO; it always signs in with its password.
* **Identity binding.** The first SSO sign-in stores the provider's subject id (`sub`) on the user. A later sign-in with the same username and a different subject (an email address reassigned to someone else) is refused. To allow it, delete the user and add them again.

Every SSO sign-in, and every refusal, is recorded in the [audit log](/audit-log) as `auth` / `login` with `method: oidc` in the metadata.

### Removing access

Disable or remove the person at your identity provider. Datris does not check back with the provider during a session, so a session already open stays valid for up to its 8-hour sliding lifetime (it is extended while the person keeps using the UI). Deleting the user on **Configuration → Users** stops further SSO sign-ins only when `OIDC_DEFAULT_ROLE` is empty.

### Trying it with Keycloak

A local walkthrough on macOS with Docker Desktop. The browser and the datris container must reach Keycloak under the same host name, because that name is part of the issuer:

```bash theme={null}
grep -q host.docker.internal /etc/hosts || echo "127.0.0.1 host.docker.internal" | sudo tee -a /etc/hosts
docker run -d --name keycloak -p 8081:8080 \
  -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:26.7.5 start-dev
KC="docker exec keycloak /opt/keycloak/bin/kcadm.sh"
$KC config credentials --server http://localhost:8080 --realm master --user admin --password admin
$KC create realms -s realm=datris -s enabled=true
# Local development only: Keycloak requires HTTPS for any host other than
# localhost, and the issuer here is http://host.docker.internal. Never do this on a real realm.
$KC update realms/datris -s sslRequired=NONE
$KC create clients -r datris -s clientId=datris -s secret=datris-local-secret -s publicClient=false \
  -s standardFlowEnabled=true -s directAccessGrantsEnabled=false \
  -s 'redirectUris=["http://localhost:4200/api/v1/auth/oidc/callback"]' \
  -s 'attributes."pkce.code.challenge.method"=S256'
$KC create users -r datris -s username=alice -s email=alice@example.com -s emailVerified=true -s enabled=true
$KC set-password -r datris --username alice --new-password alice-pass
docker compose exec -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN="$(docker compose exec -T datris cat /vault-token/token)" vault \
  vault kv put secret/oss/oidc clientSecret=datris-local-secret
```

Add to `.env`:

```bash theme={null}
USE_USER_AUTH=true
OIDC_ENABLED=true
OIDC_ISSUER=http://host.docker.internal:8081/realms/datris
OIDC_CLIENT_ID=datris
OIDC_REDIRECT_URI=http://localhost:4200/api/v1/auth/oidc/callback
```

Then `docker compose up -d --force-recreate datris`. `curl -s http://localhost:8080/api/v1/version` now reports `"oidcEnabled": "true"`. Sign in as `admin` with the password, add `alice@example.com` as editor on **Configuration → Users**, sign out, press **Sign in with SSO** and log in at Keycloak as `alice` / `alice-pass`. To clean up: `docker rm -f keycloak` and remove the `OIDC_*` lines from `.env`.

<Note>
  SSO is single-tenant: with `MULTI_TENANT=true` it is not verified and signs people in to the default environment's users. One identity provider per install.
</Note>

## Reverting

Set `USE_USER_AUTH=false` (or remove the line) and recreate the Datris container. The login screen disappears; the Configuration tab is unrestricted again. **User rows persist in the database** — they just aren't checked. Flipping back on later returns you to the same set of users with the same passwords.

## Production checklist

* Run Datris **behind HTTPS** before turning auth on. Sessions are cookie-based; without TLS, the login is exposed in transit.
* Review your CORS allow-list (`cors.allowedOrigins` in `application.yaml`) — the default `"*"` is for local dev only.
* To sign people in with your company identity provider and its MFA, turn on [single sign-on](#single-sign-on-oidc).
* Pair this with `USE_API_KEYS=true` for any shared install — user auth gates the **UI**, API-key gating handles programmatic clients (CLI, MCP, external scripts). API keys are meant to be used **alongside** user auth: enable the two together. Turning on `USE_API_KEYS=true` without `USE_USER_AUTH=true` leaves the browser with no coherent auth mechanism — see [supported configurations](/api-keys#supported-configurations); the two are designed to compose.

## Related

* [API Keys](/api-keys) — the paired flag. Required for any deployment where the UI or CLI talks to a real install.
* [Configuring Claude](/configuring-claude) — connecting AI agents via MCP. MCP clients use the API-key path, not user auth.
* [Configuration Reference](/configuration-reference) — the underlying `useUserAuth`, `useApiKeys`, `multiTenant` and `oidc.*` keys in `application.yaml`.