Changelog - Datris

September 10, 2026

Datris can now diagnose its own deployment.

Doctor. A built-in operational self-check finds the problems that used to be discovered only after an outage: a Vault token about to expire or silently capped, an AI provider secret that is missing or incomplete so the server would not come back after an upgrade, an embedding model that was never pulled, a disk about to fill, and a mixed-version stack. Every finding comes with the command that fixes it; nothing is changed automatically.
Four places to run it. The cheap checks run at every server start as DOCTOR log lines (a healthy install logs no warnings). The full set runs on demand from Configuration → Doctor, from the new run_doctor MCP tool, and from the new datris doctor CLI command.
Host checks from the CLI. On the machine running Docker, datris doctor also catches data on an anonymous volume, data left behind on a dangling volume, a container still running with an old .env, containers from a previous Compose file, a stale from-source build, and an unreachable MCP server. datris doctor --pre-upgrade works with the server stopped and is step 0 of the upgrade guide.
Optional AI probe. On request, the doctor sends a minimal request through each configured AI model to confirm the key and model id still work. Never run automatically.
Version stamps. The server reports the exact build it is running and the UI image carries its version, so drift between components is visible.
Fresh-install AI defaults refreshed. New installs seed Claude Fable 5.1 (Anthropic assistant), GPT-5.6 Sol (OpenAI) and Claude Sonnet 5 (overload fallback); code generation stays on Claude Opus 5. Existing installs are unchanged.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.29.0.

September 10, 2026

Failed loads now say which destination failed, and why.

Destination failures are named in the job status. When a run fails while writing to a destination, the job’s error identifies that destination and carries its own error message instead of a generic orchestration failure with a stack trace. The full stack trace remains in the run’s event stream.
Agents and the CLI act on the real cause. The Assistant, MCP clients and the recovery agent read the same job status field, so they see the failing destination directly. The datris CLI prints the failing destination with the error on a failed upload.
Multi-destination runs still report as a single job; when one destination fails and another succeeds, the error names the one that failed.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.28.2.

September 8, 2026

Security and documentation maintenance release.

Security update. The embedded web server that serves the API is updated to close three recently disclosed critical advisories affecting authentication and authorization handling. The UI and tap-runner images pick up the latest OS package security fixes. No functional changes.
Hardening switches now take effect. DATRIS_ENV, DATRIS_ALLOW_PLAINTEXT_DB, DATRIS_ALLOW_PRIVATE_EGRESS and TAP_MAX_OUTPUT_MB were described in .env.example but not passed through to the server by the bundled Compose files. They are now honored and documented in the Configuration Reference.
Documentation accuracy pass. Every page on docs.datris.ai was checked against the current release and corrected, including the MCP tool list, CLI flags, Assistant defaults, tap runtime behaviour, ingestion and destination options, the API reference and OpenAPI spec, and the examples.
Examples. Two examples that depended on files no longer in the repository were removed.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.28.1.

September 4, 2026

Lineage now reflects what actually ran, not just what is configured — with an interactive graph and an agent tool to traverse it.

Run-level lineage. Every pipeline run is recorded as it completes: what it read, which destinations it wrote — each with its own success or failure — how many records, and the definition version it ran under. Automatic for every pipeline, independent of provenance stamping, and never fails or delays a run.
History in the graph. A dataset a pipeline used to land into under an earlier configuration stays in the lineage graph marked historical, with the runs that wrote it.
Lineage graph. Catalog → Lineage opens an interactive Source → Tap → Pipeline → Dataset → Catalog view, filterable by catalog, tag or name. Click any node for what feeds it, what depends on it, freshness and recent runs, with links to the tap or pipeline.
Recent runs per node, plus upstream-only / downstream-only / bounded-depth traversal on the lineage neighborhood endpoint.
New MCP tool: get_lineage for impact analysis before changing or deleting something, and for seeing what a pipeline actually wrote lately.
Real sources in lineage. A tap’s source node names where the data actually comes from: the tap’s new optional source field, else the endpoint host, else the host its script calls most. Taps sharing a provider share one source node.
Evidence on every lineage hop: runs, records, failures and last run per edge (last 90 days); claims without runs draw faint; numbers on the panel chips and in find_data.
Source of authority: datasets labeled system of record / derived / undeclared, declared never inferred; single destinations authoritative by default; authoritative: false for derived copies, destination.authoritative for multi-destination pipelines; find_data prefers the authoritative copy; “Canonical only” graph filter.
Column-level lineage in the graph’s side panel and via a new endpoint: exact schema-derived mappings always, system columns for provenance stamps, and an opt-in AI-inferred tier for pipelines with a transformation that returns only evidenced mappings, cached per definition version.
Fix: AI transformations that add, remove or reorder CSV columns now land correctly; the pipeline picks up the transformed column list instead of projecting by the old one.
Provenance resolution for recorded runs now includes per-destination outputs and resolves the definition version from the record when the row’s stamp is unavailable.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. Runs are recorded from the first run after upgrading. CLI users on pip or Homebrew: upgrade to 1.28.0.

September 2, 2026

Claude Fable 5.1 is now the recommended chat model, plus security and dependency updates.

Claude Fable 5.1 support. Anthropic’s newest model is selectable for both the Assistant and code generation, on the direct Anthropic provider and on Amazon Bedrock. It is the recommended chat model on both; code generation keeps Claude Opus 5. Fresh Bedrock installs default to Fable 5.1; existing installs keep their configured models. On Bedrock it needs the same one-time data-retention opt-in as Fable 5.
Security update. The UI’s build tooling picks up fixes for several recently disclosed high-severity advisories in a third-party component. No functional changes.
UI framework update to the latest Angular patch release.
CLI and MCP server via pip. The published package now pins the MCP SDK to the supported major version, so a fresh pip install datris-mcp-server no longer pulls an incompatible SDK that fails at startup.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.26.1.

September 2, 2026

Provenance on your data: any row can say which run, which configuration, which script, and which source produced it — and agents can find datasets by meaning.

Opt-in provenance stamping per pipeline: every row, document, message, and vector chunk landed carries the run that produced it, when it landed, the definition version, the tap run that fed it, the exact script commit, and the declared source. Existing rows are never rewritten — provenance starts with the first stamped run; SQL destinations gain the columns automatically.
Every stamp resolves back to its origin — the run, the feeding tap run, the script commit, the definition version with its change note, and the source — via the platform or the new get_provenance agent tool. Vector search results carry provenance per chunk so RAG answers can cite the run and source behind each passage.
A lineage view shows source → tap → pipeline → landed dataset → catalog, derived from configuration alone. Pipeline detail pages show what feeds them, where they land, and how fresh they are, using the same staleness judgment as the Activity dashboard.
Tags on taps and pipelines: free-form labels shown in the catalog and ranked by discovery.
find_data: agents discover datasets by meaning — the best-matching pipelines their key can read, each with location, freshness, provenance handles, and a ready-to-run query hint. Discovery only: the query stays the agent’s own call, under its own permissions. AI answers can carry the provenance handles along.
Deleting a tap or pipeline from the catalog now shows progress while the platform finishes cleaning up.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; provenance stamping is off by default — enable it per pipeline in the pipeline editor’s Destination step.

September 1, 2026

The recovery agent: Datris notices broken data flows, diagnoses them, and — within limits you set — repairs and verifies them.

The opt-in recovery agent opens an incident when a scheduled tap keeps failing after its automatic retries, a pipeline load errors, a tap goes quiet past its cadence, or a pipeline’s volume swings against its baseline — then diagnoses it with the same judgment as the Ops chat and closes it with a step-by-step story.
The Agent Policy gains a recovery mode: off, propose (every repair waits for one-click approval), or autopilot (repairs follow your per-action policy) — with per-tap / per-pipeline overrides.
Repairs are narrow and verified: re-run, script fix (as a new version), test — never deletes, secrets, or schema changes. Every fix must prove itself with a clean real run, or the platform reverts it and says so. Budgets are enforced by the platform, not a prompt.
A new Incidents panel on the Activity page shows state, classification, and timelines with approval links; the Ops chat explains open incidents instead of re-diagnosing; two new agent tools read them; the audit log carries the full per-incident ledger; an optional webhook reports milestones.
The MCP page’s tool catalog now lists every available tool, including the destination-typing pair missing since v1.22.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; set USE_AGENT_POLICY=true + RECOVERY_AGENT_ENABLED=true and pick a recovery mode to opt in.

August 31, 2026

Security update for the UI and MCP server images.

The datris-ui and datris-mcp-server container images ship with updated OpenSSL packages, picking up upstream fixes for several recently disclosed issues, including one rated high severity. No functional changes.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 31, 2026

Agent Policy: decide what agents may do on their own, what waits for your approval, and what is refused.

A new opt-in Agent Policy lets an administrator set, per action — run a tap, delete a pipeline, migrate destination column types, write a secret, and every other change an agent can make — whether agents do it unattended, queue it for a person to approve, or are refused. People using the UI are never gated; they are the approvers. Nothing changes until you set a rule, and the policy applies to every agent: the built-in Assistant and any connected MCP client alike.
Queued actions appear under Activity → Approvals with who asked, the reason they gave, exactly what would run, and one-click Approve / Reject. Approving performs the original request on your behalf and records the whole chain — request, decision, and result — in the audit log. Approvals go stale if the pipeline or tap changed in the meantime, expire on their own, and can never be approved by an agent.
Agents are told the truth: a gated action comes back “waiting for approval” (never silently done), a refused one says so plainly, and the chat shows matching status cards. Agents can read the policy up front, list what they have queued, and poll for your decision; every change-making agent tool now accepts a short reason that shows up on the approval card and in the audit log.
Configuration → Agent Policy manages it all, with a recommended starting point and per-pipeline / per-tap overrides that can only tighten. The Configuration → Environment tab has been removed.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; set USE_AGENT_POLICY=true to turn the agent policy on.

August 28, 2026

Audit log: who did what, by login or API key.

A new opt-in Audit Log records every create, change, run, delete, login, and denied request — humans by their login, agents by their API key, scheduled runs as system. Admins read it under Configuration → Audit Log with filters by time, category, actor, and outcome, a detail view per entry, and CSV export. Entries are also written to the server log so an existing log aggregator picks them up.
Actions the Assistant takes for you are attributed to you — in the audit log and in pipeline and tap version history — instead of to the platform’s internal key. Every issued API key now carries a stable id, so a key that is revoked and later re-issued under the same name is never confused with its predecessor.
Reads are left out by default to keep the trail focused on changes; turn them on to see every query and search an agent runs. Reading a secret is always recorded.
Tap scripts that read platform data now work when API keys are required. Each run gets a short-lived, read-only credential that is attached automatically — scripts need no changes and no platform credential in their secret — and the audit log names the tap that made each call.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; set USE_AUDIT_LOG=true to turn the audit log on.

August 27, 2026

Set real column types on pipelines that landed as text.

Pipelines created through agents store every destination column as text. The Catalog now marks such pipelines (PostgreSQL, Snowflake, and Databricks destinations) with a quiet “text” badge — click it to review proposed column types and apply them. The same action is available from the pipeline view.
Proposed types are inferred from the data already loaded, and every column shows real sample values so you can check the proposal at a glance. When a column stays text because of a stray value, the dialog shows the offending value and the type it blocked (for example: found “N/A”, would otherwise be a number) so overriding is an informed choice.
Applying is safe by design: every loaded value is validated first, the destination table is retyped, and the pipeline definition is updated as a new version. A value that won’t convert fails the whole apply with the column named and nothing changed. From then on, incoming data is type-checked on every load.
Agents get the same capability through two new MCP tools, get_dest_types and apply_dest_types — applying always requires the user’s explicit approval.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 26, 2026

Live progress feedback across the Assistant and tap building.

The Assistant shows what it’s thinking about while it reasons — a live summary streams next to the progress indicator with an elapsed timer, instead of silent dots. The full reasoning stays available in the expandable thinking block.
AI tap-script generation shows live progress: the tap wizard and the Assistant’s tool cards report what the generation is doing and how long it has been running.
The Assistant sets time expectations before long-running operations, and when AI script generation fails for a well-known API it writes the script directly instead of retrying — turning a multi-minute failure into a quick recovery.
The Assistant only reports an action as completed when it actually ran and observed the result — tightened guarantees against overstated progress reports.
Dependency updates across the UI and build tooling.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 25, 2026

Security refresh across all container images.

All four container images rebuilt on updated bases with refreshed OS packages and Python tooling, clearing every published high- and medium-severity vulnerability finding against the images.
Every container now reports its own health status, so docker ps (and any orchestrator) shows healthy/unhealthy per service out of the box.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 25, 2026

MCP tool catalog now matches API-key permissions.

With API keys enabled, each MCP session’s tool list is filtered to what its key is actually allowed to do — agents no longer see tools that could only fail with a permission error at call time. Full-access keys and installs without API keys keep the complete catalog.
The rag-builder key template gained the read permissions its workflow was missing (list taps, manage its own tap credentials). Existing issued keys are unchanged — re-issue or edit rag-builder keys to pick up the additions.
With API keys enabled, MCP sessions that present no key are refused instead of silently connecting. Installs without API keys are unaffected.
Permission denials return a clear, structured error instead of a generic server error, so agents recognize a permission boundary rather than retrying.
Tightened API-key permission enforcement on the server.
Upgrade: docker compose pull && docker compose up -d --force-recreate. If API keys are enabled, ensure MCP clients send their key (header for remote connections, DATRIS_API_KEY environment variable for stdio).

August 24, 2026

Opt-in Postgres TLS enforcement.

Deployments marked as production now refuse to start when the platform’s Postgres connection points at an external host without TLS (sslmode=require or stricter), with an explicit opt-out for deployments that intentionally use plaintext. The bundled in-network Postgres is exempt; existing deployments are unaffected unless production mode is enabled. Plaintext connections to external databases now log a startup warning either way.

Observability: Prometheus metrics and structured logs.

Prometheus metrics are now exposed for scraping, reachable only from inside the deployment’s network (the bundled edge proxy blocks them publicly). See Audit Log for the metrics endpoint.
Production deployments emit one JSON log line per event for SIEM / log-aggregator ingestion; development logs stay human-readable.

Tap isolation on by default for docker compose.

All bundled Docker Compose configurations now default to USE_TAP_RUNNER=true. The runner token is minted automatically on first boot (including upgrades where it was missing or still the placeholder value). The server fails fast if isolation is on and no valid token exists — a minted token is accepted.
Running the server outside Compose (no runner sidecar) stays in-process and logs a warning. Isolated taps still cannot open direct connections to internal DB / object store / Vault — they return records; this is existing behavior, called out because compose now isolates by default.
Set USE_TAP_RUNNER=false to keep in-process on compose. No host allowlist in this change.

August 21, 2026

Dependency security cleanup across the server and UI.

Updated server and UI dependencies to resolve every remaining published medium-severity advisory (and one low). One low-severity advisory remains open upstream with no fixed release available. No functional changes intended.
The Activity tab’s charts moved to the current major version of the charting library as part of the security updates — same charts, slightly refreshed default colors.
Standard upgrade: docker compose pull && docker compose up -d. Existing pipelines, taps, and schedules are unaffected.

August 21, 2026

Server security updates and a friendlier vector-store default in the installer.

Upgraded the server’s embedded web framework to its current supported release line, resolving all remaining published high-severity advisories against server dependencies (and a number of medium- and low-severity ones). No functional changes intended.
The installer now presents pgvector as the default vector store. It ships inside the bundled Postgres, so vector search works out of the box with no extra container. Additional vector stores (Qdrant, Weaviate, Chroma, Milvus) remain opt-in additions at install time. Existing installs are unaffected — prompts and install summary only.
Standard upgrade: docker compose pull && docker compose up -d. Existing pipelines, taps, and schedules are unaffected.

August 20, 2026

Security updates across the UI and server, and a pipeline-delete fix.

Upgraded the web UI’s underlying framework to the current long-term-support line, resolving all published critical- and high-severity advisories affecting the UI. No visual or functional changes intended.
Updated bundled server dependencies to resolve the remaining published high-severity advisories against third-party libraries. No functional changes.
Deleting a pipeline now also removes its data from the built-in object store destination, matching every other destination type. Previously the files were left behind, and recreating a pipeline with the same output prefix silently picked them up. Shared output prefixes are detected and safely skipped. Files orphaned by deletes on earlier versions are not removed retroactively.
Hardened the UI container image configuration. No behavior change.
Standard upgrade: docker compose pull && docker compose up -d. Existing pipelines, taps, and schedules are unaffected.

August 20, 2026

Dependency security updates and an install fix.

Updated bundled server dependencies to resolve all critical published advisories against third-party libraries, and removed an unused legacy dependency tree from the server entirely. No functional changes.
Fresh installs now create the default object-store output bucket automatically — previously the first pipeline writing to the built-in object store with default settings failed until the bucket was created by hand.
Standard upgrade: refresh your checkout (or re-download docker-compose.standalone.yml), then docker compose pull && docker compose up -d. If you update images without refreshing the Compose files, create a bucket named {environment}-data (default: oss-data) once in the MinIO console.

August 19, 2026

Security hardening.

A broad security pass across authentication, authorization, credential handling, and input validation, plus updated bundled dependencies for published advisories. A standard upgrade needs no configuration changes.
Initial admin login (when user login is enabled): the first admin password is now generated and printed once to the server logs — docker compose logs datris, sign in, and change it.
Taps calling internal addresses are blocked by default, with an opt-out for endpoints that legitimately run on an internal network (see the Tap Execution & Isolation guide).
Bundled Vault uses a unique per-install token; for manual vault commands read it with docker compose exec -T datris cat /vault-token/token.
Fix: the default Docker Compose config pulls the published images again (a recent change had it building from source).
Standard upgrade: docker compose pull && docker compose up -d.

August 18, 2026

Build taps in any language — HTTP taps.

HTTP taps. A tap can now be a service you host — Rust, Go, TypeScript, a serverless function, or an existing internal system. Datris calls your endpoint on every run and everything else works exactly like a Python tap: schedules, per-run params, incremental sync state, run history, retries. Your service keeps its own upstream credentials, and no tap code runs on the platform. Wire contract and a complete Rust example: HTTP Tap Contract. Create one from the wizard, the CLI, or an agent over MCP.
A more focused tap wizard. Two choices: paste your own Python script, or point at an HTTP endpoint. AI generation stays available through the Assistant and connected agents. Structured vs. document output is detected automatically from the test run, and a passing script is kept exactly as tested — the automatic performance-rewrite pass is gone.
GitHub-stored taps open at the latest code. The editor loads the newest repo version on open (runs keep the previously saved version until you test and save). False “script was changed” warnings caused by unrelated repository commits are fixed, and the AI test diagnosis now works for repository-stored scripts.
Catalog quality of life. Drag-resizable columns on the taps and pipelines tables (layout remembered), and background refresh no longer interrupts column resizing, move-to-catalog menus, or delete confirmations.
Standard upgrade: docker compose pull && docker compose up -d — everything is additive; existing taps and schedules work unchanged.

August 17, 2026

New AI provider: Grok (xAI).

Grok for chat and CodeGen. Grok joins Anthropic Claude and OpenAI as a third model family — the Assistant, tap script generation, AI data quality rules, transformations, and natural-language → SQL can all run on it. Configure it in Configuration → AI Providers with one key from console.x.ai; Grok 4.6 is the recommended model, with Grok Code Fast 1 available for CodeGen. Mix providers freely across sections. See AI Configuration.
Install-time support. The installer prompts for an xAI key alongside Anthropic and OpenAI, and first-boot .env seeding supports Grok as the chat/CodeGen provider.
Embeddings unaffected. xAI has no embeddings API, so semantic search stays on OpenAI, Azure, or the bundled local model — a Grok-only setup works out of the box with the bundled embedder.
Standard upgrade: docker compose pull && docker compose up -d — no configuration changes required.

August 14, 2026

Assistant fix: taps that read data already in Datris no longer ask for database credentials.

No more credentials form for platform data. When a tap’s fetch logic depends on data a pipeline already maintains (e.g. an id list kept fresh in a Datris table), the Assistant previously asked for database credentials or offered to freeze a snapshot of the data into the script. It now builds the tap to read the platform data live on every run — no extra credentials, and the tap’s secret stays limited to the external source’s API key.
Same guidance for MCP agents. Agents connected over MCP (Claude Desktop, Claude Code, and others) receive the same instructions.
Standard upgrade: docker compose pull && docker compose up -d. Start a new Assistant conversation after upgrading — existing conversations keep the older behavior until reopened.

August 13, 2026

Azure OpenAI without API keys — Microsoft Entra ID authentication.

Keyless auth for Azure OpenAI. Required for resources with disableLocalAuth (API keys disabled by policy), and useful anywhere you’d rather rotate credentials centrally. Chat, CodeGen, embeddings, and the Assistant all honor it. See AI Configuration.
Service principal mode. Enter your Entra Tenant ID, Client ID, and Client Secret once in Configuration → AI Providers; tokens are acquired and refreshed automatically.
Managed identity mode. Zero stored secrets when Datris runs on Azure compute (VM, AKS, App Service) — the server authenticates as its managed identity.
Safe mode switching. A stored API key always takes precedence, so the auth-mode selector clears the mode you’re leaving — a stale key can never shadow Entra auth.
Install-time support. Keyless setup works from .env too: provide the service-principal values, or nothing at all on Azure compute. Grant the identity the “Cognitive Services OpenAI User” role on the resource.
Existing Azure OpenAI API-key setups keep working exactly as before — no configuration changes on upgrade.

August 12, 2026

Reliability: long AI generations complete, and the Assistant shows live progress.

Large script generations finish reliably. Generating a big tap script can legitimately take more than five minutes — previously these could time out or hang partway. Generation responses now stream under the hood with a much larger time budget, so long generations run to completion.
Live progress in the Assistant. Running steps show an elapsed-time counter once they take more than a few seconds, plus a note after about a minute explaining that long generations are normal and temporary provider errors retry automatically.
Unresponsive AI providers no longer hang requests. AI calls fail fast during a provider outage and feed the automatic retry ladder, instead of waiting indefinitely on a response that will never come.

Standard upgrade: docker compose pull && docker compose up -d — no configuration changes required.

August 11, 2026

Amazon Bedrock support — run Claude through your AWS account.

New AI provider. Amazon Bedrock joins Anthropic, OpenAI, Azure OpenAI, and Ollama for the chat assistants and CodeGen — Claude models served from your own AWS account with IAM authentication and AWS billing. Configure it in Configuration → AI Providers. See AI Configuration.
IAM-native auth. Enter AWS credentials once, or leave them blank on AWS infrastructure to use the server’s IAM role — zero stored secrets.
Live model discovery. The model dropdown shows what your AWS account and region can actually invoke — including cross-region inference profiles — rather than a hardcoded list.
Claude Fable 5 support. Chat recommends Claude Fable 5; CodeGen recommends Claude Opus 5. Fable 5’s one-time AWS data-retention opt-in is explained in-product and in the docs.
Deliberate opt-in. Bedrock is only ever selected explicitly — AWS credentials used for S3 destinations never silently become your AI provider. The installer offers Bedrock when you don’t have a direct provider key.
Clearer credentials panel. The Configuration → AI Providers key panel is now grouped per provider with a short explainer; credentials hide immediately after saving.
Fix: provider-scoped model lists. A model saved under one provider no longer appears in another provider’s dropdown after switching a section between providers.
Refreshed recommendations. Claude Fable 5 is now the recommended chat model for the Anthropic provider.

August 10, 2026

Azure OpenAI support — run Datris AI on your Azure resource.

New AI provider. Azure OpenAI joins Anthropic, OpenAI, and Ollama as a first-class provider for all three AI slots (AI Provider, CodeGen, Embedding) — mix it freely with the others. Configure it in Configuration → AI Providers with your key, resource endpoint, and deployment name.
Deployment-name aware. Azure addresses models by your deployment names; name deployments after the model they serve so model-specific handling applies automatically.
Installer support. The one-command installer offers Azure OpenAI when you don’t have a direct Anthropic or OpenAI key, and first-boot seeding configures everything from your Azure details. See AI Configuration.
Built for how Azure behaves. Responses stream under the hood so long reasoning calls complete reliably; both current and legacy Azure endpoint styles work.
Fix: session-expiry login. Signing back in after a session expires no longer requires a full page refresh.
Fix: per-provider key updates. Updating one provider’s API key through the API no longer clears the other providers’ stored keys.
Fix: agent-uploaded content. Data uploaded through the AI assistants can no longer be silently garbled when the model sends plain text where encoded content was expected.

August 5, 2026

Fix: the tap sync-state view is now reachable.

Sync-state button restored. v1.14.0’s per-tap sync-state view (current bookmark, commit time, one-click reset) was missing its button on the Catalog page, so it couldn’t be opened in the UI. Each tap row now has a bookmark icon that opens it. The MCP sync-state tools were unaffected.

August 4, 2026

Incremental taps — scheduled taps now fetch only what’s new.

Taps remember where they left off. After each successful run the platform stores a bookmark; the next run pulls only records added or changed since. Recurring syncs run at constant size instead of re-fetching everything.
No gaps, no duplicates. The bookmark only advances when a run succeeds — failed runs leave it in place so the retry re-fetches the same window; key-field destinations absorb overlap by updating in place.
Visible and resettable. Each tap’s sync state can be viewed and reset from the Taps page; agents get the same via two new MCP tools. Manual backfills with an explicit date range never move the bookmark.
AI-built taps are incremental by default. Described a recurring source? The generated script now uses change-detection automatically when the source supports it (modified-since filters, ID watermarks, cursors, or content hashing). See the Taps guide.
MongoDB upsert. MongoDB pipelines can declare key fields, so revised records update in place instead of duplicating — previously blocked by over-strict validation.
MCP playground fixed. All tools in the MCP tab’s playground now execute (28 previously errored when run there); 63 tools total including the two new sync-state tools.

July 28, 2026

Store your tap scripts in GitHub — every save is a commit.

GitHub script storage. Connect a repository under Configuration → Code Repository and tap Python scripts are stored there instead of built-in storage — every save from the wizard, the Assistant, or the API becomes a reviewable commit.
Runs pinned to the saved version. Scheduled runs execute the exact commit that was saved, from a local cache — they keep working even if GitHub is unreachable, and each run records which version it executed.
Edit in your own IDE. Commit changes to the repository directly; Datris detects the newer version when the tap is opened and offers to load it. Conflicting edits are stopped with a clear message instead of overwritten.
Move taps either way, one click. Existing taps stay on built-in storage until moved; each Taps row shows where its script lives. Fine-grained token scoped to the one repository; connection test built in; GitHub Enterprise Server supported. See the GitHub Script Storage guide.

July 27, 2026

Claude Opus 5 support, and a fix for a month-after-install credential expiry.

Claude Opus 5. Anthropic’s newest model is fully supported and is now the recommended Anthropic model for chat and code generation — same price as Opus 4.8 with better results. Fresh installs default to it; existing installs keep their configured model.
Fix: internal credential expired a month after install. An internal service credential was silently issued with a 32-day lifetime, after which restarting the platform could fail. Credentials are now long-lived and refreshed on every startup. Applies automatically on upgrade.

July 24, 2026

Curate the data sources your assistant recommends.

Approved data sources registry. A new Configuration → Data Sources tab lets you maintain a curated list of approved external data sources in one editable document. When the AI assistant or the tap brainstormer suggests where to pull data from, registry sources are offered first, by name; anything outside the registry is clearly flagged. Leave it empty and behavior is unchanged.
You stay in control. Registry entries don’t skip confirmation — the assistant still confirms source, scope, and destination before building anything.

July 23, 2026

Failed scheduled runs now retry themselves — and tell you how to fix what they can’t.

Automatic retry for scheduled runs. Transient failures (source briefly unreachable, rate limits, network blips) are re-run automatically a few times with a short delay before being flagged. Only runs that are provably safe to repeat are retried, so a retry can never duplicate data. On by default and configurable; manual runs pause the retry ladder.
AI fix suggestions, right where failures show. Persistent failures get a one-line suggested fix directly on the failed run in Ops → Ingestion and tap Run History, with the full diagnosis and concrete change one click away. Advisory only — nothing changes unless you apply it.
Agents see suggestions too. Agents polling pipeline status get the same one-line fix suggestion for failed jobs.
Sign-in session fix. With user login enabled, an expired session now ends cleanly and returns you to the login screen right away, instead of requiring a page refresh.

July 22, 2026

Search shows only what’s installed, plus internal consolidation.

The Search tab now lists only active data stores. Stores that aren’t installed or reachable no longer appear in the query-type menu, and the list recovers automatically after a server restart.
Clearer errors for unavailable stores. Searching a store your deployment doesn’t have returns a clear message naming the valid options.
Internal consolidation. Vector-store loading and search, AI provider integration, and tap AI workflows were restructured onto shared foundations — same behavior, less duplication, faster future vector-destination support.

July 21, 2026

Reliability and hardening.

Faster repeated queries. Postgres connections are now pooled and reused across pipeline loads and interactive queries instead of being opened fresh each time.
Connection details are masked in logs. Database connection information no longer appears in logs or job status output.
More stable long-running deployments. Fixed several resource leaks in file ingestion and data-reading paths that could slowly accumulate open connections.
Clearer diagnostics. Many background failures that were previously silent are now logged with context.
Every release is now test-gated. An automated test suite runs on every change and before every release build.

July 14, 2026

Installer hardening.

Clear guidance when a previous installation is in the way. If containers from an earlier Datris install (in another directory) would block a new one, the installer detects this before downloading anything and prints exactly what is blocking, where it lives, and how to remove or upgrade it.
API keys are no longer echoed. The installer reads provider keys with input hidden and confirms with a masked preview, so keys never land in your terminal scrollback.
Best model out of the box. Fresh installs choosing Anthropic default to Claude Opus 4.8 for chat, NL→SQL, and analysis — matching code generation. Existing installs keep their configured model.

July 13, 2026

Choose your databases at install — run less, connect what you already have.

Pick and choose at install. The installer now asks which databases and stores you want instead of installing everything. Each can run bundled, connect to a service you already operate, or be skipped — and pressing Enter at every prompt still gives the standard full install.
Bring your own infrastructure. Point Datris at an existing Postgres, Kafka, or managed vector store (Qdrant, Weaviate, Chroma, Milvus) during install, and optionally store Snowflake or Databricks destination credentials up front so pipelines can use them on day one.
Lighter installs. Skipping the bundled semantic-search server avoids a multi-gigabyte model download — the installer recommends OpenAI embeddings, which need no local container and cost pennies. Vector stores and a local Kafka test broker are now a one-line opt-in instead of commented-out YAML.
Both AI providers, each at its best. The installer asks for an Anthropic key (chat, code generation, AI data quality) and an OpenAI key (embeddings) — both optional, with sensible behavior when only one is present.
Everything knows what’s installed. The pipeline wizard greys out destinations that aren’t available, the AI assistant offers only the destinations your deployment actually has, and creating a pipeline against a missing database returns a clear explanation instead of a timeout.
Install-time validation. After first boot the installer checks every store you configured — bundled or external — and reports each by name, so a mistyped hostname or bad credential surfaces immediately.
Change your mind anytime. Enable or disable any store after install with a one-line edit and a restart. Disabling keeps the data; re-enabling brings it back.
Safer upgrades. Upgrading an existing install never changes which services run — everything you had keeps running, no action required. Data services now use durable named storage, so containers can be removed and recreated without losing data.

Upgrading: Re-run the installer, or docker compose pull && docker compose up -d --remove-orphans after refreshing the compose file. Your current services and data are preserved automatically. See Installation for the new options.

July 9, 2026

Databricks destination — load, upsert, and query your Databricks workspace.

Load into Databricks. Point any structured pipeline at your own Databricks workspace and Datris loads it as governed Delta tables in Unity Catalog: tables are created automatically, the schema evolves as new columns appear, and natural-key upserts keep scheduled re-runs duplicate-free. Loaded tables are immediately queryable across the workspace — notebooks, dashboards, and lineage all see them natively. Available in the pipeline wizard and through the AI assistant.
Secure by default. Connects as a dedicated service principal you control, with least-privilege access to a single catalog; credentials live in a platform secret, never in pipeline configs. The Databricks destination guide includes a copy-paste grant script. Works out of the box with serverless SQL warehouses — nothing runs in your cloud account and there’s no cluster to manage.
Safe full refreshes. Pipelines that replace their table on each run swap the contents atomically — readers never see a half-loaded or empty table, and a failed run leaves the previous data intact.
Forgiving setup. Pasted workspace URLs and warehouse connection paths are cleaned up automatically in whatever shape they arrive, and connection problems come back as plain-language errors that say exactly what to fix.
The assistant can verify and query Databricks. Ask it to confirm a load landed or answer questions over the data — read-only, using the pipeline’s own credentials — and it can browse your catalogs, schemas, tables, and columns along the way.
Cleaner deletes. Removing a pipeline together with its data also cleans up the table it created in Databricks.
CLI catch-up. datris ingest can now target Snowflake and Databricks directly, and pipeline listings label Snowflake and Databricks destinations.

Upgrading: Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris. Using Databricks requires a one-time setup in your workspace — see the Databricks destination guide.

July 8, 2026

Snowflake destination — load, upsert, and query your Snowflake account.

Load into Snowflake. Point any structured pipeline at your own Snowflake account and Datris bulk-loads it: tables are created automatically, the schema evolves as new columns appear, and natural-key upserts keep scheduled re-runs duplicate-free. Available in the pipeline wizard and through the AI assistant.
Secure by default. Connects with key-pair authentication under a dedicated service user and least-privilege role you control; credentials live in a platform secret, never in pipeline configs. The Snowflake destination guide includes a copy-paste setup script that creates everything — role, warehouse, database, service user — in one run.
Forgiving setup. Pasted private keys and account URLs are cleaned up automatically in whatever shape they arrive, and connection problems come back as plain-language errors that say exactly what to fix.
The assistant can verify and query Snowflake. Ask it to confirm a load landed or answer questions over the data — read-only, using the pipeline’s own credentials — and it can browse your databases, tables, and columns along the way.
Watch the assistant work. Long assistant turns now show live progress: a size counter while it composes larger pieces of work, a running glimpse of its reasoning, and a continuous activity indicator between steps — no more silent pauses.
Faster pipeline creation. The assistant sends only a small schema sample when creating pipelines, trimming minutes off longer build conversations.
Cleaner deletes. Removing a pipeline together with its data now also cleans up the table it created in Snowflake.

Upgrading: Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris. Using Snowflake requires a one-time setup in your Snowflake account — see the Snowflake destination guide.

July 1, 2026

Claude Sonnet 5 support.

Choose Claude Sonnet 5. Anthropic’s new Claude Sonnet 5 model is now available as a choice for the AI assistant and code generation. Select it in the Configuration tab like any other model.

Upgrading: Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris.

June 29, 2026

Switch AI providers freely, and an assistant that finishes the job.

Switch providers without losing your keys. Your Anthropic and OpenAI API keys are now kept per provider, so you can change the provider for any section — AI, CodeGen, or Embedding — and switch back later without re-entering the key you already saved. Enter each provider’s key once and mix providers however you like.
The assistant runs to completion. Longer assistant replies that used to stop partway — leaving you to type “continue” — now finish on their own.

Upgrading: Existing installs: docker compose pull && docker compose up -d. Keys you’ve already entered carry over; no re-entry needed. The CLI: brew upgrade datris.

June 24, 2026

Your secrets and configuration now persist across restarts and rebuilds.

Settings that stick. Connection secrets, AI provider and model choices, and other changes you make in the Configuration tab now survive a rebuild (docker compose up --build), a restart, and host reboots. Previously some of these could quietly revert on a rebuild because they were held only in memory — now they’re stored durably, and the Configuration tab is the source of truth.

Upgrading — one-time, please read: with this release the secrets store moves from in-memory to durable on-disk storage. Any secret you had set only in the app’s UI — including tap connection secrets — does not carry over and must be entered once more after upgrading. From then on your secrets persist across every future rebuild and restart. (Settings supplied through your environment configuration are restored automatically.) Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris.

June 22, 2026

Version history for taps and pipelines, plus a faster assistant.

Version history for taps and pipelines. Every create or change to a tap or pipeline is now saved as a snapshot. Open Version History to see what changed and when, compare any two versions side by side, and roll back — or forward — to an earlier version. Nothing is overwritten; restoring simply adds a new version. AI agents can review, compare, and restore versions too.
Configuration that sticks. AI provider and model changes in the Configuration tab now save reliably and persist across restarts.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. Existing taps and pipelines are seeded into version history automatically on first start; no manual migration needed. The CLI: brew upgrade datris.

June 18, 2026

Stronger isolation for tap scripts, plus a catalog readability fix.

Optional sandboxed execution for taps. Tap scripts can run in a dedicated, isolated runner that’s cut off from the platform’s credentials and internal services — defense-in-depth for the code your taps execute. Opt-in; see the Tap Execution & Isolation guide.
Tighter defaults for tap scripts. Platform credentials are kept out of tap code, and any extra packages a tap needs install in isolation rather than into the shared environment.
Catalog: full names on hover. Long tap and pipeline names in the catalog are no longer cut off — hover to see the full name.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 14, 2026

Drop a file into the Assistant and it builds the pipeline for you.

Drag a file into the chat. Attach a CSV, JSON, XML, or document right in the Assistant conversation. The assistant reads a sample, works out the shape, and proposes where to put it — then, once you confirm, creates the pipeline, loads your data, and reports how many rows landed. No wizard, no manual schema step.
It confirms before it builds. The assistant picks a sensible default destination for your file and names the alternatives, so you can steer it to Postgres, MongoDB, an object store, or a vector store before anything is created — and it asks about an upsert key if that makes sense for your data.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 12, 2026

A more decisive assistant, and taps that tell you when a credential is missing.

No more “type continue to keep going.” The in-product assistants (build, ops, catalog, and search) used to sometimes end a turn by announcing the next step and then stopping — making you nudge them to carry on. Now, when the assistant has already decided what to do next, it just does it, while still pausing where it genuinely needs your decision or approval.
Taps surface missing credentials instead of silently returning nothing. If a tap’s API key or other credential has been deleted or is missing the field the tap needs, the run is now reported as a clear failure naming what’s missing — rather than quietly completing with zero records and hiding the real cause. Runs that legitimately have no new data are unaffected.
Clearer MCP activity graph. The connection graph in the MCP Activity monitor now renders at a readable size.
“Move to catalog” shows everything. The per-item move menu in the Catalog now lists all your catalogs, not just the first one.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 11, 2026

Claude Fable 5 — Anthropic’s most capable model, now selectable.

New model option. Claude Fable 5 now appears in the model picker for both the in-product assistant and code generation, alongside the existing Claude and OpenAI choices. Pick it from Configuration when you want Anthropic’s most capable model for demanding reasoning and long-horizon work.
A note on data retention. Fable requires standard (30-day) data retention on the Anthropic account and isn’t available on zero-data-retention organizations. If you bring your own Anthropic key on a zero-data-retention plan, choose a different model — you’ll now see a clear message explaining why, instead of a generic error.
Clearer model errors. When a model declines a request — wrong account settings, a rejected key, or unsupported options — the assistant now surfaces a plain-language explanation naming the model, rather than a raw provider error.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 7, 2026

A domain-neutral assistant — guidance that fits whatever data you work with.

No more finance-flavored examples. The in-product assistant, the tap generator, and the prompt suggestions used to lean on stock-market wording — tickers, symbols, financial filings — in their examples and defaults. They’re now domain-neutral, so the guidance and sample values match the data you’re actually working with, whatever the domain.
Generic search defaults. Vector search now defaults to a documents collection (and a Documents class) instead of finance-specific names. If you already pass your own collection, table, or class name, nothing changes — only the placeholder defaults differ.
Refreshed starter prompt fragments. The built-in tap prompt fragments now describe general patterns — rate-limited APIs and APIs that require a User-Agent header — rather than specific named data providers.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 4, 2026

Organize your catalog by chatting with it.

Catalog assistant. The Catalog tab has a new side-panel assistant for tidying things up. Ask it to group your taps and pipelines into catalogs, move items around, or suggest a cleaner structure — it lays out a plan first and only makes changes once you approve them, and the catalog updates as it works. It sticks to organizing: questions about your data go to Search, and running or fixing pipelines stays in Ops.
Right where you’re working. The assistant lives in a collapsible right rail (toggle it with Cmd/Ctrl + backslash) so the catalog stays in view while you reorganize. Clicking Describe to Assistant on a catalog now opens this panel in place instead of jumping to another tab, and the conversation sticks around as you navigate away and back.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 3, 2026

Ask your data a question — conversational search comes to the Search tab.

Chat search. The Search tab has a new Chat mode, with a Traditional toggle for the structured query UI you already know. Ask a question in plain language and Datris finds the answer across all your pipelines and taps — cataloged or not — querying tables, searching documents, and replying with citations to where each answer came from. It’s read-only: it looks, it never changes anything.
Scope to a catalog. Narrow a chat to a single catalog (or to Uncataloged data) from the dropdown, or leave it on All to search everything.
Conversations survive a refresh. Your Search chat and Assistant conversations now persist across a browser refresh, so reloading the page no longer clears the transcript.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 2, 2026

Orchestrate Datris taps from Apache Airflow.

Run taps from Airflow. A new airflow-provider-datris package adds an operator that triggers a tap, waits for the pipeline to finish, streams Datris logs into the Airflow task log, and reports run tokens and row counts back to Airflow. Cancelling the DAG run cancels the Datris job.
Date-windowed backfills. Taps can now take per-run parameters, so an Airflow DAG can pass its logical date (or any window) into the tap for that run — backfills and incremental loads work without editing the tap.
No double-firing. A tap is scheduled by Datris or Airflow, never both: if a tap has a Datris cron, the Airflow operator declines to trigger it. To drive a tap from Airflow, leave its cron empty.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. Install the provider where Airflow runs: pip install airflow-provider-datris.

May 29, 2026

Write to AWS S3, query Parquet and ORC from the Assistant and Search, and stop chats actually stop.

AWS S3 as a first-class destination. The Object Store destination now writes Parquet or ORC directly to S3 alongside the built-in MinIO. Pick Object Store (MinIO or S3) in the pipeline wizard, point at your bucket, and reference a credentials secret you’ve created in Configuration → Secrets → Platform. Multiple S3 destinations with different IAM keys coexist in the same deployment — each pipeline carries its own credential reference, applied per bucket at write time.
Region lives with the credential. AWS credentials and region travel together in the credentials secret rather than on the pipeline config. One source of truth, one place to rotate, no more silent us-east-1-says-the-config-but-the-key-is-us-west-2 failures. Field names are flexible — accessKey / AWS_ACCESS_KEY / AWS_ACCESS_KEY_ID all work.
Query Parquet and ORC from the Assistant. “Show me the weather data” now works against pipelines whose destination is Object Store. The Assistant resolves the bucket and credentials from the pipeline config, reads the columnar files, and returns rows in chat.
Search tab gains an Object Store option. Pick a pipeline from the dropdown, see the resolved bucket/prefix/format, set a limit, hit Execute. Works for both MinIO and S3 destinations.
Assistant offers all three structured destinations. When you ask for structured data and no pipeline yet covers it, the Assistant now mentions MongoDB, PostgreSQL, and Object Store as choices instead of silently defaulting to one.
Assistant can discover destination credentials it shouldn’t create. Platform-tab secrets are now visible to the Assistant for reading (names and field shape only — never values). When a pipeline destination needs a credentials reference, it lists what’s available, verifies the field shape, and points you at the Secrets tab to create one when nothing fits.
Stop actually stops. Clicking Stop in the Assistant now halts the in-flight chat within a fraction of a second instead of waiting for the upstream model to finish generating. Cancelled responses cost only what was already streamed.
Pipeline failures surface fast. A failed pipeline now flips to Error in the Ops dashboard within seconds instead of staying stuck in Processing for up to ten minutes.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. Existing MinIO pipelines work unchanged. For AWS S3, create a Platform-tab secret with accessKey, secretKey, region (and optionally sessionToken), then reference it by name from your pipeline.

May 28, 2026

Ask the Ops assistant about a failing pipeline without leaving the dashboard.

New Ops chat side panel. A collapsible chat lives on the right side of Ops → Activity. The assistant has the current failures, stale taps, and volume anomalies in mind — ask “why did X fail?” and it pulls the root cause; ask it to re-run a tap and it runs and reports the outcome. The panel stays mounted as you switch between Activity and Ingestion, so the conversation survives the tab change.
“Ask” buttons on failure and volume rows. Click “Ask” next to a row to seed the chat with a row-specific question so you don’t have to retype the tap or pipeline name.
Successes are expandable like Failures. Click any row in the Successes pane to see the same event trail (begin → processing → end) you get from the Failures pane. Only one row across either pane is open at a time so the layout stays compact.
Claude Opus 4.8 is the new recommended CodeGen model. New installs seed Opus 4.8 as the codegen default. Existing installs can pick it from the model dropdown in Configuration. The older Opus versions remain selectable for anyone who wants to pin a specific version.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 27, 2026

The Assistant stops second-guessing itself when the data is already there.

No more apology loops. When the Assistant verifies platform state and the pipeline / tap it created earlier is present in the list, it now treats that as evidence the work was done — instead of retracting a prior turn’s “done” claim and rebuilding from scratch.
“Show me X” goes straight to the data. When you ask to see / list / show data and a matching pipeline already exists, the Assistant now jumps to the destination’s query tool (Mongo / Postgres / vector search) and returns the actual rows or documents. It no longer asks you which sources, providers, or schedules to use for a pipeline you already have.
Long catalogs no longer hide existing resources. The tools the Assistant uses to inventory pipelines and taps now return a compact summary with names at the top. Previously, on environments with many pipelines, an entry near the end of the list could slip past the Assistant’s scan — the new shape makes every name impossible to miss.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 27, 2026

Ops Activity gains a Successes pane, long lists scroll in place, and the dashboard’s auto-refresh no longer yanks you back to the top.

Successes pane on Ops Activity. A new pane below Failures lists every pipeline that ran successfully in the selected window, with the run count, items processed, and last-run time. Click a row to jump to that pipeline.
Long lists scroll inline. Failures, Successes, and the Per-pipeline volume table all cap at ~10 rows of height and scroll internally instead of stretching the page. Per-pipeline volume column headers stick to the top so you don’t lose context as you scroll.
Auto-refresh preserves scroll position and expansion state. The 30-second refresh no longer scrolls a long pipeline-volume list back to the top or collapses an open failure detail. Expand a failure, scroll where you want, leave the tab open — it stays put.
Numeric column headers aligned with their data on the Per-pipeline volume table — Today, 7d avg, vs avg now line up with the numbers underneath.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 26, 2026

A new Ops Activity dashboard, Postgres pipelines learn upsert, per-run tap parameters, and a safer agent workflow.

Ops Activity dashboard — at-a-glance ingestion health. A new tab under Ops pulls every tap and pipeline run in a rolling window (24h / 7d / 30d) into KPI tiles, time-series charts of runs and items ingested, a per-pipeline 7-day volume table, and a Failures pane that dedupes by item with attempt counts. Each unrecovered row has a Re-run button — for pipeline failures with an upstream tap, the button re-runs that tap to retry the load so you don’t have to hunt for it in the Catalog.
Postgres pipelines upsert on conflict when keyFields is set. Matches the semantics Mongo has always had. Backfills over already-loaded dates, incremental taps with overlap, and “load again with the same key” flows now upsert instead of failing with a duplicate-key error. If you retrofit keyFields onto an existing table, the platform adds the matching unique index for you on the next load — or surfaces a clear remediation message if existing data violates the proposed key.
Per-run tap parameters. Run a tap with caller-supplied values — date ranges, ticker lists, page cursors, batch sizes — without rewriting the script or the secret. Pass params to run_tap and the script reads them as env vars for that one run only. Scheduled cron runs see an empty params bag and fall back to script defaults, so the same script handles both manual ad-hoc calls and unattended schedules without branching.
“No records” is no longer a failure. A tap that runs cleanly and returns zero rows — a polling tap on a quiet day, an incremental tap that’s caught up, a market tap on a weekend — now records a distinct no_records status with a neutral badge. Doesn’t count in the Failures tile, doesn’t fire bogus “recovered” badges, doesn’t train agents to interpret “no new data” as “platform broken.”
Large tap outputs fail fast with an actionable error. A backfill that exceeds the size cap (default 100 MB) now stops with a clear message telling you and the Assistant to chunk the source range smaller — instead of OOM-killing the server. Multiple smaller runs all land in the same destination pipeline; with keyFields set, overlapping ranges upsert safely.
Concurrent tap runs no longer race. Two pipelines that loaded data in the same millisecond previously risked landing each other’s records in the wrong destination under specific timing. Fixed at the source; no action required.
Secret values stay masked in Configuration. The Secrets tab now masks any field whose name suggests a credential — passwords, tokens, API keys, signing keys, certificates, and named variants of those (e.g. *_API_KEY, *_SECRET_*). No action required.
Tighter Assistant workflow. Three new disciplines:
- Scheduling lives on the tap. Say “every morning” or “at market open” and the Assistant sets the CRON expression on the tap itself — instead of handing back a cron line for you to wire up yourself.
- Test before first run. A newly-created or just-edited script is validated before any real run, and before being put on a schedule — no more “guaranteed-bad nightly run” the next time the cron fires.
- No confabulated progress. If the Assistant intended to do N things and only did M of them, it tells you which M happened rather than narrating all N as complete. The credential form the Assistant pops up now asks only for true secrets — not for configuration values you already typed into chat.
Agent Monitor stays responsive during long tool calls. Running a slow tool no longer freezes the Connections viz or the Activity Log — they keep streaming. Returning to the Agent Monitor tab refreshes the log to current server state instead of showing a stale empty view.
Configurable JVM heap. The bundled datris service has a sensible default heap size for small hosts (8 GB) and a clean way to bump it on larger machines via .env. See Installation → JVM Heap Sizing for the suggested sizings per host RAM.
Ops tab moved after Search in the top navigation. Mostly cosmetic; route URLs are unchanged so bookmarks and deep links still work.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. Existing Postgres pipelines with keyFields automatically pick up the upsert path on the next load. The CLI: brew upgrade datris.

May 22, 2026

Embedding pipelines stop failing on oversized chunks, the Assistant carries long runs to completion, and the Catalog gets the missing delete buttons and run-history detail.

Embedding pipelines no longer fail when a single chunk is too big. Every embedding call is now guarded by a token-aware safety net that splits any chunk over the model’s input cap before sending — so a 10-Q ingest with one dense table can’t take down the whole batch. Works the same for OpenAI, Cohere, Voyage, BGE-M3, Nomic, Mistral, or anything else, with built-in caps for the common models and a conservative default for the rest. OpenAI families get exact token counts; everyone else gets a heuristic.
Token-aware chunking. A new maxChunkTokens option on the chunking config tells the chunker to stop merging segments before they cross a token estimate — the safety net above becomes a true last resort. Recommended for any new vector pipeline.
The Assistant carries pipeline runs to completion. When the agent kicks off a tap or upload, it now polls the run to a terminal outcome with exponential backoff instead of summarizing “still running — check back later.” You’ll see a one-line progress update each cycle and the final outcome reported in chat, including any per-document failures.
Run History now shows per-document outcomes. Expanding a tap run that fed a vector pipeline lists every document the pipeline processed with its own status, elapsed time, and — for failures — the specific stage and error. A run that fetched 28 documents but failed on 1 now shows the failure inline instead of a misleading green “success.”
Catalog: delete buttons work on individual taps and pipelines. The trash icon now shows an inline confirm right in the row — Delete / × for taps, Config & Data / Data Only / × for pipelines (Data Only wipes rows but keeps the config so the next ingest fills it again).
Catalog: last-run timestamp is back above the status badge.
Agent Monitor fits the viewport. Both Connections and Activity Log panes now size to the visible area on first load and reflow when you resize the window, not just in the pop-out.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 21, 2026

Streamlined navigation and a Catalog-centric workflow — fewer tabs, richer Catalog, and the Assistant front-and-center.

Five top-level tabs instead of ten. Datris now opens to Assistant, MCP, Catalog, Data, Configuration — plus the Help dropdown. Same capabilities, organized around how you actually work.
Catalog is the home for taps and pipelines. Each catalog card embeds the full tap and pipeline tables with inline rename and a move-to-catalog dropdown on every row. Uncataloged is always shown so day-1 users have a place to start.
Bulk move at the catalog level. Move every tap and pipeline from one catalog into another in one click. The destination auto-expands so you see the items land.
Describe to Assistant from any catalog. A button on each catalog card opens the Assistant with a fresh chat and the catalog pre-filled — the Assistant assigns the right catalog to whatever it creates.
Wizards link back to where you came from. Tap and pipeline edit screens now show the item’s name in the title, a “Back to Catalog” link at the top, and primary action buttons at both top and bottom of the form. Pipeline wizard’s JSON-review step is gone — Save fires from the Destination step.
Pop out the Agent Monitor. A new icon opens both Connections and Activity Log in a separate browser window — park it on a second monitor and watch tool calls stream while you work in the Assistant.
Catalog state persists across navigation. Open catalogs and expanded sub-sections survive refreshes and tab switches.
Fixed: tap rename no longer breaks scripts. Renaming a tap inline used to leave the new tap pointing at a deleted script file.
Fixed: catalogs containing only pipelines could lose their assignment when the pipeline edit wizard was opened.
Discovery tab removed. The Assistant covers its workflow.
Getting Started tab removed. First-run guidance lives in the Assistant’s starter prompts and inline empty states.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui. No data migration needed. The CLI: brew upgrade datris.

May 19, 2026

Scoped API keys with per-agent permissions, plus Assistant resilience and smarter onboarding.

New API-Keys tab in Configuration. Issue a dedicated key per agent, CLI, or integration with an explicit list of what it’s allowed to do — read pipelines, run taps, upload documents, query data, and so on. Each key is its own identity in the request log and can be rotated or revoked independently. Five starting templates: read-only, rag-builder, reporting, ops, and full-access. The tab only appears when you’ve set USE_API_KEYS=true in .env — issuing keys is pointless until the validation layer is on.
Keys actually constrain. When an external agent (Claude Desktop, Cursor, the CLI) connects with a scoped key and tries something outside its bundle, the platform refuses the call and tells the agent why in plain JSON — so the agent doesn’t keep retrying alternate paths. Agents stay productive within their lane; you don’t have to trust them not to wander.
UI no longer asks for a key when user authentication is on. With login enabled, your session cookie is the only thing the browser needs — paste-the-key flow goes away. The Assistant runs under your identity, audit logs show you as the actor, and your role determines what it can do (admin = full access, editor = data writes, viewer = read-only).
Assistant rides through Anthropic overload. When Claude Opus is rate-limit-shedded, the Assistant retries with backoff and, if needed, transparently switches to Sonnet for the rest of the turn — with a small inline note so you know it happened. Conversations that previously errored out now keep moving.
Assistant checks the platform before suggesting external sources. On any data-related ask (“I’m looking for X”), the agent now lists your existing pipelines and taps first, then either points you to what already exists or asks before adding more. Avoids the “let me enumerate seven public APIs” detour.
Assistant auto-runs newly created taps that have no schedule. When you build a one-shot tap, the Assistant kicks off the first run so you see real data instead of an empty pipeline. For scheduled taps it asks first, since the cron will fire on its own.
Health and version endpoints are public. Container orchestrators, status pages, and the UI’s connection check no longer trip 500s when API-key auth is required.
Configuration → Taps sub-tab removed. Prompt Fragments are unchanged and still apply to tap creation, brainstorm, auto-fix, and Discovery — they’re now managed via the API instead of a dedicated UI page.
Tap wizard pipeline link is clickable. The “Linked to: <pipeline>” pill in step 4 now navigates straight to the pipeline editor.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui. Existing API keys keep working — they’re treated as full-access until you replace them with scoped keys from the new tab. The CLI: brew upgrade datris.

May 13, 2026

Polish for the Assistant, AI Configuration, and MCP tabs.

Assistant: keep typing without clicking. After you send a question, the cursor stays in the composer — so as soon as the agent finishes (or even while it’s still working), you can type your next prompt without reaching for the mouse.
AI Configuration: provider switches no longer wipe your overrides. Switching the primary AI, codegen, or embedding provider used to clear the saved provider/model/endpoint on the next page load, so you had to re-pick them every time. Your selections now persist correctly through a provider switch (you still re-enter the API key when changing providers — that’s intentional).
Connect Your Agent: paste-and-go for local. The MCP tab’s generated config no longer asks for an API key for the default local setup — paste the snippet into your agent and it just works. A key is still required when you point the snippet at a remote instance.
Hosted trial provisioning tools removed from the agent toolset (hosted platform since retired). The Getting Started tab has also been refreshed to point at the Assistant for the common onboarding path.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris.

May 12, 2026

A new in-product Assistant — chat your way from “I need data” to a working pipeline.

The Assistant tab. A new top-level tab opens an in-product agent that finds an external data source, builds the tap, creates the pipeline, runs it, and shows you the result — all in one chat. You watch the model’s reasoning, every tool it calls, and the live status, then click straight into the tap or pipeline it created. See Assistant.
Real-time visibility while the agent works. Streaming reasoning, inline tool cards with friendly labels (“Searching the web for …”, “Creating tap …”), live success/error status, and a Stop button that aborts the loop at the next checkpoint. Conversations survive navigating to other tabs and back.
Credentials never enter the chat. When the agent needs an API key or other credentials, it opens an inline credentials form right in the chat. Values go straight to the vault — they don’t appear in the conversation log, your screenshots, or the model’s context. The form also lets you reuse an existing tap secret instead of creating a new one.
Pipelines can be edited in place. Calling pipeline-create again with the same name now upserts the configuration without dropping the destination data. Two new knobs in the same call: a natural-key list for dedupe/upsert on every run, and a flag to wipe the destination before each run for full-snapshot workflows. Both work on PostgreSQL and MongoDB destinations.
External agents get the same new tools. Claude Desktop, Cursor, and any other MCP client connected to your Datris server now see two new tools for discovering existing tap secrets (names and field shapes, never values) plus the new dedupe and reset knobs on pipeline create.
Pipeline delete now describes what it actually does. The tool description used to claim it kept your destination data; it never has. It now correctly says it removes both the configuration and the destination data, with an explicit opt-in flag for the “keep the schema, clear the data” reset case.
MCP transport upgraded — both protocols at once. The bundled MCP server now serves the streamable-HTTP transport alongside the existing SSE transport on the same port, so the in-product Assistant and external agents connect to one running process. No configuration change needed.
Discovery tab is hidden in this release. The Assistant supersedes the Discovery tab for the common path.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris. The Assistant uses your existing codegen AI configuration — with Anthropic you’ll see full chain-of-thought reasoning streamed inline; on OpenAI you’ll see reasoning summaries instead.

May 11, 2026

Web search for AI tap workflows, plus simpler MCP authentication.

AI tap workflows can consult the live web. When enabled in Configuration → AI Providers, tap brainstorm, dataset discovery, tap diagnosis, and tap auto-fix look up current API documentation, free-tier limits, current package names, and recent deprecation notices before recommending sources or generating fixes. Pick your web-search provider independently of AI Primary — the platform uses each provider’s native search tool and routes accordingly. First-pass tap script generation stays fast and uses the model’s training data only. See AI Configuration for setup.
AI Configuration changes survive Docker restarts. Saving from the Configuration UI now mirrors the relevant keys back to your .env file so changes aren’t lost when the local Vault container restarts. Provider switches also clear any stale credential preserved from the prior provider, so a wrong-provider key can’t silently 401 the next call.
Cleaner MCP authentication. The bundled MCP server is now a transparent forwarder — each connecting agent provides its own API key per session and the MCP server passes it through to the Datris REST API on every tool call. The Configuration → Connect Your Agent panel generates the new configuration snippet automatically; paste your key from the Configuration UI into your agent’s MCP config. Existing remote-instance setups see no change in behavior.
Tap brainstorm asks about sources first. When you describe data without naming where to fetch it from, the AI now lists 3-5 candidate sources (with free vs paid and key-required info) before drilling into parameters — instead of asking for filtering details up front.
Tap script generation is more resilient. The platform now validates that a generated script actually defines a fetch() function before storing it, and the JSON extractor handles model responses that contain narrative braces without falling back to the raw-text path.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris. If you run the MCP server standalone outside Docker, the connection-target environment variable was renamed for consistency — see the MCP server docs for the new variable name.

May 6, 2026

Reliable job-status polling, re-ingest that doesn’t overwrite your config, and first-class catalog tooling.

Polling an upload’s job status now gives you a clear answer. Previously, when a file finished processing successfully, the response was a raw stream of progress events with no terminal status — agents and scripts couldn’t tell “still running” from “done.” Job status now returns a rollup with a single allDone flag and an aggregate outcome (success, warning, error), plus per-job error detail when something fails. Poll the rollup; act on the outcome.
Re-ingesting a file preserves your pipeline’s config. datris ingest against an existing pipeline used to silently rewrite the config from CLI flags only — wiping out the catalog, custom validation rules, and any other fields you’d set through the UI or via an agent. Re-ingest now uploads into the existing pipeline as-is. To start over with a different config, delete the pipeline first.
Catalogs without the read-modify-write dance. New --catalog flag on datris ingest for new pipelines, new optional catalog argument on the create_pipeline MCP tool, and a new set_catalog MCP tool that retags an existing pipeline or tap in one call. Empty catalog clears the label back to Uncataloged. See Data Catalog.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris.

May 5, 2026

Choose your embedding provider independently of your chat provider.

Mix-and-match AI providers. The embedding slot is now configured separately from the chat and code-generation slots, so you can keep Claude for chat and code generation while pointing embeddings at OpenAI (or vice-versa). Useful when the bundled embedder is too heavy for your host, or when you want a different model family for vector quality vs chat quality. See AI Configuration for the full list of options.
Existing installs keep their current behavior. If you don’t set an embedding override, the embedding slot continues to follow your chat provider exactly as before — Claude installs keep using the bundled embedder, OpenAI installs keep using OpenAI embeddings. The override is purely opt-in.

Upgrading: docker compose pull && docker compose up -d --remove-orphans. No data migration needed. If you switch embedding providers on an existing deployment, vector destinations built on the previous embedder will fail-fast with a dimension-mismatch message on the next run — drop the affected destination tables or collections and re-ingest.

May 5, 2026

User authentication, roles, and an admin-only Configuration tab.

Optional username/password login. Datris can now require a login before any tab is reachable. Three roles ship out of the box — admin (full access), editor (read + edit pipelines, taps, secrets), and viewer (read-only). Off by default, so existing installs are unaffected. See the new User Authentication doc to enable it.
Configuration is admin-only. When auth is on, only admins see the Configuration tab (Secrets, AI Providers, Taps, Users, Environment). Editors and viewers continue to use everything else.
New Users sub-tab. Admins can add, remove, and reassign roles. A built-in 16-character password generator with a reveal toggle makes handing out credentials painless. The last admin can’t be deleted.
Self-service password change. Users can change their own password from the top-right user menu.
Reveal toggle on the login screen. Easier to see what you’re typing on a new device.
Clear the Agents activity log. The trash icon now wipes the server-side activity buffer (with an inline confirm) so the cleared state survives a refresh.

Upgrading: docker compose pull && docker compose up -d. No data migration needed; auth defaults to off.

May 4, 2026

AI-agent RAG ingestion no longer burns through the conversation context.

Creating a vector-store pipeline no longer requires sample content. Asking an agent to ingest a PDF into pgvector previously forced it to base64-encode the entire document just to register the pipeline — wasting tens of thousands of tokens before any work began. Vector pipelines now register from a name plus destination alone, freeing budget for the actual upload.
Agents are guided to send each document in a single upload. The MCP server now makes explicit that vector destinations chunk server-side, preventing agents from needlessly splitting documents into many small uploads.
Clearer Claude setup docs. The “Configuring Claude” guide shows the recommended SSE / mcp-remote setup first, and steers large-file ingestion to the CLI rather than dragging files directly into the chat — which can overflow the conversation context on sizable PDFs.
README accuracy pass. Corrected tool counts, AI-provider model defaults, and license badge.

Upgrading: docker compose pull && docker compose up -d. No data migration needed.

May 4, 2026

UI cleanup, friendlier Help menu, and a more reliable bundled embedding service.

Bundled embedding handles large ingest batches without errors. The bundled embedding service no longer rejects large batches submitted by the platform, which previously surfaced as ingestion failures on long documents.
Secrets is now a tab inside Configuration. Instead of a separate top-level tab, Secrets lives under Configuration alongside AI Providers, Taps, and Environment. Existing /secrets links continue to work — they redirect to the new location.
Help menu in the top bar. The Docs link is replaced with a Help dropdown that exposes both the docs and a direct link to file an issue on GitHub.
Easier setup with Claude. A new “Configuring Claude” page in the docs walks through Claude Desktop and Claude Code setup end-to-end, including a first-prompts walkthrough for an empty install.
Structured issue reporting. GitHub issues now use forms that capture version, component, deployment mode, and reproduction steps, making bug reports easier to triage and faster to fix.

Upgrading: docker compose pull && docker compose up -d. No data migration needed.

April 30, 2026

Smaller, faster default install. Lighter download, opt-in Kafka, vector ingestion fixes.

~58% smaller docker compose pull. The bundled platform now downloads roughly 11 GB less out of the box. Fresh installs come up dramatically faster.
Bundled embeddings are faster on the same bge-m3 model. No configuration changes needed; existing vector collections built with the previous bundled embedder continue to work without re-embedding.
Kafka is now opt-in. Most local installs don’t need it, so Kafka, Zookeeper, and the Kafka UI ship commented out in docker-compose.yml. Uncomment the bundled blocks (and the related volumes at the top of the file) to enable them. Pipelines that point at external Kafka brokers are unaffected.
Vector ingestion no longer fails with a “duplicate key” error when many documents land at once. Concurrent document loaders previously raced on creating the pgvector extension; the race is now serialized.
Vector ingestion no longer fails on embedding providers that limit batch size. The chunk batch size is now configurable per embedding secret (batchSize), with a cross-provider-safe default. OpenAI users who want to maximize throughput can set this higher.
Configuration tab clarifies optional providers. The bundled embedding option is labeled bge-m3 (bundled). The AI Provider, CodeGen Provider, and Embedding Provider dropdowns each indicate that local Ollama is opt-in.
Service Health no longer shows “Down” for optional services that were never enabled. Kafka and the optional vector databases (Qdrant, Weaviate, Milvus, Chroma) now correctly report “Not Configured” until you turn them on. Existing installs may still show “Down” until their stale Vault secrets are removed.

Upgrading from earlier versions: the bundled embedding service moved off Ollama on the same host port 11434, so existing installs must pass --remove-orphans to release the port: docker compose pull && docker compose up -d --remove-orphans. Without it the upgrade fails with Bind for 0.0.0.0:11434 failed: port is already allocated. Your data and the cached Ollama model are preserved.

April 29, 2026

Reliable run-completion signals, smarter agent calls, cleaner run history.

Agents get a single “are we done?” signal when watching a pipeline load. Polling a publisher token returns a rollup with a clear allDone boolean and a per-job outcome (success / warning / error / processing / timed out), so agents no longer have to interpret the raw event stream to figure out whether a run is complete.
Pipeline status by publisher token works reliably for completed runs. A storage path that occasionally hid completed runs from the publisher-token query is fixed; the query is now backed by an indexed top-level field, with a fallback for older rows so existing data resolves without a migration.
run_tap no longer ships the records array back to the agent. A push run returns recordCount, publisherToken, and the persisted / persistedReason flags — enough to verify ingestion via get_pipeline_status without bloating the agent’s context. Use test_tap to preview a script’s output (capped at 20 sample rows with a recordsTruncated flag).
Duplicate run_tap calls are suppressed. The agent skips a run_tap for a tap that’s already in flight in the same session, and the platform debounces push runs to one per tap per 5 seconds. Prevents accidental duplicate ingestion from parallel tool calls, double-clicks, and transport retries (persistedReason: already_running or debounced).
Every tap run now produces visible logs. The script wrapper emits start / fetch / record-count lifecycle lines on every run, so run history shows useful output even when the user’s script never calls print().
Secret values are masked in stored tap logs and exception messages. A tap script that incidentally printed an API key or Vault-loaded credential would previously have surfaced the raw value in run history; those values are now redacted in the persisted log and the error string.
Deleting a tap now cleans up its run history. Previously, run-history rows accumulated indefinitely and could resurface under a recreated tap with the same name.
Agent activity log shows full requests and responses. The expanded view in the agent monitor no longer truncates request arguments or response bodies.

April 24, 2026

MCP tab and documentation sync.

MCP tab now lists every agent tool. The in-app MCP reference was missing get_pipeline_status, create_tap_secret, delete_tap_secret, get_tap_ledger, and query_natural. All five are now in the catalog and the Try-It playground, and create_tap exposes the tap_type parameter.
Recommended Agent Workflow rewritten to match the platform’s actual flow. The in-app workflow and the MCP docs had drifted — they started with profile_data (which the platform explicitly says not to use for pipeline generation), skipped the persisted / persistedReason check after run_tap, and didn’t show the publisherToken poll that confirms records actually landed. All three are now canonical: check-before-create, verify-via-publisher-token, and tap credentials managed via create_tap_secret.
Documentation: updated agent workflow examples. RAG over external documents is now shown as a document tap (tap_type="document" + get_tap_ledger), onboarding an external source uses create_tap_secret for credentials, and quality monitoring of scheduled taps shows the get_tap_logs → get_pipeline_status(publisher_token=...) pivot.

April 24, 2026

Tap wizard reliability, iteration history, and cleaner vector-search errors.

Tap wizards learn from their own retries. When the AI fixes, optimizes, or reviews a tap script, it now carries forward up to the last three attempts — what was tried, what went wrong, and what changed — into the next call. The wizard stops cycling through the same failed approaches.
Saved tap scripts always match the tap. Saving a tap now pushes the in-memory script to object storage before writing the tap config, and the create/update call verifies the stored script is actually there. No more “missing script” banners from an interrupted save, and auto-revert no longer strands a tap with a deleted script.
Run Tap stays on the page when nothing was ingested. If a manual run finishes without persisting records, the wizard keeps you on the run step and shows an inline reason (test mode, no records, run error) instead of navigating away and hiding the diagnostic.
Tap logs now carry the publisher token. Every tap run that submitted records records its publisher token in the log. Agents reading get_tap_logs can pivot directly to get_pipeline_status to confirm a scheduled run actually landed in the destination — not just that the script ran.
Vector search fails cleanly when the embedding dimension doesn’t match the collection. If you change embedding providers on a pipeline whose vector collection already has vectors of a different dimension, search queries now return a clear 400 with a user-actionable message instead of leaking a JVM stack trace.
Sturdier local-dev startup. Kafka and Zookeeper now use named volumes (no more corruption races on rebuild), Kafka waits for Zookeeper’s request processor to actually be ready (not just its listener bound), and Vault init picks up an explicit AI-provider override so a stray shell env var can’t silently flip providers.

April 22, 2026

Agent-native tap observability, scheduler fix, and agent-owned tap secrets.

Agents can watch a tap load to completion. Running a tap now reports back whether the data was actually persisted — and names the reason when it wasn’t (test mode, no target pipeline, no records, run error). Every persisted run returns a single publisher token covering the whole run, even for document taps that spawn many ingestion jobs. A new get_pipeline_status MCP tool lets an agent poll that one token until the entire load reaches its final state, so it can report “done” with real numbers instead of guessing from a response body.
Scheduled taps no longer need a manual kickoff. Taps saved with a cron schedule now fire on their next scheduled time automatically. Previously, a newly saved scheduled tap would wait indefinitely until you ran it once by hand.
Self-diagnosing tap scripts. If a tap’s generated script goes missing from object storage, the Edit Tap page now shows an amber banner explaining the state and pointing you to Regenerate, instead of a cryptic mid-run “key does not exist” error. Test Tap surfaces the same state with actionable wording.
Agents can manage their own tap secrets. Via MCP, agents can now create and delete the secrets their taps need (API keys, tokens). Scope is strictly tap-owned — agents cannot create, overwrite, or delete human-owned Platform secrets (DB creds, AI keys, vector-store creds).
Secrets page split into Platform and Taps sub-tabs. Platform lists the built-in Datris secret slots; Taps lists agent-authored tap secrets. Creating a secret from the Taps sub-tab auto-tags it so it stays agent-editable.
Honest Test Tap banner. The run-result banner on Test Tap now reflects what actually happened on the server — “sent to pipeline” only when the run was truly persisted, otherwise “not persisted” with the reason — rather than whatever the pre-request checkbox said.
BYO-code taps can declare pip dependencies. If you paste your own fetch script into Create Tap, you can now list the Python packages it needs. Previously only AI-generated taps could declare dependencies.
Example agent refactored onto taps. The bundled market-macro-agent example now drives ingestion through taps instead of ad-hoc fetch scripts, demonstrating the full agent-native tap flow (provisioning, secrets, publisher-token watching).

April 21, 2026

Tap prompt fragments, post-run script review, BYO code, and Configuration page reorg.

Tap prompt fragments (new). A new Configuration → Taps sub-tab lets you save reusable context snippets — things like API conventions, required headers, rate limits, and preferred libraries for a given source. When the key or any of its aliases appears in a Create Tap description, brainstorm, auto-fix, optimize, or Discovery chat, the fragment’s content is automatically added to the system prompt. Includes an AI Suggest button, a Load Examples catalog (AWS, Polygon, Stripe, SEC EDGAR), JSON import/export, and an “Extra context applied” chip row in the tap wizard showing which fragments hit.
Post-run script review. After a tap’s first successful test, the AI now scans the captured stderr/stdout for signals that the script should change — rate-limit or burst warnings, deprecation hints, pagination cues, schema drift, auth warnings — and regenerates the script if needed. On a rewrite the wizard auto-retests; the performance optimizer runs only when the logs are clean. The optimizer’s prompt was also tightened so rate-limit markers push it toward throttling instead of more concurrency.
I Have My Own Code (new). A third Tap Type on the Create Tap wizard lets you paste a fetch() script directly instead of having AI generate one. Step 1 switches to a code textarea with a Use My Code button; after upload the button flips to Re-upload My Code and Step 2 gates on the text matching what’s on disk, so edits force a fresh upload before running the test.
Configuration page reorganized into three sub-tabs — Environment, AI Providers, and Taps — with a prominent “Highly recommended: Anthropic with the latest coding model” tip on the CodeGen Provider section.
Tap Name collision warning. If the name you type in the Create Tap wizard matches an existing tap, an amber banner appears under the field warning that continuing will overwrite the existing tap’s configuration and script.
Auto-fix retries bumped to 3. When a tap script fails its first test, the AI now gets up to three repair attempts (was two) before giving up.
Cron Custom preset no longer blocked by AI formatting. AI-generated cron expressions wrapped in code fences, brackets, or quotes are now cleaned automatically, so the Next button is enabled on valid output.

April 20, 2026

OpenAI Codex models now work for code generation. Previously, selecting a codex-family model for CodeGen (e.g., the recommended GPT-5.3-Codex) caused tap-script generation and AI data quality / transformation to fail immediately with a 404. Datris now routes codex models to the right OpenAI endpoint automatically.
Vector-store dimension changes now fail fast with a clear message. If you switch embedding providers (for example, Ollama bge-m3 → OpenAI text-embedding-3-small) on a pipeline whose destination table or collection already has vectors of the old dimension, the job stops up front and tells you exactly what to do instead of blowing up mid-ingest with a cryptic database error. Applies to pgvector, Qdrant, Weaviate, Milvus, and Chroma destinations.
Configuration save is honest about missing API keys. Changing the AI Provider to Anthropic or OpenAI without entering that provider’s API key no longer silently skips the save while reporting success — the Configuration page now flags the missing key and tells you which one to add.
Create Tap brainstorm wraps up sooner. For document taps, the AI assistant no longer drills for optional date filters once you’ve supplied the source, auth, and a broad scope — the tap ledger already dedupes by content, so those extra questions were just noise.

April 20, 2026

OpenAI GPT-5 and reasoning-model support. AI-backed features (Discovery chat, Create Tap brainstorm, AI data quality, AI transformation, script generation) now work with OpenAI’s newer model families. Previously, selecting one of these models caused chat panels and wizards to hang without a response.

April 19, 2026

Document Taps — a new tap type for feeding vector stores with unstructured files.

Document Taps (new). A purpose-built tap for ingesting PDFs, Word docs, HTML, and other unstructured files into a vector-store pipeline. Describe the source in plain English (“ingest all PDFs from our SharePoint legal folder”, “pull every DOCX from legal-contracts/2026/ in S3”) and the generated tap discovers the files and hands their raw bytes to the pipeline. Text extraction, chunking, embedding, and loading are all handled downstream — you don’t configure any of that on the tap.
Tap type toggle in Create Tap. Choose Document Ingestion or Structured/Semi-Structured on the first step of the wizard; the prompts, placeholders, and example instructions adapt to the choice.
Ingestion ledger. Every discovered file is tracked by URI and content hash. Re-running the tap skips files that are already up to date — no re-embedding, no duplicates. Changed files flow through normally.
Pre-flight validation. Document taps linked to a pipeline are checked at save time to confirm the pipeline is shaped for document ingestion (unstructured source, vector-store destination). Misconfigurations surface as an actionable error instead of a cryptic mid-run failure.
Safe defaults for local paths. Document taps refuse to silently walk arbitrary host directories if a requested path isn’t mounted into the container — they fail loudly instead of ingesting unintended files.
Faster embeddings. The bundled Ollama sidecar now handles concurrent embedding requests in parallel and keeps the embedding model warm between pipeline runs, eliminating cold-start delays when a pipeline resumes after an idle period.
More Python libraries pre-installed for taps. AWS S3, Google Cloud Storage, Azure Blob, Excel, YAML, and date/timezone helpers are now baked into the image. Taps that fetch from these sources no longer need a per-run pip install.

April 18, 2026

Live MCP agent monitor and pipeline status self-healing.

Agents tab — new live view of connected MCP agents. See every tool call as it happens, with agent name, arguments, record count, response size, status, and latency. Click any row to expand the full request and response.
Pipeline status now self-heals when a job completes but the summary gets stuck showing “processing” — completed, warned, and errored jobs resolve to their correct final state.
Example agent (market-macro-agent) automatically reconnects with backoff if the MCP connection drops, and degrades gracefully while offline instead of crashing.

April 17, 2026

New AI models from OpenAI and Anthropic now appear in the Configuration dropdowns automatically, without a Datris upgrade.
Claude Opus 4.7 is the recommended Anthropic model for CodeGen (tap script generation, AI data quality rules, AI transformations, JSON Schema / XSD generation, and natural-language → SQL).
Fresh instances are seeded with the latest recommended models by default.

April 16, 2026

Two-pass AI tap optimization and Data Catalog UX.

AI optimize pass. After a tap script passes its initial test, the platform sends it back to the LLM for a performance rewrite and re-tests. Auto-reverts on regression (>=20% slower) or failure. Also available through the API; tap test results now report run duration.
Discovery wizard. Auto-optimize per tap with before/after timing banner, per-row stop button, on-demand AI fix panel for failed items, chat auto-scroll, and re-shown Discover Datasets button after new messages.
Create Tap. Configurable test sample size (defaults to 20 records).
Data Catalog. Kebab menu on Uncataloged items with Edit, Delete, and Move to Catalog (with name-clash detection).

April 15, 2026

Database lockdown, tap script hardening, Create Tap UX.

Server-controlled database name. The UI no longer edits the Postgres or MongoDB database name — the server reports the configured names and the UI submits them unchanged.
MongoDB internal vs user split. mongodb.database (user-facing, default datris) holds pipeline data; mongodb.internalDatabase (default oss) holds platform state. Existing installs keep oss for platform state; new user pipelines land in datris.
Unlimited reads for tap scripts. Tap scripts can now request every matching row from the MongoDB and Postgres query APIs instead of a capped preview. Preview defaults (20 Mongo / 100 Postgres) are unchanged for UI/MCP callers.
Tap test sampling. New “Limit test sample to 20 records” checkbox in Create Tap step 2 caps how much a test run reads. Cron and manual runs read everything.
Codegen + diagnosis hardening. Generated scripts treat platform response shapes as contractual (no shape-probing, no candidate-key iteration). Diagnosis quotes the actual traceback and respects in-script guards.
Create Tap UX. Ask button, auto-apply diagnosis (capped at 2 attempts), Stop Test, copy-to-clipboard, scrollable-JSON test results, destination collision check on Generate Pipeline, full destination shown on step 5.
Data Catalog. One-click delete of the Uncataloged group; per-item trash icon on taps and pipelines inside every catalog.

April 14, 2026

Input sanitization hardening and minor UI polish.

Pipeline creation sanitizes destination identifiers before writing the config — Postgres dbName/schema/table, MongoDB dbName/table, Kafka topic, ActiveMQ queueName, vector collectionName/tableName/schemaName, and the DQ schema name.
Name sanitization is now applied consistently across the UI, so labels and identifiers are cleaned the same way everywhere.

April 14, 2026

Discovery wizard, Data Catalog, and bring-your-own AI keys.

Discovery wizard. Six-step AI-guided onboarding that turns “yfinance daily prices for the S&P 500” into running taps and pipelines. Available from the UI, the API, and the new discover_source MCP tool.
Data Catalog. Group related taps and pipelines into named catalogs. New Data Catalog tab with expandable contents and an Uncataloged group.
Per-pipeline DQ + transformation editor. Inline editor inside the pipeline view.
Bring-your-own AI keys. Hosted trial signups could supply their own Anthropic or OpenAI key, which took effect on the next AI call (hosted platform since retired).

April 11, 2026

Hosted platform release.

Hosted dedicated-instance provisioning and hosted-aware Configuration UI (since retired).
Improved multi-user session handling.

April 11, 2026

Full tap MCP tool suite and user-supplied tap scripts.

Four new MCP tools: get_tap, test_tap, update_tap, get_tap_logs.
create_tap accepts an optional script parameter for user-supplied Python fetch(), plus secret_name for Vault-injected credentials.
CLI: datris tap create --script path/to/script.py and new datris tap show.

April 10, 2026

Ollama for all AI slots and hot-reload on save.

Configuration UI offers Ollama (local) as a provider for AI Primary, CodeGen, and Embedding. Bundled Ollama sidecar option pre-fills bge-m3 for embedding.
Saving AI configuration takes effect immediately — no container restart required.

April 9, 2026

Hosted trial hardening.

Hosted trial cost tuning and Configuration tab polish (since retired). Self-hosted installs continue to recommend Opus for codegen on their own keys.

April 8, 2026

Breaking: AI configuration restructured into three independent Vault secrets.

ai.aiSecretName and ai.provider are replaced by three top-level slots — ai.aiPrimary.secretName, ai.codegen.secretName, ai.embedding.secretName.
Each Vault secret is self-describing (provider, endpoint, model, apiKey, optionally version). No path derivation from YAML.
v1.5.x deployments must update their server configuration and re-seed Vault on upgrade.

April 8, 2026

Onboarding and tap scheduling UX.

Getting Started tab as the new first-run landing page; Docs ↗ top-nav link.
Inline cron editor on the Taps page: preset buttons, AI prompt field (“Every weekday at 4pm”), Quartz field validation, human-readable description.
Truncate-before-run toggle in the tap wizard (sets truncateBeforeWrite: true on the generated destination).
Hosted trial bring-your-own AI key banner on the Configuration tab (since retired).
Fix: scheduled taps now honor the configured date timezone.

April 7, 2026

Taps (Beta). AI-generated Python scripts that fetch data on a schedule and stream it into a pipeline.

4-step tap creation wizard (Describe → Edit & Test → Schedule → Review) with brainstorm chat and proactive env-var suggestions.
AI script generation with JSON-parse retry and raw-script fallback; AI diagnosis with explicit (a)/(b)/(c) options; “Apply Diagnosis” rewrites the script in place.
Tap secrets stored in Vault and tagged as tap-owned; per-run history is retained and viewable.
CRON scheduling (Quartz format), AI-generated from natural language.
JSON-to-CSV pipeline feed (union of keys) with column-name normalization via spell-out table (% → percent, # → num, & → and, …).
Pipelines page shows the feeding tap; pipeline wizard has “Create from Tap”.

April 3, 2026

Schema Evolution.

Additive evolution: new CSV columns are auto-added as string, schemaVersion increments, and ALTER TABLE runs on Postgres.
Dropped columns are left out of the Postgres load so they default to NULL (previously failed on typed columns).
Missing key fields raise a clear error.
Schema evolution now behaves identically for streamed and file-based ingestion.
Query results now include NULL columns.

April 3, 2026

Remote MCP endpoint.

Per-session API key forwarding on the MCP server, with an option to reject unauthenticated remote connections.
Hosted trial signup and dedicated-upgrade agent tools and remote registry entry (since retired).

April 3, 2026

Hosted trial instances: pipeline wizard auto-populated database names (since retired).

April 2, 2026

Shared-instance hosting.

Hosted multi-tenant isolation on a shared instance (since retired).
Vector DB secret isolation across Qdrant, Weaviate, Milvus, Chroma, pgvector.
Batch upload for compressed files (.zip, .gz, .tar, .jar) processes contents inline — no MinIO webhook dependency.

March 30, 2026

Default AI provider changed to Anthropic (Claude Sonnet 4.6). Customers can still switch to OpenAI or Ollama in configuration.
Generated Python scripts for DQ and transformation are logged in full to the server logs for debugging.

March 29, 2026

New unified datris analyze command replaces ask-sql and ask. Auto-picks the right approach based on --dest (Postgres → SQL generation, MongoDB → document fetch, vector stores → RAG).
--ai-analyze flag on ingest.

March 27, 2026

Mintlify documentation and accuracy pass.

Migrated docs from .md to .mdx with a two-tab Mintlify layout.
Removed deprecated docs (row rules, column rules, JavaScript row functions, REST transformations, deduplication, column trimming).
Page-by-page accuracy review against the codebase.

March 26, 2026

Server fails fast at startup with a clear error if an AI provider is not configured (CodeGen DQ and transformation require it).
Removed stale NEVER rules and deprecated-feature references from the MCP server instructions.

March 26, 2026

CodeGen DQ and transformation.

aiRule replaces the prior AI DQ approach: the LLM generates a self-contained Python validation script from a plain-English instruction, which runs locally. Cost drops from $25–40/file to ~$ 0.003/rule.
aiTransformation uses the same CodeGen approach.
Works for CSV, JSON, and XML.
Removed: columnRules (regex), JavaScript row rules, REST endpoint row rules, and the legacy AI data-quality path. JavaScript row functions and REST transformations removed from UI and CLI.

March 26, 2026

Atomic create_pipeline. generate_schema is removed; create_pipeline accepts base64-encoded sample data, auto-detects schema, and creates the pipeline in one call.
Content-based uploads across upload_data (renamed from upload_file), profile_data, upload_config.
update_secret MCP tool for AI provider keys only.
All-string schema on MCP pipeline creation; pipeline registration verified by read-back.

March 25, 2026

update_secret MCP tool added (scoped to AI secrets: anthropic, openai, ollama, embedding).
Published to the MCP Registry (io.github.datris/datris) and PyPI (datris-mcp-server).

March 24, 2026

New UI tabs.

MCP Tab — agent view, service health for 10 backend services, browsable tool grid, config generator for Claude Desktop/Code and Cursor, and a Tool Playground that executes against the live API.
Secrets Tab — full CRUD for Vault secrets with sensitive-field masking.
New REST endpoints for vector store metadata discovery (Qdrant collections and friends).