Changelog - Datris

October 20, 2026

The MCP server and CLI move to the current MCP SDK, with no change for the agents that use them.

MCP server and CLI on the current MCP SDK. The datris-mcp-server package and the MCP server image now run on the current major version of the MCP Python SDK. Agents see no difference: the same tools, resources, results, error messages and version, over stdio, SSE and streamable HTTP. pip install datris-mcp-server now works alongside other packages that need the current SDK. If you install the CLI into a shared Python environment that pins the previous SDK major version, install it in its own environment with pipx or uvx instead.
Oversized requests are refused at once. A request to the MCP server over HTTP that is larger than the server’s upload limit (about 1.3 GB with the default DATRIS_MAX_UPLOAD_MB) is now refused immediately with a clear error instead of being read in full. Anything that fit before still fits.
Refused tool calls show in the Agents tab. A tool call turned away for invalid arguments now appears in the Agents tab activity as an error with the reason, where nothing was recorded before. The call never reaches the platform, as before.
Security update to a UI dependency. The web UI picks up a security fix in one of its framework packages. No user-facing change.

Upgrading: docker compose pull && docker compose up -d; the UI and MCP server images changed. No configuration changes are required. CLI users: pip install -U datris-mcp-server or brew upgrade datris.

October 19, 2026

The recovery agent shows its work: evidence before you approve, a rehearsal before anything lands, and a history of every script it touches.

See what a fix will do before you approve it. Incidents and their approvals show an evidence card: what the resource feeds, columns added and removed, rows and data-quality pass rate before and after, the biggest null-rate changes, each threshold with a pass or fail, and a link to compare the script before and after. Column values are never shown. The incident webhook now includes a one-line summary of the same numbers, with no values.
Fixes are rehearsed in a scratch lane before they land. When the recovery agent proposes a fix for a failing tap, the platform first runs it against a scratch copy of the pipeline and compares the result with the last good run. A fix that clearly breaks the data is stopped with nothing written; one that changes the data’s shape waits for a person, who can review the evidence and choose “Proceed to real run”. Installs with the recovery agent in propose or autopilot will see one extra verification step per incident, and some proposals may now end as failed or waiting for a person on the evidence alone. Taps that fetch documents skip the rehearsal and are verified with the real run, as before. Set recovery.verifyMode to real in the agent policy to keep the previous behaviour for every tap. A rehearsal that passes counts as the fix’s test run, so an approved fix can be saved on a scheduled tap; while the fix awaits approval, the tap’s “Last test” shows the rehearsed fix, and the tap’s own last test comes back if the fix is rejected.
Column changes, rejected credentials and platform problems are named for you. When a scheduled tap run succeeds but its columns changed, an incident opens that names the added and removed columns and what is downstream; it is diagnosis-only unless you set recovery.allowSchemaDriftActions and run the tap on autopilot. A run that fails because the source rejected its credentials is marked as a credential problem, never retried or rewritten by the agent, and its card shows the name of the secret to check. A failure the doctor traces to the platform is marked as a platform problem with the doctor’s fix, before any AI is used. While diagnosing, the agent can now also read lineage, provenance and pipeline version history.
Every AI script change is kept; compare and restore. Each change to the script a pipeline runs for its AI data-quality rule or AI transformation (generated at save or at a run, regenerated, adopted from the code repository, or imported by a catalog restore) is now kept as a numbered version with its text. Open Script history on a pipeline’s run page, or History beside the transformation script in the Lineage panel, to see who changed it and when, compare any two versions line by line, and restore one as the new live script with no model call; the pipeline’s own version does not change. A version written for a different instruction or source schema than the pipeline has now is refused with the reason. Agents get the same through four new MCP tools. Scripts stored before this release appear as version 1 the first time their history is opened or they change.
Changing a pipeline’s AI script now needs the right to update pipelines. Regenerating a script, loading the latest version from the code repository and restoring a script version need pipeline:update instead of pipeline:create. The editor role and full-access keys are unaffected. Keys that hold pipeline:create but not pipeline:update, including keys made from the rag-builder template, can no longer regenerate or load scripts; add pipeline:update to any key that should. Under an agent policy, regenerating and restoring a script are the new pipeline:update:codegen-scripts action, which the Recommended template sends for approval.
The recovery agent can repair a failing AI script. When a pipeline fails because its AI data-quality or transformation script fails, the recovery agent can regenerate that script from the pipeline’s current instruction, within your agent policy. For a pipeline fed by a tap, the new script is proven in the scratch lane and then by a real run of the tap; if either fails, the previous script is restored. For a pipeline fed by uploads or other sources, the incident says plainly that the fix could not be verified and the next run will tell. The incident’s card names the script versions before and after and opens their line-by-line comparison. Destination and schema problems still go to a person. The recovery-agent key gains the right to update pipelines the next time the server starts, recorded in the audit log; a revoked key is left alone.
Script fixes show up in a tap’s history. A script fix applied by the recovery agent, a script generated, fixed, reviewed or optimized with AI, or a script replaced through the API or the tap script editor, now appears in the tap’s version history with who made it. A save that changes nothing records no version, even with a change note.
Recovery settings that fit the environment. The Agent Policy page shows the recovery settings recommended for the install’s DATRIS_ENV label next to the current ones, and an admin can apply them with one click: on dev, autopilot with looser limits (row-count changes up to 50 %, removed columns and type changes allowed); on qa, production or an unlabelled install, propose with the strict limits. The recovery mode never changes by itself and stays off until you set it. The new verification limits default the same way when you have not set them, so a dev install uses the looser limits and every other install the strict ones; saving the recovery settings fixes the values you see.
Choose the recovery mode on the Agent Policy page. The Recovery panel now lets an admin pick off, propose or autopilot, choose scratch-first or real verification, and allow schema-drift fixes, with a one-line explanation of each choice; switching to autopilot asks for confirmation. Thresholds are shown there and still set through the API or the Configuration assistant.
The recovery agent learns source quirks, with your say-so. When an incident is fixed by changing a tap’s script and the agent noted a quirk of the source worth remembering, the note is saved as a suggested tap prompt fragment. It is not used until a person accepts it, after which the tap generator takes it into account for that source. Suggestions can be listed through the tap prompts API; agents can read them but cannot accept them.
An incident closes itself once the tap or pipeline runs healthy again. An incident waiting on a person now closes as resolved when the tap or pipeline next runs healthy, whether you re-ran it after fixing the cause or its next scheduled run succeeded. Any actions it had queued for approval are withdrawn. Resolved incidents no longer hold off a new incident for the same tap or pipeline.
An incident that hits an error after an approval now closes right away. If the recovery agent’s work after an approved step fails with an error, the incident closes as failed with that error instead of waiting for the runtime limit.

October 16, 2026

Run Datris across dev, QA and production: promote what you tested, and always know where you are.

A guide to dev, QA and production. A new Dev, QA and production page under Production explains how to run one install per environment, promote a catalog archive from one to the next, keep secrets and GitHub branches per environment, drive promotion from CI, and point agents and the CLI at the right install.
See which environment you are on. Set DATRIS_ENV to dev, qa, production or any other word and the UI header shows it (red for production, amber for anything else), the version check and datris doctor report it, and catalog archives record where they were exported from. Restoring an archive into an install with a different label adds a warning naming both; the restore still goes ahead. Only production changes behaviour, exactly as before.
Agents know the environment too. A connected agent is told which environment it is on before it does anything, and responses from tools that change or remove things name the environment they changed, so a call to the wrong install stands out. On production, agents are asked to confirm with you before a change. datris version prints the label as well.
Promoted pipelines run the AI scripts you tested. Restoring a catalog archive now makes each pipeline’s archived AI data-quality and transformation scripts the live scripts, with the model and generation time they were recorded with, so production runs exactly what QA tested and the restore makes no model call. The restore result lists which scripts were imported, regenerated or kept, and why. Pass --regenerate-scripts to regenerate them on the target instead. With GitHub script storage, a restore that would place a new pipeline’s AI script on a different file already in the repository is refused and nothing is written.
Deleting never removes a hand-edited script. With GitHub script storage, deleting a pipeline, a tap or a whole catalog keeps any script file that was edited in the repository since Datris last wrote it. The item is still deleted, and the result names each kept file so you can remove it by hand. Saving a pipeline without an AI instruction it used to have keeps an edited script file the same way and says so.
Clearer “not found” answers. Looking up or deleting a pipeline or tap, or running or setting the state of a tap, that does not exist now answers “not found” instead of a server error or a false “deleted”.

Upgrading: docker compose pull && docker compose up -d; the server, UI and MCP server images changed. No configuration changes are required. Installs without DATRIS_ENV look and behave as before, and agents see the same instructions and responses as before. Archives exported earlier restore as before, and their AI scripts are regenerated on the target as before (re-export to carry the scripts’ records). Scripts that delete a tap unconditionally should expect a “not found” answer for a missing tap.

October 15, 2026

Busy hosts stay up, and Claude Opus 5.5 becomes the default chat model.

Busy hosts stay up. Tap scripts and AI rule scripts now run a bounded number at a time, inside CPU, memory and process limits. When the runner is full, a run waits its turn instead of piling on; its run history shows how long it queued, and datris doctor warns when runs had to wait or were turned away. All limits are adjustable in .env.
Claude Opus 5.5 is the default chat model for Anthropic. New installs on the Anthropic and Bedrock paths seed Claude Opus 5.5 for the Assistant and other chat features (it was already the code-generation default), and the model picker recommends it. Existing installs keep the model they have; change it any time on Configuration → AI Providers.

Upgrading: docker compose pull && docker compose up -d; the server, UI, MCP server and script runner images changed. The new compose file gives both script runners CPU, memory and process ceilings and each script a 1 GB memory limit (ceilings, not reservations). Raise them in .env for scripts that need more; see Tap Execution & Isolation. No other configuration changes are required.

October 14, 2026

Bring a restored catalog back to life in one click, and let assistants read whole scripts.

Enable a restored catalog in the right order. Taps that read tables produced by other taps in the same catalog are recognised from their scripts (or declared by you), and each catalog can show the order to enable its taps in, with what still blocks each one. Disabled taps on the Catalog tab show their level or the sources they need, Run now lands a tested tap’s data without enabling it, and the restore result says how many restored taps depend on others. Agents get the same enable order through a new read-only MCP tool.
Enable a whole catalog in order with one click. A new Enable in order action on a catalog’s card tests, enables and runs each tap in its enable order, waiting for each tap’s data to land before starting the taps that read it. Taps whose sources failed are skipped with the reason, progress shows on the card, and the job can be cancelled. Agents can start the same job, and an approval-required agent policy asks for one approval for the whole catalog.
Updated object store image. The bundled MinIO image moves to a newer build with current security patches. Run docker compose pull && docker compose up -d --force-recreate minio minio-init; buckets and data are kept.
In-app assistants read whole tap scripts. In-app assistants can now read whole tap scripts and larger tool results, so they can review and fix a long script without asking you to paste it. A tap version’s script now also shows for taps stored in a code repository.

Upgrading: docker compose pull && docker compose up -d --force-recreate minio minio-init; the server, UI, MCP server and object store images changed. No configuration changes are required; the new endpoints, tools and settings are additive.

October 13, 2026

Back up a catalog as one file and restore it anywhere.

Download a catalog. The download icon on a catalog’s card in the Catalog tab saves the whole catalog as a zip archive: every tap and pipeline definition, each tap’s script and incremental state, and the validation schemas its pipelines reference. Secret values, destination data, run history and version history are never included.
Import a catalog. The Import catalog button restores an archive on the same install or another one, under the original name or a new one. A restore never merges into an existing catalog: clashes are refused and listed with nothing changed. Choose Overwrite catalog and type its name to replace an existing catalog’s contents, or skip the incremental tap state to make a clone that starts from scratch.
Secrets stay yours. The archive lists the secret names the catalog needs, and the restore result names any missing on the target. Inline REST bearer tokens and API keys on a pipeline are removed on export; re-enter them after the restore.
Restored taps start disabled. Taps come back with a Disabled badge in place of their schedule; click it to enable the tap or to test it first.
CLI and agents. datris catalog export and datris catalog restore, plus the export_catalog and restore_catalog MCP tools, with the same refuse-on-clash and confirm-to-overwrite rules.

Upgrading: docker compose pull && docker compose up -d; the server, UI and MCP server images changed. No configuration changes are required; the new endpoints and tools are additive.

October 12, 2026

See which AI model is answering.

The Assistant names its model. The top of the Assistant conversation shows the chat provider and model in use (for example “Anthropic · claude-fable-5-1 · extended thinking on”), and the line stays visible after the first message. Administrators get a link to change it on Configuration → AI Providers; the line updates as soon as the configuration is saved.
Generated tap scripts say who wrote them. When the Assistant or an agent generates a tap script, the result reports which CodeGen provider and model produced it. The tap generation status carries the provider and model from the first poll.
Consistent provider names. The Assistant, tap generation and the Configuration tab use the same display names for AI providers.

Upgrading: docker compose pull && docker compose up -d; the server, UI and MCP server images changed. No configuration changes are required; the additions are new fields only.

October 11, 2026

Server framework update.

Current, supported server framework. The server now runs on Spring Boot 4, Spring Framework 7 and Tomcat 11, a line that still receives security fixes. This closes two open vulnerability reports against the web framework; neither affected path is used by Datris. The API, responses, sign-in sessions, Assistant streaming, metrics and audit log behave exactly as before.

Upgrading: docker compose pull && docker compose up -d; only the server image changed. No configuration changes are required.

October 9, 2026

Single sign-on through your identity provider, and security updates.

Sign in with SSO. With user authentication on, people can sign in through your identity provider (Okta, Microsoft Entra ID, Google Workspace, Keycloak, Auth0, or any OpenID Connect provider) from a new Sign in with SSO button on the login screen. OpenID Connect authorization code flow with PKCE, so MFA, password rules and lockout come from your provider. SSO users are ordinary Datris users with one of the three roles; sessions, role checks, the Users screen and the audit log work as for password users. API keys, the CLI and MCP clients are unchanged. See Single sign-on (OIDC).
You decide who gets in. Users already on Configuration → Users sign in with their role; anyone else is refused unless OIDC_DEFAULT_ROLE is viewer or editor. The built-in admin account is never reachable through SSO. The first sign-in binds the user to the provider’s subject id; a different identity with the same username is refused.
Never in the way of password login. A wrong or incomplete setting switches SSO off with one ERROR line and the server starts as usual; an unreachable provider fails the SSO button with a clear message while password login keeps working, and SSO recovers on its own.
Audited. Every SSO sign-in and refusal is an auth / login entry with method: oidc.
Security updates. Refreshed server and UI build dependencies to clear the open vulnerability reports. No behaviour change.
MinIO health check no longer depends on a config file inside the container.

Upgrading: refresh the compose file (git pull or re-download docker-compose.standalone.yml), then docker compose pull && docker compose up -d; the server and UI images changed. SSO is off by default. To turn it on, set USE_USER_AUTH=true, OIDC_ENABLED=true, OIDC_ISSUER, OIDC_CLIENT_ID and OIDC_REDIRECT_URI, put the client secret in Vault under oss/oidc (key clientSecret), and recreate the server. A secret named oidc is now platform-managed (taps and pipelines cannot reference it). The version endpoint gains oidcEnabled. SSO is single-tenant; no SAML or provider group-to-role mapping yet.

October 8, 2026

Strict evidence mode, a complete record of password changes, a data-flow matrix, and graceful shutdown.

Strict evidence mode. PROVENANCE_STRICT=true makes a stamping pipeline fail its run instead of loading rows it could not stamp (XML sources are exempt and say so). AUDIT_LOG_STRICT=true refuses an audited request with a clear 503 while the audit log cannot record it, instead of dropping an older entry; bucket drops, Kafka messages and scheduled tap runs wait the same way, and everything replays in order when the store returns. Both default to off. datris doctor reports the mode; the audit-log status endpoint shows acceptingWrites. See Audit Log and Provenance.
Password resets and rotations are audited. reset-password names the account and the acting session or key; role changes carry the new and previous role; the startup bootstrap rotation is rotate-password with the system actor (the password never enters the audit log); the first-boot admin seed is create with reason bootstrap. See Recovering admin access.
Data Flows. A new page lists, for each kind of data, every place it can go and the switch that changes it, with a no-cloud-exit recipe. Evidence claims across the docs now say when they hold.
Graceful shutdown. docker stop now reaches the server: requests drain, the Kafka consumer closes cleanly, queued audit entries are written, exit code 143. Previously every stop was a hard kill after ten seconds.

Upgrading: docker compose pull && docker compose up -d; only the server image changed. Refresh the compose file for the new 30 s stop grace period. Stops are now graceful (exit 143, usually 1-3 s; up to ~12 s with the audit log on and the store down). Strict mode is opt-in; with both switches off, behaviour is unchanged. Audit action names changed: password reset is reset-password (was update), first-boot seed is create + reason bootstrap (was seed-admin); existing rows unchanged. New: status fields strict/unrecorded/acceptingWrites, doctor checks audit.strict/provenance.strict, metric datris_audit_unrecorded_total.

October 8, 2026

AI rule and transformation scripts are generated once, stored with your pipeline, and run in an isolated container.

One script per instruction, written when you save. Saving a pipeline with an AI data-quality rule or an AI transformation generates its Python script then and stores it. Every run executes that stored script: an unchanged instruction behaves the same on every run, runs make no model call, and a model outage no longer stops ingestion. A later save regenerates only when the instruction or the source schema changed; pipelines created before the upgrade generate and store on their first run. See AI rules and AI transformation.
Read, review and replace the script. The save response, a new API endpoint and the Lineage panel show each script, when it was generated and by which model. Regenerate on demand; a failing script is never silently replaced, and the run error says when it was generated and how to replace it.
Scripts in your code repository. With a code repository configured, pipeline scripts are committed under pipelines/<pipeline>/ next to your taps, and runs execute the recorded commit. Hand edits are adopted deliberately with one call; a save that would overwrite one is refused with a warning, and runs keep the recorded commit until you resolve it. See Pipeline scripts.
Generated scripts run outside the server. On compose, they execute in a new hardened container, datris-codegen-runner: no platform secrets, no server filesystem, no route to the database, object store, secrets store, internet or the Datris API; a dedicated scratch volume emptied after every run. datris doctor reports the mode. See Execution isolation.
Blank AI instructions are ignored instead of calling the model.
Doctor covers the new container in the environment-drift check and a new codegen.isolation row.

Upgrading: refresh docker-compose.yml (git pull or re-download docker-compose.standalone.yml), then docker compose pull && docker compose up -d. All four images changed; one new container, one new internal network (codegen-net) and one new named volume (codegen-scratch) appear; no .env change. Pulling images without refreshing the compose file keeps scripts in-process with a startup and doctor warning; refreshing the compose file without pulling fails loudly on every AI rule and transformation, never silently in-process. Scripts are now written from column names and types, not sample rows. A failing script stays until you change the instruction or regenerate. Scripts that reached the network or server files now fail; USE_CODEGEN_RUNNER=false restores in-process execution. Keep codegen-net internal: true.

October 6, 2026

datris doctor tells you when a newer version is available.

Know when you are behind. A new version.update row reads “up to date” or names the newer version and the upgrade command. To find out, doctor tells datris.ai your Datris version, operating system and CPU architecture, and nothing else, each time you run it. Never from the server, on a timer, or during --pre-upgrade; nothing is stored on your machine. See Doctor.
Off with one line: DATRIS_UPDATE_CHECK=0 in .env (run doctor from the install directory or pass --project-dir, or set it in the environment). A fresh install prints a short notice, and the installer writes the line when given DATRIS_UPDATE_CHECK=0.
Never in the way. If datris.ai cannot be reached within three seconds the row is skipped and doctor’s result is unaffected.
How to verify: Security Architecture lists both outbound requests Datris can make and commands to confirm there are no others.
Security updates. Refreshed the server, UI and tap-runner dependencies and the tap-runner base image to clear the open vulnerability reports. No behaviour change.

Upgrading: run docker compose pull && docker compose up -d --force-recreate (all four images changed). The update check is on after you upgrade the CLI (pip install -U datris-mcp-server / brew upgrade datris), so each datris doctor run now makes one request to datris.ai; add DATRIS_UPDATE_CHECK=0 to .env to keep it off. On a deployment behind the latest release, datris doctor now exits 1 where it exited 0. datris doctor --pre-upgrade is unchanged.

October 6, 2026

One switch to turn on the governance controls, a doctor check that reports when they are off, and corrected first-login instructions.

Switch on the governance controls in one step. A fresh install takes DATRIS_GOVERNED=1 to turn on user login, API keys, the audit log and the agent policy; an interactive install asks once, defaulting to no. An existing install adds four documented lines to its .env. Nothing changes for installs that do not ask for it. See Quick Start.
datris doctor reports the governance controls. A new governance.controls row names any control that is off, with the lines to add. A warning, never an error. See Doctor.
First-login instructions corrected. admin is created with a bootstrap password printed once to the server log, not a blank password. The docs and login screen say where to read it and to save it before recreating the container. A new Recovering admin access section covers the case where it is gone.
Fixed: a headless install on Debian or Ubuntu stopped silently right after fetching the runtime files. It now continues.
Governed installs skip a check that needs a key. The installer no longer waits on the post-boot store check when API keys are enforced; it prints the command to run once a key exists.
Docs and website wording now say the controls are switched on for production rather than “on by default”.

Upgrading: docker compose pull && docker compose up -d --force-recreate. The server, UI and MCP server images changed. No defaults change. datris doctor now exits 1 on an install with any of the four controls off, where it exited 0. Before turning the controls on, save the admin bootstrap password from the server log (docker compose logs datris | grep "Bootstrap login"); if nothing is printed, see Recovering admin access.

October 5, 2026

A HIPAA Safe Harbor preset, a switch that keeps row values away from AI helpers, clearer errors, and an installer agents can run.

HIPAA Safe Harbor preset for field protection. Datris recognises the Safe Harbor identifier classes from field names with a fixed table and no AI call, proposes a method for each, and lists what needs a person’s review. When enforced, a pipeline cannot be saved with an unprotected identifier and new columns are protected before they land. Available in the wizard, on the pipeline page and to agents. An aid, not a certification. See Field Protection.
One switch so helpers send no row values to a model. With DATRIS_AI_SAMPLE_VALUES=false, schema generation, profiling, Assistant attachments and AI rule and transformation generation work from structure only. The default is unchanged, and datris doctor reports the setting.
A new mask option keeps the first three characters.
Schema generation and profiling no longer fail when the model declines. You get a usable result with a note.
An invalid pipeline is reported as an invalid request, with the same message, shown as plain text.
Failed runs show the error, not a Java stack trace. The trace is behind a toggle in the Ops activity view and in the server log.
Fixed: a column the destination already declares is no longer added twice when it later arrives as a new source column.
Fixed: editing a pipeline in the wizard keeps its protection settings.
Install Datris from an agent. See Install for Agents. The installer also stops clearly when no AI provider is given and cleans up after a failed first run.

Upgrading: docker compose pull && docker compose up -d --force-recreate. The server, UI and MCP server images changed. A refused pipeline save now returns HTTP 400 where it returned 500, with the same message. A failed run’s stack trace moved from the status description to a separate detail field. To use DATRIS_AI_SAMPLE_VALUES on an existing install, add it to your .env.

October 3, 2026

Field protection for PII and PHI, reveal for authorized users, and tighter rules for secrets.

Protect sensitive columns before they land anywhere. Mark a field and its original value never reaches a destination, a preview, or an AI step. Choose a stable one-way pseudonym, a mask that keeps the part you pick, a fixed redaction, dropping the column, or encryption. Protection runs on your server before data quality and transformation, and the raw copies Datris held for the run are removed afterwards. See Field Protection.
Encrypt what an authorized person may need to see again. Encrypted fields are stored as ciphertext. Someone with the reveal permission can get the original back; every reveal is audited and no user ever holds a key. The permission is in no built-in role or key template. Keys can be rotated without losing older rows.
Reveal from the Search tab. Encrypted values show as a lock in results for Databricks, Snowflake, object store, PostgreSQL and MongoDB, and an admin can reveal the current result with one click. Nothing is kept afterwards. Assistants and agents cannot reveal.
Suggestions for what to protect. Datris proposes a method per field from its name and type alone. No data is sent and nothing is applied until you accept it.
Protection in the wizard and for agents. The schema step has a Protect column with an explanation of each option. Agents can set protection and ask for suggestions, and confirm the fields with you first. Lineage shows protected and dropped columns.
Security: stronger protection for stored secrets. Taps can now only use tap secrets, the keys Datris manages for itself cannot be used by a tap or pipeline, and secret names are checked more strictly. We recommend upgrading.
Databricks loads are no longer reported as failed when they succeeded.
MongoDB results read as a table in Search when the documents are flat.
datris doctor checks tap secrets.

Upgrading: docker compose pull && docker compose up -d --force-recreate. The server, UI and MCP server images changed. A tap whose secret is not a tap secret now fails its next run and cannot be saved; create a tap secret for it, or set DATRIS_TAP_SECRET_SCOPE=any to keep the previous behaviour. Secret names with spaces or URL-special characters are refused. Protected pipelines delete their source file after a run unless they opt out. See Taps.

October 2, 2026

Security dependency updates and a skill that teaches your coding agents to use Datris.

Security: updated third-party libraries. Dependency updates close several published vulnerabilities, including high-severity ones, in libraries the server and the UI build use. No Datris behaviour changes. Shared installs should upgrade soon.
A skill for your project’s coding agents. The new datris-platform skill tells Claude Code, Codex, Hermes, OpenClaw and similar agents to use Datris for data work in your own projects, with MCP tools first and the REST API only for code written into the project. See Agent Skill for install commands.

Upgrading: docker compose pull && docker compose up -d --force-recreate. The server, UI and MCP server images changed. No configuration changes are required.

October 2, 2026

Iceberg tables that work on Databricks, Unity Catalog metadata on by default, and a security fix for API keys.

Iceberg pipelines can now target Databricks Unity Catalog. A new managed mode lets the catalog choose where the table lives while Datris writes to it through the catalog, so Iceberg output on Databricks is governed, queryable and kept current after every run. The pipeline status shows the table and its location. Deleting the pipeline leaves the catalog’s data in place.
Unity Catalog metadata is on by default for Databricks pipelines. Descriptions, tags, properties and lineage are written without a per-pipeline opt-in. A pipeline can still turn it off explicitly, and a single environment setting restores the previous opt-in behaviour. The status endpoint tells you where the setting came from.
Clearer guidance when a catalog cannot do what you asked. Deleting a catalog reports any Unity Catalog cleanup it could not complete, and saving a pipeline that points an Iceberg register mode at Databricks warns right away and names the mode that works.
Safer Iceberg history for adopted tables. Tables Datris adopts from an existing catalog keep a longer metadata history and record where they came from.
Security: revoked API keys are now rejected immediately. Revoke a key and every request made with it is refused from that moment, and the refusal is recorded in the audit log under the key’s name. We recommend upgrading.
API errors are returned as JSON. Server errors no longer come back as raw text; clients, the UI and the CLI get a short, readable message. Agents calling tools over MCP see failures flagged as errors instead of ordinary results, and the CLI explains when its key was rejected.

UpgradingRun docker compose pull && docker compose up -d --force-recreate. The server, UI and MCP server images changed. Databricks pipelines that never configured Unity Catalog now get metadata written after each load; set DATRIS_UNITY_CATALOG_DEFAULT=disabled to keep the old opt-in behaviour, or add unityCatalog: {enabled: false} to an individual pipeline. See Unity Catalog.

October 1, 2026

Unity Catalog integration for Databricks, warehouse queries in the Search tab, and less to configure for Databricks pipelines.

Datris annotates your Databricks tables in Unity Catalog. Turn on Unity Catalog for a pipeline and, after every successful load, its table gets a description, stable tags (which pipeline loaded it, its data-quality status, managed by Datris) and run details in the table properties, all visible in Catalog Explorer. Nothing is written unless a pipeline opts in, and a metadata problem is reported as a warning while the load still succeeds.
Lineage appears in Unity Catalog too. The same opt-in publishes source → pipeline → table lineage, with column mappings, to the table’s Lineage tab.
Agents can look around Unity Catalog before building a pipeline. A new tool lists the catalogs, schemas, tables and columns a connection can see, and data discovery can include warehouse tables in its results, marking the ones Datris loaded.
Iceberg tables can be registered with, or committed through, an Iceberg REST catalog. Object-store pipelines writing Iceberg can keep an external catalog current. Databricks Unity Catalog does not yet accept tables at a location you choose; Datris detects this, warns, and keeps writing safely. Use the Databricks destination for governed Databricks tables.
Query Databricks and Snowflake from the Search tab. Pick a pipeline, run read-only SQL and see the results in place. Error messages across every query type are now short and specific.
Less to configure for Databricks. A pipeline no longer needs the SQL warehouse in its configuration when the connection secret provides it, and assistants stop asking for it.
Deleting a pipeline tells you when a catalog still references its table. The CLI shows the same notice. Its keep-data option, which never kept data, now refuses instead of deleting silently.
Editors and viewers can run read-only queries against object-store, Snowflake and Databricks destinations.
Fix: uploads to pipelines that leave out the CSV delimiter no longer fail.
Upgrade: docker compose pull && docker compose up -d --force-recreate. All four images changed. No configuration changes are required; Unity Catalog features stay off until a pipeline opts in. To write tags and lineage, the Databricks service principal needs two additional grants. See Unity Catalog.

September 29, 2026

Security maintenance release.

Security update. The JSON processing library used by the server is updated to close three recently disclosed advisories, one rated high. No functional changes.
Upgrade: docker compose pull && docker compose up -d --force-recreate. Only the server image changed. No configuration changes required.

September 28, 2026

Catalogs can now be renamed, and deleting one asks whether to keep or delete its items.

Rename a catalog in place. A named catalog can be renamed from the Catalog tab; every tap and pipeline in it moves to the new name in one step. If API keys were limited to the old name, you are told which ones to update.
Deleting a catalog keeps its items by default. You choose between keeping the items (they move to Uncataloged) and deleting them together with their data, which requires typing the catalog name. Items that could not be changed are listed with the reason.
Catalog names may now use capital letters, when created or renamed.
Renaming to an existing name is refused. Renaming a catalog to a name that already exists is refused; use Move all contents to combine catalogs.
No near-duplicate names. Creating a catalog whose name differs from an existing one only by capitalisation is refused.
Uncataloged is protected. It can no longer be renamed or deleted as a group; its items are deleted one at a time.
The assistant can rename catalogs. Ask the Catalog assistant, or any connected agent, to rename or remove a catalog and it does so in one step. Agents can only remove a catalog while keeping its items. See Data Catalog.
Upgrade: docker compose pull && docker compose up -d --force-recreate. The server and UI images must be updated together: an older server does not support catalog rename or delete from the new UI.

September 27, 2026

Clearer errors for files whose names cannot be parsed.

The real ingest error is reported. When an uploaded file’s name cannot be matched to a pipeline, the log now says so instead of reporting an unrelated internal error. Files that fail later in ingestion still record their status exactly as before.
Upgrade: docker compose pull && docker compose up -d --force-recreate. Only the server image changed. See Object-store ingestion.

September 26, 2026

Object-store ingestion is more forgiving: a bad file no longer blocks its neighbours, and root-level uploads work.

One bad file no longer stops the other files uploaded at the same time. When several files land in the raw bucket together and one of them cannot be read, the others still run in the same pass. The failure is logged with its bucket and file name.
Uploads dropped in the root of the raw bucket are now ingested. A file placed directly in the bucket, with no folder prefix, is picked up like any other instead of failing with a naming-convention error.
Every file in a multi-file event is ingested. An event carrying several files for the same bucket now processes each distinct file; previously only one of them was kept.
Upgrade: docker compose pull && docker compose up -d --force-recreate. Only the server image changed. See Object-store ingestion.

September 25, 2026

Installs no longer depend on a third-party registry staying public.

MinIO ships from the same place as the rest of the platform. The object store image is now published alongside the other Datris images, so a fresh install or docker compose pull no longer depends on a third-party registry staying public. Existing object-storage data is kept.
Upgrade: docker compose pull && docker compose up -d --force-recreate. The minio and minio-init containers are recreated because the image changed; object-storage data is kept.

September 25, 2026

Change your platform’s settings by asking: every change waits for your Confirm, and secrets never touch the chat.

A Configuration assistant. The Configuration tab has an Ask panel at its right edge that reads and changes what the sub-tabs cover: AI providers, secrets, data sources, the code repository, users, API keys and agent policy. It proposes one change at a time on a card, and nothing is written until you click Confirm; a confirmation works once and expires after ten minutes. Credentials are entered in a masked form on the card, never in the conversation. A new user’s temporary password or a new API key is shown once on the card, with a copy button. The panel is for admins, needs a signed-in session when user auth is on, and is not shown on trial installs. See Configuration Assistant.
Ask the audit log. Ask the assistant who changed what and when in plain language, and it answers from the audit entries themselves.
Doctor from chat. Ask the assistant to run the health checks, optionally with a test request to each AI provider, and it summarizes what needs attention.
Deep links to Code Repository. A link that opens Configuration on the Code Repository sub-tab now lands there instead of on the default sub-tab.
Changes made through the assistant are marked in the audit log. With user auth on, changes made through the assistant are attributed to the signed-in admin and marked via: config-chat in the audit log. See Audit Log.
More chats at once. Several chats can stream at the same time without one failing on a connection-pool timeout; the limits are configurable, see Configuration Reference.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. Update all images together, including the UI. No configuration changes. With API keys and user auth on, the assistant is available in the UI only.

September 24, 2026

Installs and upgrades work again after MinIO closed its public image registry.

MinIO images now come from Chainguard. MinIO withdrew its images from Docker Hub earlier this month and, as of September 24, put its quay.io images behind a login, so every fresh install and every docker compose pull failed with “unauthorized: access to the requested resource is not authorized”. The Compose files now pull a multi-arch MinIO build from Chainguard’s public registry, pinned to a digest. Running deployments were never affected. The installer, the standalone Compose file and a fresh clone all pick this up automatically, and existing object-storage data is kept.
Upgrade: docker compose pull && docker compose up -d --force-recreate.

September 24, 2026

Agents get live, validated reads from any source, and only land what’s worth keeping.

The scratch destination is now called Live Read. Same behaviour, same configuration; only the name changed in the wizard, the Pipelines list, the run detail page, the docs, the Assistant and the CLI. The first mention on each surface still says which JSON key it maps to, so nothing you have written needs to change. See Live Read.
The Assistant offers Live Read. When it asks where data should go, it now names Live Read alongside the structured destinations, with a one-line description, and still never picks it unless you do. It leaves Live Read out when you have asked for a schedule.
The docs page moved. Live Read has its own page under Destinations; the old address redirects.
The CLI signs in to the MCP server. On installs with API keys turned on, every CLI command now sends the key you already export for datris doctor. A missing key gives a one-line message instead of a connection error, and a rejected key gives a one-line message and a non-zero exit instead of raw output and success. The query-string form still works. See CLI.
A clean exit on every command. The CLI no longer prints a stray traceback after a successful command, and commands that make several calls no longer reuse a stale connection.
MCP clients see the real version. The MCP server now reports its own version instead of the library it is built on.
Claude Opus 5.5 and Grok 4.7. Both are in the model catalog. Opus 5.5 is the recommended and first-boot default for code generation on Anthropic and Bedrock; Grok 4.7 is the recommended and first-boot default for Grok. The chat default is unchanged.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. Update all images together, including the UI and the MCP server. Existing installs keep their models; the new defaults apply only on first boot. If you use API keys and run the CLI, export DATRIS_API_KEY once.

September 23, 2026

Taps handle sources of tens of millions of rows: faster batches, honest timeouts, and an estimate before anything is built.

A tap can yield batches. fetch() may yield DataFrames or Arrow record batches as well as single records. Batches are written natively, roughly ten times faster than one record at a time, and land exactly what yielding the same rows one by one would land: integers stay integers, nulls stay null, timestamps keep their precision. Frames read with the pyarrow backend are the fastest of all. Scripts that yield or return records are unchanged. See Taps.
Tests preview, runs run. A tap test started from the Assistant, an MCP client or the CLI now stops after 20 records, exactly like the UI’s Test button, instead of streaming the whole source and timing out. The tools say plainly that a test’s record count is not what a run will produce. Pass a limit of 0 to test the whole source.
Real runs get their own timeout. Tests still stop at five minutes; real and scheduled runs now get an hour by default. Both are settable, and a timed-out run says which mode it ran in and which setting to raise. A mistyped value warns at startup and falls back instead of stopping the server.
A timed-out run keeps its logs. The error now ends with how many records were streamed before the kill and the last lines the script printed, and the run log holds the script’s full output. A script that yields prints progress every 100,000 records, so a slow-but-healthy run reads as slow, not broken.
Agents estimate before they build. For a large source the Assistant and the MCP tools show rows, bytes per record, disk and minutes, compare them against the budgets actually set on your install rather than the documented defaults, and when one is exceeded stop and offer three choices: raise it, narrow the window, or chunk the range. Column projection is named as the lever that cuts both numbers.
Malformed schedules are refused at save. A 5-field Unix cron is rejected with the 6-field equivalent to use, with weekday numbers translated. Taps already stored with a bad schedule log once instead of every 30 seconds.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. Update all images together, including the UI and the MCP server. Two new optional settings, TAP_SCRIPT_TIMEOUT_SECONDS (300) and TAP_RUN_TIMEOUT_SECONDS (3600); see Configuration.

September 21, 2026

GPT-6 Astra support, and an Assistant that stays connected through long tool calls.

GPT-6 Astra works everywhere on the OpenAI provider. Pick or type gpt-6-astra for the Assistant or for code generation and it works for chat, tap script generation, AI data-quality rules, AI transformations, schema generation, brainstorming and natural-language queries; previously every non-chat call to it failed. It appears in the model dropdowns; GPT-5.6 Sol remains the recommended OpenAI model. See AI Configuration.
OpenAI reasoning models show their reasoning. Every Assistant turn on an OpenAI reasoning model now asks for a reasoning summary, so a thinking block appears where before there was none. Models or accounts that decline the summary fall back to the previous behaviour.
The Assistant no longer stops silently during long tool calls. A tool call that ran for several minutes could end the conversation mid-turn with no message. The connection now stays alive for as long as the work runs, and if it is ever cut, the chat says so. The same applies to the Ops, Catalog and Search chats.
Tap scripts stream their sources instead of downloading them. The AI helpers, the Assistant and the documentation now state that a tap has only a small amount of scratch space and must read a remote file directly rather than save it first, and that a run’s size should be estimated against the per-run disk budget before anything is built. See Taps.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. Update all images together, including the UI and the MCP server. No configuration changes required.

September 20, 2026

Taps stream large sources, and failures say what went wrong.

A tap can yield its records. fetch() may return a generator or any iterator instead of a list; records are written to disk as they are produced, so a tap over millions of rows runs in a few tens of megabytes of memory. Scripts that return a list are unchanged. See Taps.
A tap killed for memory says so. The error explains that the script was killed, almost certainly out of memory, and that fetch() should yield records instead of building the whole result. The Assistant and MCP guidance carry the same rule, stop to report after two failed tests, and no longer probe the runner environment.
Rows returned as lists land in CSV pipelines. A tap returning each row as a list with a header row now loads into a CSV pipeline with the header applied and names normalised, or fails with a message naming the problem, instead of corrupting the pipeline schema.
A bad first line no longer changes the pipeline. A CSV upload or object-store pickup whose header does not parse fails before schema evolution, leaving the schema and version untouched. Quoted header names resolve to their plain names.
A blank schedule is no schedule. Saving a tap with an empty schedule stores none and the scheduler stays quiet; taps already saved that way are quiet on upgrade.
Testing a tap that yields stops at the test limit instead of draining the source.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. Update all images together, including the MCP server. No configuration changes required.

September 20, 2026

A pipeline run is no longer limited by memory.

Payload size no longer limits a run. Files uploaded over the API, archives uploaded as a batch, files picked up from the object store and records returned by taps all stream to disk as they arrive instead of being held in the server’s memory. A multi-GB upload or tap run now completes on the default 2 GB heap, and memory use no longer grows with the size of the data passing through.
The tap output cap becomes a per-run disk budget. PIPELINE_MAX_PAYLOAD_MB (default 4096 MB, 0 = unlimited) is the most one run may stage on disk; a run that exceeds it stops with a message naming the variable and leaves nothing behind. Installs that never set the old TAP_MAX_OUTPUT_MB go from a 100 MB in-memory ceiling to 4 GB of disk. The old name is honoured for one more release, with a warning at startup. See Taps.
Uploads honour the same budget. An uploaded file, or the combined contents of a batch archive, above the run budget fails with the same message. The upload ceiling stays at 1 GB by default and is now configurable with DATRIS_MAX_UPLOAD_MB.
Features that must read the whole payload say so. Deduplication, JavaScript row functions, JSON/XML schema validation, single-call REST preprocessors and destinations, and single-message Kafka, ActiveMQ and scratch destinations keep working exactly as before below PIPELINE_MATERIALIZE_MAX_MB (256 MB). Above it the run stops with a message naming the feature and the setting, instead of running out of memory. CodeGen rules and transformations, provenance stamping and every database, warehouse and object-store destination stream and are never subject to it.
REST preprocessors and destinations can batch. A batchSize sends the rows in batches of that size, each call carrying its batch number and the batch count; a REST row function in batch mode accepts a batch size as its sixth parameter. Without a batch size, both keep today’s single call. See Preprocessor.
Record counts are right for more shapes of data. Newline-delimited JSON input reports its record count instead of 1; a quoted value with an embedded newline counts and loads as one row, including through a CodeGen transformation; a quoted value containing the delimiter lands in one column of an object-store destination instead of being split.
Bad input fails at the door. Invalid JSON on a JSON pipeline is rejected when it is received, before a run is created; a file whose header contains none of the pipeline’s columns fails with a message naming both sides instead of loading blank rows.
Testing a tap returns a sample. A tap test returns the first 20 records and says whether more were cut, instead of every record.
datris doctor watches the staging area. Two new checks report whether the staging area is writable with room for one run, and whether run directories left behind by a crash are accumulating. See Doctor.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. Update the server and tap-runner images together; an older tap-runner still works but keeps its previous in-memory path. The compose file adds one named volume, datris-staging, for staged payloads — it is created on first start and needs no action. The bundled compose file now needs Docker Compose v2. No configuration changes are required; if .env sets TAP_MAX_OUTPUT_MB, rename it to PIPELINE_MAX_PAYLOAD_MB before the next release.

September 18, 2026

A schedule only ever runs code that has passed a test.

Test before you schedule. A tap can carry a schedule only once the exact script that will run on it has passed a server-side test run. Saving a schedule on a never-tested or just-edited script is refused with a message naming the tap and the remedy: save unscheduled, test, then add the schedule. Cadence changes, clearing the schedule, and metadata edits on a tested tap need no re-test. See Taps.
Editing a scheduled tap never repoints the schedule at untested code — by hand, by an agent over MCP, through the AI generate, fix, review and optimize helpers, or by restoring an earlier version. The scheduled script keeps running as it was until the new one is tested and the schedule re-applied.
The wizard handles it for you. Saving a scheduled tap saves it unscheduled, runs the test, and applies the schedule on success; on failure the tap is kept without its schedule and the wizard says so.
Existing scheduled taps are untouched. They keep running and stay editable; their first schedule or script change is accepted if the tap has ever tested or run successfully.
Fix: the CLI works on Python 3.14 (every command previously failed with an event-loop error).
Fix: upload, generate and profile return a plain error message instead of a stack trace.
Fix: the MCP Registry listing now names the settings the server actually reads; a Glama directory listing was added.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. No configuration changes required. Agents editing a scheduled tap’s script must clear the schedule first and re-test (the tools explain this on refusal). CLI users on pip or Homebrew: upgrade for the Python 3.14 fix.

September 18, 2026

Run a pipeline and get the rows back, without landing them.

Scratch destination. For when you, or an agent, want an answer now rather than a dataset: what a source returns for these ids, whether today’s file passes its rules, what a transformation produces. The run goes through the same preprocessing, data quality and transformation chain as any other pipeline; the rows are held briefly so the caller can read them back. Nothing is landed, catalogued or added to lineage, and the result expires after a retention window you control. Structured sources only; cannot be combined with another destination. See Scratch.
Reading the result. The first rows ride on the run’s status. Larger results page from the run detail page’s View result panel, from the CLI (datris pipeline result), and from the new get_pipeline_result MCP tool. Paging never re-runs the source.
Agents know when to keep and when to scratch. The MCP server and the Assistant follow a keep-or-scratch rule: data to be kept, observed, queried later or refreshed on a schedule goes to a real destination; a one-off answer goes to scratch. Fetch-only taps target a scratch pipeline; promote it later by changing the destination — the tap, its schedule and its incremental position do not move.
Scratch in the UI. The pipeline wizard offers Scratch under Destination; the Pipelines list badges scratch pipelines.
Fix: editing a secret keeps the values you did not change — saving one field no longer blanks the masked sensitive fields you left alone, in the UI and over the API.
Fix: the destination’s authoritative flag set over the REST API was silently ignored; it is now applied and validated.
Hardening. S3 object-store endpoints pass the same network safeguards as other outbound connections; pipelines writing to the built-in object store can be restricted to an allowlist of buckets (off by default, reported by the doctor); query and status error responses no longer include internal diagnostic detail.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. No configuration changes required. Scratch results are held for 24 hours by default; shorten it on shared hosts. CLI users on pip or Homebrew: upgrade to pick up datris pipeline result.

September 17, 2026

Pipelines can now land data as Apache Iceberg tables.

Iceberg as an output format. Set the file format to iceberg on an object-store or S3 destination and each run commits a real Iceberg table at the pipeline’s location: Parquet data files plus the table metadata any Iceberg-aware engine reads. One atomic commit per run; a failed run leaves the previous snapshot intact. No new destination type, no extra service. See Iceberg.
Upserts by key. A new merge write mode updates rows matching the configured key fields and inserts the rest; re-running the same data leaves the row count unchanged.
Partitioning and schema evolution. Partition by columns; new nullable columns are added automatically, while type changes, dropped columns, and new required columns are refused before anything is written.
Readable without Datris. Standard Iceberg tooling opens the table by path. Time travel, catalog registration, table maintenance, and Iceberg as a tap source come later; tables written today will not need rewriting.
Snapshot in query results for Iceberg tables; the wizard, MCP tools, and Assistant know the new format and modes; lineage labels object-store destinations with their format.
Fix: querying an object-store pipeline that has never run returns an empty result instead of an error (a pipeline whose bucket does not exist also reads as empty).
Overlapping runs writing the same object-store location now run one after the other, for every object-store format.
S3 credential changes apply immediately. Correcting or rotating S3 credentials in a secret now takes effect on the next run without a restart.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. No configuration changes required; existing Parquet and ORC pipelines are unaffected. Switching an existing pipeline to Iceberg in place is refused unless deleteBeforeWrite is set; use a new prefix to keep the old files. The server image is about 50 MB larger.

September 15, 2026

A smaller install, and docs written for the way people search.

Minimal install. The installation guide now leads with a minimal path that skips the bundled embedding server and bundled PostgreSQL and fits in a 4 GB Docker memory allocation; the full stack needs about 8 GB. Numbers measured on a live stack.
OpenAI embeddings by default when you have a key. Fresh installs with an OpenAI key present use OpenAI embeddings instead of the local embedding server, whichever provider handles chat. Existing installs are unchanged; set the embedding provider explicitly to keep embeddings local.
New recipe: Give Claude Code a safe way to load data into PostgreSQL — MCP connection, approval gates, a load, and the audit and lineage records it leaves.
Docs pages named for the task. Destination, MCP, policy, audit, lineage, and Airflow pages are titled for what you are trying to do. URLs unchanged.
More reliable release builds for the arm64 UI image.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

September 14, 2026

The doctor now runs itself.

Upgrades check themselves first. Re-running the installer on an existing install runs the pre-upgrade self-check before pulling, when the datris CLI is on the machine. A warning continues; an error stops the upgrade with the fix on screen (and can be bypassed deliberately). Without the CLI the installer says so and continues as before.
Continuous checks. Set an interval and the server re-runs its self-checks on that cadence, posting to the incident webhook when a check flips to an error or recovers. Off by default; AI probes never run on the timer. See Doctor.
The recovery agent can consult the doctor. When an incident looks like the platform rather than the tap, the agent runs the quick self-check and hands the incident to a person with the doctor’s fix.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users: upgrade to 1.29 or later (pip install -U datris-mcp-server / brew upgrade datris) — the installer’s pre-upgrade check needs it and will keep reminding you until you do.

September 14, 2026

Installs and upgrades work again.

MinIO images now come from quay.io. MinIO removed its images from Docker Hub, so every fresh install and every docker compose pull failed with “pull access denied for minio/minio”. The Compose files now pull the identical images from quay.io; running deployments were never affected. The installer, the standalone file and a fresh clone pick this up automatically.
Docs: the production trail. A new Production section walks a deployment from first install to a governed, audited, agent-operated setup; the Quick Start now ends on an approval and a provenance lookup.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. Compose recreates minio on the new image name; the data volume re-attaches. No configuration changes required.

September 10, 2026

Datris can now diagnose its own deployment.

Doctor. A built-in operational self-check finds the problems that used to be discovered only after an outage: a Vault token about to expire or silently capped, an AI provider secret that is missing or incomplete so the server would not come back after an upgrade, an embedding model that was never pulled, a disk about to fill, and a mixed-version stack. Every finding comes with the command that fixes it; nothing is changed automatically.
Four places to run it. The cheap checks run at every server start as DOCTOR log lines (a healthy install logs no warnings). The full set runs on demand from Configuration → Doctor, from the new run_doctor MCP tool, and from the new datris doctor CLI command.
Host checks from the CLI. On the machine running Docker, datris doctor also catches data on an anonymous volume, data left behind on a dangling volume, a container still running with an old .env, containers from a previous Compose file, a stale from-source build, and an unreachable MCP server. datris doctor --pre-upgrade works with the server stopped and is step 0 of the upgrade guide.
Optional AI probe. On request, the doctor sends a minimal request through each configured AI model to confirm the key and model id still work. Never run automatically.
Version stamps. The server reports the exact build it is running and the UI image carries its version, so drift between components is visible.
Fresh-install AI defaults refreshed. New installs seed Claude Fable 5.1 (Anthropic assistant), GPT-5.6 Sol (OpenAI) and Claude Sonnet 5 (overload fallback); code generation stays on Claude Opus 5. Existing installs are unchanged.
Upgrade: datris doctor --pre-upgrade, then docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.29.0.

September 10, 2026

Failed loads now say which destination failed, and why.

Destination failures are named in the job status. When a run fails while writing to a destination, the job’s error identifies that destination and carries its own error message instead of a generic orchestration failure with a stack trace. The full stack trace remains in the run’s event stream.
Agents and the CLI act on the real cause. The Assistant, MCP clients and the recovery agent read the same job status field, so they see the failing destination directly. The datris CLI prints the failing destination with the error on a failed upload.
Multi-destination runs still report as a single job; when one destination fails and another succeeds, the error names the one that failed.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.28.2.

September 8, 2026

Security and documentation maintenance release.

Security update. The embedded web server that serves the API is updated to close three recently disclosed critical advisories affecting authentication and authorization handling. The UI and tap-runner images pick up the latest OS package security fixes. No functional changes.
Hardening switches now take effect. DATRIS_ENV, DATRIS_ALLOW_PLAINTEXT_DB, DATRIS_ALLOW_PRIVATE_EGRESS and TAP_MAX_OUTPUT_MB were described in .env.example but not passed through to the server by the bundled Compose files. They are now honored and documented in the Configuration Reference.
Documentation accuracy pass. Every page on docs.datris.ai was checked against the current release and corrected, including the MCP tool list, CLI flags, Assistant defaults, tap runtime behaviour, ingestion and destination options, the API reference and OpenAPI spec, and the examples.
Examples. Two examples that depended on files no longer in the repository were removed.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.28.1.

September 4, 2026

Lineage now reflects what actually ran, not just what is configured — with an interactive graph and an agent tool to traverse it.

Run-level lineage. Every pipeline run is recorded as it completes: what it read, which destinations it wrote — each with its own success or failure — how many records, and the definition version it ran under. Automatic for every pipeline, independent of provenance stamping, and never fails or delays a run.
History in the graph. A dataset a pipeline used to land into under an earlier configuration stays in the lineage graph marked historical, with the runs that wrote it.
Lineage graph. Catalog → Lineage opens an interactive Source → Tap → Pipeline → Dataset → Catalog view, filterable by catalog, tag or name. Click any node for what feeds it, what depends on it, freshness and recent runs, with links to the tap or pipeline.
Recent runs per node, plus upstream-only / downstream-only / bounded-depth traversal on the lineage neighborhood endpoint.
New MCP tool: get_lineage for impact analysis before changing or deleting something, and for seeing what a pipeline actually wrote lately.
Real sources in lineage. A tap’s source node names where the data actually comes from: the tap’s new optional source field, else the endpoint host, else the host its script calls most. Taps sharing a provider share one source node.
Evidence on every lineage hop: runs, records, failures and last run per edge (last 90 days); claims without runs draw faint; numbers on the panel chips and in find_data.
Source of authority: datasets labeled system of record / derived / undeclared, declared never inferred; single destinations authoritative by default; authoritative: false for derived copies, destination.authoritative for multi-destination pipelines; find_data prefers the authoritative copy; “Canonical only” graph filter.
Column-level lineage in the graph’s side panel and via a new endpoint: exact schema-derived mappings always, system columns for provenance stamps, and an opt-in AI-inferred tier for pipelines with a transformation that returns only evidenced mappings, cached per definition version.
Fix: AI transformations that add, remove or reorder CSV columns now land correctly; the pipeline picks up the transformed column list instead of projecting by the old one.
Provenance resolution for recorded runs now includes per-destination outputs and resolves the definition version from the record when the row’s stamp is unavailable.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. Runs are recorded from the first run after upgrading. CLI users on pip or Homebrew: upgrade to 1.28.0.

September 2, 2026

Claude Fable 5.1 is now the recommended chat model, plus security and dependency updates.

Claude Fable 5.1 support. Anthropic’s newest model is selectable for both the Assistant and code generation, on the direct Anthropic provider and on Amazon Bedrock. It is the recommended chat model on both; code generation keeps Claude Opus 5. Fresh Bedrock installs default to Fable 5.1; existing installs keep their configured models. On Bedrock it needs the same one-time data-retention opt-in as Fable 5.
Security update. The UI’s build tooling picks up fixes for several recently disclosed high-severity advisories in a third-party component. No functional changes.
UI framework update to the latest Angular patch release.
CLI and MCP server via pip. The published package now pins the MCP SDK to the supported major version, so a fresh pip install datris-mcp-server no longer pulls an incompatible SDK that fails at startup.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required. CLI users on pip or Homebrew: upgrade to 1.26.1.

September 2, 2026

Provenance on your data: any row can say which run, which configuration, which script, and which source produced it — and agents can find datasets by meaning.

Opt-in provenance stamping per pipeline: every row, document, message, and vector chunk landed carries the run that produced it, when it landed, the definition version, the tap run that fed it, the exact script commit, and the declared source. Existing rows are never rewritten — provenance starts with the first stamped run; SQL destinations gain the columns automatically.
Every stamp resolves back to its origin — the run, the feeding tap run, the script commit, the definition version with its change note, and the source — via the platform or the new get_provenance agent tool. Vector search results carry provenance per chunk so RAG answers can cite the run and source behind each passage.
A lineage view shows source → tap → pipeline → landed dataset → catalog, derived from configuration alone. Pipeline detail pages show what feeds them, where they land, and how fresh they are, using the same staleness judgment as the Activity dashboard.
Tags on taps and pipelines: free-form labels shown in the catalog and ranked by discovery.
find_data: agents discover datasets by meaning — the best-matching pipelines their key can read, each with location, freshness, provenance handles, and a ready-to-run query hint. Discovery only: the query stays the agent’s own call, under its own permissions. AI answers can carry the provenance handles along.
Deleting a tap or pipeline from the catalog now shows progress while the platform finishes cleaning up.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; provenance stamping is off by default — enable it per pipeline in the pipeline editor’s Destination step.

September 1, 2026

The recovery agent: Datris notices broken data flows, diagnoses them, and — within limits you set — repairs and verifies them.

The opt-in recovery agent opens an incident when a scheduled tap keeps failing after its automatic retries, a pipeline load errors, a tap goes quiet past its cadence, or a pipeline’s volume swings against its baseline — then diagnoses it with the same judgment as the Ops chat and closes it with a step-by-step story.
The Agent Policy gains a recovery mode: off, propose (every repair waits for one-click approval), or autopilot (repairs follow your per-action policy) — with per-tap / per-pipeline overrides.
Repairs are narrow and verified: re-run, script fix (as a new version), test — never deletes, secrets, or schema changes. Every fix must prove itself with a clean real run, or the platform reverts it and says so. Budgets are enforced by the platform, not a prompt.
A new Incidents panel on the Activity page shows state, classification, and timelines with approval links; the Ops chat explains open incidents instead of re-diagnosing; two new agent tools read them; the audit log carries the full per-incident ledger; an optional webhook reports milestones.
The MCP page’s tool catalog now lists every available tool, including the destination-typing pair missing since v1.22.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; set USE_AGENT_POLICY=true + RECOVERY_AGENT_ENABLED=true and pick a recovery mode to opt in.

August 31, 2026

Security update for the UI and MCP server images.

The datris-ui and datris-mcp-server container images ship with updated OpenSSL packages, picking up upstream fixes for several recently disclosed issues, including one rated high severity. No functional changes.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 31, 2026

Agent Policy: decide what agents may do on their own, what waits for your approval, and what is refused.

A new opt-in Agent Policy lets an administrator set, per action — run a tap, delete a pipeline, migrate destination column types, write a secret, and every other change an agent can make — whether agents do it unattended, queue it for a person to approve, or are refused. People using the UI are never gated; they are the approvers. Nothing changes until you set a rule, and the policy applies to every agent: the built-in Assistant and any connected MCP client alike.
Queued actions appear under Activity → Approvals with who asked, the reason they gave, exactly what would run, and one-click Approve / Reject. Approving performs the original request on your behalf and records the whole chain — request, decision, and result — in the audit log. Approvals go stale if the pipeline or tap changed in the meantime, expire on their own, and can never be approved by an agent.
Agents are told the truth: a gated action comes back “waiting for approval” (never silently done), a refused one says so plainly, and the chat shows matching status cards. Agents can read the policy up front, list what they have queued, and poll for your decision; every change-making agent tool now accepts a short reason that shows up on the approval card and in the audit log.
Configuration → Agent Policy manages it all, with a recommended starting point and per-pipeline / per-tap overrides that can only tighten. The Configuration → Environment tab has been removed.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; set USE_AGENT_POLICY=true to turn the agent policy on.

August 28, 2026

Audit log: who did what, by login or API key.

A new opt-in Audit Log records every create, change, run, delete, login, and denied request — humans by their login, agents by their API key, scheduled runs as system. Admins read it under Configuration → Audit Log with filters by time, category, actor, and outcome, a detail view per entry, and CSV export. Entries are also written to the server log so an existing log aggregator picks them up.
Actions the Assistant takes for you are attributed to you — in the audit log and in pipeline and tap version history — instead of to the platform’s internal key. Every issued API key now carries a stable id, so a key that is revoked and later re-issued under the same name is never confused with its predecessor.
Reads are left out by default to keep the trail focused on changes; turn them on to see every query and search an agent runs. Reading a secret is always recorded.
Tap scripts that read platform data now work when API keys are required. Each run gets a short-lived, read-only credential that is attached automatically — scripts need no changes and no platform credential in their secret — and the audit log names the tap that made each call.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required; set USE_AUDIT_LOG=true to turn the audit log on.

August 27, 2026

Set real column types on pipelines that landed as text.

Pipelines created through agents store every destination column as text. The Catalog now marks such pipelines (PostgreSQL, Snowflake, and Databricks destinations) with a quiet “text” badge — click it to review proposed column types and apply them. The same action is available from the pipeline view.
Proposed types are inferred from the data already loaded, and every column shows real sample values so you can check the proposal at a glance. When a column stays text because of a stray value, the dialog shows the offending value and the type it blocked (for example: found “N/A”, would otherwise be a number) so overriding is an informed choice.
Applying is safe by design: every loaded value is validated first, the destination table is retyped, and the pipeline definition is updated as a new version. A value that won’t convert fails the whole apply with the column named and nothing changed. From then on, incoming data is type-checked on every load.
Agents get the same capability through two new MCP tools, get_dest_types and apply_dest_types — applying always requires the user’s explicit approval.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 26, 2026

Live progress feedback across the Assistant and tap building.

The Assistant shows what it’s thinking about while it reasons — a live summary streams next to the progress indicator with an elapsed timer, instead of silent dots. The full reasoning stays available in the expandable thinking block.
AI tap-script generation shows live progress: the tap wizard and the Assistant’s tool cards report what the generation is doing and how long it has been running.
The Assistant sets time expectations before long-running operations, and when AI script generation fails for a well-known API it writes the script directly instead of retrying — turning a multi-minute failure into a quick recovery.
The Assistant only reports an action as completed when it actually ran and observed the result — tightened guarantees against overstated progress reports.
Dependency updates across the UI and build tooling.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 25, 2026

Security refresh across all container images.

All four container images rebuilt on updated bases with refreshed OS packages and Python tooling, clearing every published high- and medium-severity vulnerability finding against the images.
Every container now reports its own health status, so docker ps (and any orchestrator) shows healthy/unhealthy per service out of the box.
Upgrade: docker compose pull && docker compose up -d --force-recreate. No configuration changes required.

August 25, 2026

MCP tool catalog now matches API-key permissions.

With API keys enabled, each MCP session’s tool list is filtered to what its key is actually allowed to do — agents no longer see tools that could only fail with a permission error at call time. Full-access keys and installs without API keys keep the complete catalog.
The rag-builder key template gained the read permissions its workflow was missing (list taps, manage its own tap credentials). Existing issued keys are unchanged — re-issue or edit rag-builder keys to pick up the additions.
With API keys enabled, MCP sessions that present no key are refused instead of silently connecting. Installs without API keys are unaffected.
Permission denials return a clear, structured error instead of a generic server error, so agents recognize a permission boundary rather than retrying.
Tightened API-key permission enforcement on the server.
Upgrade: docker compose pull && docker compose up -d --force-recreate. If API keys are enabled, ensure MCP clients send their key (header for remote connections, DATRIS_API_KEY environment variable for stdio).

August 24, 2026

Opt-in Postgres TLS enforcement.

Deployments marked as production now refuse to start when the platform’s Postgres connection points at an external host without TLS (sslmode=require or stricter), with an explicit opt-out for deployments that intentionally use plaintext. The bundled in-network Postgres is exempt; existing deployments are unaffected unless production mode is enabled. Plaintext connections to external databases now log a startup warning either way.

Observability: Prometheus metrics and structured logs.

Prometheus metrics are now exposed for scraping, reachable only from inside the deployment’s network (the bundled edge proxy blocks them publicly). See Audit Log for the metrics endpoint.
Production deployments emit one JSON log line per event for SIEM / log-aggregator ingestion; development logs stay human-readable.

Tap isolation on by default for docker compose.

All bundled Docker Compose configurations now default to USE_TAP_RUNNER=true. The runner token is minted automatically on first boot (including upgrades where it was missing or still the placeholder value). The server fails fast if isolation is on and no valid token exists — a minted token is accepted.
Running the server outside Compose (no runner sidecar) stays in-process and logs a warning. Isolated taps still cannot open direct connections to internal DB / object store / Vault — they return records; this is existing behavior, called out because compose now isolates by default.
Set USE_TAP_RUNNER=false to keep in-process on compose. No host allowlist in this change.

August 21, 2026

Dependency security cleanup across the server and UI.

Updated server and UI dependencies to resolve every remaining published medium-severity advisory (and one low). One low-severity advisory remains open upstream with no fixed release available. No functional changes intended.
The Activity tab’s charts moved to the current major version of the charting library as part of the security updates — same charts, slightly refreshed default colors.
Standard upgrade: docker compose pull && docker compose up -d. Existing pipelines, taps, and schedules are unaffected.

August 21, 2026

Server security updates and a friendlier vector-store default in the installer.

Upgraded the server’s embedded web framework to its current supported release line, resolving all remaining published high-severity advisories against server dependencies (and a number of medium- and low-severity ones). No functional changes intended.
The installer now presents pgvector as the default vector store. It ships inside the bundled Postgres, so vector search works out of the box with no extra container. Additional vector stores (Qdrant, Weaviate, Chroma, Milvus) remain opt-in additions at install time. Existing installs are unaffected — prompts and install summary only.
Standard upgrade: docker compose pull && docker compose up -d. Existing pipelines, taps, and schedules are unaffected.

August 20, 2026

Security updates across the UI and server, and a pipeline-delete fix.

Upgraded the web UI’s underlying framework to the current long-term-support line, resolving all published critical- and high-severity advisories affecting the UI. No visual or functional changes intended.
Updated bundled server dependencies to resolve the remaining published high-severity advisories against third-party libraries. No functional changes.
Deleting a pipeline now also removes its data from the built-in object store destination, matching every other destination type. Previously the files were left behind, and recreating a pipeline with the same output prefix silently picked them up. Shared output prefixes are detected and safely skipped. Files orphaned by deletes on earlier versions are not removed retroactively.
Hardened the UI container image configuration. No behavior change.
Standard upgrade: docker compose pull && docker compose up -d. Existing pipelines, taps, and schedules are unaffected.

August 20, 2026

Dependency security updates and an install fix.

Updated bundled server dependencies to resolve all critical published advisories against third-party libraries, and removed an unused legacy dependency tree from the server entirely. No functional changes.
Fresh installs now create the default object-store output bucket automatically — previously the first pipeline writing to the built-in object store with default settings failed until the bucket was created by hand.
Standard upgrade: refresh your checkout (or re-download docker-compose.standalone.yml), then docker compose pull && docker compose up -d. If you update images without refreshing the Compose files, create a bucket named {environment}-data (default: oss-data) once in the MinIO console.

August 19, 2026

Security hardening.

A broad security pass across authentication, authorization, credential handling, and input validation, plus updated bundled dependencies for published advisories. A standard upgrade needs no configuration changes.
Initial admin login (when user login is enabled): the first admin password is now generated and printed once to the server logs — docker compose logs datris, sign in, and change it.
Taps calling internal addresses are blocked by default, with an opt-out for endpoints that legitimately run on an internal network (see the Tap Execution & Isolation guide).
Bundled Vault uses a unique per-install token; for manual vault commands read it with docker compose exec -T datris cat /vault-token/token.
Fix: the default Docker Compose config pulls the published images again (a recent change had it building from source).
Standard upgrade: docker compose pull && docker compose up -d.

August 18, 2026

Build taps in any language — HTTP taps.

HTTP taps. A tap can now be a service you host — Rust, Go, TypeScript, a serverless function, or an existing internal system. Datris calls your endpoint on every run and everything else works exactly like a Python tap: schedules, per-run params, incremental sync state, run history, retries. Your service keeps its own upstream credentials, and no tap code runs on the platform. Wire contract and a complete Rust example: HTTP Tap Contract. Create one from the wizard, the CLI, or an agent over MCP.
A more focused tap wizard. Two choices: paste your own Python script, or point at an HTTP endpoint. AI generation stays available through the Assistant and connected agents. Structured vs. document output is detected automatically from the test run, and a passing script is kept exactly as tested — the automatic performance-rewrite pass is gone.
GitHub-stored taps open at the latest code. The editor loads the newest repo version on open (runs keep the previously saved version until you test and save). False “script was changed” warnings caused by unrelated repository commits are fixed, and the AI test diagnosis now works for repository-stored scripts.
Catalog quality of life. Drag-resizable columns on the taps and pipelines tables (layout remembered), and background refresh no longer interrupts column resizing, move-to-catalog menus, or delete confirmations.
Standard upgrade: docker compose pull && docker compose up -d — everything is additive; existing taps and schedules work unchanged.

August 17, 2026

New AI provider: Grok (xAI).

Grok for chat and CodeGen. Grok joins Anthropic Claude and OpenAI as a third model family — the Assistant, tap script generation, AI data quality rules, transformations, and natural-language → SQL can all run on it. Configure it in Configuration → AI Providers with one key from console.x.ai; Grok 4.6 is the recommended model, with Grok Code Fast 1 available for CodeGen. Mix providers freely across sections. See AI Configuration.
Install-time support. The installer prompts for an xAI key alongside Anthropic and OpenAI, and first-boot .env seeding supports Grok as the chat/CodeGen provider.
Embeddings unaffected. xAI has no embeddings API, so semantic search stays on OpenAI, Azure, or the bundled local model — a Grok-only setup works out of the box with the bundled embedder.
Standard upgrade: docker compose pull && docker compose up -d — no configuration changes required.

August 14, 2026

Assistant fix: taps that read data already in Datris no longer ask for database credentials.

No more credentials form for platform data. When a tap’s fetch logic depends on data a pipeline already maintains (e.g. an id list kept fresh in a Datris table), the Assistant previously asked for database credentials or offered to freeze a snapshot of the data into the script. It now builds the tap to read the platform data live on every run — no extra credentials, and the tap’s secret stays limited to the external source’s API key.
Same guidance for MCP agents. Agents connected over MCP (Claude Desktop, Claude Code, and others) receive the same instructions.
Standard upgrade: docker compose pull && docker compose up -d. Start a new Assistant conversation after upgrading — existing conversations keep the older behavior until reopened.

August 13, 2026

Azure OpenAI without API keys — Microsoft Entra ID authentication.

Keyless auth for Azure OpenAI. Required for resources with disableLocalAuth (API keys disabled by policy), and useful anywhere you’d rather rotate credentials centrally. Chat, CodeGen, embeddings, and the Assistant all honor it. See AI Configuration.
Service principal mode. Enter your Entra Tenant ID, Client ID, and Client Secret once in Configuration → AI Providers; tokens are acquired and refreshed automatically.
Managed identity mode. Zero stored secrets when Datris runs on Azure compute (VM, AKS, App Service) — the server authenticates as its managed identity.
Safe mode switching. A stored API key always takes precedence, so the auth-mode selector clears the mode you’re leaving — a stale key can never shadow Entra auth.
Install-time support. Keyless setup works from .env too: provide the service-principal values, or nothing at all on Azure compute. Grant the identity the “Cognitive Services OpenAI User” role on the resource.
Existing Azure OpenAI API-key setups keep working exactly as before — no configuration changes on upgrade.

August 12, 2026

Reliability: long AI generations complete, and the Assistant shows live progress.

Large script generations finish reliably. Generating a big tap script can legitimately take more than five minutes — previously these could time out or hang partway. Generation responses now stream under the hood with a much larger time budget, so long generations run to completion.
Live progress in the Assistant. Running steps show an elapsed-time counter once they take more than a few seconds, plus a note after about a minute explaining that long generations are normal and temporary provider errors retry automatically.
Unresponsive AI providers no longer hang requests. AI calls fail fast during a provider outage and feed the automatic retry ladder, instead of waiting indefinitely on a response that will never come.

Standard upgrade: docker compose pull && docker compose up -d — no configuration changes required.

August 11, 2026

Amazon Bedrock support — run Claude through your AWS account.

New AI provider. Amazon Bedrock joins Anthropic, OpenAI, Azure OpenAI, and Ollama for the chat assistants and CodeGen — Claude models served from your own AWS account with IAM authentication and AWS billing. Configure it in Configuration → AI Providers. See AI Configuration.
IAM-native auth. Enter AWS credentials once, or leave them blank on AWS infrastructure to use the server’s IAM role — zero stored secrets.
Live model discovery. The model dropdown shows what your AWS account and region can actually invoke — including cross-region inference profiles — rather than a hardcoded list.
Claude Fable 5 support. Chat recommends Claude Fable 5; CodeGen recommends Claude Opus 5. Fable 5’s one-time AWS data-retention opt-in is explained in-product and in the docs.
Deliberate opt-in. Bedrock is only ever selected explicitly — AWS credentials used for S3 destinations never silently become your AI provider. The installer offers Bedrock when you don’t have a direct provider key.
Clearer credentials panel. The Configuration → AI Providers key panel is now grouped per provider with a short explainer; credentials hide immediately after saving.
Fix: provider-scoped model lists. A model saved under one provider no longer appears in another provider’s dropdown after switching a section between providers.
Refreshed recommendations. Claude Fable 5 is now the recommended chat model for the Anthropic provider.

August 10, 2026

Azure OpenAI support — run Datris AI on your Azure resource.

New AI provider. Azure OpenAI joins Anthropic, OpenAI, and Ollama as a first-class provider for all three AI slots (AI Provider, CodeGen, Embedding) — mix it freely with the others. Configure it in Configuration → AI Providers with your key, resource endpoint, and deployment name.
Deployment-name aware. Azure addresses models by your deployment names; name deployments after the model they serve so model-specific handling applies automatically.
Installer support. The one-command installer offers Azure OpenAI when you don’t have a direct Anthropic or OpenAI key, and first-boot seeding configures everything from your Azure details. See AI Configuration.
Built for how Azure behaves. Responses stream under the hood so long reasoning calls complete reliably; both current and legacy Azure endpoint styles work.
Fix: session-expiry login. Signing back in after a session expires no longer requires a full page refresh.
Fix: per-provider key updates. Updating one provider’s API key through the API no longer clears the other providers’ stored keys.
Fix: agent-uploaded content. Data uploaded through the AI assistants can no longer be silently garbled when the model sends plain text where encoded content was expected.

August 5, 2026

Fix: the tap sync-state view is now reachable.

Sync-state button restored. v1.14.0’s per-tap sync-state view (current bookmark, commit time, one-click reset) was missing its button on the Catalog page, so it couldn’t be opened in the UI. Each tap row now has a bookmark icon that opens it. The MCP sync-state tools were unaffected.

August 4, 2026

Incremental taps — scheduled taps now fetch only what’s new.

Taps remember where they left off. After each successful run the platform stores a bookmark; the next run pulls only records added or changed since. Recurring syncs run at constant size instead of re-fetching everything.
No gaps, no duplicates. The bookmark only advances when a run succeeds — failed runs leave it in place so the retry re-fetches the same window; key-field destinations absorb overlap by updating in place.
Visible and resettable. Each tap’s sync state can be viewed and reset from the Taps page; agents get the same via two new MCP tools. Manual backfills with an explicit date range never move the bookmark.
AI-built taps are incremental by default. Described a recurring source? The generated script now uses change-detection automatically when the source supports it (modified-since filters, ID watermarks, cursors, or content hashing). See the Taps guide.
MongoDB upsert. MongoDB pipelines can declare key fields, so revised records update in place instead of duplicating — previously blocked by over-strict validation.
MCP playground fixed. All tools in the MCP tab’s playground now execute (28 previously errored when run there); 63 tools total including the two new sync-state tools.

July 28, 2026

Store your tap scripts in GitHub — every save is a commit.

GitHub script storage. Connect a repository under Configuration → Code Repository and tap Python scripts are stored there instead of built-in storage — every save from the wizard, the Assistant, or the API becomes a reviewable commit.
Runs pinned to the saved version. Scheduled runs execute the exact commit that was saved, from a local cache — they keep working even if GitHub is unreachable, and each run records which version it executed.
Edit in your own IDE. Commit changes to the repository directly; Datris detects the newer version when the tap is opened and offers to load it. Conflicting edits are stopped with a clear message instead of overwritten.
Move taps either way, one click. Existing taps stay on built-in storage until moved; each Taps row shows where its script lives. Fine-grained token scoped to the one repository; connection test built in; GitHub Enterprise Server supported. See the GitHub Script Storage guide.

July 27, 2026

Claude Opus 5 support, and a fix for a month-after-install credential expiry.

Claude Opus 5. Anthropic’s newest model is fully supported and is now the recommended Anthropic model for chat and code generation — same price as Opus 4.8 with better results. Fresh installs default to it; existing installs keep their configured model.
Fix: internal credential expired a month after install. An internal service credential was silently issued with a 32-day lifetime, after which restarting the platform could fail. Credentials are now long-lived and refreshed on every startup. Applies automatically on upgrade.

July 24, 2026

Curate the data sources your assistant recommends.

Approved data sources registry. A new Configuration → Data Sources tab lets you maintain a curated list of approved external data sources in one editable document. When the AI assistant or the tap brainstormer suggests where to pull data from, registry sources are offered first, by name; anything outside the registry is clearly flagged. Leave it empty and behavior is unchanged.
You stay in control. Registry entries don’t skip confirmation — the assistant still confirms source, scope, and destination before building anything.

July 23, 2026

Failed scheduled runs now retry themselves — and tell you how to fix what they can’t.

Automatic retry for scheduled runs. Transient failures (source briefly unreachable, rate limits, network blips) are re-run automatically a few times with a short delay before being flagged. Only runs that are provably safe to repeat are retried, so a retry can never duplicate data. On by default and configurable; manual runs pause the retry ladder.
AI fix suggestions, right where failures show. Persistent failures get a one-line suggested fix directly on the failed run in Ops → Ingestion and tap Run History, with the full diagnosis and concrete change one click away. Advisory only — nothing changes unless you apply it.
Agents see suggestions too. Agents polling pipeline status get the same one-line fix suggestion for failed jobs.
Sign-in session fix. With user login enabled, an expired session now ends cleanly and returns you to the login screen right away, instead of requiring a page refresh.

July 22, 2026

Search shows only what’s installed, plus internal consolidation.

The Search tab now lists only active data stores. Stores that aren’t installed or reachable no longer appear in the query-type menu, and the list recovers automatically after a server restart.
Clearer errors for unavailable stores. Searching a store your deployment doesn’t have returns a clear message naming the valid options.
Internal consolidation. Vector-store loading and search, AI provider integration, and tap AI workflows were restructured onto shared foundations — same behavior, less duplication, faster future vector-destination support.

July 21, 2026

Reliability and hardening.

Faster repeated queries. Postgres connections are now pooled and reused across pipeline loads and interactive queries instead of being opened fresh each time.
Connection details are masked in logs. Database connection information no longer appears in logs or job status output.
More stable long-running deployments. Fixed several resource leaks in file ingestion and data-reading paths that could slowly accumulate open connections.
Clearer diagnostics. Many background failures that were previously silent are now logged with context.
Every release is now test-gated. An automated test suite runs on every change and before every release build.

July 14, 2026

Installer hardening.

Clear guidance when a previous installation is in the way. If containers from an earlier Datris install (in another directory) would block a new one, the installer detects this before downloading anything and prints exactly what is blocking, where it lives, and how to remove or upgrade it.
API keys are no longer echoed. The installer reads provider keys with input hidden and confirms with a masked preview, so keys never land in your terminal scrollback.
Best model out of the box. Fresh installs choosing Anthropic default to Claude Opus 4.8 for chat, NL→SQL, and analysis — matching code generation. Existing installs keep their configured model.

July 13, 2026

Choose your databases at install — run less, connect what you already have.

Pick and choose at install. The installer now asks which databases and stores you want instead of installing everything. Each can run bundled, connect to a service you already operate, or be skipped — and pressing Enter at every prompt still gives the standard full install.
Bring your own infrastructure. Point Datris at an existing Postgres, Kafka, or managed vector store (Qdrant, Weaviate, Chroma, Milvus) during install, and optionally store Snowflake or Databricks destination credentials up front so pipelines can use them on day one.
Lighter installs. Skipping the bundled semantic-search server avoids a multi-gigabyte model download — the installer recommends OpenAI embeddings, which need no local container and cost pennies. Vector stores and a local Kafka test broker are now a one-line opt-in instead of commented-out YAML.
Both AI providers, each at its best. The installer asks for an Anthropic key (chat, code generation, AI data quality) and an OpenAI key (embeddings) — both optional, with sensible behavior when only one is present.
Everything knows what’s installed. The pipeline wizard greys out destinations that aren’t available, the AI assistant offers only the destinations your deployment actually has, and creating a pipeline against a missing database returns a clear explanation instead of a timeout.
Install-time validation. After first boot the installer checks every store you configured — bundled or external — and reports each by name, so a mistyped hostname or bad credential surfaces immediately.
Change your mind anytime. Enable or disable any store after install with a one-line edit and a restart. Disabling keeps the data; re-enabling brings it back.
Safer upgrades. Upgrading an existing install never changes which services run — everything you had keeps running, no action required. Data services now use durable named storage, so containers can be removed and recreated without losing data.

Upgrading: Re-run the installer, or docker compose pull && docker compose up -d --remove-orphans after refreshing the compose file. Your current services and data are preserved automatically. See Installation for the new options.

July 9, 2026

Databricks destination — load, upsert, and query your Databricks workspace.

Load into Databricks. Point any structured pipeline at your own Databricks workspace and Datris loads it as governed Delta tables in Unity Catalog: tables are created automatically, the schema evolves as new columns appear, and natural-key upserts keep scheduled re-runs duplicate-free. Loaded tables are immediately queryable across the workspace — notebooks, dashboards, and lineage all see them natively. Available in the pipeline wizard and through the AI assistant.
Secure by default. Connects as a dedicated service principal you control, with least-privilege access to a single catalog; credentials live in a platform secret, never in pipeline configs. The Databricks destination guide includes a copy-paste grant script. Works out of the box with serverless SQL warehouses — nothing runs in your cloud account and there’s no cluster to manage.
Safe full refreshes. Pipelines that replace their table on each run swap the contents atomically — readers never see a half-loaded or empty table, and a failed run leaves the previous data intact.
Forgiving setup. Pasted workspace URLs and warehouse connection paths are cleaned up automatically in whatever shape they arrive, and connection problems come back as plain-language errors that say exactly what to fix.
The assistant can verify and query Databricks. Ask it to confirm a load landed or answer questions over the data — read-only, using the pipeline’s own credentials — and it can browse your catalogs, schemas, tables, and columns along the way.
Cleaner deletes. Removing a pipeline together with its data also cleans up the table it created in Databricks.
CLI catch-up. datris ingest can now target Snowflake and Databricks directly, and pipeline listings label Snowflake and Databricks destinations.

Upgrading: Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris. Using Databricks requires a one-time setup in your workspace — see the Databricks destination guide.

July 8, 2026

Snowflake destination — load, upsert, and query your Snowflake account.

Load into Snowflake. Point any structured pipeline at your own Snowflake account and Datris bulk-loads it: tables are created automatically, the schema evolves as new columns appear, and natural-key upserts keep scheduled re-runs duplicate-free. Available in the pipeline wizard and through the AI assistant.
Secure by default. Connects with key-pair authentication under a dedicated service user and least-privilege role you control; credentials live in a platform secret, never in pipeline configs. The Snowflake destination guide includes a copy-paste setup script that creates everything — role, warehouse, database, service user — in one run.
Forgiving setup. Pasted private keys and account URLs are cleaned up automatically in whatever shape they arrive, and connection problems come back as plain-language errors that say exactly what to fix.
The assistant can verify and query Snowflake. Ask it to confirm a load landed or answer questions over the data — read-only, using the pipeline’s own credentials — and it can browse your databases, tables, and columns along the way.
Watch the assistant work. Long assistant turns now show live progress: a size counter while it composes larger pieces of work, a running glimpse of its reasoning, and a continuous activity indicator between steps — no more silent pauses.
Faster pipeline creation. The assistant sends only a small schema sample when creating pipelines, trimming minutes off longer build conversations.
Cleaner deletes. Removing a pipeline together with its data now also cleans up the table it created in Snowflake.

Upgrading: Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris. Using Snowflake requires a one-time setup in your Snowflake account — see the Snowflake destination guide.

July 1, 2026

Claude Sonnet 5 support.

Choose Claude Sonnet 5. Anthropic’s new Claude Sonnet 5 model is now available as a choice for the AI assistant and code generation. Select it in the Configuration tab like any other model.

Upgrading: Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris.

June 29, 2026

Switch AI providers freely, and an assistant that finishes the job.

Switch providers without losing your keys. Your Anthropic and OpenAI API keys are now kept per provider, so you can change the provider for any section — AI, CodeGen, or Embedding — and switch back later without re-entering the key you already saved. Enter each provider’s key once and mix providers however you like.
The assistant runs to completion. Longer assistant replies that used to stop partway — leaving you to type “continue” — now finish on their own.

Upgrading: Existing installs: docker compose pull && docker compose up -d. Keys you’ve already entered carry over; no re-entry needed. The CLI: brew upgrade datris.

June 24, 2026

Your secrets and configuration now persist across restarts and rebuilds.

Settings that stick. Connection secrets, AI provider and model choices, and other changes you make in the Configuration tab now survive a rebuild (docker compose up --build), a restart, and host reboots. Previously some of these could quietly revert on a rebuild because they were held only in memory — now they’re stored durably, and the Configuration tab is the source of truth.

Upgrading — one-time, please read: with this release the secrets store moves from in-memory to durable on-disk storage. Any secret you had set only in the app’s UI — including tap connection secrets — does not carry over and must be entered once more after upgrading. From then on your secrets persist across every future rebuild and restart. (Settings supplied through your environment configuration are restored automatically.) Existing installs: docker compose pull && docker compose up -d. The CLI: brew upgrade datris.

June 22, 2026

Version history for taps and pipelines, plus a faster assistant.

Version history for taps and pipelines. Every create or change to a tap or pipeline is now saved as a snapshot. Open Version History to see what changed and when, compare any two versions side by side, and roll back — or forward — to an earlier version. Nothing is overwritten; restoring simply adds a new version. AI agents can review, compare, and restore versions too.
Configuration that sticks. AI provider and model changes in the Configuration tab now save reliably and persist across restarts.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. Existing taps and pipelines are seeded into version history automatically on first start; no manual migration needed. The CLI: brew upgrade datris.

June 18, 2026

Stronger isolation for tap scripts, plus a catalog readability fix.

Optional sandboxed execution for taps. Tap scripts can run in a dedicated, isolated runner that’s cut off from the platform’s credentials and internal services — defense-in-depth for the code your taps execute. Opt-in; see the Tap Execution & Isolation guide.
Tighter defaults for tap scripts. Platform credentials are kept out of tap code, and any extra packages a tap needs install in isolation rather than into the shared environment.
Catalog: full names on hover. Long tap and pipeline names in the catalog are no longer cut off — hover to see the full name.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 14, 2026

Drop a file into the Assistant and it builds the pipeline for you.

Drag a file into the chat. Attach a CSV, JSON, XML, or document right in the Assistant conversation. The assistant reads a sample, works out the shape, and proposes where to put it — then, once you confirm, creates the pipeline, loads your data, and reports how many rows landed. No wizard, no manual schema step.
It confirms before it builds. The assistant picks a sensible default destination for your file and names the alternatives, so you can steer it to Postgres, MongoDB, an object store, or a vector store before anything is created — and it asks about an upsert key if that makes sense for your data.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 12, 2026

A more decisive assistant, and taps that tell you when a credential is missing.

No more “type continue to keep going.” The in-product assistants (build, ops, catalog, and search) used to sometimes end a turn by announcing the next step and then stopping — making you nudge them to carry on. Now, when the assistant has already decided what to do next, it just does it, while still pausing where it genuinely needs your decision or approval.
Taps surface missing credentials instead of silently returning nothing. If a tap’s API key or other credential has been deleted or is missing the field the tap needs, the run is now reported as a clear failure naming what’s missing — rather than quietly completing with zero records and hiding the real cause. Runs that legitimately have no new data are unaffected.
Clearer MCP activity graph. The connection graph in the MCP Activity monitor now renders at a readable size.
“Move to catalog” shows everything. The per-item move menu in the Catalog now lists all your catalogs, not just the first one.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 11, 2026

Claude Fable 5 — Anthropic’s most capable model, now selectable.

New model option. Claude Fable 5 now appears in the model picker for both the in-product assistant and code generation, alongside the existing Claude and OpenAI choices. Pick it from Configuration when you want Anthropic’s most capable model for demanding reasoning and long-horizon work.
A note on data retention. Fable requires standard (30-day) data retention on the Anthropic account and isn’t available on zero-data-retention organizations. If you bring your own Anthropic key on a zero-data-retention plan, choose a different model — you’ll now see a clear message explaining why, instead of a generic error.
Clearer model errors. When a model declines a request — wrong account settings, a rejected key, or unsupported options — the assistant now surfaces a plain-language explanation naming the model, rather than a raw provider error.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 7, 2026

A domain-neutral assistant — guidance that fits whatever data you work with.

No more finance-flavored examples. The in-product assistant, the tap generator, and the prompt suggestions used to lean on stock-market wording — tickers, symbols, financial filings — in their examples and defaults. They’re now domain-neutral, so the guidance and sample values match the data you’re actually working with, whatever the domain.
Generic search defaults. Vector search now defaults to a documents collection (and a Documents class) instead of finance-specific names. If you already pass your own collection, table, or class name, nothing changes — only the placeholder defaults differ.
Refreshed starter prompt fragments. The built-in tap prompt fragments now describe general patterns — rate-limited APIs and APIs that require a User-Agent header — rather than specific named data providers.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 4, 2026

Organize your catalog by chatting with it.

Catalog assistant. The Catalog tab has a new side-panel assistant for tidying things up. Ask it to group your taps and pipelines into catalogs, move items around, or suggest a cleaner structure — it lays out a plan first and only makes changes once you approve them, and the catalog updates as it works. It sticks to organizing: questions about your data go to Search, and running or fixing pipelines stays in Ops.
Right where you’re working. The assistant lives in a collapsible right rail (toggle it with Cmd/Ctrl + backslash) so the catalog stays in view while you reorganize. Clicking Describe to Assistant on a catalog now opens this panel in place instead of jumping to another tab, and the conversation sticks around as you navigate away and back.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 3, 2026

Ask your data a question — conversational search comes to the Search tab.

Chat search. The Search tab has a new Chat mode, with a Traditional toggle for the structured query UI you already know. Ask a question in plain language and Datris finds the answer across all your pipelines and taps — cataloged or not — querying tables, searching documents, and replying with citations to where each answer came from. It’s read-only: it looks, it never changes anything.
Scope to a catalog. Narrow a chat to a single catalog (or to Uncataloged data) from the dropdown, or leave it on All to search everything.
Conversations survive a refresh. Your Search chat and Assistant conversations now persist across a browser refresh, so reloading the page no longer clears the transcript.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

June 2, 2026

Orchestrate Datris taps from Apache Airflow.

Run taps from Airflow. A new airflow-provider-datris package adds an operator that triggers a tap, waits for the pipeline to finish, streams Datris logs into the Airflow task log, and reports run tokens and row counts back to Airflow. Cancelling the DAG run cancels the Datris job.
Date-windowed backfills. Taps can now take per-run parameters, so an Airflow DAG can pass its logical date (or any window) into the tap for that run — backfills and incremental loads work without editing the tap.
No double-firing. A tap is scheduled by Datris or Airflow, never both: if a tap has a Datris cron, the Airflow operator declines to trigger it. To drive a tap from Airflow, leave its cron empty.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. Install the provider where Airflow runs: pip install airflow-provider-datris.

May 29, 2026

Write to AWS S3, query Parquet and ORC from the Assistant and Search, and stop chats actually stop.

AWS S3 as a first-class destination. The Object Store destination now writes Parquet or ORC directly to S3 alongside the built-in MinIO. Pick Object Store (MinIO or S3) in the pipeline wizard, point at your bucket, and reference a credentials secret you’ve created in Configuration → Secrets → Platform. Multiple S3 destinations with different IAM keys coexist in the same deployment — each pipeline carries its own credential reference, applied per bucket at write time.
Region lives with the credential. AWS credentials and region travel together in the credentials secret rather than on the pipeline config. One source of truth, one place to rotate, no more silent us-east-1-says-the-config-but-the-key-is-us-west-2 failures. Field names are flexible — accessKey / AWS_ACCESS_KEY / AWS_ACCESS_KEY_ID all work.
Query Parquet and ORC from the Assistant. “Show me the weather data” now works against pipelines whose destination is Object Store. The Assistant resolves the bucket and credentials from the pipeline config, reads the columnar files, and returns rows in chat.
Search tab gains an Object Store option. Pick a pipeline from the dropdown, see the resolved bucket/prefix/format, set a limit, hit Execute. Works for both MinIO and S3 destinations.
Assistant offers all three structured destinations. When you ask for structured data and no pipeline yet covers it, the Assistant now mentions MongoDB, PostgreSQL, and Object Store as choices instead of silently defaulting to one.
Assistant can discover destination credentials it shouldn’t create. Platform-tab secrets are now visible to the Assistant for reading (names and field shape only — never values). When a pipeline destination needs a credentials reference, it lists what’s available, verifies the field shape, and points you at the Secrets tab to create one when nothing fits.
Stop actually stops. Clicking Stop in the Assistant now halts the in-flight chat within a fraction of a second instead of waiting for the upstream model to finish generating. Cancelled responses cost only what was already streamed.
Pipeline failures surface fast. A failed pipeline now flips to Error in the Ops dashboard within seconds instead of staying stuck in Processing for up to ten minutes.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. Existing MinIO pipelines work unchanged. For AWS S3, create a Platform-tab secret with accessKey, secretKey, region (and optionally sessionToken), then reference it by name from your pipeline.

May 28, 2026

Ask the Ops assistant about a failing pipeline without leaving the dashboard.

New Ops chat side panel. A collapsible chat lives on the right side of Ops → Activity. The assistant has the current failures, stale taps, and volume anomalies in mind — ask “why did X fail?” and it pulls the root cause; ask it to re-run a tap and it runs and reports the outcome. The panel stays mounted as you switch between Activity and Ingestion, so the conversation survives the tab change.
“Ask” buttons on failure and volume rows. Click “Ask” next to a row to seed the chat with a row-specific question so you don’t have to retype the tap or pipeline name.
Successes are expandable like Failures. Click any row in the Successes pane to see the same event trail (begin → processing → end) you get from the Failures pane. Only one row across either pane is open at a time so the layout stays compact.
Claude Opus 4.8 is the new recommended CodeGen model. New installs seed Opus 4.8 as the codegen default. Existing installs can pick it from the model dropdown in Configuration. The older Opus versions remain selectable for anyone who wants to pin a specific version.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 27, 2026

The Assistant stops second-guessing itself when the data is already there.

No more apology loops. When the Assistant verifies platform state and the pipeline / tap it created earlier is present in the list, it now treats that as evidence the work was done — instead of retracting a prior turn’s “done” claim and rebuilding from scratch.
“Show me X” goes straight to the data. When you ask to see / list / show data and a matching pipeline already exists, the Assistant now jumps to the destination’s query tool (Mongo / Postgres / vector search) and returns the actual rows or documents. It no longer asks you which sources, providers, or schedules to use for a pipeline you already have.
Long catalogs no longer hide existing resources. The tools the Assistant uses to inventory pipelines and taps now return a compact summary with names at the top. Previously, on environments with many pipelines, an entry near the end of the list could slip past the Assistant’s scan — the new shape makes every name impossible to miss.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 27, 2026

Ops Activity gains a Successes pane, long lists scroll in place, and the dashboard’s auto-refresh no longer yanks you back to the top.

Successes pane on Ops Activity. A new pane below Failures lists every pipeline that ran successfully in the selected window, with the run count, items processed, and last-run time. Click a row to jump to that pipeline.
Long lists scroll inline. Failures, Successes, and the Per-pipeline volume table all cap at ~10 rows of height and scroll internally instead of stretching the page. Per-pipeline volume column headers stick to the top so you don’t lose context as you scroll.
Auto-refresh preserves scroll position and expansion state. The 30-second refresh no longer scrolls a long pipeline-volume list back to the top or collapses an open failure detail. Expand a failure, scroll where you want, leave the tab open — it stays put.
Numeric column headers aligned with their data on the Per-pipeline volume table — Today, 7d avg, vs avg now line up with the numbers underneath.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 26, 2026

A new Ops Activity dashboard, Postgres pipelines learn upsert, per-run tap parameters, and a safer agent workflow.

Ops Activity dashboard — at-a-glance ingestion health. A new tab under Ops pulls every tap and pipeline run in a rolling window (24h / 7d / 30d) into KPI tiles, time-series charts of runs and items ingested, a per-pipeline 7-day volume table, and a Failures pane that dedupes by item with attempt counts. Each unrecovered row has a Re-run button — for pipeline failures with an upstream tap, the button re-runs that tap to retry the load so you don’t have to hunt for it in the Catalog.
Postgres pipelines upsert on conflict when keyFields is set. Matches the semantics Mongo has always had. Backfills over already-loaded dates, incremental taps with overlap, and “load again with the same key” flows now upsert instead of failing with a duplicate-key error. If you retrofit keyFields onto an existing table, the platform adds the matching unique index for you on the next load — or surfaces a clear remediation message if existing data violates the proposed key.
Per-run tap parameters. Run a tap with caller-supplied values — date ranges, ticker lists, page cursors, batch sizes — without rewriting the script or the secret. Pass params to run_tap and the script reads them as env vars for that one run only. Scheduled cron runs see an empty params bag and fall back to script defaults, so the same script handles both manual ad-hoc calls and unattended schedules without branching.
“No records” is no longer a failure. A tap that runs cleanly and returns zero rows — a polling tap on a quiet day, an incremental tap that’s caught up, a market tap on a weekend — now records a distinct no_records status with a neutral badge. Doesn’t count in the Failures tile, doesn’t fire bogus “recovered” badges, doesn’t train agents to interpret “no new data” as “platform broken.”
Large tap outputs fail fast with an actionable error. A backfill that exceeds the size cap (default 100 MB) now stops with a clear message telling you and the Assistant to chunk the source range smaller — instead of OOM-killing the server. Multiple smaller runs all land in the same destination pipeline; with keyFields set, overlapping ranges upsert safely.
Concurrent tap runs no longer race. Two pipelines that loaded data in the same millisecond previously risked landing each other’s records in the wrong destination under specific timing. Fixed at the source; no action required.
Secret values stay masked in Configuration. The Secrets tab now masks any field whose name suggests a credential — passwords, tokens, API keys, signing keys, certificates, and named variants of those (e.g. *_API_KEY, *_SECRET_*). No action required.
Tighter Assistant workflow. Three new disciplines:
- Scheduling lives on the tap. Say “every morning” or “at market open” and the Assistant sets the CRON expression on the tap itself — instead of handing back a cron line for you to wire up yourself.
- Test before first run. A newly-created or just-edited script is validated before any real run, and before being put on a schedule — no more “guaranteed-bad nightly run” the next time the cron fires.
- No confabulated progress. If the Assistant intended to do N things and only did M of them, it tells you which M happened rather than narrating all N as complete. The credential form the Assistant pops up now asks only for true secrets — not for configuration values you already typed into chat.
Agent Monitor stays responsive during long tool calls. Running a slow tool no longer freezes the Connections viz or the Activity Log — they keep streaming. Returning to the Agent Monitor tab refreshes the log to current server state instead of showing a stale empty view.
Configurable JVM heap. The bundled datris service has a sensible default heap size for small hosts (8 GB) and a clean way to bump it on larger machines via .env. See Installation → JVM Heap Sizing for the suggested sizings per host RAM.
Ops tab moved after Search in the top navigation. Mostly cosmetic; route URLs are unchanged so bookmarks and deep links still work.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. Existing Postgres pipelines with keyFields automatically pick up the upsert path on the next load. The CLI: brew upgrade datris.

May 22, 2026

Embedding pipelines stop failing on oversized chunks, the Assistant carries long runs to completion, and the Catalog gets the missing delete buttons and run-history detail.

Embedding pipelines no longer fail when a single chunk is too big. Every embedding call is now guarded by a token-aware safety net that splits any chunk over the model’s input cap before sending — so a 10-Q ingest with one dense table can’t take down the whole batch. Works the same for OpenAI, Cohere, Voyage, BGE-M3, Nomic, Mistral, or anything else, with built-in caps for the common models and a conservative default for the rest. OpenAI families get exact token counts; everyone else gets a heuristic.
Token-aware chunking. A new maxChunkTokens option on the chunking config tells the chunker to stop merging segments before they cross a token estimate — the safety net above becomes a true last resort. Recommended for any new vector pipeline.
The Assistant carries pipeline runs to completion. When the agent kicks off a tap or upload, it now polls the run to a terminal outcome with exponential backoff instead of summarizing “still running — check back later.” You’ll see a one-line progress update each cycle and the final outcome reported in chat, including any per-document failures.
Run History now shows per-document outcomes. Expanding a tap run that fed a vector pipeline lists every document the pipeline processed with its own status, elapsed time, and — for failures — the specific stage and error. A run that fetched 28 documents but failed on 1 now shows the failure inline instead of a misleading green “success.”
Catalog: delete buttons work on individual taps and pipelines. The trash icon now shows an inline confirm right in the row — Delete / × for taps, Config & Data / Data Only / × for pipelines (Data Only wipes rows but keeps the config so the next ingest fills it again).
Catalog: last-run timestamp is back above the status badge.
Agent Monitor fits the viewport. Both Connections and Activity Log panes now size to the visible area on first load and reflow when you resize the window, not just in the pop-out.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui mcp-server. No data migration needed. The CLI: brew upgrade datris.

May 21, 2026

Streamlined navigation and a Catalog-centric workflow — fewer tabs, richer Catalog, and the Assistant front-and-center.

Five top-level tabs instead of ten. Datris now opens to Assistant, MCP, Catalog, Data, Configuration — plus the Help dropdown. Same capabilities, organized around how you actually work.
Catalog is the home for taps and pipelines. Each catalog card embeds the full tap and pipeline tables with inline rename and a move-to-catalog dropdown on every row. Uncataloged is always shown so day-1 users have a place to start.
Bulk move at the catalog level. Move every tap and pipeline from one catalog into another in one click. The destination auto-expands so you see the items land.
Describe to Assistant from any catalog. A button on each catalog card opens the Assistant with a fresh chat and the catalog pre-filled — the Assistant assigns the right catalog to whatever it creates.
Wizards link back to where you came from. Tap and pipeline edit screens now show the item’s name in the title, a “Back to Catalog” link at the top, and primary action buttons at both top and bottom of the form. Pipeline wizard’s JSON-review step is gone — Save fires from the Destination step.
Pop out the Agent Monitor. A new icon opens both Connections and Activity Log in a separate browser window — park it on a second monitor and watch tool calls stream while you work in the Assistant.
Catalog state persists across navigation. Open catalogs and expanded sub-sections survive refreshes and tab switches.
Fixed: tap rename no longer breaks scripts. Renaming a tap inline used to leave the new tap pointing at a deleted script file.
Fixed: catalogs containing only pipelines could lose their assignment when the pipeline edit wizard was opened.
Discovery tab removed. The Assistant covers its workflow.
Getting Started tab removed. First-run guidance lives in the Assistant’s starter prompts and inline empty states.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui. No data migration needed. The CLI: brew upgrade datris.

May 19, 2026

Scoped API keys with per-agent permissions, plus Assistant resilience and smarter onboarding.

New API-Keys tab in Configuration. Issue a dedicated key per agent, CLI, or integration with an explicit list of what it’s allowed to do — read pipelines, run taps, upload documents, query data, and so on. Each key is its own identity in the request log and can be rotated or revoked independently. Five starting templates: read-only, rag-builder, reporting, ops, and full-access. The tab only appears when you’ve set USE_API_KEYS=true in .env — issuing keys is pointless until the validation layer is on.
Keys actually constrain. When an external agent (Claude Desktop, Cursor, the CLI) connects with a scoped key and tries something outside its bundle, the platform refuses the call and tells the agent why in plain JSON — so the agent doesn’t keep retrying alternate paths. Agents stay productive within their lane; you don’t have to trust them not to wander.
UI no longer asks for a key when user authentication is on. With login enabled, your session cookie is the only thing the browser needs — paste-the-key flow goes away. The Assistant runs under your identity, audit logs show you as the actor, and your role determines what it can do (admin = full access, editor = data writes, viewer = read-only).
Assistant rides through Anthropic overload. When Claude Opus is rate-limit-shedded, the Assistant retries with backoff and, if needed, transparently switches to Sonnet for the rest of the turn — with a small inline note so you know it happened. Conversations that previously errored out now keep moving.
Assistant checks the platform before suggesting external sources. On any data-related ask (“I’m looking for X”), the agent now lists your existing pipelines and taps first, then either points you to what already exists or asks before adding more. Avoids the “let me enumerate seven public APIs” detour.
Assistant auto-runs newly created taps that have no schedule. When you build a one-shot tap, the Assistant kicks off the first run so you see real data instead of an empty pipeline. For scheduled taps it asks first, since the cron will fire on its own.
Health and version endpoints are public. Container orchestrators, status pages, and the UI’s connection check no longer trip 500s when API-key auth is required.
Configuration → Taps sub-tab removed. Prompt Fragments are unchanged and still apply to tap creation, brainstorm, auto-fix, and Discovery — they’re now managed via the API instead of a dedicated UI page.
Tap wizard pipeline link is clickable. The “Linked to: <pipeline>” pill in step 4 now navigates straight to the pipeline editor.

Upgrading: docker compose pull && docker compose up -d --force-recreate datris ui. Existing API keys keep working — they’re treated as full-access until you replace them with scoped keys from the new tab. The CLI: brew upgrade datris.

May 13, 2026

Polish for the Assistant, AI Configuration, and MCP tabs.

Assistant: keep typing without clicking. After you send a question, the cursor stays in the composer — so as soon as the agent finishes (or even while it’s still working), you can type your next prompt without reaching for the mouse.
AI Configuration: provider switches no longer wipe your overrides. Switching the primary AI, codegen, or embedding provider used to clear the saved provider/model/endpoint on the next page load, so you had to re-pick them every time. Your selections now persist correctly through a provider switch (you still re-enter the API key when changing providers — that’s intentional).
Connect Your Agent: paste-and-go for local. The MCP tab’s generated config no longer asks for an API key for the default local setup — paste the snippet into your agent and it just works. A key is still required when you point the snippet at a remote instance.
Hosted trial provisioning tools removed from the agent toolset (hosted platform since retired). The Getting Started tab has also been refreshed to point at the Assistant for the common onboarding path.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris.

May 12, 2026

A new in-product Assistant — chat your way from “I need data” to a working pipeline.

The Assistant tab. A new top-level tab opens an in-product agent that finds an external data source, builds the tap, creates the pipeline, runs it, and shows you the result — all in one chat. You watch the model’s reasoning, every tool it calls, and the live status, then click straight into the tap or pipeline it created. See Assistant.
Real-time visibility while the agent works. Streaming reasoning, inline tool cards with friendly labels (“Searching the web for …”, “Creating tap …”), live success/error status, and a Stop button that aborts the loop at the next checkpoint. Conversations survive navigating to other tabs and back.
Credentials never enter the chat. When the agent needs an API key or other credentials, it opens an inline credentials form right in the chat. Values go straight to the vault — they don’t appear in the conversation log, your screenshots, or the model’s context. The form also lets you reuse an existing tap secret instead of creating a new one.
Pipelines can be edited in place. Calling pipeline-create again with the same name now upserts the configuration without dropping the destination data. Two new knobs in the same call: a natural-key list for dedupe/upsert on every run, and a flag to wipe the destination before each run for full-snapshot workflows. Both work on PostgreSQL and MongoDB destinations.
External agents get the same new tools. Claude Desktop, Cursor, and any other MCP client connected to your Datris server now see two new tools for discovering existing tap secrets (names and field shapes, never values) plus the new dedupe and reset knobs on pipeline create.
Pipeline delete now describes what it actually does. The tool description used to claim it kept your destination data; it never has. It now correctly says it removes both the configuration and the destination data, with an explicit opt-in flag for the “keep the schema, clear the data” reset case.
MCP transport upgraded — both protocols at once. The bundled MCP server now serves the streamable-HTTP transport alongside the existing SSE transport on the same port, so the in-product Assistant and external agents connect to one running process. No configuration change needed.
Discovery tab is hidden in this release. The Assistant supersedes the Discovery tab for the common path.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris. The Assistant uses your existing codegen AI configuration — with Anthropic you’ll see full chain-of-thought reasoning streamed inline; on OpenAI you’ll see reasoning summaries instead.

May 11, 2026

Web search for AI tap workflows, plus simpler MCP authentication.

AI tap workflows can consult the live web. When enabled in Configuration → AI Providers, tap brainstorm, dataset discovery, tap diagnosis, and tap auto-fix look up current API documentation, free-tier limits, current package names, and recent deprecation notices before recommending sources or generating fixes. Pick your web-search provider independently of AI Primary — the platform uses each provider’s native search tool and routes accordingly. First-pass tap script generation stays fast and uses the model’s training data only. See AI Configuration for setup.
AI Configuration changes survive Docker restarts. Saving from the Configuration UI now mirrors the relevant keys back to your .env file so changes aren’t lost when the local Vault container restarts. Provider switches also clear any stale credential preserved from the prior provider, so a wrong-provider key can’t silently 401 the next call.
Cleaner MCP authentication. The bundled MCP server is now a transparent forwarder — each connecting agent provides its own API key per session and the MCP server passes it through to the Datris REST API on every tool call. The Configuration → Connect Your Agent panel generates the new configuration snippet automatically; paste your key from the Configuration UI into your agent’s MCP config. Existing remote-instance setups see no change in behavior.
Tap brainstorm asks about sources first. When you describe data without naming where to fetch it from, the AI now lists 3-5 candidate sources (with free vs paid and key-required info) before drilling into parameters — instead of asking for filtering details up front.
Tap script generation is more resilient. The platform now validates that a generated script actually defines a fetch() function before storing it, and the JSON extractor handles model responses that contain narrative braces without falling back to the raw-text path.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris. If you run the MCP server standalone outside Docker, the connection-target environment variable was renamed for consistency — see the MCP server docs for the new variable name.

May 6, 2026

Reliable job-status polling, re-ingest that doesn’t overwrite your config, and first-class catalog tooling.

Polling an upload’s job status now gives you a clear answer. Previously, when a file finished processing successfully, the response was a raw stream of progress events with no terminal status — agents and scripts couldn’t tell “still running” from “done.” Job status now returns a rollup with a single allDone flag and an aggregate outcome (success, warning, error), plus per-job error detail when something fails. Poll the rollup; act on the outcome.
Re-ingesting a file preserves your pipeline’s config. datris ingest against an existing pipeline used to silently rewrite the config from CLI flags only — wiping out the catalog, custom validation rules, and any other fields you’d set through the UI or via an agent. Re-ingest now uploads into the existing pipeline as-is. To start over with a different config, delete the pipeline first.
Catalogs without the read-modify-write dance. New --catalog flag on datris ingest for new pipelines, new optional catalog argument on the create_pipeline MCP tool, and a new set_catalog MCP tool that retags an existing pipeline or tap in one call. Empty catalog clears the label back to Uncataloged. See Data Catalog.

Upgrading: docker compose pull && docker compose up -d. No data migration needed. The CLI: brew upgrade datris.

May 5, 2026

Choose your embedding provider independently of your chat provider.

Mix-and-match AI providers. The embedding slot is now configured separately from the chat and code-generation slots, so you can keep Claude for chat and code generation while pointing embeddings at OpenAI (or vice-versa). Useful when the bundled embedder is too heavy for your host, or when you want a different model family for vector quality vs chat quality. See AI Configuration for the full list of options.
Existing installs keep their current behavior. If you don’t set an embedding override, the embedding slot continues to follow your chat provider exactly as before — Claude installs keep using the bundled embedder, OpenAI installs keep using OpenAI embeddings. The override is purely opt-in.

Upgrading: docker compose pull && docker compose up -d --remove-orphans. No data migration needed. If you switch embedding providers on an existing deployment, vector destinations built on the previous embedder will fail-fast with a dimension-mismatch message on the next run — drop the affected destination tables or collections and re-ingest.

May 5, 2026

User authentication, roles, and an admin-only Configuration tab.

Optional username/password login. Datris can now require a login before any tab is reachable. Three roles ship out of the box — admin (full access), editor (read + edit pipelines, taps, secrets), and viewer (read-only). Off by default, so existing installs are unaffected. See the new User Authentication doc to enable it.
Configuration is admin-only. When auth is on, only admins see the Configuration tab (Secrets, AI Providers, Taps, Users, Environment). Editors and viewers continue to use everything else.
New Users sub-tab. Admins can add, remove, and reassign roles. A built-in 16-character password generator with a reveal toggle makes handing out credentials painless. The last admin can’t be deleted.
Self-service password change. Users can change their own password from the top-right user menu.
Reveal toggle on the login screen. Easier to see what you’re typing on a new device.
Clear the Agents activity log. The trash icon now wipes the server-side activity buffer (with an inline confirm) so the cleared state survives a refresh.

Upgrading: docker compose pull && docker compose up -d. No data migration needed; auth defaults to off.

May 4, 2026

AI-agent RAG ingestion no longer burns through the conversation context.

Creating a vector-store pipeline no longer requires sample content. Asking an agent to ingest a PDF into pgvector previously forced it to base64-encode the entire document just to register the pipeline — wasting tens of thousands of tokens before any work began. Vector pipelines now register from a name plus destination alone, freeing budget for the actual upload.
Agents are guided to send each document in a single upload. The MCP server now makes explicit that vector destinations chunk server-side, preventing agents from needlessly splitting documents into many small uploads.
Clearer Claude setup docs. The “Configuring Claude” guide shows the recommended SSE / mcp-remote setup first, and steers large-file ingestion to the CLI rather than dragging files directly into the chat — which can overflow the conversation context on sizable PDFs.
README accuracy pass. Corrected tool counts, AI-provider model defaults, and license badge.

Upgrading: docker compose pull && docker compose up -d. No data migration needed.

May 4, 2026

UI cleanup, friendlier Help menu, and a more reliable bundled embedding service.

Bundled embedding handles large ingest batches without errors. The bundled embedding service no longer rejects large batches submitted by the platform, which previously surfaced as ingestion failures on long documents.
Secrets is now a tab inside Configuration. Instead of a separate top-level tab, Secrets lives under Configuration alongside AI Providers, Taps, and Environment. Existing /secrets links continue to work — they redirect to the new location.
Help menu in the top bar. The Docs link is replaced with a Help dropdown that exposes both the docs and a direct link to file an issue on GitHub.
Easier setup with Claude. A new “Configuring Claude” page in the docs walks through Claude Desktop and Claude Code setup end-to-end, including a first-prompts walkthrough for an empty install.
Structured issue reporting. GitHub issues now use forms that capture version, component, deployment mode, and reproduction steps, making bug reports easier to triage and faster to fix.

Upgrading: docker compose pull && docker compose up -d. No data migration needed.

April 30, 2026

Smaller, faster default install. Lighter download, opt-in Kafka, vector ingestion fixes.

~58% smaller docker compose pull. The bundled platform now downloads roughly 11 GB less out of the box. Fresh installs come up dramatically faster.
Bundled embeddings are faster on the same bge-m3 model. No configuration changes needed; existing vector collections built with the previous bundled embedder continue to work without re-embedding.
Kafka is now opt-in. Most local installs don’t need it, so Kafka, Zookeeper, and the Kafka UI ship commented out in docker-compose.yml. Uncomment the bundled blocks (and the related volumes at the top of the file) to enable them. Pipelines that point at external Kafka brokers are unaffected.
Vector ingestion no longer fails with a “duplicate key” error when many documents land at once. Concurrent document loaders previously raced on creating the pgvector extension; the race is now serialized.
Vector ingestion no longer fails on embedding providers that limit batch size. The chunk batch size is now configurable per embedding secret (batchSize), with a cross-provider-safe default. OpenAI users who want to maximize throughput can set this higher.
Configuration tab clarifies optional providers. The bundled embedding option is labeled bge-m3 (bundled). The AI Provider, CodeGen Provider, and Embedding Provider dropdowns each indicate that local Ollama is opt-in.
Service Health no longer shows “Down” for optional services that were never enabled. Kafka and the optional vector databases (Qdrant, Weaviate, Milvus, Chroma) now correctly report “Not Configured” until you turn them on. Existing installs may still show “Down” until their stale Vault secrets are removed.

Upgrading from earlier versions: the bundled embedding service moved off Ollama on the same host port 11434, so existing installs must pass --remove-orphans to release the port: docker compose pull && docker compose up -d --remove-orphans. Without it the upgrade fails with Bind for 0.0.0.0:11434 failed: port is already allocated. Your data and the cached Ollama model are preserved.

April 29, 2026

Reliable run-completion signals, smarter agent calls, cleaner run history.

Agents get a single “are we done?” signal when watching a pipeline load. Polling a publisher token returns a rollup with a clear allDone boolean and a per-job outcome (success / warning / error / processing / timed out), so agents no longer have to interpret the raw event stream to figure out whether a run is complete.
Pipeline status by publisher token works reliably for completed runs. A storage path that occasionally hid completed runs from the publisher-token query is fixed; the query is now backed by an indexed top-level field, with a fallback for older rows so existing data resolves without a migration.
run_tap no longer ships the records array back to the agent. A push run returns recordCount, publisherToken, and the persisted / persistedReason flags — enough to verify ingestion via get_pipeline_status without bloating the agent’s context. Use test_tap to preview a script’s output (capped at 20 sample rows with a recordsTruncated flag).
Duplicate run_tap calls are suppressed. The agent skips a run_tap for a tap that’s already in flight in the same session, and the platform debounces push runs to one per tap per 5 seconds. Prevents accidental duplicate ingestion from parallel tool calls, double-clicks, and transport retries (persistedReason: already_running or debounced).
Every tap run now produces visible logs. The script wrapper emits start / fetch / record-count lifecycle lines on every run, so run history shows useful output even when the user’s script never calls print().
Secret values are masked in stored tap logs and exception messages. A tap script that incidentally printed an API key or Vault-loaded credential would previously have surfaced the raw value in run history; those values are now redacted in the persisted log and the error string.
Deleting a tap now cleans up its run history. Previously, run-history rows accumulated indefinitely and could resurface under a recreated tap with the same name.
Agent activity log shows full requests and responses. The expanded view in the agent monitor no longer truncates request arguments or response bodies.

April 24, 2026

MCP tab and documentation sync.

MCP tab now lists every agent tool. The in-app MCP reference was missing get_pipeline_status, create_tap_secret, delete_tap_secret, get_tap_ledger, and query_natural. All five are now in the catalog and the Try-It playground, and create_tap exposes the tap_type parameter.
Recommended Agent Workflow rewritten to match the platform’s actual flow. The in-app workflow and the MCP docs had drifted — they started with profile_data (which the platform explicitly says not to use for pipeline generation), skipped the persisted / persistedReason check after run_tap, and didn’t show the publisherToken poll that confirms records actually landed. All three are now canonical: check-before-create, verify-via-publisher-token, and tap credentials managed via create_tap_secret.
Documentation: updated agent workflow examples. RAG over external documents is now shown as a document tap (tap_type="document" + get_tap_ledger), onboarding an external source uses create_tap_secret for credentials, and quality monitoring of scheduled taps shows the get_tap_logs → get_pipeline_status(publisher_token=...) pivot.

April 24, 2026

Tap wizard reliability, iteration history, and cleaner vector-search errors.

Tap wizards learn from their own retries. When the AI fixes, optimizes, or reviews a tap script, it now carries forward up to the last three attempts — what was tried, what went wrong, and what changed — into the next call. The wizard stops cycling through the same failed approaches.
Saved tap scripts always match the tap. Saving a tap now pushes the in-memory script to object storage before writing the tap config, and the create/update call verifies the stored script is actually there. No more “missing script” banners from an interrupted save, and auto-revert no longer strands a tap with a deleted script.
Run Tap stays on the page when nothing was ingested. If a manual run finishes without persisting records, the wizard keeps you on the run step and shows an inline reason (test mode, no records, run error) instead of navigating away and hiding the diagnostic.
Tap logs now carry the publisher token. Every tap run that submitted records records its publisher token in the log. Agents reading get_tap_logs can pivot directly to get_pipeline_status to confirm a scheduled run actually landed in the destination — not just that the script ran.
Vector search fails cleanly when the embedding dimension doesn’t match the collection. If you change embedding providers on a pipeline whose vector collection already has vectors of a different dimension, search queries now return a clear 400 with a user-actionable message instead of leaking a JVM stack trace.
Sturdier local-dev startup. Kafka and Zookeeper now use named volumes (no more corruption races on rebuild), Kafka waits for Zookeeper’s request processor to actually be ready (not just its listener bound), and Vault init picks up an explicit AI-provider override so a stray shell env var can’t silently flip providers.

April 22, 2026

Agent-native tap observability, scheduler fix, and agent-owned tap secrets.

Agents can watch a tap load to completion. Running a tap now reports back whether the data was actually persisted — and names the reason when it wasn’t (test mode, no target pipeline, no records, run error). Every persisted run returns a single publisher token covering the whole run, even for document taps that spawn many ingestion jobs. A new get_pipeline_status MCP tool lets an agent poll that one token until the entire load reaches its final state, so it can report “done” with real numbers instead of guessing from a response body.
Scheduled taps no longer need a manual kickoff. Taps saved with a cron schedule now fire on their next scheduled time automatically. Previously, a newly saved scheduled tap would wait indefinitely until you ran it once by hand.
Self-diagnosing tap scripts. If a tap’s generated script goes missing from object storage, the Edit Tap page now shows an amber banner explaining the state and pointing you to Regenerate, instead of a cryptic mid-run “key does not exist” error. Test Tap surfaces the same state with actionable wording.
Agents can manage their own tap secrets. Via MCP, agents can now create and delete the secrets their taps need (API keys, tokens). Scope is strictly tap-owned — agents cannot create, overwrite, or delete human-owned Platform secrets (DB creds, AI keys, vector-store creds).
Secrets page split into Platform and Taps sub-tabs. Platform lists the built-in Datris secret slots; Taps lists agent-authored tap secrets. Creating a secret from the Taps sub-tab auto-tags it so it stays agent-editable.
Honest Test Tap banner. The run-result banner on Test Tap now reflects what actually happened on the server — “sent to pipeline” only when the run was truly persisted, otherwise “not persisted” with the reason — rather than whatever the pre-request checkbox said.
BYO-code taps can declare pip dependencies. If you paste your own fetch script into Create Tap, you can now list the Python packages it needs. Previously only AI-generated taps could declare dependencies.
Example agent refactored onto taps. The bundled market-macro-agent example now drives ingestion through taps instead of ad-hoc fetch scripts, demonstrating the full agent-native tap flow (provisioning, secrets, publisher-token watching).

April 21, 2026

Tap prompt fragments, post-run script review, BYO code, and Configuration page reorg.

Tap prompt fragments (new). A new Configuration → Taps sub-tab lets you save reusable context snippets — things like API conventions, required headers, rate limits, and preferred libraries for a given source. When the key or any of its aliases appears in a Create Tap description, brainstorm, auto-fix, optimize, or Discovery chat, the fragment’s content is automatically added to the system prompt. Includes an AI Suggest button, a Load Examples catalog (AWS, Polygon, Stripe, SEC EDGAR), JSON import/export, and an “Extra context applied” chip row in the tap wizard showing which fragments hit.
Post-run script review. After a tap’s first successful test, the AI now scans the captured stderr/stdout for signals that the script should change — rate-limit or burst warnings, deprecation hints, pagination cues, schema drift, auth warnings — and regenerates the script if needed. On a rewrite the wizard auto-retests; the performance optimizer runs only when the logs are clean. The optimizer’s prompt was also tightened so rate-limit markers push it toward throttling instead of more concurrency.
I Have My Own Code (new). A third Tap Type on the Create Tap wizard lets you paste a fetch() script directly instead of having AI generate one. Step 1 switches to a code textarea with a Use My Code button; after upload the button flips to Re-upload My Code and Step 2 gates on the text matching what’s on disk, so edits force a fresh upload before running the test.
Configuration page reorganized into three sub-tabs — Environment, AI Providers, and Taps — with a prominent “Highly recommended: Anthropic with the latest coding model” tip on the CodeGen Provider section.
Tap Name collision warning. If the name you type in the Create Tap wizard matches an existing tap, an amber banner appears under the field warning that continuing will overwrite the existing tap’s configuration and script.
Auto-fix retries bumped to 3. When a tap script fails its first test, the AI now gets up to three repair attempts (was two) before giving up.
Cron Custom preset no longer blocked by AI formatting. AI-generated cron expressions wrapped in code fences, brackets, or quotes are now cleaned automatically, so the Next button is enabled on valid output.

April 20, 2026

OpenAI Codex models now work for code generation. Previously, selecting a codex-family model for CodeGen (e.g., the recommended GPT-5.3-Codex) caused tap-script generation and AI data quality / transformation to fail immediately with a 404. Datris now routes codex models to the right OpenAI endpoint automatically.
Vector-store dimension changes now fail fast with a clear message. If you switch embedding providers (for example, Ollama bge-m3 → OpenAI text-embedding-3-small) on a pipeline whose destination table or collection already has vectors of the old dimension, the job stops up front and tells you exactly what to do instead of blowing up mid-ingest with a cryptic database error. Applies to pgvector, Qdrant, Weaviate, Milvus, and Chroma destinations.
Configuration save is honest about missing API keys. Changing the AI Provider to Anthropic or OpenAI without entering that provider’s API key no longer silently skips the save while reporting success — the Configuration page now flags the missing key and tells you which one to add.
Create Tap brainstorm wraps up sooner. For document taps, the AI assistant no longer drills for optional date filters once you’ve supplied the source, auth, and a broad scope — the tap ledger already dedupes by content, so those extra questions were just noise.

April 20, 2026

OpenAI GPT-5 and reasoning-model support. AI-backed features (Discovery chat, Create Tap brainstorm, AI data quality, AI transformation, script generation) now work with OpenAI’s newer model families. Previously, selecting one of these models caused chat panels and wizards to hang without a response.

April 19, 2026

Document Taps — a new tap type for feeding vector stores with unstructured files.

Document Taps (new). A purpose-built tap for ingesting PDFs, Word docs, HTML, and other unstructured files into a vector-store pipeline. Describe the source in plain English (“ingest all PDFs from our SharePoint legal folder”, “pull every DOCX from legal-contracts/2026/ in S3”) and the generated tap discovers the files and hands their raw bytes to the pipeline. Text extraction, chunking, embedding, and loading are all handled downstream — you don’t configure any of that on the tap.
Tap type toggle in Create Tap. Choose Document Ingestion or Structured/Semi-Structured on the first step of the wizard; the prompts, placeholders, and example instructions adapt to the choice.
Ingestion ledger. Every discovered file is tracked by URI and content hash. Re-running the tap skips files that are already up to date — no re-embedding, no duplicates. Changed files flow through normally.
Pre-flight validation. Document taps linked to a pipeline are checked at save time to confirm the pipeline is shaped for document ingestion (unstructured source, vector-store destination). Misconfigurations surface as an actionable error instead of a cryptic mid-run failure.
Safe defaults for local paths. Document taps refuse to silently walk arbitrary host directories if a requested path isn’t mounted into the container — they fail loudly instead of ingesting unintended files.
Faster embeddings. The bundled Ollama sidecar now handles concurrent embedding requests in parallel and keeps the embedding model warm between pipeline runs, eliminating cold-start delays when a pipeline resumes after an idle period.
More Python libraries pre-installed for taps. AWS S3, Google Cloud Storage, Azure Blob, Excel, YAML, and date/timezone helpers are now baked into the image. Taps that fetch from these sources no longer need a per-run pip install.

April 18, 2026

Live MCP agent monitor and pipeline status self-healing.

Agents tab — new live view of connected MCP agents. See every tool call as it happens, with agent name, arguments, record count, response size, status, and latency. Click any row to expand the full request and response.
Pipeline status now self-heals when a job completes but the summary gets stuck showing “processing” — completed, warned, and errored jobs resolve to their correct final state.
Example agent (market-macro-agent) automatically reconnects with backoff if the MCP connection drops, and degrades gracefully while offline instead of crashing.

April 17, 2026

New AI models from OpenAI and Anthropic now appear in the Configuration dropdowns automatically, without a Datris upgrade.
Claude Opus 4.7 is the recommended Anthropic model for CodeGen (tap script generation, AI data quality rules, AI transformations, JSON Schema / XSD generation, and natural-language → SQL).
Fresh instances are seeded with the latest recommended models by default.

April 16, 2026

Two-pass AI tap optimization and Data Catalog UX.

AI optimize pass. After a tap script passes its initial test, the platform sends it back to the LLM for a performance rewrite and re-tests. Auto-reverts on regression (>=20% slower) or failure. Also available through the API; tap test results now report run duration.
Discovery wizard. Auto-optimize per tap with before/after timing banner, per-row stop button, on-demand AI fix panel for failed items, chat auto-scroll, and re-shown Discover Datasets button after new messages.
Create Tap. Configurable test sample size (defaults to 20 records).
Data Catalog. Kebab menu on Uncataloged items with Edit, Delete, and Move to Catalog (with name-clash detection).

April 15, 2026

Database lockdown, tap script hardening, Create Tap UX.

Server-controlled database name. The UI no longer edits the Postgres or MongoDB database name — the server reports the configured names and the UI submits them unchanged.
MongoDB internal vs user split. mongodb.database (user-facing, default datris) holds pipeline data; mongodb.internalDatabase (default oss) holds platform state. Existing installs keep oss for platform state; new user pipelines land in datris.
Unlimited reads for tap scripts. Tap scripts can now request every matching row from the MongoDB and Postgres query APIs instead of a capped preview. Preview defaults (20 Mongo / 100 Postgres) are unchanged for UI/MCP callers.
Tap test sampling. New “Limit test sample to 20 records” checkbox in Create Tap step 2 caps how much a test run reads. Cron and manual runs read everything.
Codegen + diagnosis hardening. Generated scripts treat platform response shapes as contractual (no shape-probing, no candidate-key iteration). Diagnosis quotes the actual traceback and respects in-script guards.
Create Tap UX. Ask button, auto-apply diagnosis (capped at 2 attempts), Stop Test, copy-to-clipboard, scrollable-JSON test results, destination collision check on Generate Pipeline, full destination shown on step 5.
Data Catalog. One-click delete of the Uncataloged group; per-item trash icon on taps and pipelines inside every catalog.

April 14, 2026

Input sanitization hardening and minor UI polish.

Pipeline creation sanitizes destination identifiers before writing the config — Postgres dbName/schema/table, MongoDB dbName/table, Kafka topic, ActiveMQ queueName, vector collectionName/tableName/schemaName, and the DQ schema name.
Name sanitization is now applied consistently across the UI, so labels and identifiers are cleaned the same way everywhere.

April 14, 2026

Discovery wizard, Data Catalog, and bring-your-own AI keys.

Discovery wizard. Six-step AI-guided onboarding that turns “yfinance daily prices for the S&P 500” into running taps and pipelines. Available from the UI, the API, and the new discover_source MCP tool.
Data Catalog. Group related taps and pipelines into named catalogs. New Data Catalog tab with expandable contents and an Uncataloged group.
Per-pipeline DQ + transformation editor. Inline editor inside the pipeline view.
Bring-your-own AI keys. Hosted trial signups could supply their own Anthropic or OpenAI key, which took effect on the next AI call (hosted platform since retired).

April 11, 2026

Hosted platform release.

Hosted dedicated-instance provisioning and hosted-aware Configuration UI (since retired).
Improved multi-user session handling.

April 11, 2026

Full tap MCP tool suite and user-supplied tap scripts.

Four new MCP tools: get_tap, test_tap, update_tap, get_tap_logs.
create_tap accepts an optional script parameter for user-supplied Python fetch(), plus secret_name for Vault-injected credentials.
CLI: datris tap create --script path/to/script.py and new datris tap show.

April 10, 2026

Ollama for all AI slots and hot-reload on save.

Configuration UI offers Ollama (local) as a provider for AI Primary, CodeGen, and Embedding. Bundled Ollama sidecar option pre-fills bge-m3 for embedding.
Saving AI configuration takes effect immediately — no container restart required.

April 9, 2026

Hosted trial hardening.

Hosted trial cost tuning and Configuration tab polish (since retired). Self-hosted installs continue to recommend Opus for codegen on their own keys.

April 8, 2026

Breaking: AI configuration restructured into three independent Vault secrets.

ai.aiSecretName and ai.provider are replaced by three top-level slots — ai.aiPrimary.secretName, ai.codegen.secretName, ai.embedding.secretName.
Each Vault secret is self-describing (provider, endpoint, model, apiKey, optionally version). No path derivation from YAML.
v1.5.x deployments must update their server configuration and re-seed Vault on upgrade.

April 8, 2026

Onboarding and tap scheduling UX.

Getting Started tab as the new first-run landing page; Docs ↗ top-nav link.
Inline cron editor on the Taps page: preset buttons, AI prompt field (“Every weekday at 4pm”), Quartz field validation, human-readable description.
Truncate-before-run toggle in the tap wizard (sets truncateBeforeWrite: true on the generated destination).
Hosted trial bring-your-own AI key banner on the Configuration tab (since retired).
Fix: scheduled taps now honor the configured date timezone.

April 7, 2026

Taps (Beta). AI-generated Python scripts that fetch data on a schedule and stream it into a pipeline.

4-step tap creation wizard (Describe → Edit & Test → Schedule → Review) with brainstorm chat and proactive env-var suggestions.
AI script generation with JSON-parse retry and raw-script fallback; AI diagnosis with explicit (a)/(b)/(c) options; “Apply Diagnosis” rewrites the script in place.
Tap secrets stored in Vault and tagged as tap-owned; per-run history is retained and viewable.
CRON scheduling (Quartz format), AI-generated from natural language.
JSON-to-CSV pipeline feed (union of keys) with column-name normalization via spell-out table (% → percent, # → num, & → and, …).
Pipelines page shows the feeding tap; pipeline wizard has “Create from Tap”.

April 3, 2026

Schema Evolution.

Additive evolution: new CSV columns are auto-added as string, schemaVersion increments, and ALTER TABLE runs on Postgres.
Dropped columns are left out of the Postgres load so they default to NULL (previously failed on typed columns).
Missing key fields raise a clear error.
Schema evolution now behaves identically for streamed and file-based ingestion.
Query results now include NULL columns.

April 3, 2026

Remote MCP endpoint.

Per-session API key forwarding on the MCP server, with an option to reject unauthenticated remote connections.
Hosted trial signup and dedicated-upgrade agent tools and remote registry entry (since retired).

April 3, 2026

Hosted trial instances: pipeline wizard auto-populated database names (since retired).

April 2, 2026

Shared-instance hosting.

Hosted multi-tenant isolation on a shared instance (since retired).
Vector DB secret isolation across Qdrant, Weaviate, Milvus, Chroma, pgvector.
Batch upload for compressed files (.zip, .gz, .tar, .jar) processes contents inline — no MinIO webhook dependency.

March 30, 2026

Default AI provider changed to Anthropic (Claude Sonnet 4.6). Customers can still switch to OpenAI or Ollama in configuration.
Generated Python scripts for DQ and transformation are logged in full to the server logs for debugging.

March 29, 2026

New unified datris analyze command replaces ask-sql and ask. Auto-picks the right approach based on --dest (Postgres → SQL generation, MongoDB → document fetch, vector stores → RAG).
--ai-analyze flag on ingest.

March 27, 2026

Mintlify documentation and accuracy pass.

Migrated docs from .md to .mdx with a two-tab Mintlify layout.
Removed deprecated docs (row rules, column rules, JavaScript row functions, REST transformations, deduplication, column trimming).
Page-by-page accuracy review against the codebase.

March 26, 2026

Server fails fast at startup with a clear error if an AI provider is not configured (CodeGen DQ and transformation require it).
Removed stale NEVER rules and deprecated-feature references from the MCP server instructions.

March 26, 2026

CodeGen DQ and transformation.

aiRule replaces the prior AI DQ approach: the LLM generates a self-contained Python validation script from a plain-English instruction, which runs locally. Cost drops from $25–40/file to ~$0.003/rule.
aiTransformation uses the same CodeGen approach.
Works for CSV, JSON, and XML.
Removed: columnRules (regex), JavaScript row rules, REST endpoint row rules, and the legacy AI data-quality path. JavaScript row functions and REST transformations removed from UI and CLI.

March 26, 2026

Atomic create_pipeline. generate_schema is removed; create_pipeline accepts base64-encoded sample data, auto-detects schema, and creates the pipeline in one call.
Content-based uploads across upload_data (renamed from upload_file), profile_data, upload_config.
update_secret MCP tool for AI provider keys only.
All-string schema on MCP pipeline creation; pipeline registration verified by read-back.

March 25, 2026

update_secret MCP tool added (scoped to AI secrets: anthropic, openai, ollama, embedding).
Published to the MCP Registry (io.github.datris/datris) and PyPI (datris-mcp-server).

March 24, 2026

New UI tabs.

MCP Tab — agent view, service health for 10 backend services, browsable tool grid, config generator for Claude Desktop/Code and Cursor, and a Tool Playground that executes against the live API.
Secrets Tab — full CRUD for Vault secrets with sensitive-field masking.
New REST endpoints for vector store metadata discovery (Qdrant collections and friends).